Control method, device, and apparatus for virtual object, and storage medium

Abstract

The embodiment of the present disclosure provides a virtual object control method, device and equipment and a storage medium, a virtual object is constructed based on a target object in an original image; wherein the virtual object comprises a plurality of virtual parts, and the virtual part comprises at least one virtual element; a facial action and / or a body action of the target object are acquired; and the virtual part in the virtual object is controlled to perform a corresponding action according to the facial action and / or the body action. The virtual object control method provided by the embodiment of the present disclosure can drive the facial and body movement of the virtual object composed of a plurality of elements based on the facial action and / or the body action of the target object, and the flexibility of virtual object control can be improved.

Description

Technical Field

[0001] This disclosure relates to the field of image processing technology, and in particular to a method, apparatus, device, and storage medium for controlling virtual objects. Background Technology

[0002] Targeted character-driven virtual avatars are a popular special effects feature, primarily involving facial and limb manipulation. Current virtual objects or models use high-polygon facial reconstruction and facial algorithms to reproduce facial movements onto the virtual object's face. Limb manipulation typically employs a skeletal skinning method. However, these methods are not suitable for driving other styles of virtual models (such as "block" styles), making the exploration of new manipulation technologies for driving virtual objects in these new styles particularly important. Summary of the Invention

[0003] This disclosure provides a method, apparatus, device, and storage medium for controlling virtual objects, which can drive the face and limbs of virtual objects composed of multiple components, thereby improving the flexibility of virtual object control.

[0004] In a first aspect, embodiments of this disclosure provide a method for controlling a virtual object, including:

[0005] A virtual object is constructed based on the target object in the original image; wherein the virtual object includes multiple virtual parts, and each virtual part includes at least one virtual element; the virtual element is a virtual three-dimensional polyhedron.

[0006] Acquire the facial and / or limb movements of the target object;

[0007] The virtual parts of the virtual object are controlled to perform corresponding actions based on the facial and / or limb movements.

[0008] Secondly, embodiments of this disclosure also provide a control device for a virtual object, comprising:

[0009] A virtual object construction module is used to construct virtual objects based on target objects in the original image; wherein, the virtual object includes multiple virtual parts, and each virtual part includes at least one virtual element; the virtual element is a virtual three-dimensional polyhedron;

[0010] An action acquisition module is used to acquire facial and / or limb movements of the target object;

[0011] The virtual object control module is used to control virtual parts in the virtual object to perform corresponding actions based on the facial and / or limb movements.

[0012] Thirdly, embodiments of this disclosure also provide an electronic device, the electronic device comprising:

[0013] One or more processors;

[0014] Storage device for storing one or more programs.

[0015] When the one or more programs are executed by the one or more processors, the one or more processors implement the virtual object control method as described in the embodiments of this disclosure.

[0016] Fourthly, embodiments of this disclosure also provide a storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a control method for a virtual object as described in embodiments of this disclosure.

[0017] This disclosure provides a method, apparatus, device, and storage medium for controlling virtual objects. The virtual object is constructed based on a target object in an original image. Each virtual object includes multiple virtual parts, and each virtual part includes at least one virtual element. Facial and / or limb movements of the target object are acquired. The virtual parts in the virtual object are controlled to perform corresponding actions based on the facial and / or limb movements. The virtual object control method provided in this disclosure improves the flexibility of virtual object control by driving the facial and limb movements of a virtual object composed of multiple elements based on the facial and / or limb movements of the target object. Attached Figure Description

[0018] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.

[0019] Figure 1 This is a flowchart illustrating a method for controlling a virtual object provided in an embodiment of this disclosure;

[0020] Figure 2 This is an example diagram of a virtual object provided in an embodiment of this disclosure;

[0021] Figure 3 This is an example diagram of a parent-child relationship provided in an embodiment of this disclosure;

[0022] Figure 4 This is an example diagram of facial key points provided in an embodiment of this disclosure;

[0023] Figure 5 This is an example diagram illustrating how to determine the degree of eye opening provided in an embodiment of this disclosure;

[0024] Figure 6 This is an example diagram illustrating the control of eye movement of a virtual object according to an embodiment of this disclosure.

[0025] Figure 7 This is an example diagram illustrating the control of the eye movement of a virtual object according to an embodiment of this disclosure;

[0026] Figure 8 This is an example diagram illustrating how to determine the amplitude of eyebrow movement according to an embodiment of this disclosure;

[0027] Figure 9 This is an example diagram illustrating the control of eyebrow movement in a virtual object, provided in an embodiment of this disclosure.

[0028] Figure 10 This is an example diagram illustrating the control of virtual mouth movement provided in an embodiment of this disclosure;

[0029] Figure 11 This is an example diagram of the head movement of a virtual object provided in an embodiment of this disclosure;

[0030] Figure 12 This is an example diagram of limb skeletal points provided in an embodiment of this disclosure;

[0031] Figure 13 This is an example diagram of the arm movement of a virtual object provided in an embodiment of this disclosure;

[0032] Figure 14 This is a schematic diagram of the structure of a control device for a virtual object provided in an embodiment of this disclosure;

[0033] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Detailed Implementation

[0034] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0035] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0036] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0037] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0038] It should be noted that the terms "a" and "a plurality of" used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as "one or more".

[0039] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0040] It is understood that before using the technical solutions disclosed in the various embodiments of this disclosure, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in this disclosure in an appropriate manner in accordance with relevant laws and regulations, and user authorization should be obtained.

[0041] For example, upon receiving a user's active request, a prompt message is sent to the user to explicitly inform them that the requested operation will require the acquisition and use of the user's personal information. This allows the user to independently choose whether to provide personal information to the software or hardware, such as the electronic device, application, server, or storage medium performing the operations of this disclosed technical solution, based on the prompt message.

[0042] As an optional but non-limiting implementation, in response to a user's active request, sending a prompt message to the user can be done via a pop-up window, where the prompt message can be presented in text format. Furthermore, the pop-up window can also include a selection control allowing the user to choose "agree" or "disagree" to provide personal information to the electronic device.

[0043] It is understood that the above notification and user authorization process are merely illustrative and do not constitute a limitation on the implementation of this disclosure. Other methods that comply with relevant laws and regulations may also be applied to the implementation of this disclosure.

[0044] It is understood that the data involved in this technical solution (including but not limited to the data itself, the acquisition or use of the data) shall comply with the requirements of relevant laws, regulations and related provisions.

[0045] Figure 1 This is a flowchart illustrating a method for controlling a virtual object according to an embodiment of the present disclosure. The embodiments of the present disclosure are applicable to situations where virtual objects are controlled. The method can be executed by a control device for the virtual object, which can be implemented in the form of software and / or hardware, or optionally, by an electronic device, such as a mobile terminal, a PC, or a server.

[0046] like Figure 1 As shown, the method includes:

[0047] S110, construct a virtual object based on the target object in the original image.

[0048] The virtual object comprises multiple virtual parts, and each virtual part includes at least one virtual element. The virtual element is a virtual three-dimensional polyhedron, such as a virtual cuboid or a virtual triangular prism, assembled into a virtual object with a "building block" style. If the virtual object is a 3D object, then the virtual element is a 3D element. The virtual parts can be represented as components. The original image can be an image captured by the terminal's camera, or an authorized image stored locally or downloaded from a server. The target object can be a real person, a real animal, or a real plant, etc., without limitation. In this embodiment, a person is used as an example to illustrate the solution.

[0049] Specifically, the process of constructing a virtual object based on a target object in the original image can be as follows: identifying the set of target parts contained in the target object; obtaining a set of virtual parts with a corresponding set of styles based on the set of target parts; then obtaining the splicing method of each virtual part; and assembling the set of virtual parts based on the splicing method to obtain the virtual object. In this embodiment, when the virtual object is constructed, the identification information, index information, and component information of each virtual element constituting the virtual object are stored to facilitate the grouping of virtual elements. The virtual object can be in a "building block" style. For example, Figure 2 This is an example diagram of the virtual object in this embodiment, such as... Figure 2 As shown, the virtual object consists of multiple virtual parts, including: virtual hair, virtual head, virtual eyebrows, virtual eyes, virtual mouth, virtual upper limbs, virtual upper body, and virtual legs.

[0050] S120, acquire the facial and / or limb movements of the target object.

[0051] Facial movements can include the movement of the five facial features, such as the movement of eyebrows, the opening and closing of eyes, the opening and closing of the mouth, and the turning of the head. Limb movements can include the movement of the upper limbs, the movement of the upper body, and the movement of the legs.

[0052] In this embodiment, facial movements are represented by multiple frames of facial key point information, and limb movements are represented by multiple frames of limb skeletal point information.

[0053] Specifically, the method for acquiring the facial and / or limb movements of the target object can be as follows: A camera is used to capture images of the target object in real time, and the facial key point information and limb skeletal point information of the target object in each frame of the image are identified, thereby obtaining the facial and limb movements of the target object. The facial key point recognition can be implemented using any facial recognition algorithm; the limb skeletal point recognition can be implemented using any pose recognition algorithm, and no limitation is made here. The facial key points are represented by 2D coordinates, and the limb skeletal points are represented by 3D coordinates.

[0054] S130, control the virtual parts in the virtual object to perform corresponding actions based on facial and / or limb movements.

[0055] One method for controlling virtual parts in a virtual object to perform corresponding actions based on facial and / or limb movements is as follows: For each frame, obtain the facial key point information and / or limb skeletal point information of the target object in that frame, then determine the pose information of the corresponding virtual part in each frame based on the facial key point information and / or limb skeletal point information, and render the virtual object based on the pose information, thereby obtaining a video of the virtual object moving according to the action of the target object.

[0056] In this embodiment, the virtual parts include virtual eyes, virtual mouth, virtual eyebrows, virtual head, and virtual upper limbs. That is, only the virtual eyes, virtual mouth, virtual eyebrows, virtual head, and virtual upper limbs are controlled, and it is not necessary to control all virtual parts. This can greatly reduce the amount of computation and thus improve processing efficiency.

[0057] In this embodiment, the process of controlling virtual parts in a virtual object to perform corresponding actions based on facial and / or limb movements can be as follows: grouping virtual objects into parts to obtain multiple virtual parts; establishing parent-child relationships between multiple virtual parts; obtaining the target part action from facial and / or limb movements; and controlling the corresponding virtual part to perform the corresponding action based on the target part action and the parent-child relationship.

[0058] In this embodiment, the parent-child relationship is represented by virtual eyes, virtual mouth, and virtual eyebrows, all of which are child parts of the virtual head. Each virtual object is composed of multiple virtual elements, each belonging to a different virtual part. When grouping virtual objects, at least one of the following information can be used: identification information, index information, component information, and vertex position information.

[0059] Specifically, the process of grouping virtual objects into parts to obtain multiple virtual parts can be as follows: traverse the virtual elements that constitute the virtual object, and obtain at least one of the identification information, index information, and component information of the traversed virtual elements; if the identification information of the traversed virtual element is an eye, then group the traversed virtual element into a virtual eye; if the identification information of the traversed virtual element is an eyebrow, then group the traversed virtual element into a virtual eyebrow; if the index information of the traversed virtual element is a mouth, then group the traversed virtual element into a virtual mouth; if the component information of the traversed virtual element is an upper body component, then group the traversed virtual element into a virtual head.

[0060] The virtual eyes include a virtual left eye and a virtual right eye; the virtual eyebrows include a virtual left eyebrow and a virtual right eyebrow; and the virtual head includes virtual eyes, virtual eyebrows, and a virtual mouth.

[0061] The identification information can be the name of the virtual element. Each virtual element is named during the construction of the virtual object, so its identification information can be directly obtained. In this embodiment, the identification information of each virtual element is traversed. If the identification information contains fields related to "eyes," the virtual element is assigned to the virtual eye; if the identification information contains fields related to "eyebrows," the virtual element is assigned to the virtual eyebrow. For example, if the identification information of a virtual element contains the strings "u," "EYES," "ab," "i," and "1," the virtual element is assigned to the virtual left eye. If it contains the strings "u," "EYES," "ab," "i," and "2," the virtual element is assigned to the virtual right eye. The same method is used to obtain the virtual elements contained in the virtual eyebrow, which will not be elaborated further here.

[0062] In this embodiment, when constructing a virtual mouth, a square facet is created as the mouth of the virtual object. This standard square facet is set at a predetermined position, and its index information is recorded as "mouth". Therefore, if the index information of a virtual element is "mouth", the virtual element is grouped into the virtual mouth category.

[0063] In this embodiment, when constructing a virtual object, the virtual header is a component category, namely the upper body component. Therefore, virtual elements whose component information is the upper body component are directly assigned to the virtual header.

[0064] In this embodiment, virtual objects are grouped by part based on at least one of the identification information, index information, and component information of the virtual element. This can improve the accuracy of the grouping and facilitate accurate control of each virtual part in the future.

[0065] In this embodiment, the segmentation of the virtual upper limb needs to be based on the vertex position information of the virtual elements. Grouping the virtual object into parts to obtain multiple virtual parts can be achieved by: traversing the virtual elements constituting the virtual object and obtaining the vertex position information of the traversed virtual elements; determining whether the traversed virtual elements are within a defined bounding box based on the vertex position information; and grouping the traversed virtual elements into the virtual upper limb if they are within the defined bounding box.

[0066] The virtual upper limbs include a virtual left upper limb and a virtual right upper limb. The virtual element has multiple vertices. The bounding boxes can be defined as those constructed for the virtual left and right upper limbs during the virtual object creation process; the positions of these two bounding boxes are fixed.

[0067] The process of obtaining the vertex position information of the traversed virtual element can be as follows: first, obtain the coordinate information of the vertex of the traversed virtual element in the model coordinate system, and then transform the coordinate information to the world coordinate system.

[0068] Specifically, for the virtual left upper limb, if all vertices of the traversed virtual element fall within the bounding box corresponding to the virtual left upper limb, then the virtual element is assigned to the virtual left upper limb. For the virtual right upper limb, if all vertices of the traversed virtual element fall within the bounding box corresponding to the virtual right upper limb, then the virtual element is assigned to the virtual right upper limb. In this embodiment, since some virtual elements have chamfers, determining whether all vertices of the virtual element fall within the bounding box would result in high computational complexity. Therefore, if even one vertex of a virtual element falls within the bounding box, the virtual element is assigned to the virtual upper limb. This greatly reduces the computational load. This embodiment segments the virtual upper limbs using vertex position information, accurately separating the two virtual upper limbs.

[0069] In this embodiment, the parent-child relationship between the multiple virtual parts can be established based on the ownership relationship between each virtual part. For example, Figure 3 This is an example diagram of a parent-child relationship in this embodiment, such as... Figure 3 As shown, the eyes, eyebrows, and mouth are all located on the head. Therefore, the virtual eyes, virtual mouth, and virtual eyebrows are all considered as sub-parts of the virtual head. The virtual hands and the virtual head are independent of each other and have no parent-child relationship.

[0070] In this embodiment, acquiring the target body part movements in the facial and / or limb movements can be understood as: acquiring the movements of the target eyes, the target mouth, the target eyebrows, the target head, and the target upper limbs. Specifically, the target eye movements can be represented by multiple frames of eye keypoint information, the target mouth movements by multiple frames of mouth keypoint information, the target eyebrow movements by multiple frames of eyebrow keypoint information, the target head movements by multiple frames of head keypoint information, and the target upper limb movements by multiple frames of upper limb skeletal point information.

[0071] In this embodiment, the process of controlling the corresponding virtual part to perform corresponding actions based on the target part's action and parent-child relationship can be as follows: First, determine the pose information of the corresponding virtual part in each frame based on the target part's action. If the part has a parent part, then correct its position information based on the parent part's pose information, so that the child part moves with its parent part while moving itself. For example, the virtual eye is a child part of the virtual head. If the virtual head rotates, the virtual eye must rotate with the head while opening or closing. In this embodiment, controlling the corresponding virtual part to perform corresponding actions based on the target part's action and parent-child relationship can make the movement of the virtual object more realistic.

[0072] The technical solution of this disclosure embodiment constructs a virtual object based on a target object in an original image; wherein the virtual object includes multiple virtual parts, and each virtual part includes at least one virtual element; facial movements and / or limb movements of the target object are acquired; and the virtual parts in the virtual object are controlled to perform corresponding actions based on the facial movements and / or limb movements. The virtual object control method provided by this disclosure embodiment, which drives the facial and limb movements of a virtual object composed of multiple elements based on the facial movements and / or limb movements of the target object, can improve the flexibility of virtual object control.

[0073] Specifically, if the virtual parts are virtual eyes, virtual mouth, and virtual eyebrows, the method for controlling the corresponding virtual parts to perform corresponding actions based on the target part's movement and parent-child relationship can be as follows: For each frame, determine the movement amplitude of the target part based on the facial key points of the current frame; determine the correction coefficient based on the head's pose information; correct the movement amplitude based on the correction coefficient to obtain the initial pose offset; smooth the initial pose offset based on the historical pose offset to obtain the target pose offset; and adjust the pose information of the virtual parts based on the target pose offset.

[0074] The pose information includes at least one of translation, rotation, and scaling information, and the scaling information includes horizontal scaling sub-information and vertical scaling sub-information. Facial key points can be 106 facial key points, for example... Figure 4This is an example image of facial key points in this embodiment, such as... Figure 4 As shown, 106 points on the face were marked, with each key point corresponding to a different location on the face.

[0075] In this embodiment, for the virtual eyes, the range of motion is the degree of eye opening, which can be determined based on key eye points. For the virtual eyebrows, the range of motion is the degree of eyebrow movement, which can be determined based on key eyebrow points. For the virtual mouth, the range of motion is the degree of mouth opening, which can be determined based on key mouth points.

[0076] The head attitude information consists of roll, pitch, and yaw. The correction coefficients can be determined based on the head attitude information as follows: First, take the maximum of the absolute values ​​of the pitch and yaw angles. Then, normalize this maximum value to obtain a value between 0 and 1. Finally, process this normalized value using a set easing function (e.g., the easing in expo function) to obtain the correction coefficients.

[0077] In this embodiment, the method of correcting the motion amplitude based on the correction coefficient can be as follows: multiply the correction coefficient by the motion amplitude to obtain the initial pose offset. Adjusting the pose information of the virtual part according to the target pose offset can be understood as: accumulating the target pose offset with the current pose information.

[0078] Here, the historical pose offset can be understood as the target pose offset of a set number of historical frames. The process of smoothing the initial pose offset based on the historical pose offset can be: calculating the average of the historical pose offset and the initial pose offset to obtain the target pose offset.

[0079] In this embodiment, the motion amplitude is corrected based on the correction coefficient, and the initial pose offset is smoothed based on the historical pose offset, which can improve the accuracy of adjusting pose information and ensure that the virtual image's movements are more realistic.

[0080] Optionally, for virtual eyes, the method for determining the motion amplitude of the target part based on the facial key points of the current frame can be: determining the first reference information corresponding to the eyes based on the facial key points; determining the eye opening information based on the eye key points in the facial key points; and determining the eye opening amplitude based on the eye opening information and the first reference information.

[0081] The first reference information can be determined by key facial points located on the bridge of the nose, such as... Figure 4As shown, the facial key points located on the bridge of the nose are labeled 52, 53, 54, and 55. The first reference information is the distance between the geometric center points of 52 and 53 and the geometric center points of 54 and 55. The eye opening information can be characterized by the distance between the upper and lower eyelids, which can be represented by the distance between the center points of the upper and lower eyelids. For example, for the left eye, the center points of the upper eyelid are the geometric center points labeled 68, 69, and 70, and the center points of the lower eyelid are the center points labeled 73, 73, and 74; for the right eye, the center points of the upper eyelid are the geometric center points labeled 77, 78, and 79, and the center points of the lower eyelid are the center points labeled 81, 82, and 83. The method for determining the degree of eye opening based on the eye opening information and the first reference information can be: the ratio of the eye opening information to the first reference information can be used as the degree of eye opening. For example, Figure 5 This is an example diagram illustrating the determination of the degree of eye opening in this embodiment, such as... Figure 5 As shown, taking the left eye as an example, the length of vector pv1 represents the eye opening information, and the length of vector av1 represents the first reference information. In this embodiment, the ratio of the eye opening information to the first reference information is used as the degree of eye opening, which can correct the distance between the target object and the camera, thereby improving the accuracy of subsequent pose information adjustments.

[0082] In this embodiment, adjusting the pose information of the virtual part according to the target pose offset can be understood as adjusting the scaling information of each virtual element in the virtual eye according to the target pose offset. Specifically, the method of adjusting the pose information of the virtual part according to the target pose offset can be: determining whether the eye opening amplitude exceeds a first preset threshold; if the eye opening amplitude does not exceed the first preset threshold, then adjusting the vertical scaling sub-information of the virtual eye according to the target pose offset; if the eye opening amplitude exceeds the first preset threshold, then simultaneously adjusting the vertical scaling sub-information and the horizontal scaling sub-information of the virtual eye according to the target pose offset.

[0083] The vertical scaling sub-information can be understood as the y-component (i.e., the scaling amount along the y-direction), and the horizontal scaling sub-information can be understood as the x-component (i.e., the scaling amount along the x-direction). Before determining whether the eye opening amplitude exceeds a first preset threshold, an upper limit needs to be set on abnormal values ​​of the eye opening amplitude using the clamp function. This upper limit value is user-defined. The first preset threshold is less than this upper limit value.

[0084] In this embodiment, when the eye opening amplitude does not exceed a first preset threshold, the vertical scaling sub-information of the virtual eye is adjusted according to the target pose offset. Specifically, when the eye opening amplitude does not exceed the first preset threshold, the eye opening amplitude is first linearly mapped to a value within a first preset range. Then, the mapped value is corrected based on a correction coefficient. Next, the corrected value is smoothed based on historical pose offsets to obtain the target pose offset. Finally, the target pose offset is added to the vertical scaling sub-information of the virtual eye. For example, Figure 6 Example diagram for controlling the eye movement of a virtual object, such as Figure 6 As shown, compared to the right eye, the virtual object's left eye is magnified vertically.

[0085] In this embodiment, if the eye opening amplitude exceeds a first preset threshold, the vertical scaling sub-information and horizontal scaling sub-information of the virtual eye are simultaneously adjusted based on the target pose offset. Specifically, when the eye opening amplitude exceeds the first preset threshold, the eye opening amplitude is first linearly mapped to a value within a second preset interval. Then, the mapped value is corrected based on a correction coefficient. Next, the corrected value is smoothed based on historical pose offsets to obtain the target pose offset. Finally, the target pose offset is accumulated into the vertical scaling sub-information and the horizontal scaling information. The second preset interval is different from the first preset interval mentioned above. For example, Figure 7 Example diagram for controlling the eye movement of a virtual object, such as Figure 7 As shown, the left eye of the virtual object is magnified both vertically and horizontally compared to the right eye.

[0086] In this embodiment, determining the adjustment direction of eye scaling information based on the degree of eye opening can make the movement of the virtual object's eyes more realistic.

[0087] Optionally, for virtual eyebrows, the method for determining the motion amplitude of the target area based on the facial key points of the current frame can be: determining the second reference information corresponding to the eyebrows based on the facial key points; determining the eyebrow motion information based on the eyebrow key points in the facial key points; and determining the eyebrow motion amplitude based on the eyebrow information and the second reference information.

[0088] The second reference information can be determined by facial key points located on the nose, such as... Figure 4The facial key points located on the nose include: 57, 58, 55, 64, and 65. Eyebrow movement information can be represented by the direction vector of the eyebrows. For the virtual left eyebrow, the second reference information is the vector pointing from the geometric center points of points 55, 64, and 65 to the geometric center points of points 57, 58, and 55. The movement information of the left eyebrow is the vector pointing from the geometric center points of points 38, 39, and 40 to the geometric center points of points 35, 36, 41, and 42. For the virtual right eyebrow, the second reference information is the vector pointing from the geometric center points of points 55, 57, and 58 to the geometric center points of points 64, 65, and 55. The movement information of the right eyebrow is the vector pointing from the geometric center points of points 43, 51, and 50 to the geometric center points of points 45, 46, 48, and 49.

[0089] Specifically, the method for determining the eyebrow movement amplitude based on eyebrow movement information and second reference information can be as follows: calculate the angle between the vector corresponding to the eyebrow movement information and the vector corresponding to the second reference information, and use this angle as the eyebrow movement amplitude. For example, Figure 8 This is an example diagram illustrating the determination of eyebrow movement amplitude in this embodiment, such as... Figure 8 As shown, taking the left eyebrow as an example, vector pv2 represents eyebrow movement information, and vector av1 represents the second reference information. In this embodiment, the angle between the vector corresponding to the eyebrow movement information and the vector corresponding to the second reference information is used as the eyebrow movement amplitude, which can accurately control the eyebrow movement.

[0090] In this embodiment, adjusting the pose information of the virtual part based on the target pose offset can be understood as adjusting the translation and rotation information of each virtual element in the virtual eyebrow based on the target position offset. The method for adjusting the pose information of the virtual part based on the target pose offset can be: determining the translation and rotation offsets based on the target pose offset; adjusting the translation information of the virtual eyebrow based on the translation offset; and adjusting the rotation information of the virtual eyebrow based on the rotation offset.

[0091] Specifically, the target pose offset is converted according to a first predetermined mapping relationship to obtain a translation offset, and the target pose offset is converted according to a second predetermined mapping relationship to obtain a rotational translation. Finally, the translation offset is superimposed on the longitudinal component (i.e., the y-component) of the translation information, and the rotational translation is superimposed on the z-component (i.e., the rotation around the z-axis) of the rotation information. For example, Figure 9 This is an example diagram illustrating the control of eyebrow movement in a virtual object. For example... Figure 9 As shown, with Figure 2 In contrast, the virtual object's two eyebrows were translated upwards and rotated along the z-axis.

[0092] In this embodiment, the translation and rotation information of the virtual eyebrow are adjusted according to the target pose offset, so that the virtual eyebrow translates longitudinally and rotates along the z-axis, making the movement of the virtual object's eyebrow more realistic.

[0093] Optionally, for a virtual mouth, the method for determining the motion amplitude of the target part based on the facial key points of the current frame can be: determining the third reference information corresponding to the mouth based on the facial key points; determining the mouth opening information based on the mouth key points in the facial key points; and determining the mouth opening amplitude based on the mouth opening information and the third reference information.

[0094] The third reference information can be determined by facial key points located on the bridge of the nose, such as... Figure 4 As shown, the facial key points located on the bridge of the nose are labeled 52, 53, 54, and 55. The third reference information is the distance between the geometric center points of 52 and 53 and the geometric center points of 54 and 55. Mouth opening information can be represented by the distance between the upper and lower lips, which is represented by the distance between the center points of the upper and lower lips. The center points of the upper lip are represented by the geometric center points 98, 99, and 100, and the center points of the lower lip are represented by the geometric center points 102, 103, and 104. The method for determining the mouth opening amplitude based on the mouth opening information and the third reference information can be: the ratio between the mouth opening information and the third reference information can be used as the mouth opening amplitude. In this embodiment, using the ratio between the mouth opening information and the third reference information as the mouth opening amplitude can improve the accuracy of mouth movement control for the virtual object.

[0095] In this embodiment, adjusting the pose information of the virtual part according to the target pose offset can be understood as adjusting the scaling information of each virtual element in the virtual mouth according to the target pose offset. Specifically, the method for adjusting the pose information of the virtual part according to the target pose offset can be: determining whether the mouth opening exceeds a second preset threshold; if the mouth opening does not exceed the second preset threshold, adjusting the vertical scaling sub-information of the virtual eye according to the target pose offset; if the mouth opening exceeds the second preset threshold, simultaneously adjusting the vertical and horizontal scaling sub-information of the virtual eye according to the target pose offset.

[0096] Specifically, before determining whether the eye opening exceeds the second preset threshold, an upper limit needs to be set on abnormal values ​​of mouth opening using the clamp function. This upper limit is user-defined. The second preset threshold is a value less than this upper limit.

[0097] In this embodiment, when the mouth opening amplitude does not exceed the second preset threshold, the vertical scaling sub-information of the virtual mouth is adjusted according to the target pose offset. Specifically, when the mouth opening amplitude does not exceed the first preset threshold, the mouth opening amplitude is first linearly mapped to a value in a third preset interval. Then, the corrected value is smoothed based on historical pose offsets to obtain the target pose offset. Finally, the target pose offset is added to the vertical scaling sub-information of the virtual mouth.

[0098] In this embodiment, if the mouth opening exceeds a second preset threshold, the vertical scaling sub-information and horizontal scaling sub-information of the virtual mouth are adjusted simultaneously based on the target pose offset. Specifically, when the mouth opening exceeds the second preset threshold, the mouth opening is first linearly mapped to a value in a third preset interval. Then, the corrected value is smoothed based on historical pose offsets to obtain the target pose offset. Finally, the target pose offset is added to the vertical scaling sub-information and horizontal scaling information of the virtual mouth. The fourth preset interval is different from the third preset interval described above.

[0099] In this embodiment, the correction coefficient for the virtual mouth is 1, meaning there is no need to correct the mouth opening amplitude based on the head's posture information. For example, Figure 10 An example diagram for controlling the movement of a virtual mouth, with Figure 2 As shown, Figure 10 The mouth in the middle is open even wider.

[0100] In this embodiment, determining the adjustment direction of mouth scaling information based on the mouth opening amplitude can make the movement of the virtual object's mouth more realistic.

[0101] Optionally, if the virtual part is a virtual head, the way to control the corresponding virtual part to perform the corresponding action according to the target part's action and parent-child relationship can be as follows: for each frame, determine the pose information of the target head based on the facial key points of the current frame; smooth the pose information based on historical pose information to obtain the target pose information; and adjust the rotation information of the virtual head according to the target pose information.

[0102] The head pose information consists of roll, pitch, and yaw angles. Historical pose information can be understood as the target pose information in historical frames. Smoothing the pose information based on historical pose information can be achieved by calculating the average value between the historical pose information and the pose information in the current frame. Specifically, after obtaining the target head pose information, the pose information is first converted into three pose angles. Then, based on the historical pose information, each of the three pose angles is smoothed to obtain the target pose information. Finally, the target pose information is superimposed on the virtual head's rotation information. For example, Figure 11This is an example diagram of the head movement of the virtual object in this embodiment, such as... Figure 11 As shown, the virtual object tilts its head. In this embodiment, the pose information of the current frame is smoothed based on historical pose information, which makes the virtual object's head move more smoothly.

[0103] Optionally, if the virtual part is a virtual upper limb, the method of controlling the corresponding virtual part to perform the corresponding action according to the target part's action and parent-child relationship can be as follows: For each frame, determine the upper limb orientation information based on the upper limb bone points in the limb bone points of the current frame; determine the initial pose offset based on the upper limb orientation information and the set orientation information; smooth the initial pose offset based on the historical pose offset of the upper limb to obtain the target pose offset; and adjust the pose information of the virtual upper limb based on the target pose offset.

[0104] The upper limb skeletal points can include upper arm skeletal points and wrist skeletal points. The positional information of these skeletal points is represented by 3D coordinates. For example, Figure 12 This is an example diagram of limb skeletal points in this embodiment, such as... Figure 12 As shown, bone point 17 is the left upper arm bone point, and bone point 21 is the left wrist bone point; bone point 16 is the left upper arm bone point, and bone point 20 is the left wrist bone point. Vector L1 represents the orientation information of the left upper limb, and vector L2 represents the orientation information of the right upper limb.

[0105] The orientation information can be the direction vector of the upper limb when it is naturally hanging down, for example, [0,-1,0]. The pose offset can be understood as rotation information, which can be represented by quaternion information or Rogögs rotation information. The method for determining the initial pose offset based on the upper limb orientation information and the orientation information can be: determining the quaternion information from the vector corresponding to the upper limb orientation information to the vector corresponding to the orientation information. The method for smoothing the initial pose offset based on the historical pose offset of the upper limb can be: performing spherical linear interpolation processing on the initial pose offset based on the historical pose offset of the upper limb. For example, Figure 13 This is an example diagram of the arm movement of the virtual object in this embodiment, such as... Figure 13 As shown, with Figure 2 In contrast, the virtual object's left arm is bent and raised. In this embodiment, the upper limb movement is controlled based on the orientation information of the upper limb skeletal points, which can accurately control the upper limb of the virtual object, allowing the virtual object to simulate the movement of the target object.

[0106] In this embodiment, the movement of each virtual part can be performed simultaneously, or several virtual parts can move; this is not limited here.

[0107] Optionally, the method of controlling the corresponding virtual part to perform the corresponding action based on the target part's action and parent-child relationship can be: if no upper limb bone point of the target object is detected, control the virtual upper limb to move from the current pose state to the pose corresponding to the set orientation information with a certain step length.

[0108] Specifically, the process of controlling the virtual upper limb to move from its current pose to the pose corresponding to the set orientation information with a certain step size can be as follows: Spherical linear interpolation is performed between the current pose information and the set orientation information with a certain step size to obtain the pose information of the virtual upper limb in each frame during the movement, and the movement of the virtual upper limb is controlled based on this pose information. In this embodiment, when no upper limb skeletal points of the target object are detected, the virtual upper limb is controlled to gradually move back to a naturally drooping state, improving the flexibility of virtual object control.

[0109] Figure 14 This is a schematic diagram of a control device structure for a virtual object provided in an embodiment of the present disclosure, as shown below. Figure 14 As shown, the device includes:

[0110] The virtual object construction module 210 is used to construct a virtual object based on a target object in the original image; wherein the virtual object includes multiple virtual parts, and each virtual part includes at least one virtual element; the virtual element is a virtual three-dimensional polyhedron.

[0111] The motion acquisition module 220 is used to acquire facial and / or limb movements of the target object;

[0112] The virtual object control module 230 is used to control virtual parts in a virtual object to perform corresponding actions based on facial and / or limb movements.

[0113] Optionally, the virtual object control module 230 is also used for:

[0114] Group virtual objects by body parts to obtain multiple virtual body parts;

[0115] Construct parent-child relationships between multiple virtual parts;

[0116] Acquire target body movements from facial and / or limb movements;

[0117] Based on the target part's actions and parent-child relationships, the corresponding virtual parts are controlled to perform the appropriate actions.

[0118] Optionally, facial movements are represented by multiple frames of facial key point information; limb movements are represented by multiple frames of limb skeletal point information; virtual parts include virtual eyes, virtual mouth, virtual eyebrows, virtual head, and virtual upper limbs; parent-child relationships are defined as at least one of the virtual eyes, virtual mouth, and virtual eyebrows being a child part of the virtual head.

[0119] Optionally, the virtual object control module 230 is also used for:

[0120] Traverse the virtual components that constitute the virtual object and obtain at least one of the following: the identification information, the index information, and the component information to which the traversed virtual component belongs;

[0121] If the identifier of a virtual element encountered is an eye, then the encountered virtual elements are grouped into virtual eyes; wherein, virtual eyes include virtual left eye and virtual right eye;

[0122] If the identifier of the virtual element being traversed is an eyebrow, then the traversed virtual elements are grouped into virtual eyebrows; whereby virtual eyebrows include virtual left eyebrow and virtual right eyebrow.

[0123] If the index information of the virtual element being traversed is a mouth, then the traversed virtual elements are grouped into virtual mouths;

[0124] If the component information of the virtual element being traversed is the upper body component, then the traversed virtual element will be grouped into the virtual head; the virtual head includes virtual eyes, virtual eyebrows and virtual mouth.

[0125] Optionally, the virtual object control module 230 is also used for:

[0126] Traverse the virtual elements that make up the virtual object and obtain the vertex position information of the traversed virtual elements;

[0127] Determine whether the traversed virtual element is within the defined bounding box based on the vertex position information;

[0128] If a virtual element is within a defined bounding box, the traversed virtual elements will be grouped into virtual upper limbs.

[0129] Optionally, if the virtual parts are virtual eyes, virtual mouth, or virtual eyebrows, the virtual object control module 230 is also used for:

[0130] For each frame, determine the range of motion of the target area based on the facial key points of the current frame;

[0131] The correction coefficient is determined based on the head's posture information;

[0132] The motion amplitude is corrected based on the correction factor to obtain the initial pose offset;

[0133] The initial pose offset is smoothed based on the historical pose offset to obtain the target pose offset.

[0134] The pose information of the virtual part is adjusted according to the target pose offset; the pose information includes at least one of translation information, rotation information and scaling information, and the scaling information includes horizontal scaling sub-information and vertical scaling sub-information.

[0135] Optionally, for virtual eyes, virtual object control module 230:

[0136] Determine the first reference information corresponding to the eyes based on facial features;

[0137] Determine eye-opening information based on eye key points in facial key points;

[0138] The degree of eye opening is determined based on information about eye opening and the first reference information.

[0139] Optionally, the virtual object control module 230 is also used for:

[0140] Determine whether the degree of eye opening exceeds a first preset threshold;

[0141] If the eye opening does not exceed the first set threshold, the vertical scaling sub-information of the virtual eye is adjusted according to the target pose offset.

[0142] If the eye opening exceeds the first set threshold, the vertical scaling sub-information and horizontal scaling sub-information of the virtual eye are adjusted simultaneously according to the target pose offset.

[0143] Optionally, for virtual eyebrows, the virtual object control module 230 is also used for:

[0144] Determine the second reference information corresponding to the eyebrows based on facial features;

[0145] Eyebrow movement information is determined based on eyebrow key points in facial key points;

[0146] The amplitude of eyebrow movement is determined based on eyebrow movement information and a second reference.

[0147] Optionally, the virtual object control module 230 is also used for:

[0148] Determine the translational and rotational offsets based on the target pose offset;

[0149] The translation information of the virtual eyebrows is adjusted based on the translation offset.

[0150] The rotation information of the virtual eyebrows is adjusted based on the rotation offset.

[0151] Optionally, for the virtual mouth, the virtual object control module 230 is also used for:

[0152] Determine the third reference information corresponding to the mouth based on facial features;

[0153] Determine mouth opening information based on the mouth key point in the facial key points;

[0154] The degree of mouth opening is determined based on information about mouth opening and third reference information.

[0155] Optionally, the virtual object control module 230 is also used for:

[0156] Determine whether the mouth opening exceeds a second preset threshold;

[0157] If the mouth opening does not exceed the second set threshold, the vertical scaling sub-information of the virtual eye is adjusted according to the target pose offset.

[0158] If the mouth opening exceeds the second set threshold, the vertical scaling sub-information and horizontal scaling sub-information of the virtual eye are adjusted simultaneously according to the target pose offset.

[0159] Optionally, if the virtual part is a virtual head, the virtual object control module 230 is also used for:

[0160] For each frame, determine the pose information of the target head based on the facial key points of the current frame;

[0161] The target attitude information is obtained by smoothing the attitude information based on historical attitude information.

[0162] The rotation information of the virtual head is adjusted based on the target posture information.

[0163] Optionally, if the virtual part is a virtual upper limb, the virtual object control module 230 is also used for:

[0164] For each frame, determine the upper limb orientation information based on the upper limb bone points in the limb bone points of the current frame;

[0165] The initial pose offset is determined based on the upper limb orientation information and the set orientation information;

[0166] The initial pose offset is smoothed based on the historical pose offset of the upper limb to obtain the target pose offset.

[0167] The pose information of the virtual upper limb is adjusted based on the target pose offset.

[0168] Optionally, the virtual object control module 230 is also used for:

[0169] If no upper limb skeletal points of the target object are detected, the virtual upper limb is controlled to move from the current pose state to the pose corresponding to the set orientation information with a certain step length.

[0170] The virtual object control device provided in this disclosure can execute the virtual object control method provided in any embodiment of this disclosure, and has the corresponding functional modules and beneficial effects of the method execution.

[0171] It is worth noting that the various units and modules included in the above-mentioned device are only divided according to functional logic, but are not limited to the above division, as long as the corresponding functions can be realized; in addition, the specific names of each functional unit are only for easy differentiation and are not used to limit the protection scope of the embodiments of this disclosure.

[0172] Figure 15 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this disclosure. Reference is made below. Figure 15 It illustrates an electronic device suitable for implementing embodiments of the present disclosure (e.g., Figure 15 The diagram below shows the structure of the terminal device or server 500. The terminal device in this embodiment may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), and vehicle terminals (e.g., vehicle navigation terminals), as well as fixed terminals such as digital TVs and desktop computers. Figure 15 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments disclosed herein.

[0173] like Figure 15 As shown, electronic device 500 may include a processing unit (e.g., central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from storage device 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. An edit / output (I / O) interface 505 is also connected to bus 504.

[0174] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 15An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively.

[0175] In particular, according to embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of this disclosure include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a storage device 508, or installed from a ROM 502. When the computer program is executed by the processing device 501, it performs the functions defined in the methods of embodiments of this disclosure.

[0176] The names of messages or information exchanged between multiple devices in the embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of such messages or information.

[0177] The electronic device provided in this embodiment and the virtual object control method provided in the above embodiments belong to the same inventive concept. Technical details not described in detail in this embodiment can be found in the above embodiments, and this embodiment has the same beneficial effects as the above embodiments.

[0178] This disclosure provides a computer storage medium storing a computer program that, when executed by a processor, implements the virtual object control method provided in the above embodiments.

[0179] It should be noted that the computer-readable medium described in this disclosure can be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium can be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this disclosure, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in connection with an instruction execution system, apparatus, or device. In this disclosure, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals can take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0180] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and peer-to-peer networks (e.g., ad hoc peer-to-peer networks), as well as any currently known or future-developed networks.

[0181] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device.

[0182] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to:

[0183] The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: construct a virtual object based on a target object in an original image; wherein the virtual object includes multiple virtual parts, and each virtual part includes at least one virtual element; acquire facial and / or limb movements of the target object; and control the virtual parts in the virtual object to perform corresponding actions based on the facial and / or limb movements.

[0184] Computer program code for performing the operations of this disclosure can be written in one or more programming languages ​​or a combination thereof, including but not limited to object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0185] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0186] The units described in the embodiments of this disclosure can be implemented in software or in hardware. The name of a unit does not necessarily limit the unit itself; for example, the first acquisition unit can also be described as "a unit that acquires at least two Internet Protocol addresses".

[0187] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0188] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0189] The above description is merely a preferred embodiment of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features disclosed in this disclosure that have similar functions.

[0190] Furthermore, while the operations are described in a specific order, this should not be construed as requiring these operations to be performed in the specific order shown or in a sequential order. In certain environments, multitasking and parallel processing may be advantageous. Similarly, while several specific implementation details are included in the above discussion, these should not be construed as limiting the scope of this disclosure. Certain features described in the context of individual embodiments may also be implemented in combination in a single embodiment. Conversely, various features described in the context of a single embodiment may also be implemented individually or in any suitable sub-combination in multiple embodiments.

[0191] Although the subject matter has been described using language specific to structural features and / or methodological logic, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. Rather, the specific features and actions described above are merely illustrative examples of implementing the claims.

Claims

1. A method for controlling a virtual object, characterized in that, include: A virtual object is constructed based on the target object in the original image; wherein the virtual object includes multiple virtual parts, and each virtual part includes at least one virtual element; the virtual element is a virtual three-dimensional polyhedron; the virtual parts are obtained by grouping according to at least one of the following information of the virtual element: identification information, index information, component information, and vertex position information; Acquire the facial and / or limb movements of the target object; Control the virtual parts in the virtual object to perform corresponding actions based on the facial and / or limb movements; The method of controlling virtual parts of the virtual object to perform corresponding actions based on facial and / or limb movements includes: The virtual object is grouped by parts to obtain multiple virtual parts; Construct parent-child relationships among the multiple virtual parts; Acquire the target body part movement from the facial and / or limb movements; Based on the target part's action and the parent-child relationship, the corresponding virtual part is controlled to perform the corresponding action.

2. The method according to claim 1, characterized in that, The facial movements are represented by multiple frames of facial key point information; the limb movements are represented by multiple frames of limb skeletal point information; the virtual parts include virtual eyes, virtual mouth, virtual eyebrows, virtual head, and virtual upper limbs; the parent-child relationship is that at least one of the virtual eyes, virtual mouth, and virtual eyebrows is a child part of the virtual head.

3. The method according to claim 2, characterized in that, The virtual object is grouped by body parts to obtain multiple virtual body parts, including: Traverse the virtual elements that constitute the virtual object and obtain at least one of the following: identification information, index information, and component information of the traversed virtual elements; If the identifier of a virtual element encountered is an eye, then the encountered virtual elements are grouped into virtual eyes; wherein, the virtual eyes include a virtual left eye and a virtual right eye; If the identifier of a virtual element encountered is an eyebrow, then the encountered virtual elements are grouped into virtual eyebrows; wherein, the virtual eyebrows include a virtual left eyebrow and a virtual right eyebrow; If the index information of the virtual element being traversed is a mouth, then the traversed virtual elements are grouped into virtual mouths; If the component information of the virtual element being traversed is an upper body component, then the traversed virtual element is grouped into a virtual head; wherein, the virtual head includes the virtual eyes, virtual eyebrows and virtual mouth.

4. The method according to claim 2, characterized in that, The virtual object is grouped by body parts to obtain multiple virtual body parts, including: Traverse the virtual elements that constitute the virtual object and obtain the vertex position information of the traversed virtual elements; Determine whether the traversed virtual element is within the set bounding box based on the vertex position information; If the virtual element is within the defined bounding box, the traversed virtual elements are grouped into virtual upper limbs.

5. The method according to claim 2, characterized in that, If the virtual part is a virtual eye, virtual mouth, or virtual eyebrow, then the corresponding virtual part is controlled to perform corresponding actions based on the target part's action and the parent-child relationship, including: For each frame, determine the range of motion of the target area based on the facial key points of the current frame; The correction coefficient is determined based on the head's posture information; The motion amplitude is corrected based on the correction coefficient to obtain the initial pose offset. The initial pose offset is smoothed based on the historical pose offset to obtain the target pose offset. The pose information of the virtual part is adjusted according to the target pose offset; the pose information includes at least one of translation information, rotation information and scaling information, and the scaling information includes horizontal scaling sub-information and vertical scaling sub-information.

6. The method according to claim 5, characterized in that, For virtual eyes, the range of motion of the target area is determined based on the facial key points of the current frame, including: Determine the first reference information corresponding to the eyes based on the facial key features; Determine eye-opening information based on eye key points in facial key points; The degree of eye opening is determined based on the eye opening information and the first reference information.

7. The method according to claim 6, characterized in that, Adjusting the pose information of the virtual part based on the target pose offset includes: Determine whether the degree of eye opening exceeds a first preset threshold; If the eye opening amplitude does not exceed the first preset threshold, the vertical scaling sub-information of the virtual eye is adjusted according to the target pose offset. If the eye opening exceeds a first preset threshold, the vertical scaling sub-information and horizontal scaling sub-information of the virtual eye are adjusted simultaneously according to the target pose offset.

8. The method according to claim 5, characterized in that, For virtual eyebrows, the range of motion of the target area is determined based on the facial key points of the current frame, including: Determine the second reference information corresponding to the eyebrows based on the facial features; Eyebrow movement information is determined based on eyebrow key points in facial key points; The eyebrow movement amplitude is determined based on the eyebrow movement information and the second reference information.

9. The method according to claim 8, characterized in that, Adjusting the pose information of the virtual part based on the target pose offset includes: The translational offset and rotational offset are determined based on the target pose offset. The translation information of the virtual eyebrows is adjusted based on the translation offset. The rotation information of the virtual eyebrow is adjusted based on the rotation offset.

10. The method according to claim 5, characterized in that, For a virtual mouth, the range of motion of the target area is determined based on the facial key points of the current frame, including: Determine the third reference information corresponding to the mouth based on the facial key features; Determine mouth opening information based on the mouth key point in the facial key points; The mouth opening range is determined based on the mouth opening information and the third reference information.

11. The method according to claim 10, characterized in that, Adjusting the pose information of the virtual part based on the target pose offset includes: Determine whether the mouth opening exceeds a second preset threshold; If the mouth opening does not exceed the second preset threshold, the vertical scaling sub-information of the virtual eye is adjusted according to the target pose offset. If the mouth opening exceeds the second preset threshold, the vertical scaling sub-information and horizontal scaling sub-information of the virtual eye are adjusted simultaneously according to the target pose offset.

12. The method according to claim 1, characterized in that, If the virtual part is a virtual head, then the corresponding virtual part is controlled to perform corresponding actions based on the target part's action and the parent-child relationship, including: For each frame, determine the pose information of the target head based on the facial key points of the current frame; The attitude information is smoothed based on historical attitude information to obtain the target attitude information; The rotation information of the virtual head is adjusted based on the target posture information.

13. The method according to claim 2, characterized in that, If the virtual body part is a virtual upper limb, then the corresponding virtual body part is controlled to perform corresponding actions based on the target body part's action and the parent-child relationship, including: For each frame, determine the upper limb orientation information based on the upper limb bone points in the limb bone points of the current frame; The initial pose offset is determined based on the upper limb orientation information and the set orientation information; The initial pose offset is smoothed based on the historical pose offset of the upper limb to obtain the target pose offset. The pose information of the virtual upper limb is adjusted based on the target pose offset.

14. The method according to claim 13, characterized in that, Based on the target part's action and the parent-child relationship, the corresponding virtual part is controlled to perform corresponding actions, including: If no upper limb skeletal points of the target object are detected, the virtual upper limb is controlled to move from the current pose state to the pose corresponding to the set orientation information with a certain step size.

15. A control device for a virtual object, characterized in that, The apparatus is used to execute the control method for the virtual object according to any one of claims 1-14, comprising: A virtual object construction module is used to construct virtual objects based on target objects in the original image; wherein, the virtual object includes multiple virtual parts, and each virtual part includes at least one virtual element; the virtual element is a virtual three-dimensional polyhedron; the virtual parts are obtained by grouping according to at least one of the following information of the virtual element: identification information, index information, component information, and vertex position information; An action acquisition module is used to acquire facial and / or limb movements of the target object; The virtual object control module is used to control virtual parts in the virtual object to perform corresponding actions based on the facial and / or limb movements.

16. An electronic device, characterized in that, The electronic device includes: One or more processors; Storage device for storing one or more programs. When the one or more programs are executed by the one or more processors, the one or more processors implement the control method for the virtual object as described in any one of claims 1-14.

17. A storage medium containing computer-executable instructions, which, when executed by a computer processor, are used to perform a method for controlling a virtual object as described in any one of claims 1-14.

Images