Execution optimization method and device for agent expression simulation, equipment and medium

Abstract

The embodiment of the invention provides an execution optimization method for agent expression simulation, which can be applied to the technical field of artificial intelligence. The execution optimization method for agent expression simulation comprises the following steps: generating an inverse kinematics target optimization task based on control point simulation information and target epidermis vertex information of an agent expression grid; performing optimization processing on the inverse kinematics target optimization task to obtain an optimal joint angle parameter corresponding to the control point simulation information; and performing execution optimization of agent expression simulation according to the optimal joint angle parameter. The embodiment of the invention further provides an execution optimization device and equipment for agent expression simulation, a storage medium and a program product.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, specifically to the field of image processing technology, and more specifically to an execution optimization method, device, equipment, medium and product for intelligent body expression imitation. Background Art

[0002] Artificial Intelligence (AI) is an important driving force for the new round of scientific and technological revolution and industrial transformation. It is a new key technical science that studies and develops theories, methods, technologies and application systems for simulating, extending and expanding human intelligence. As an important part of intelligent science, artificial intelligence attempts to understand the essence of intelligence and produce a new type of intelligent machine that can respond in a similar way to human intelligence (such as a highly bionic expression robot).

[0003] As an important part of artificial intelligence technology, the recognition and reproduction of human expressions (such as expression imitation) has received widespread attention in the fields of human-computer interaction, security identification, robot manufacturing, automation, medical treatment, communication and driving, and has quickly become a research hotspot in the industry. Expression imitation technology is mainly divided into three stages: expression perception, mapping transfer and drive execution. It aims to accurately capture the dynamic features of human facial expressions in the expression perception stage, realize the mapping of embodied intelligent faces such as highly bionic expression robots in the mapping transfer stage, and finally control the expression actuator in the drive execution stage to achieve accurate expression reproduction drive. However, the traditional expression imitation technology in the prior art has problems such as difficulty in ensuring the accuracy of inverse kinematics solution in the drive execution stage, failure to fully consider the physical constraints of the robot actuator, and poor expression control accuracy and stability. Summary of the invention

[0004] In view of at least one of the technical problems existing in the above-mentioned execution scheme of expression imitation, the embodiments of the present invention provide an execution optimization method, device, equipment, medium and product for intelligent body expression imitation, so that in the expression-driven execution stage, the physical constraints of the intelligent body actuator can be explicitly modeled to optimize the expression-driven parameters, thereby ensuring that the generated control instructions are both physically executable and meet the requirements of high precision and real-time performance.

[0005] One aspect of an embodiment of the present invention provides an execution optimization method for intelligent agent expression imitation, which includes: generating an inverse kinematics target optimization task based on control point simulation information and target surface vertex information of the intelligent agent expression mesh; performing optimization processing on the inverse kinematics target optimization task to obtain optimal joint angle parameters corresponding to the control point simulation information; and performing execution optimization of the intelligent agent expression imitation according to the optimal joint angle parameters.

[0006] According to one embodiment of the present invention, before generating the inverse kinematics target optimization task based on the control point simulation information and the target epidermal vertex information of the agent expression mesh, it also includes: generating an agent expression mesh corresponding to the agent based on the mixed shape basis of the agent's face; generating a distance weight matrix based on the control point simulation information and the target epidermal vertex information of the agent expression mesh.

[0007] According to one embodiment of the present invention, before generating a distance weight matrix based on the control point simulation information and the target epidermal vertex information of the intelligent agent expression mesh, it also includes: generating control point simulation information according to a preset positive kinematics optimization task of the intelligent agent expression mesh and a joint angle parameter vector corresponding to the simulation control point; generating target epidermal vertex information according to the deformation weight matrix of the epidermal mesh vertices of the intelligent agent expression mesh and the control point simulation information.

[0008] According to one embodiment of the present invention, in generating an inverse kinematics target optimization task based on control point simulation information and target epidermal vertex information of an intelligent agent expression mesh, it includes: extending a distance weight matrix into a weight amplification matrix; generating control point driven epidermal mesh vertices according to the control point simulation information and the weight amplification matrix; generating an inverse kinematics target optimization task according to the control point driven epidermal mesh vertices and target epidermal vertex information.

[0009] According to one embodiment of the present invention, in performing optimization processing on the inverse kinematics target optimization task to obtain the optimal joint angle parameters corresponding to the control point simulation information, it includes: performing minimization target optimization iterative processing on the inverse kinematics target optimization task according to the joint angle constraint conditions to generate the optimal joint angle parameters.

[0010] According to one embodiment of the present invention, in the execution optimization of intelligent body expression imitation according to the optimal joint angle parameters, it includes: obtaining the control point driving information of the intelligent body expression imitation through the control point simulation information and the optimal joint angle parameters; executing the driving of the intelligent body expression imitation according to the control point driving information to complete the execution optimization.

[0011] Another aspect of the embodiments of the present invention provides an execution optimization device for imitating the expression of an intelligent body, which includes a target task generation module, a task optimization module and an imitation execution module. The target task generation module is used to generate an inverse kinematics target optimization task based on the control point simulation information and the target surface vertex information of the intelligent body expression grid; the task optimization module is used to perform optimization processing on the inverse kinematics target optimization task to obtain the optimal joint angle parameters corresponding to the control point simulation information; and the imitation execution module is used to perform execution optimization of the imitation of the expression of the intelligent body according to the optimal joint angle parameters.

[0012] Another aspect of an embodiment of the present invention provides an electronic device, comprising one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned execution optimization method of intelligent agent expression imitation.

[0013] Another aspect of an embodiment of the present invention provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned execution optimization method for imitating the expression of an intelligent agent.

[0014] Another aspect of an embodiment of the present invention provides a computer program product, including a computer program, which, when executed by a processor, implements the above-mentioned execution optimization method of intelligent agent expression imitation.

[0015] The execution optimization method for agent expression imitation provided by the embodiment of the present invention can at least partially solve the problems in the related art, and thus can achieve at least one of the following technical effects: Through the execution optimization method of the above-mentioned intelligent body expression imitation of Example 3 of the present invention, the optimal position of the control point under the target expression can be solved by defining the control point area, using inverse kinematics, and generating accurate actuator motion instructions, so that the motion state of the control point area is close to the target epidermal movement. In this case, the fineness and naturalness of expression restoration can be further improved by simply increasing the density of control points without changing the algorithms in the perception and mapping stages. Therefore, not only the accuracy of expression imitation is greatly improved, but also the physical executable nature of the generated instructions is ensured by explicit modeling of the physical constraints of the actuator, avoiding the execution failure problem caused by ignoring physical constraints in traditional methods. In addition, the independence of control point optimization also allows the execution stage to flexibly adapt to different types of intelligent body facial structures, providing technical guarantees for the scalability and modular design of the system, and has extremely high engineering application value and commercial application value.

[0016] It should be understood that the above general description and the following detailed description are merely exemplary and illustrative and are not intended to limit the scope of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The above contents and other objects, features and advantages of the present invention will become more apparent through the following description of the embodiments of the present invention with reference to the accompanying drawings, in which: Figure 1 Schematically showing the application scenario diagram of the optimization method, device, equipment, medium and program product for agent expression imitation according to embodiments 1-3 of the present invention; Figure 2AA flowchart of a method for optimizing the perception of an intelligent agent's facial expression imitation according to Embodiment 1 of the present invention is schematically shown; Figure 2B The structure block diagram of the perception optimization device for imitating the expression of an intelligent agent according to Embodiment 1 of the present invention is schematically shown; Figure 3A A flowchart of a method for optimizing the mapping of an intelligent agent's facial expression imitation according to Embodiment 2 of the present invention is schematically shown; Figure 3B A schematic block diagram of a mapping optimization device for agent expression imitation according to Embodiment 2 of the present invention is shown; Figure 4A A flowchart of an execution optimization method for agent expression imitation according to Embodiment 3 of the present invention is schematically shown; Figure 4B A schematic diagram showing a structural block diagram of an execution optimization device for agent expression imitation according to Embodiment 3 of the present invention; Figure 5A Schematically showing the expression semantic alignment diagram from the human face expression mixed space to the robot expression mixed space in Embodiments 1-3 of the present invention; Figure 5B Schematically showing the kinematic design diagram of the robot expression in Embodiment 1-3 of the present invention; Figure 5C Schematically showing the electromechanical design diagram of the robot expression in Embodiments 1-3 of the present invention; and Figure 6 The block diagram of an electronic device according to the method of embodiments 1 to 3 of the present invention is schematically shown.

[0018] The above-mentioned drawings are part of the specification of the embodiments of the present invention, which illustrate exemplary embodiments of the present invention. The attached drawings and the description of the specification are used together to illustrate the principles of the embodiments of the present invention. It should be understood that the above general description of the drawings and the following specific implementations are only exemplary and illustrative, and they cannot limit the scope of the present invention. DETAILED DESCRIPTION

[0019] In order to make the objectives, technical solutions and advantages of the embodiments of the present invention more clearly understood, the spirit of the contents disclosed by the present invention will be clearly explained with the accompanying drawings and detailed descriptions below. After understanding the embodiments of the contents of the present invention, any technician in the relevant technical field can change and modify the techniques taught by the contents of the present invention without departing from the spirit and scope of the contents of the present invention.

[0020] The exemplary embodiments and descriptions of the present invention are used to explain the present invention, but are not intended to limit the present invention. In addition, elements / components with the same or similar reference numerals used in the drawings and embodiments are used to represent the same or similar parts.

[0021] The terms “first”, “second”, etc. used in the present invention do not particularly refer to an order or sequence, nor are they used to limit the present invention. They are only used to distinguish elements or operations described with the same technical terms.

[0022] The directional terms used in the present invention, such as up, down, left, right, front or back, etc., are only used to refer to the directions of the drawings. Therefore, the directional terms used are used to illustrate and not to limit the present invention.

[0023] The words “include,” “including,” “have,” “contain,” etc. used in the present invention are open-ended terms, meaning including but not limited to.

[0024] The term "and / or" used in the present invention includes any or all combinations of items.

[0025] Regarding the present invention, "plurality" includes "two" and "more than two"; regarding the present invention, "plurality of groups" includes "two groups" and "more than two groups".

[0026] The terms "substantially" and "approximately" used in the present invention are used to modify any quantity or error that may vary slightly, but these slight changes or errors do not change their essence. Generally speaking, the range of slight changes or errors modified by such terms may be 20% in some embodiments, 10% in some embodiments, 5% in some embodiments, or other values. Those skilled in the art should understand that the aforementioned values ​​can be adjusted according to actual needs and are not limited thereto.

[0027] All terms (including technical and scientific terms) used herein have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0028] In the case of using expressions such as "at least one of A, B, and C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, and C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). In the case of using expressions such as "at least one of A, B, or C, etc.", it should generally be interpreted in accordance with the meaning of the expression generally understood by those skilled in the art (for example, "a system having at least one of A, B, or C" should include but not be limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or a system having A, B, C, etc.). Those skilled in the art should also understand that any transitional conjunctions and / or phrases that substantially represent two or more optional items, whether in the specification, claims, or drawings, should be understood to give the possibility of including one of these items, either of these items, or both of these items. For example, the phrase "A or B" should be understood to include the possibilities of "A" or "B", or "A and B".

[0029] In addition, all actions of acquiring information, signals or data in the present invention are carried out in compliance with the corresponding data protection laws, regulations and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0030] In interpersonal communication in daily human life, the communication effect can be enhanced by controlling facial expressions. For humans, facial expressions are the result of one or more movements or states of facial muscles. These movements express the emotional state of the human individual to the external environment. Facial expressions are a form of non-verbal communication, and it is the main means of expressing social information between humans.

[0031] For artificial intelligence, in order to achieve the goal that intelligent machines can make real human interactions in a similar way to human intelligence, it is necessary to consider creating a real humanoid machine from all aspects, including limbs, appearance, posture, expression, even voice dialogue, action ability and human-like thinking mode. Among them, expression imitation technology provides strong technical support for today's humanoid robots to enter the real world and provide emotional value and non-verbal power for humans.

[0032] Facial expression imitation technology aims to achieve accurate mapping and physical driving of highly bionic facial expressions into robots by accurately capturing the dynamic features of human facial expressions. Its overall technical framework covers three links: expression perception, mapping migration, and drive execution. Problems in any link will directly limit the realization of high-precision facial expression imitation.

[0033] In order to solve at least one of the prior art problems existing in the three links of expression perception, mapping migration and drive execution in the process of realizing intelligent agent expression imitation in the prior art, the present invention provides embodiments 1-3 to perform targeted optimization on the above three links of the process of realizing intelligent agent expression imitation, in order to significantly improve the intelligence level of intelligent agent expression imitation.

[0034] Figure 1 The application scenario diagram schematically shows the optimization method, device, equipment, medium and program product for intelligent agent expression imitation according to embodiments 1-3 of the present invention.

[0035] like Figure 1 As shown, the application scenario 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for a communication link between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links or optical fiber cables, etc.

[0036] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social platform software, etc. (only examples).

[0037] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0038] The server 105 may be a server that provides various services, such as a background management server (only as an example) that provides support for websites browsed by users using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal device.

[0039] It should be noted that the optimization method for the imitation of the expression of the intelligent agent provided in the embodiment of the present invention can generally be executed by the server 105. Accordingly, the optimization device for the imitation of the expression of the intelligent agent provided in the embodiment of the present invention can generally be set in the server 105. The optimization method for the imitation of the expression of the intelligent agent provided in the embodiment of the present invention can also be executed by a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the optimization device for the imitation of the expression of the intelligent agent provided in the embodiment of the present invention can also be set in a server or a server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0040] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to the implementation requirements.

[0041] Among them, the optimization method for intelligent body expression imitation provided by the embodiment of the present invention may include at least one of the above-mentioned perception optimization method, mapping optimization method and execution optimization method, and the optimization device for intelligent body expression imitation provided by the embodiment of the present invention may include at least one of the above-mentioned perception optimization device, mapping optimization device and execution optimization device.

[0042] In order to further reflect the complete technical process of the three key stages of perception, mapping and execution in the imitation of intelligent body expression, and to ensure that the full-link expression imitation from the original face input to the accurate reproduction of the robot expression can be achieved, the following is combined with the embodiments 1-3 of the present invention and Figure 1-5C Further explanation is as follows: It should be noted that the intelligent agent may be the execution subject of at least one of the perception optimization method, mapping optimization method and execution optimization method provided in the above-mentioned embodiments 1-3 of the present invention, or may be the execution party controlled by these methods, such as a humanoid intelligent robot or other AI device with a simulated human face skin that can imitate human facial expressions. Specifically, Figure 5B and Figure 5C The electromechanical design of the intelligent body with facial expressions shown in the figure covers the surface with "muscle" and "skeleton" with a simulated skin (silicone mask, i.e. skin). Thus, the movement of the simulated skin can be realized through the motor (such as LFD-01M servomotor) and steel cable and Teflon conduit matched with the electrical control system, thereby achieving the imitation of facial expressions.

[0043] The following will be combined Figure 2A-Figure 5C The perception optimization method, mapping optimization method and execution optimization method provided in Examples 1-3 of the present invention are described in detail respectively.

[0044] Example 1

[0045] The expression perception stage involves facial expression recognition. Achieving accurate expression recognition perception, obtaining more complex expression descriptions, and ensuring the accuracy and versatility of expression perception are technical issues that need to be urgently resolved in the embodiments of the present invention.

[0046] Specifically, in the perception stage, traditional methods usually use sparse landmarks to perform structured representation of faces. This method relies on the position and displacement of landmarks to describe the dynamic changes of facial expressions. Although simple and efficient, its main problem is that the complexity of expression description is insufficient. Since sparse landmarks can only capture limited local dynamic features, they cannot fully represent the subtle changes and complex characteristics of human facial expressions, resulting in oversimplification of expression description. In addition, there is a coupling relationship between face shape and expression semantics, which makes it impossible for the same geometric displacement to maintain consistent expression semantics in individuals with different face shapes. This coupling further limits the accuracy and versatility of expression perception.

[0047] Therefore, in view of at least one of the technical problems existing in the above-mentioned prior art, the embodiments of the present invention aim to provide a perception optimization method, device, equipment, medium and product for intelligent body expression imitation, thereby providing a perception stage optimization scheme for intelligent body expression imitation, in order to achieve effective decoupling of individual facial features and expression dynamics, significantly improve the ability to capture subtle expressions through dense expression representation, and overcome the limitations of traditional schemes.

[0048] The following will be based on Figure 1 The scene described by Figure 2A to Figure 5C The perceptual optimization method for intelligent agent expression imitation in the disclosed embodiment is described in detail.

[0049] like Figure 2A As shown, one aspect of an embodiment of the present invention provides a perceptual optimization method for intelligent agent expression imitation, which includes operations S201 to S204.

[0050] In operation S201, facial expression motion information of a target facial expression image is extracted; In operation S202, a dense face reconstruction task is generated by using the expression motion information and the preset joint optimization information; In operation S203, performing joint optimization on the dense face reconstruction task to generate three-dimensional face reconstruction information, and the three-dimensional face reconstruction information is used to generate a target three-dimensional face; and In operation S204, dense expression motion information of the standardized average face is extracted according to the target three-dimensional human face to achieve perceptual optimization.

[0051] In the embodiment of the present invention, the intelligent agent expression imitation can be that the intelligent agent obtains a facial expression image by recognizing and using the above-mentioned facial expression in the facial expression image to imitate the facial expression of the intelligent agent. Figure 5C The process of reproducing the electromechanical design of the object. Among them, the facial expression image as the object to be recognized can be the image information directly detected and obtained by the intelligent agent through its own image detection equipment (such as camera image sensor, etc.), or it can be the image information transmitted to the intelligent agent through network transmission equipment (wireless or wired), and the image information can be video or picture data.

[0052] For example, a humanoid robot, as a type of intelligent entity, conducts a video call with a real human user. The humanoid robot can directly or indirectly obtain the video data of the video call, pre-process the video content of the human user in the video data, and extract the expression features of each frame of the image containing the human user's expression in the video content. Among them, the target expression image can be one of these frame images, and the single frame image contains the facial expression of the expression imitation object (human user) at a certain moment. In addition, the humanoid robot can also directly interact with the human user face to face, and obtain and recognize the real-time expression of the human user in real time.

[0053] The expression motion information may be facial expression feature parameters extracted from the facial expression in the target expression image, and these expression feature parameters may involve expression base parameters and facial expression motion parameters (such as facial shape coefficients, dynamic expression motion coefficients, and facial posture coefficients in the camera coordinate system) of the target facial expression in the target expression image. The so-called expression base parameters may include more (such as 50) different expression bases, wherein each expression base coefficient represents the displacement of each corner point on the dense mesh of the standardized average face of the current facial image; in addition, the facial expression motion parameters are motion field parameters relative to the standardized average face in the standardized average face coordinate system.

[0054] The preset joint optimization information may be information for performing three-dimensional face reconstruction based on the standardized average face and expression motion information according to the target three-dimensional face reconstruction requirements, wherein the target three-dimensional face reconstruction requirements are also related to the target three-dimensional face reconstruction model adopted by the three-dimensional face reconstruction. The standardized average face is the average shape of a three-dimensional face obtained through statistical learning (i.e., a three-dimensional average face, Mean Shape). The preset joint optimization information is mainly used to generate a dense face reconstruction task based on the expression motion information extracted from the target expression image. The dense face reconstruction task may be a joint optimization task for restoring three-dimensional face shape and expression motion parameters according to the target three-dimensional face reconstruction requirements, for example, it may be embodied as a joint optimization problem created for three-dimensional face shape recovery.

[0055] By performing joint optimization processing on the dense face reconstruction task, three-dimensional face reconstruction information can be generated according to the processing result of the dense face reconstruction task, wherein the three-dimensional face reconstruction information may include the corresponding facial expression base And the corresponding facial expression coefficient The facial expression base can usually be used to describe the dynamic expression changes of the face, and the facial expression coefficient can usually be used as a parameterized representation of the facial expression. The facial expression base and the facial expression coefficient can be configured to construct a three-dimensional facial expression model based on a standardized average face.

[0056] The target 3D face can be understood as a 3D facial expression model based on a standardized average face. For example, a 3D morphable model (such as a 3D Morphable Model, 3DMM for short) can be a statistical model for 3D face modeling. It mainly learns a large amount of 3D face data and represents any 3D face as a combination of linear bases. It can be represented by a 3D face shape based on a standardized average face. The 3D face shape can be represented by the standardized average face. (i.e. three-dimensional average face, Mean Shape), face shape base 、Facial expression base , face shape coefficient , facial expression coefficient Therefore, the facial expression in the two-dimensional image can be mapped into a parameterized three-dimensional representation, and with the help of its ability to decompose the face shape, expression and posture space, the effective decoupling of individual facial features and expression dynamics can be achieved.

[0057] Among them, dense expression motion information can be information related to the dynamic changes of facial expressions that are directly related to the extracted expression motion information, and can be specifically represented by a linear combination of a facial expression base and a facial expression coefficient. Dense expression motion information can most directly reflect the dense expression motion information of the user extracted by the agent on the standardized average face, thereby realizing facial expression representation.

[0058] Therefore, in the perception stage, compared with the traditional sparse key point expression representation scheme, the perception optimization method of the above-mentioned intelligent agent expression imitation of the embodiment of the present invention can map the facial expression in the two-dimensional image into a parameterized three-dimensional representation by adopting the target three-dimensional face to realize the representation of the facial expression motion information. With the help of its ability to decompose the face shape, expression and posture space, it can realize the effective decoupling of individual facial features and expression dynamics, and generate a dense expression motion under the standardized average face topology, thereby overcoming the abstract defects of the traditional sparse key point scheme and realizing the densification of expression description. At the same time, by normalizing the expression to the standardized average face, the coupling problem between the expression semantics and the individual face shape is eliminated, ensuring the consistency of the expression semantics in individuals with different face shapes. Therefore, with the help of the high precision and density of the three-dimensional facial expression model based on the standardized average face of the above-mentioned target three-dimensional face, the accuracy and delicacy of expression perception can be significantly improved, and the ability to capture subtle expressions can be improved, thereby laying a high-quality foundation for the subsequent mapping and execution stages.

[0059] like Figure 2A-Figure 5C As shown, according to an embodiment of the present invention, in operation S201 , extracting the expression motion information of the target expression image includes: performing expression feature recognition on the target expression image to extract the expression motion information.

[0060] The target expression image can be a single-frame two-dimensional image (such as an RGB image), which can be selected from a time-series-based two-dimensional image set (such as a video), and can be selected one by one according to the time sequence. Each expression feature recognition mainly extracts expression features from the selected single-frame two-dimensional image, and the extracted content mainly includes the characteristic parameters of the facial expression of the target face in the above single-frame two-dimensional image. These characteristic parameters may include expression base parameters and facial expression motion parameters.

[0061] For a complete video, it can record the facial image at each moment in a certain period of time, and at the same time, the facial image at each moment records the instantaneous expression image of the face at that moment. These instantaneous expression images are the above-mentioned target expression images, which serve as the target of expression feature extraction. Therefore, the expression feature recognition is completed one by one for all the target expression images in the above video, and the representation of the dense expression movement of each target expression image is realized one by one, so that the perception and acquisition of the facial expression in the corresponding time period of the entire video can be realized.

[0062] Therefore, through the above-mentioned expression feature recognition, accurate expression feature extraction of each frame of two-dimensional image can be achieved, which can lay the foundation for capturing subtle expressions and improving the accuracy and delicacy of expression perception.

[0063] like Figure 2A-Figure 5CAs shown, according to one embodiment of the present invention, in operation S201, a dense face reconstruction task is generated by using expression motion information and preset joint optimization information, including: obtaining task reconstruction information of the corresponding dense face reconstruction task according to the expression parameters of the expression motion information; and constructing the dense face reconstruction task according to the task reconstruction information and the preset joint optimization information.

[0064] The expression parameters of the expression motion information can be expression feature parameters extracted from the facial expression of the target expression image, and can specifically involve expression base parameters and facial expression motion parameters, wherein the facial expression motion parameters can be the facial shape coefficient, dynamic expression motion coefficient and facial posture coefficient in the camera coordinate system.

[0065] In order to perceive the facial expression motion information of the human face and to achieve representation based on the standardized average face, a three-dimensional face reconstruction can be performed based on the standardized average face and the expression motion information. Among them, the task reconstruction information can be used to construct the optimization variable information and objective function information of the joint optimization problem corresponding to the above-mentioned dense face reconstruction task, which can be determined according to the construction requirements of the dense face reconstruction task. Among them, the preset joint optimization information can define the construction rules of the dense face reconstruction task, and the dense face reconstruction task can be generated according to the task reconstruction information through the task construction rules.

[0066] Specifically, for the above-mentioned single-frame target expression image, a dense face reconstruction task is performed to construct a dense face reconstruction task, and a joint optimization problem of dense three-dimensional face reconstruction is generated.

[0067] Among them, a single-frame target expression image can be expressed as , is the width of the image, is the height of the image. The reconstruction goal of the dense face reconstruction task is to restore the corresponding 3D face shape and its expression motion parameters .

[0068] According to the task construction rules defined by the preset joint optimization information, the dense face reconstruction task can be formalized as the following joint optimization problem (Formula 1) based on the optimization variable information and objective function information corresponding to the task reconstruction information: (1) Among them, the optimized variable information includes face shape coefficients for reconstructing individual static facial features, face expression coefficients for reconstructing dense dynamic expression movements, face rotation matrices and translation vectors for representing face postures, and illumination parameters for describing face illumination conditions.

[0069] In addition, the objective function information can include projection error, photometric consistency error, shape and regularization constraint information, etc. Among them, the objective function of the projection error (Landmark Projection Loss) It can be expressed as the following formula 2: (2) in, is the camera intrinsic parameter matrix; , is the 3D vertex corresponding to the target 3D face; , are the two-dimensional key points detected in the target expression image (such as sparse corner points such as eye corners and mouth corners). It can be the index of a two-dimensional landmark. ;in, It can be the number of 2D key points of a preset grid, which can be determined according to the application system of different dense grid corner points, such as 146 key points of the Mediapipe standard or 68 key points of Dlib.

[0070] Objective Function of Photometric Loss It can be expressed as the following formula 3: (3) in, is the face area in the target expression image; is the pixel color in the input target expression image; Pixel colors generated for rendering models (e.g. face shape based on 3DMM, lighting parameters , materials and camera parameters); It can be represented as the pixel position of the target expression image. The objective function can be The weight of =0.05.

[0071] The regularization constraint information may include regularization terms of face shape and facial expression, where: A face shape coefficient constraint term may be provided for constraining the size of the shape coefficient to prevent overfitting; A facial expression coefficient constraint term may be used to constrain the size of the expression coefficient; It can be an additional regularization term (such as the sparsity of the expression coefficient); Can be an additional regularization term The weight of =1.0.

[0072] Therefore, with the help of the construction of the above-mentioned dense face reconstruction task, the foundation can be laid for the subsequent reconstruction of the target three-dimensional face based on the standardized average face, thereby replacing the traditional sparse key point expression perception scheme, and using the three-dimensional deformation model as the expression representation means in the perception stage to ensure that the facial expression movement can be decomposed into pose, shape and expression, and to ensure that the reconstructed target three-dimensional face can achieve refined facial expression description.

[0073] like Figure 2A-Figure 5C As shown, according to one embodiment of the present invention, in operation S202, the dense face reconstruction task is jointly optimized to generate three-dimensional face reconstruction information, including: generating an initial three-dimensional face corresponding to the dense face reconstruction task; performing joint optimization processing on the dense face reconstruction task according to the initial three-dimensional face to generate three-dimensional face reconstruction information.

[0074] When constructing an initial three-dimensional face based on the above-mentioned standardized average face, the initial three-dimensional face can be a standardized three-dimensional face that has undergone initial lighting condition processing on the standardized average face, and can be used as a basis for constructing a target three-dimensional face, ensuring that the target three-dimensional face can accurately represent the facial expression information of the target expression image.

[0075] The 3D face reconstruction information is used to construct the target 3D face based on the initial 3D face. It can be generated mainly based on the joint optimization processing results of the dense face reconstruction task, which can include the facial expression base and facial expression coefficient And other basic information for 3D face reconstruction.

[0076] As shown in the above formula 1, the joint optimization process of the dense face reconstruction task can be expressed as a process of solving the formula 1, by solving the joint optimization problem constructed according to the preset joint optimization information on the basis of the above initial three-dimensional face and expression motion information to solve the above facial expression basis and facial expression coefficient , thus achieving the acquisition of three-dimensional face reconstruction information.

[0077] Therefore, by acquiring the three-dimensional face reconstruction information, accurate modeling of the face geometry and expression movement can be ensured.

[0078] like Figure 2A-Figure 5C As shown, according to one embodiment of the present invention, in generating an initial three-dimensional face corresponding to a dense face reconstruction task, it includes: detecting two-dimensional key point information in a target expression image through a preset key point detection model; and obtaining initial posture parameters of a standardized average face based on the two-dimensional key point information.

[0079] The preset key point detection model can be a pre-trained model for detecting the key points of a two-dimensional face of a target expression image, such as a MediaPipe model or a Dlib model. The target expression image can represent the facial expression based on the topological structure of a two-dimensional face, and the two-dimensional key point information can be a two-dimensional information expression of sparse corner points (such as eyebrows, eyelids, etc.) on the topological structure of the two-dimensional face. Specifically, To characterize.

[0080] The two-dimensional key point information (face mesh) may include the position information (such as two-dimensional coordinates) of the above two-dimensional key points in the two-dimensional topological structure. With the help of the position information, the initial pose parameters can be estimated in combination with the face pose parameters of the expression motion information. Among them, the initial pose parameters may include the face pose parameters of the target expression image of the current frame relative to the standardized average face, such as the rotation matrix of the face pose relative to the standardized average face. and translation vectors .

[0081] Therefore, with the help of obtaining the initial posture parameters, the mapping relationship between the two-dimensional face topology and the standardized average face can be effectively established, so that the extracted two-dimensional key point information can be accurately mapped to the standardized average face, which is conducive to the subsequent more comprehensive and accurate representation of the subtle changes and complex characteristics of facial expressions.

[0082] like Figure 2A-Figure 5C As shown, according to one embodiment of the present invention, in generating an initial three-dimensional face corresponding to a dense face reconstruction task, it also includes: initializing the optimized variable parameters of the dense face reconstruction task based on the initial posture parameters; based on the initialization result of the optimized variable parameters, generating the initial three-dimensional face by standardizing the average face and default lighting conditions.

[0083] The rotation matrix of the face pose of the normalized average face corresponding to the initial pose parameters and translation vectors , can initialize the corresponding optimization variable parameters of the dense face reconstruction task, where the optimization variable parameters can include the face shape coefficient , facial expression coefficient and lighting parameters Etc. Among them, the above parameter initialization is mainly used to convert the above face shape coefficients , facial expression coefficient and lighting parameters Initialized to the zero vector.

[0084] Therefore, the initialization result of the above optimization variable parameters is the face shape coefficient of the zero vector , facial expression coefficient and lighting parameters According to the above zero vector face shape coefficient , facial expression coefficient and lighting parameters The initial three-dimensional face is constructed based on the standardized average face according to the default lighting conditions. The default lighting conditions can be used as the default lighting data for the standardized average face in the real-time face alignment process, which can ensure the detection and alignment accuracy of the generated initial three-dimensional face and eliminate the influence of changes in lighting conditions. The default lighting conditions can be the lighting information of a point light source that is consistent with the center of the camera. When acquiring the above-mentioned target expression image, the lighting conditions are generated according to the lighting information of the camera or estimated according to the target expression image.

[0085] Therefore, it can be ensured that the facial expression of the target expression image can be normalized to the standardized average face, thereby eliminating the coupling problem between expression semantics and individual face shape, and ensuring the consistency of expression semantics on individuals with different face shapes.

[0086] like Figure 2A-Figure 5C As shown, according to one embodiment of the present invention, in operation S203, a joint optimization process is performed on the dense face reconstruction task according to the initial three-dimensional face to generate three-dimensional face reconstruction information, including: iterative optimization is performed by minimizing the projection error, photometric consistency error and regularization constraint in the dense face reconstruction task to generate three-dimensional face reconstruction information after convergence.

[0087] The joint optimization process of the dense face reconstruction task can be based on the objective function of the projection error in Formula 1 defined by iterative optimization methods (such as the Levenberg-Marquardt algorithm or gradient descent scheme) for the above joint optimization problem. , objective function of photometric consistency error And regularization constraint information (such as face shape coefficient constraint , facial expression coefficient constraint , additional regularization term etc.) to perform iterative optimization to minimize the loss, so that the final optimization process can converge to meet the following three loss conditions: (1) The objective function based on the projection error Key point alignment, such as the alignment of the standardized average face and the two-dimensional detected face, to ensure that the difference in facial key points between the two is minimized; (2) objective function based on photometric consistency error (3) Regularization terms control the stability and physical rationality of the optimization, such as the face shape coefficient and facial expression coefficient minimize.

[0088] This enables joint optimization of the above-mentioned dense face reconstruction task, thereby ensuring that a standardized average face of the input target expression image can be reconstructed using a three-dimensional deformation model.

[0089] The target 3D face can be topologically constructed based on a 3D deformable model. The 3D deformable model can represent any 3D face as a combination of linear bases by learning a large amount of 3D face data. Specifically, the 3D deformable model can be expressed as the following formula 4: (4) Among them, the 3D face shape of the target 3D face can be expressed as , is the number of mesh vertices of the 3D face topology, is the real number space. Furthermore, It can be used to express the standardized average face (i.e. three-dimensional average face, Mean Shape), which represents the average shape of the face obtained by statistical learning. It represents the identity basis of the face, which can be used to represent the principal component basis of individual face differences, and s is the number of shape basis. Indicates the facial expression basis (Expression Basis), which can be used to describe the dynamic expression changes of the face. is the number of expression bases. It represents the face shape coefficient (Identity Coefficients), which can represent the facial features of a specific individual. Represents facial expression coefficients (Expression Coefficients), which can be used as a parameterized representation of facial expressions.

[0090] Through the above joint optimization process for dense face reconstruction tasks, the facial expression base can be generated and facial expression coefficient 3D face reconstruction information.

[0091] The facial expression base of the three-dimensional face reconstruction information and face shape base Based on the linear combination representation of Formula 4 of the above three-dimensional deformation model, dense expression motion representation of any facial shape can be achieved to construct the target three-dimensional face.

[0092] Therefore, through the construction process of the target three-dimensional face, the three-dimensional face reconstruction of the standardized average face of the input target expression image can be performed through the three-dimensional deformation model, and the decomposition of the three subspaces of posture, face shape and expression of facial expression movement can be realized, which correspond to the posture subspace for representing the global rotation and translation of facial expression, the face shape subspace for representing individual facial features and the expression subspace for representing the dynamic expression changes of the face, thereby realizing the mapping of the facial expression of a single-frame two-dimensional expression image into a parameterized three-dimensional representation. With the help of the decomposition ability of face shape, expression and posture subspaces, the effective decoupling of individual facial features and expression dynamics is realized, ensuring that the ability to capture subtle expressions is significantly improved through dense expression representation, thereby overcoming the limitations of traditional sparse feature key point schemes.

[0093] like Figure 2A-Figure 5C As shown, according to one embodiment of the present invention, in operation S204, dense expression motion information of the standardized average face is extracted according to the target three-dimensional face to complete the perceptual optimization, including: optimizing the shape parameters of the target three-dimensional face to complete the extraction of dense expression motion information.

[0094] Three-dimensional face reconstruction of a single-frame target expression image can be achieved based on methods such as global photometric consistency optimization and end-to-end reconstruction schemes based on deep learning (such as using CNN and Transformer models). Specifically, the facial expression basis and facial shape basis based on the three-dimensional deformation model mentioned above can restore dense and high-precision three-dimensional facial shape and its dynamic expression parameters, thereby achieving accurate modeling of facial geometry and expression movement.

[0095] According to the facial expression basis for the dense face reconstruction task of formula 1 and facial expression coefficient The joint optimization result of can express the dense expression motion information based on the formula 4 of the above three-dimensional deformation model, as shown in the following formula 5: (5) Furthermore, the three-dimensional face shape of the above target three-dimensional face can be further expressed as the following formula 6: (6) Therefore, after completing the reconstruction of the three-dimensional face shape of the target three-dimensional face, the generated three-dimensional face shape may include the individualized face shape and facial expression movement information The two parts can be specifically expressed as the above formula 4.

[0096] In order to eliminate individual facial shape features To ensure the consistency of expression semantics, the face shape coefficient can be Set to zero, only retain the expression-related parts, that is, complete the optimization of the shape parameters of the target 3D face, and generate a 3D face shape based on the standardized average face corresponding to the above formula 4 , the three-dimensional face shape can be used to characterize the topological structure of the dense expression mesh and define the vertex positions of the dense expression mesh, which is specifically expressed as the following formula 7: (7) in, It is the facial shape of the standardized average face; It describes the dense expression motion information.

[0097] Therefore, the generated 3D face shape is the three-dimensional normalized average face The three-dimensional expression shape under the condition has eliminated individual differences and only retained the dynamic expression features, so that it can provide a consistent basic representation for semantic mapping in subsequent expression mapping. Among them, the dense expression motion information is composed of It means that, therefore, it can be expressed as the result of dense expression extraction as the following formula 8: (8) Among them, dense expression motion information It is a dense three-dimensional vertex motion field that can be used to represent the expression offset of each vertex. The significance of extracting dense expression motion information is that it can provide a fine vertex-level representation of the entire expression movement, which can reflect extremely subtle expression changes, while eliminating the interference of individual shapes, and providing high-precision input for subsequent expression operations.

[0098] Therefore, in the above-mentioned standardization and dense expression extraction process, in order to eliminate the influence of individual face shape on expression description, the shape parameters are fixed to zero in the post-processing stage, and only the expression and posture parameters are retained, thereby generating a dense expression motion under the standardized average face. This process ensures the unity of expression semantics. Therefore, a high-resolution dense expression motion (Dense Expression Motion) information can be generated through the linear combination of the facial expression base of the three-dimensional deformation model and the facial expression coefficient, which can provide accurate and standardized expression description for the subsequent mapping and execution stages.

[0099] Furthermore, in the perception stage, the three-dimensional deformation model is used to decompose facial motion into three independent subspaces: posture, face shape and expression, generating a standardized and dense representation of expression motion. This not only improves the refinement of expression description, but also eliminates the coupling problem between expression semantics and individual face shape through standardized average face topology, thereby achieving consistency of expression semantics among different individuals.

[0100] In summary, based on the perception optimization method of the above-mentioned intelligent agent expression imitation according to the embodiment of the present invention, in the perception stage of expression imitation, a three-dimensional deformation model can be used as the core technical means to replace the traditional sparse key point expression representation method, and the movement of the face is decomposed into three independent subspaces of posture, face shape and expression, so as to achieve a refined description of facial expression. In the specific implementation, the above-mentioned three-dimensional face reconstruction information is further solved by solving the single-frame image, and then the shape parameters are fixed in the post-processing stage, and only the expression and posture parameters are retained, so as to generate a dense expression motion information under the standardized average face topology. Therefore, it can significantly overcome the abstract defects of the traditional sparse key point method, realize the densification of expression description, and at the same time, by normalizing the expression to the standardized average face, eliminate the coupling problem between expression semantics and individual face shape, and ensure the consistency of expression semantics in individuals with different face shapes.

[0101] Compared with the traditional coefficient key point method, the high precision and density of the dense expression motion representation of the above-mentioned three-dimensional deformable model of the embodiment of the present invention can significantly improve the accuracy and delicacy of expression perception, laying a high-quality foundation for the subsequent mapping and execution stages. Specifically, in the perception stage of expression, the facial expressions in the two-dimensional video can be mapped into parameterized three-dimensional representations by adopting a three-dimensional deformable model. With its ability to decompose the face shape, expression and posture subspace, it can effectively decouple individual facial features from expression dynamics. This method significantly improves the ability to capture subtle expressions through dense expression representation, overcoming the limitations of the traditional sparse feature point method.

[0102] Based on the above-mentioned perception optimization method for intelligent agent expression imitation, the present invention also provides a perception optimization device for intelligent agent expression imitation. Figure 2B The device is described in detail.

[0103] Figure 2B The structural block diagram of the perception optimization device for intelligent agent expression imitation according to an embodiment of the present invention is schematically shown.

[0104] like Figure 2B As shown, the perception optimization device 200 for intelligent agent expression imitation of this embodiment includes a motion extraction module 210, a task generation module 220, a joint optimization module 230 and an information extraction module 240.

[0105] The motion extraction module 210 is used to extract the expression motion information of the target expression image. In one embodiment, the motion extraction module 210 can be used to perform the operation S201 described above, which will not be described in detail here.

[0106] The task generation module 220 is used to generate a dense face reconstruction task through expression motion information and preset joint optimization information. In one embodiment, the task generation module 220 can be used to perform the operation S202 described above, which will not be repeated here.

[0107] The joint optimization module 230 is used to perform joint optimization on the dense face reconstruction task to generate 3D face reconstruction information, and the 3D face reconstruction information is used to generate a target 3D face. In one embodiment, the joint optimization module 230 can be used to perform the operation S203 described above, which will not be repeated here.

[0108] The information extraction module 240 is used to extract the dense expression motion information of the standardized average face according to the target three-dimensional human face to achieve perceptual optimization. In one embodiment, the information extraction module 240 can be used to perform the operation S204 described above, which will not be repeated here.

[0109] According to an embodiment of the present invention, any multiple modules of the motion extraction module 210, the task generation module 220, the joint optimization module 230 and the information extraction module 240 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the motion extraction module 210, the task generation module 220, the joint optimization module 230 and the information extraction module 240 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in a suitable combination of any of them. Alternatively, at least one of the motion extraction module 210 , the task generation module 220 , the joint optimization module 230 , and the information extraction module 240 may be at least partially implemented as a computer program module, and when the computer program module is executed, the corresponding function may be performed.

[0110] Example 2

[0111] In the expression mapping stage, existing methods mainly rely on two technical paths: geometric displacement migration based on sparse key points and expression semantic mapping based on image generation. Among them, the geometric displacement migration method using sparse key points attempts to directly map the key point displacement of the human face to the robot face. However, due to the significant differences in topological structure and geometric characteristics between human faces and robot faces, simple displacement copying or regularization methods such as linear interpolation and affine transformation cannot effectively transfer expression semantics, and may even cause expression features to be mismapped or lost, thereby causing serious semantic distortion. In some cases, this method is completely unable to migrate specific expressions. Another image generation method based on a generative model (GenerativeModel) can visually generate images that conform to the expression semantics, but these images only achieve expression consistency in pixel space, without considering the physical constraints of the robot actuator, and therefore cannot be directly used to drive the actual physical actuator.

[0112] Therefore, in view of at least one of the technical problems existing in the above-mentioned prior art, the embodiments of the present invention aim to provide a mapping optimization method, device, equipment, medium and product for intelligent body expression imitation, thereby providing an optimization scheme for the mapping stage of intelligent body expression imitation, in order to achieve a cross-modal mapping mechanism from standardized facial expressions to robot expression space by introducing a unified expression parameter space, thereby solving the differences in geometric structure between human faces and robot faces and ensuring the consistency of expression semantics during the migration process.

[0113] The following will be based on Figure 1 The scene described by Figure 3A to Figure 5C The mapping optimization method for intelligent agent expression imitation in the disclosed embodiment is described in detail.

[0114] like Figure 3A As shown, one aspect of an embodiment of the present invention provides a mapping optimization method for intelligent agent expression imitation, which includes operations S301 to S303.

[0115] In operation S301, a first blend shape basis of a standardized average face is generated based on a preset facial action coding rule; In operation S302, dense expression motion information of the normalized average face is mapped to the first blend shape basis to generate a second blend shape basis; and In operation S303, the blend shape coefficients of the second blend shape basis are transferred to the third blend shape basis of the agent face, completing the mapping optimization.

[0116] The preset facial action coding rules can be coding rules for facial behaviors formed by facial muscle movement states, thereby realizing the coding of facial expressions, which can improve the processing accuracy and efficiency of facial expressions during expression recognition and processing. For example, this can be achieved by using a Facial Action Coding System (FACS).

[0117] The standardized average face is the three-dimensional average face shape of a static face under the standardized face topology, which can usually be obtained based on big data statistics. For details, please refer to the standardized average face in Example 1. .

[0118] By presetting the facial action coding rules, the semantic processing of the expression shape offset is performed on the basis of the face topology structure of the standardized average face, thereby forming a first blended shape base. The first blended shape base can describe the vertex offset of the corresponding semantic expression (such as "open mouth" or "wink left eye", etc.) relative to the standardized average face, and can be used to describe the expression change through the weighted combination of predefined semantic expression bases, which can be specifically realized by the Blendshape expression representation technology.

[0119] Dense expression motion information can be information related to dynamic changes of facial expressions that are directly related to the extracted expression motion information, and can be specifically represented by a linear combination of a facial expression base and a facial expression coefficient. Dense expression motion information can most directly reflect the dense expression motion information of the user extracted by the agent on the standardized average face to achieve facial expression representation. Specifically, reference can be made to the extraction of dense expression motion information in Example 1.

[0120] By mapping the dense expression motion information onto the first blended shape base, the expression motion information of the face of each frame of the target expression image extracted in the perception stage in Example 1 can be mapped onto the standardized average face. At the same time, the second blended shape base thus generated can also exclude the influence of individual facial shape features, ensuring the consistency of expression semantics in the mapping stage and the execution stage, thereby completing the dense representation of expression semantics. The second blended shape base can be a semantic expression base expressed by the dense expression motion information obtained through perception, which can be used to describe the vertex offset of the semantic expression of the corresponding dense expression motion information relative to the standardized average face, which can be specifically achieved by using the Blendshape expression representation technology.

[0121] Accordingly, the third blended shape base is a semantic expression base that performs semantic processing of expression shape offset based on the topological structure of the face of the agent, and can be used to describe the vertex offset of the corresponding semantic expression relative to the topological structure of the face of the agent. The semantics of the third blended shape base corresponds one-to-one to the semantics of the second blended shape base and remains consistent.

[0122] The blend shape coefficient can be understood as the blend shape weight (such as the blend shape coefficient) of the second blend shape basis, which can be used to express the vertex offset of the expression action of the dense expression motion information obtained in the perception stage (refer to the above embodiment 1). By transferring the blend shape coefficient to the third blend shape basis of the intelligent body face, direct cross-topology migration of human dense expression motion information can be achieved.

[0123] Therefore, in the mapping stage, the mapping optimization method for imitating the expression of the intelligent agent in the embodiment of the present invention solves the problem of the differences in topological structure and geometric characteristics between the human face and the intelligent agent face by using a mixed shape semantic space based on a second mixed shape basis of the standardized average face as an intermediate bridge. Specifically, a first mixed shape basis is first defined on the standardized average face for dense representation of expression semantics. The first mixed shape basis can map the dense expression motion generated in the perception stage (refer to the above embodiment 1) to the mixed shape coefficient space through dense semantic key points to form a second mixed shape basis. In this process, the expression motion is mapped from the topological structure of the standardized average face to the topological structure of the intelligent agent face by transferring the mixed shape coefficients, thereby realizing semantically consistent expression migration across topological structures.

[0124] Therefore, this conversion from geometric domain to semantic domain solves the semantic distortion problem caused by the topological difference between human and robot faces in traditional solutions. In addition, the limited number of mixed shape coefficients facilitates real-time transmission, which greatly improves the efficiency and robustness of the mapping stage. At the same time, the generated mixed shape coefficients directly correspond to the robot skin (such as Figure 5C The motion state of the skin (shown in Figure 1) ensures the physical executableness of the generated expression. It can be seen that in the cross-entity mapping process of expression imitation, the mixed shape basis designed based on the above preset facial action encoding rules can be used to accurately depict the expression semantics. By introducing a unified expression parameter space, a cross-modal mapping mechanism from the dense human facial expression of the standardized average face to the robot expression space can be established, thereby solving the difference in geometric structure between the human face and the robot face and ensuring the consistency of the expression semantics during the migration process.

[0125] The blend shape base is mainly generated based on semantic expression modeling of the blend shape base technology. After the standardization processing and extraction of dense expression motion information in the above embodiment 1, in order to further realize semantic expression modeling, an expression representation method based on the blend shape base is further introduced.

[0126] The blendshape technique can be understood as describing expression changes through a weighted combination of a set of predefined semantic expression bases. Corresponding 3D face shape It can be expressed as the following formula 9: (9) in, It can be expressed as a dense grid containing semantic expressions. It can be a three-dimensional average face shape, which is consistent with the definition of the standardized average face in Example 1, and represents a static face shape under a standardized face topology structure. Can be the Human Blendshape base, used to represent semantic expression shape offsets, which can describe the vertex offsets of corresponding semantic expressions (such as "open mouth" or "wink left eye") relative to the standardized average face. Can be the The weight of the blended shape basis of a face can represent the intensity coefficient of the expression. The weight of the blended shape basis of the face can usually be normalized to the interval On, for example Indicates no such expression, This means that the expression is fully applied. is the number of blend shape bases defined, each blend shape base Can correspond to a semantic expression action.

[0127] Combining the mathematical representation of the mixed shape basis in equation 9 above, we can combine Figure 3A-Figure 5C The generation of the first mixed shape substrate in Example 2 of the present invention is further described as follows.

[0128] like Figure 3A to Figure 5C As shown, according to one embodiment of the present invention, in operation 301, generating a first blended shape basis of a standardized average face based on a preset facial action coding rule includes: Generate a mixed shape expression basis matrix according to a preset facial action coding rule; A first blend shape basis for a normalized average face is generated based on a preset blend shape weight vector and a blend shape expression basis matrix.

[0129] As mentioned above, the first blend shape basis can be based on the three-dimensional average face shape (i.e., the normalized average face shape). ) can be designed to have the same topological structure (the connection relationship between vertices and faces) as the standardized average face, and combined with the expression semantics implementation definition of preset facial action coding rules (such as Facial Action Coding System, FACS), such as "open mouth" or "wink left eye".

[0130] As defined in Formula 9 above, the face blend shape base as a component of the first blend shape base It can be designed in a predefined way, specifically based on the standardized average face The facial topology structure is constructed. The first blend shape base is designed according to the expression semantics of the preset facial action coding rules, and can be divided according to the dominant area to ensure that the semantics of each blend shape base is clear and consistent. For example, the expression action of "right corner of mouth raised" can correspond to the movement of an actuator in FACS.

[0131] Specifically, the three-dimensional face deformation represented by the first mixed shape basis can be defined on the basis of the standardized average face, and generated by a weighted combination of a set of predefined semantic expression basis. Combined with the above formula 9, the first mixed shape basis can be specifically The expression combination based on the mixed shape basis is formed by expressing it in the matrix form in the following formula 10: (10) Among them, the mixed shape expression basis matrix The semantic face mixed shape basis matrix that can constitute the first mixed shape basis can be specifically composed of Predefined facial expression bases In addition, the blend shape weights are The preset weight vector of the mixed shape corresponding to the mixed shape expression basis matrix can be expressed to represent the intensity of each expression. and the blend shape expression basis matrix The standardized average face can be constructed The corresponding first blend shape base .

[0132] So by adjusting the blend shape weights , that is, the contribution of each component blend shape basis in the first blend shape basis can be flexibly controlled to generate expression shapes of different strengths and combinations. This predefined modeling method ensures that the generated expression shapes have clear semantics, dense geometric details, and a topological structure consistent with the average face.

[0133] It can be seen that through semantic expression modeling of blend shape bases, a set of blend shape bases corresponding to dense expression movements can be designed on the facial topology (Mesh) of the standardized average face based on the facial action coding system for compact representation of expression semantics (such as ARKit's 51 standard Blendshapes).

[0134] like Figure 3A to Figure 5C As shown, according to one embodiment of the present invention, before the operation 302 maps the dense expression motion information of the standardized average face onto the first mixed shape base to generate the second mixed shape base, it also includes: Dense expression motion information of the normalized average face is generated according to the extracted expression motion information of the target expression image.

[0135] As mentioned in Example 1, the target expression image may be a single-frame image containing human facial expressions, and the expression motion information may include facial expression feature parameters extracted for the facial expressions in the target expression image. Accordingly, the dense expression motion information may be information related to the dynamic changes of facial expressions directly related to the extracted expression motion information, which may be specifically represented by a linear combination of a facial expression base and a facial expression coefficient, as shown in Formula 5 in Example 1. The dense expression motion information may most directly reflect the dense expression motion information of the user extracted by the intelligent agent on the standardized average face, thereby realizing facial expression representation.

[0136] Therefore, based on the extraction of the dense expression motion, a facial expression grid based on the standardized average face corresponding to the dense expression motion information can be constructed. (As shown in Formula 7 in Example 1), the facial expression mesh can be used as a three-dimensional facial shape with dense expression motion information to characterize the dense expression mesh topology structure and define the vertex positions of the dense expression mesh.

[0137] With the help of the above-mentioned extraction process of dense expression motion information in the perception stage, effective decoupling of individual facial features and expression dynamic information can be achieved, and the ability to capture subtle expressions can be significantly improved through dense expression representation.

[0138] In the mapping stage of expression imitation in implementation 2 of the present invention, in order to migrate the dense expression motion generated in the perception stage of implementation example 1 to the topological structure of the intelligent body's face, realize cross-topological semantic migration, and ensure cross-topological semantic consistency, it is necessary to consider the mapping of dense expression motion to mixed shape coefficients.

[0139] like Figure 3A to Figure 5C As shown, according to one embodiment of the present invention, in operation 302, mapping the dense expression motion information of the standardized average face onto the first mixed shape basis to generate a second mixed shape basis includes: Extracting first dense annotation points on the dense expression grid and second dense annotation points on the average face grid corresponding to the dense expression motion information; Obtain a dense annotation basis matrix according to the mixed shape expression basis matrix; A hybrid shape weight optimization task is generated according to the densely labeled basis matrix, the first densely labeled points, and the second densely labeled points.

[0140] The dense expression grid can be understood as the facial expression grid based on the standardized average face corresponding to the above dense expression motion information. (as shown in Formula 7 in Example 1), used to represent the normalized dense expression mesh vertex position, including vertices, each with three-dimensional coordinates (a total of degrees of freedom), which can be understood as a three-dimensional geometric representation of the entire facial network.

[0141] The average face grid can be understood as the standardized average face The standardized face mesh of can represent the three-dimensional vertex position of the corresponding face mesh.

[0142] The first dense annotation point can be represented as the standardized dense annotation point position , which is usually obtained from the above dense expression grid in dense expression modeling A specific set of points selected from the face, such as those distributed in key areas of the face (such as the mouth, eyes, and nose), can be specifically represented as a dense expression grid Selected on The positions of densely labeled points (a total of degrees of freedom).

[0143] The position of the first densely labeled point can be obtained by using the densely labeled matrix From this dense expression grid Extraction can be specifically expressed as the following formula 11: (11) The dense label matrix is a sparse selection matrix that can be used to select from a dense expression grid Extract the GIA first densely marked points.

[0144] Correspondingly, the second dense annotation point can be expressed as the average face position of the dense annotation points , defined as the average face mesh Through the dense label matrix The extracted dense annotation points can be specifically expressed as the following formula 12: (12) The average face mesh It can represent the 3D vertex positions of the face mesh defined by the standardized average face.

[0145] Therefore, with the help of the above formulas (11) and (12), the correspondence between dense annotation points and face meshes can be constructed, where the position of the first dense annotation point is and the position of the second densely labeled point With dense expression grid (full grid expression) and average face mesh Directly associated.

[0146] As mentioned above, the blend shape expression basis matrix The full-grid basis matrix can be used to form a semantic face blended shape basis matrix of the first blended shape basis, and specifically can be composed of K predefined expression basis.

[0147] The densely labeled basis matrix can be understood as the densely labeled mixed shape basis matrix , which is mainly used to represent the mixed shape basis on densely labeled points. Specifically, it can be obtained from the full grid basis matrix It can be extracted from the formula 13: (13) Therefore, according to the densely labeled basis matrix defined in the above formula 13 , the first dense annotation point defined by formula 11 and the second dense annotation point defined by formula 12 , the mixed shape weight corresponding to the above mixed shape weight optimization task can be defined as the optimization problem expressed by formula 14 as follows: (14) in, It can be understood as a mixed shape weight, which can make the reconstructed dense annotation point position The following relationship defined by formula 15 is satisfied: (15) Therefore, modeling based on mixed shape technology can use the geometric information of densely labeled points to solve the optimization problem corresponding to the mixed shape weight optimization task of the above formula 14 and obtain the mixed shape weight , thereby ensuring that the positions of the reconstructed dense annotation points are as close as possible to the input dense annotation points.

[0148] like Figure 3A to Figure 5C As shown, according to one embodiment of the present invention, in operation 302, mapping the dense expression motion information of the standardized average face onto the first mixed shape base to generate the second mixed shape base further includes: The mixed shape coefficients corresponding to the mixed shape weight optimization task are obtained by minimizing the preset error vector and the preset coefficient constraints, so as to realize mapping the dense expression motion information of the standardized average face to the first mixed shape basis to generate a second mixed shape basis.

[0149] The preset error vector can be the error vector objective function for performing the optimization process for the above-mentioned mixed shape weight optimization task. , which can be specifically expressed as the following formula 16: (16) The preset error vector may be a first densely marked point and the second densely labeled points Based on densely labeled basis matrix The error vector is set to minimize the sum of squared errors as the optimization objective for the preset error vector, and the following formula 17 can be defined: (17) Expanding the above formula 17, it can be expressed as the following formula 18: (18) The preset coefficient constraint condition can be used as a constraint condition for performing the optimization process for the above-mentioned mixed shape weight optimization task. For example, the value of the mixed shape weight can be limited to In the range of , the mixed shape coefficient corresponding to the final mixed shape weight optimization task is Therefore, the above blend shape weight optimization task corresponds to a quadratic programming problem with boundary constraints (preset coefficient constraints).

[0150] Therefore, by means of the solution process of the least squares problem of the above-mentioned mixed shape coefficient, the dense expression motion information can be mapped to the low-dimensional first mixed shape basis to form a second mixed shape basis in real time, that is, by selecting dense semantic key points (based on centroid coordinates, not vertex coordinates) on the surface of the average face mesh, the dense expression motion generated in the perception stage is mapped to the mixed shape coefficient space, thereby completing the coefficient mapping from dense expression to mixed shape space.

[0151] In summary, through an overdetermined least squares problem of the above blend shape weight optimization task, it is possible to compress dense expression motion information into a low-dimensional blend shape space. This process can generate blend shape coefficients in real time while ensuring the density and accuracy of the expression.

[0152] like Figure 3A to Figure 5C As shown, according to one embodiment of the present invention, before the operation 303 transfers the blended shape coefficients of the second blended shape basis to the third blended shape basis of the face of the agent, and before completing the mapping optimization, it also includes: A third blended shape basis of the agent's face is generated based on preset facial action coding rules, wherein the blended shape semantics of the third blended shape basis is the same as that of the first blended shape basis.

[0153] The third blend shape base is actually the blend shape space for the face of the intelligent agent. Among them, the intelligent agent blend shape space needs to ensure that it is consistent with the dense expression grid of the human face. The corresponding first blend shape basis space remains semantically consistent, enabling the intelligent agent to accurately express the same basic expression semantics as humans (such as "open mouth", "blink", etc.).

[0154] Like the first blended shape basis of humans, the third blended shape basis space of each agent can correspond one-to-one and remain consistent with the semantics of the first blended shape basis space of the standardized average face in accordance with the principle of semantic consistency (for example, "open mouth" should express the same semantics in the facial topology of the agent and in the blended basis space in the standardized average face model of humans).

[0155] Therefore, according to the semantic set of the first blend shape basis space of the face (such as "open mouth", "blink left eye", etc.), based on the above preset facial action encoding rules, a set of corresponding third mixed shape bases can also be defined for the agent based on the agent's facial topology structure, and the semantic set in the third mixed shape base space is , which matches each semantic agent blend shape basis ,in is the number of mesh vertices in the agent's facial topology, Can be semantically numbered.

[0156] Therefore, based on the first mixed shape basis of the standardized average face corresponding to the dense expression motion information, a semantically consistent third mixed shape basis of the intelligent agent's face can be constructed according to the topological structure of the intelligent agent's face, ensuring that the semantics of the two mixed shape basis spaces are completely consistent, thereby ensuring efficient and accurate dense expression migration from the standardized average face to the intelligent agent's facial topology.

[0157] like Figure 3A to Figure 5C As shown, according to one embodiment of the present invention, in operation 303, the blended shape coefficients of the second blended shape basis are transferred to the third blended shape basis of the face of the agent to complete the mapping optimization, including: The blend shape coefficients are applied to the third blend shape base to generate an agent expression mesh corresponding to the agent.

[0158] The mixed shape coefficients generated by the standardized average face are directly transferred to the third mixed shape basis corresponding to the facial topology of the intelligent agent, which can realize the cross-topological semantic migration of dense expressions and generate the intelligent agent expression mesh, thereby completing the mapping process of the mixed shape coefficients.

[0159] The core of the blend shape coefficient transfer is the blend shape coefficient generated directly from the human standardized average face Third blend shape base applied to the agent , realizing cross-topological semantic transfer of dense expressions.

[0160] Specifically, through a third blend shape basis with semantically consistent agent facial topology , the semantics of each base expression is fully aligned with the first blend shape basis of the human standardized average face topology. Therefore, the blend shape coefficients of the first blend shape basis are It can be directly transferred to the third blend shape base without any conversion, satisfying the following formula 19: (19) Therefore, the blend shape coefficients of the first blend shape base are Applied to the facial topology of intelligent agents and Agent Third Blend Shape Base , you can generate the agent expression grid corresponding to the agent It is expressed as the following formula 20: (20) Therefore, it is possible to ensure that the semantics of the robot's expression is consistent with that of a human, while adapting to the geometric characteristics of the agent's facial topology. For details, see Figure 5AA semantic alignment operation of a second blend shape base to a third blend shape base is shown.

[0161] In summary, based on the mapping optimization method for the above-mentioned intelligent agent expression imitation provided in Example 2 of the present invention, a set of mixed shape semantic spaces based on the standardized average face can be provided as an intermediate bridge in the mapping stage to solve the differences in topological structure and geometric characteristics between human faces and intelligent agent faces. Specifically, a set of mixed shape bases are first defined on the standardized average face for dense representation of expression semantics. These mixed shape bases map the dense expression motion generated in the perception stage to the mixed shape coefficient space through dense semantic key points. This process maps the dense expression motion to low-dimensional mixed shape coefficients in real time by solving the least squares problem.

[0162] Afterwards, a set of blend shape bases with semantic consistency with the average face is provided based on the agent face to ensure that the two have the same semantic definition. By transferring the blend shape coefficients, dense expression motion can be mapped from the standardized average face to the agent face, thus achieving efficient and accurate cross-topology and semantically consistent expression transfer.

[0163] Therefore, by converting from the geometric domain to the semantic domain, the semantic distortion problem caused by the topological difference between the human face and the agent's face in traditional methods is solved. In addition, the limited number of mixed shape coefficients facilitates real-time transmission, greatly improving the efficiency and robustness of the mapping stage. At the same time, the generated mixed shape coefficients directly correspond to the motion state of the agent's epidermis, ensuring the physical executableness of the generated expressions.

[0164] In the cross-entity mapping of expressions, a hybrid shape base designed based on preset facial action encoding rules is used to accurately characterize the semantics of expressions. By introducing a unified expression parameter space, a cross-modal mapping mechanism from standardized human facial expressions to the intelligent agent expression space is established, which solves the difference in geometric structure between human faces and intelligent agent faces and ensures the consistency of expression semantics during the migration process.

[0165] Therefore, the mapping optimization method for agent expression imitation provided in Embodiment 2 of the present invention can at least achieve the following technical effects: (1) Directness: No need to mix shape coefficients for humans Do any additional conversion or mapping.

[0166] (2) Efficiency: Using semantically consistent hybrid shape bases, we can quickly achieve cross-topology migration of expressions.

[0167] (3) Universality: Applicable to any agent hybrid shape substrate that conforms to the semantically consistent design.

[0168] Based on the above-mentioned mapping optimization method for imitating the expression of an intelligent agent, the present invention also provides a mapping optimization device for imitating the expression of an intelligent agent. Figure 3B The device is described in detail.

[0169] Figure 3B The structural block diagram of the mapping optimization device for agent expression imitation according to an embodiment of the present invention is schematically shown.

[0170] like Figure 3B As shown, the mapping optimization device 300 for agent expression imitation in this embodiment includes a base generation module 310, an information mapping module 320 and a coefficient transfer module 330.

[0171] The base generation module 310 is used to generate a first blend shape base of a standardized average face based on a preset facial action coding rule. In one embodiment, the base generation module 310 can be used to perform the operation S301 described above, which will not be described in detail here.

[0172] The information mapping module 320 is used to map the dense expression motion information of the standardized average face to the first mixed shape basis to generate a second mixed shape basis. In one embodiment, the information mapping module 320 can be used to perform the operation S302 described above, which will not be repeated here.

[0173] The coefficient transfer module 330 is used to transfer the blended shape coefficients of the second blended shape basis to the third blended shape basis of the agent's face to complete the mapping optimization. In one embodiment, the coefficient transfer module 330 can be used to perform the operation S303 described above, which will not be repeated here.

[0174] According to an embodiment of the present invention, any multiple modules of the base generation module 310, the information mapping module 320 and the coefficient transfer module 330 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the base generation module 310, the information mapping module 320 and the coefficient transfer module 330 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the base generation module 310, the information mapping module 320 and the coefficient transfer module 330 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be performed.

[0175] Example 3

[0176] In the execution stage, traditional methods convert epidermal motion representation into actuator instructions in two ways. One is the model-based inverse kinematics method, which converts the movement of the epidermal traction point into specific instructions for the actuator through inverse kinematics solution. However, this method has extremely high requirements on the accuracy of the geometric model and needs to accurately describe the topological structure and actuator position of the robot's face, otherwise it is difficult to ensure the accuracy of the inverse kinematics solution. Another method is the model-free neural network method based on deep learning, which uses neural networks to directly inversely calculate actuator instructions from epidermal motion (such as pixel space). Although this method has higher flexibility, the generated execution instructions often do not fully consider the physical constraints of the robot actuator, which makes the instructions difficult to implement in practical applications and causes problems with the control accuracy and stability of expressions.

[0177] In view of at least one of the technical problems existing in the above-mentioned execution scheme of facial expression imitation, an embodiment of the present invention provides an execution optimization method, device, equipment, medium and product for intelligent body facial expression imitation, so as to optimize the facial expression driving parameters by explicitly modeling the physical constraints of the intelligent body actuator, so as to ensure that the generated control instructions are both physically executable and meet the requirements of high precision and real-time performance.

[0178] The following will be based on Figure 1 The scene described by Figure 4A to Figure 5CThe execution optimization method of intelligent agent expression imitation in the disclosed embodiment is described in detail.

[0179] like Figure 4A As shown, one aspect of an embodiment of the present invention provides an execution optimization method for intelligent agent expression imitation, which includes operations S401 to S403.

[0180] In operation S401, an inverse kinematics target optimization task is generated based on control point simulation information and target surface vertex information of the agent expression mesh; In operation S402, an optimization process is performed on the inverse kinematics target optimization task to obtain optimal joint angle parameters corresponding to the control point simulation information; and In operation S403, the execution optimization of the intelligent agent expression imitation is performed according to the optimal joint angle parameters.

[0181] The agent expression mesh may include the vertex relationship of the face topology mesh structure of the agent as mentioned in the above-mentioned embodiment 2, wherein the agent expression mesh is the robot facial topology that has completed the cross-topology migration of dense expression motion information in the perception stage in embodiment 1, and can be specifically embodied as the expression mesh in formula 20 in embodiment 2 .

[0182] like Figure 5B As shown in Figure 2, there are many key control points on the agent's expression grid, such as eyebrow, eyelid, eyeball, nose, cheek, mouth and jaw, etc. These control points can be Figure 5C The steel cable / PTFE catheter shown is pulled by a motor (such as an LFD-01M servo motor) to drive the epidermis in the area surrounding the control point (such as Figure 5C The silicone mask shown in the figure is deformed. In traditional technical solutions, it is very difficult to calculate the deformation of the area around the control point, resulting in a stiff imitation of the intelligent body's expression and failure to achieve a delicate and natural "skin" performance.

[0183] The control point simulation information can be the position information formed by simulating the control point positions of the intelligent body expression mesh containing dense expression motion information through a simulation tool (such as Blender), for example, the simulated positions of each control point in the topological network of the intelligent body expression mesh containing dense expression motion information. The control point simulation information can be used to define Figure 5C The driving angle parameters of the driving motor rocker arm of the steel cable / polytetrafluoroethylene tube shown are also the joint angle parameters.

[0184] The target epidermal vertex information may be the target epidermal mesh vertex position that each control point on the agent expression mesh should reach when the facial expression represented by the dense expression motion information is to be achieved. The target epidermal vertex information may include the relative displacement distance and displacement direction information of the simulation control point position of the simulation calculated control point and the epidermal mesh vertex in the surrounding area, and may be used to define the epidermal shape (i.e., the target epidermal shape) that should be achieved when the agent facial mesh is driven by the agent expression mesh relative to the dense expression motion information.

[0185] The inverse kinematics target optimization task can be an optimization task that continuously optimizes the joint angle parameters corresponding to the control point simulation information according to the target epidermal vertex information. Through the continuous optimization of the inverse kinematics target optimization task, during the actual driving execution, according to the optimization result of the inverse kinematics target optimization task, the motion corresponding to the facial epidermal mesh of the intelligent body can be as close as possible to the target epidermal shape defined by the dense expression motion information. The inverse kinematics target optimization task can be expressed in the form of an inverse kinematics objective function.

[0186] By performing the above joint angle parameter optimization on the inverse kinematics target optimization task, the intelligent body skin mesh can be made as close to the target skin shape as possible until the optimal joint angle parameter is obtained. Figure 5C When the corresponding driving motor of the facial structure of the intelligent body shown is used as a driving parameter to perform epidermal drive, the corresponding driving control point and its surrounding epidermal area can present an expression imitation effect that is most consistent with the dense expression motion information, making the expression more delicate and natural, while ensuring that the epidermal skin around the control point presents the most natural epidermal changes at the same time.

[0187] Therefore, the optimal joint angle parameter can be the motor rocker joint angle parameter that can achieve the closest target skin deformation when the corresponding control point and its surrounding skin area are driven by the motor. Figure 5CAs shown, the "skin" (Skeleton) architecture supports the silicone mask's "skin", "muscle" (Muscle) architecture and motor control (Electrical Control) drive architecture, wherein the motor control architecture includes various types of drive motors or servo motors, which can perform rotation or even pulling actions according to the optimal joint angle parameters under the control of the controller, thereby driving the connected steel cable (Steel Cable) / polytetrafluoroethylene conduit (Teflon Conduit) to produce the optimal displacement in the specified direction to the specified position (control point) on the silicone mask ("skin") to which it is connected, so as to achieve the optimal position of the control point under the target expression, thereby generating epidermal deformation of the intelligent body's face, completing the execution optimization of the expression action, and realizing the natural deformation of the intelligent body's facial expression. Among them, the control point can be the fixing point of the above-mentioned steel cable / polytetrafluoroethylene conduit on the inner surface of the silicone mask (such as bonding fixation). In addition, the above-mentioned "skeleton" (Skeleton) architecture, "skin" (Skin), "muscle" (Muscle) architecture and motor control (Electrical Control) drive architecture can constitute the actuator of the intelligent body.

[0188] Among them, since the optimal joint angle parameters are obtained by optimizing the inverse kinematics target optimization task based on the intelligent body expression mesh of dense motion expression information, the execution process of the above expression action corresponds to the execution process of intelligent body expression imitation. In short, the generation of intelligent body facial expression depends on the kinematic modeling of subcutaneous control points. At the same time, the joint angle parameters obtained by inverse kinematics are used to drive the control point movement, and then the dense expression motion information obtained in the perception stage is transmitted to the vertices of the intelligent body's facial epidermal mesh, ensuring the natural deformation of the expression restoration process.

[0189] Therefore, through the execution optimization method of the above-mentioned intelligent body expression imitation of Example 3 of the present invention, the optimal position of the control point under the target expression can be solved by defining the control point area, using inverse kinematics, and generating accurate actuator motion instructions, so that the motion state of the control point area is close to the target epidermal movement. In this case, the fineness and naturalness of expression restoration can be further improved by simply increasing the density of control points without changing the algorithms in the perception and mapping stages. Therefore, not only the accuracy of expression imitation is greatly improved, but also the physical executable nature of the generated instructions is ensured by explicit modeling of the physical constraints of the actuator, avoiding the execution failure problem caused by ignoring physical constraints in traditional methods. In addition, the independence of control point optimization also allows the execution stage to flexibly adapt to different types of intelligent body facial structures, providing technical guarantees for the scalability and modular design of the system, and has extremely high engineering application value and commercial application value.

[0190] like Figures 4A-5C As shown, according to one embodiment of the present invention, before the operation S401 generates the inverse kinematics target optimization task based on the control point simulation information and the target epidermis vertex information of the intelligent body expression grid, it also includes: Generate an agent expression mesh corresponding to the agent according to the mixed shape basis of the agent's face; A distance weight matrix is ​​generated based on the control point simulation information of the agent's expression mesh and the target skin vertex information.

[0191] The hybrid shape base of the intelligent body face can be a semantic expression space that performs semantic processing of expression situation offset based on the intelligent body face topology structure, which is used to describe the vertex offset of the semantic expression corresponding to the dense expression motion information relative to the intelligent body face topology structure. For details, please refer to the third hybrid shape base provided in the above embodiment 2. The dense expression motion information extracted in the perception stage can be transferred to the cross-topological semantics of the blended shape basis of the above-mentioned intelligent face through the mapping process of the blended shape coefficients in the mapping stage.

[0192] Furthermore, an agent expression grid can be constructed based on the mixed shape base of the agent face. , which can be specifically embodied in Formula 20 in the above-mentioned Embodiment 2. In this way, the semantics of the robot's expression can be ensured to be consistent with that of the human, and the geometric characteristics of the facial topology of the intelligent body can be adapted to ensure that the semantics of the perceived human facial expression can be aligned, thereby ensuring the accuracy of subsequent physical drive execution, guaranteeing the natural deformation of the expression restoration process, and ensuring a better restoration of the perceived human facial expression.

[0193] Each component matrix element in the distance weight matrix Can be used to represent a single control point on the agent's expression grid Corresponding to the epidermal vertex Therefore, the distance weight matrix can define the distance-based weight distribution between the epidermis control points and the epidermis mesh vertices. Among them, these matrix elements It can be specifically expressed as the following formula 21: (twenty one) in, The epidermal apex With control point The Euclidean distance between is the distance from the epidermal vertex to the control point; is the three-dimensional position coordinate of the subcutaneous controller; It is the control parameter of distance effect, which is used to adjust the attenuation range of weight; weight normalization ensures (The sum of the weights of each epidermal vertex is 1), is the number of control points.

[0194] Among them, a single control point The position information may be formed by performing position simulation calculation on the intelligent body expression grid by a simulation tool, and the set of these control points may constitute the above control point simulation information. It can be used to define the three-dimensional coordinates of the subcutaneous traction position. Correspondingly, a single epidermal vertex It may be the expected position of the dense expression motion information reached by the corresponding grid position vertex, and the set of these epidermal vertices may constitute the above-mentioned target epidermal vertex information.

[0195] Therefore, the dense expression motion information of the agent’s expression mesh can be mapped to the agent’s facial structure topology based on the control point simulation information and the target epidermal vertex information, and the accurate description of the agent’s facial topology and actuator positions can be achieved through the distance weight matrix.

[0196] like Figures 4A-5C As shown, according to one embodiment of the present invention, before generating a distance weight matrix based on control point simulation information of the intelligent body expression grid and target epidermal vertex information, it also includes: Generate control point simulation information according to the preset positive kinematics optimization task of the agent's expression grid and the joint angle parameter vector corresponding to the simulation control point; The target skin vertex information is generated according to the deformation weight matrix of the skin mesh vertices of the intelligent body expression mesh and the control point simulation information.

[0197] In the simulation tool based on the intelligent expression grid Based on the corresponding geometric model and kinematic structure of the intelligent agent's face (such as joint control points and their associations), the preset forward kinematics optimization task can be expressed as a forward kinematics function based on the motor kinematics joint angle parameters corresponding to the intelligent agent's kinematic structure.

[0198] The simulation control points can be the subcutaneous control points defined by the above-mentioned intelligent agent facial geometry model and kinematic structure. The subcutaneous control point set is defined as , each control point Represents the three-dimensional coordinates of a certain location under the skin, is the number of control points. Each simulation control point corresponds to a different joint angle parameter, which can be used as a sub-element of the above joint angle parameter vector.

[0199] Therefore, the simulation tool can generate forward kinematic information of the control point position as the control point simulation information according to the input parameter vectors such as joint angles, etc., combined with the kinematic relationship and geometric constraints.

[0200] Specifically, the forward kinematics implementation operation for the control point simulation information can firstly construct the geometric model and kinematic structure of the face of the intelligent agent through a simulator (such as Blender), including joints, control points and their associations, and then use the simulator to calculate the input joint angle parameters. , combining kinematic relationships and geometric constraints to directly calculate the position of the control points .

[0201] Furthermore, the control point position corresponding to the control point simulation information is determined by the kinematic joint angle parameter The forward kinematics model of the driving and control point simulation information is defined as follows: (twenty two) in, is the forward kinematics function; is the joint angle parameter vector; is the number of kinematic degrees of freedom; is the number of control points. The control point simulation information can be used to complete the construction of the control point model.

[0202] The skin mesh vertices of the agent expression mesh can be based on the agent expression mesh The corresponding facial geometry model and the vertices of the facial mesh of the kinematic structure of the agent can be expressed as a set of epidermal mesh vertices: , where each skin mesh vertex Represents the three-dimensional coordinates of a point on the epidermis, is the number of vertices in the skin mesh.

[0203] Correspondingly, the control point model of the control point simulation information The corresponding control point movement can be obtained through the deformation weight matrix Driver corresponding to the skin mesh vertex The deformation weight matrix can represent the influence of the control point on the deformation of the skin mesh vertex. Specifically, the target skin vertex information can be expressed as a linear relationship through the deformation weight matrix and the control point simulation information as follows: (twenty three) in, is the deformation weight matrix, representing the control points Vertex of epidermis The influence relationship; is the global position of the control points, stacked in order as ; is the global position of the skin mesh vertices, stacked in order as Therefore, the construction of the epidermis mesh vertex model can be completed. Among them, the deformation weight matrix can be a weight construction model of the distance weight matrix, which is used to generate the distance weight matrix according to the expression mesh of the intelligent body.

[0204] This allows us to accurately define the geometric model of the control points and their surrounding areas, as well as the mesh skin vertices, so that we can subsequently calculate the optimal position of the control points under the target expression, generate accurate actuator motion instructions, and ensure that the motion state of the control point area is close to the target skin motion. Moreover, it is also helpful to further improve the precision and naturalness of expression restoration by increasing the density of control points without changing the algorithms in the perception and mapping stages.

[0205] like Figures 4A-5C As shown, according to one embodiment of the present invention, in operation S401, based on the control point simulation information of the intelligent body expression grid and the target skin vertex information, an inverse kinematics target optimization task is generated, including: The extended distance weight matrix is ​​a weight augmentation matrix; Generate control point driven skin mesh vertices according to control point simulation information and weight amplification matrix; The inverse kinematics target optimization task is generated based on the control point driven skin mesh vertices and the target skin vertex information.

[0206] The weight expansion matrix can be formed by three-dimensionally expanding each sub-element in the distance weight matrix. and the skin mesh vertices All have three-dimensional coordinates ( ), the weight augmentation matrix The dimension can be The specific form is as follows: Formula 24: (twenty four) Each matrix element Indicates the corresponding control point Vertices of the skin mesh The influence of is extended to each coordinate component ( ).

[0207] The inverse kinematics target optimization task can be reflected as a target optimization function that optimizes the joint angle parameters corresponding to the control point in combination with the position information of the target epidermal vertices, so as to realize the targeted inverse kinematic solution of the joint angle parameters and obtain the optimal joint angle parameters required for the control point to perform epidermal control.

[0208] Specifically, the goal of the inverse kinematics optimization task is to obtain the target surface shape according to , optimize the joint angle parameters , so that the skin mesh is as close to the target shape as possible. Among them, the inverse kinematics target optimization task can be defined in the form of an objective function as follows: (25) in, is the position of the target epidermis mesh vertex defined by the target epidermis vertex information; is the control point position calculated by simulation as defined by the aforementioned control point simulation information; The skin mesh vertices generated by the control point drive can be linearly combined through the control point simulation information and the weight amplification matrix.

[0209] like Figures 4A-5C As shown, according to one embodiment of the present invention, in operation S402, performing optimization processing on the inverse kinematics target optimization task to obtain the optimal joint angle parameters corresponding to the control point simulation information includes: According to the joint angle constraints, the inverse kinematics target optimization task is iteratively processed to minimize the target optimization and generate the optimal joint angle parameters.

[0210] The joint angle constraints may include joint angle limit constraints and geometric constraints of the kinematic structure of the intelligent body's face. Among them, the joint angle limit constraints may define the range of motion of the joint angle between the minimum joint angle and the maximum joint angle of the motor rocker arm, and the geometric constraints of the kinematic structure may include the pulling range of the steel cable / polytetrafluoroethylene catheter, the deformation range of the "skin" of the silicone mask, etc. By iteratively processing the minimization target of the inverse kinematics target optimization task of the above formula 25 (such as nonlinear optimization), the control deformation position of the control point is continuously approached to the optimal control point position under the target expression, wherein the joint angle of the motor rocker arm corresponding to the optimal control point position (for example, relative to the starting joint angle) can be used as the optimal joint angle parameter. .

[0211] Therefore, by combining the above-mentioned kinematic modeling and inverse kinematics solution, the actuator instructions can be solved through inverse kinematics under physical constraints for the control point area defined on the face of the intelligent agent, so that it approaches the target epidermal motion generated in the mapping stage.

[0212] like Figures 4A-5C As shown, according to one embodiment of the present invention, in operation S403, the optimization of the execution of the imitation of the expression of the intelligent agent according to the optimal joint angle parameters is performed, including: The control point driving information of the intelligent body's expression imitation is obtained through the control point simulation information and the optimal joint angle parameters; According to the control point driving information, the driving of the intelligent body's expression imitation is executed to complete the execution optimization.

[0213] The forward kinematics function corresponding to the control point simulation information obtained by the simulation tool , further combined with the above inverse kinematics target optimization task optimization processing to obtain the optimal joint angle parameters , the control point driving information of the intelligent agent facial motor control system to achieve the imitation of the intelligent agent's expression can be defined as follows: (26) Among them, the control point driving information can define the deformation position of the facial epidermis corresponding to the control point and the deformation position information of the epidermis mesh in the surrounding area when the motor rocker arm that needs to be controlled by the motor control system is driven according to the above-mentioned optimal joint angle parameters.

[0214] Specifically, the skin mesh shape can be generated by driving the corresponding control points, and the skin mesh shape can be expressed as the following formula 27: (27) Therefore, if Figure 5C As shown, the control point driving information defined by the above formulas 26 and 27 can be converted into executable instructions for physical actuators (such as motors) to achieve the fit of epidermal drive and control points, so that the mixed shape coefficients based on dense expression motion information generated in the mapping stage can be converted into physically executable intelligent facial epidermal motion, and accurate expression imitation can be achieved by driving the epidermal motion through the actuator, thereby realizing high-precision physical drive optimization for expression imitation.

[0215] In summary, according to the execution optimization method of the intelligent body expression imitation according to the embodiment of the present invention, in the execution stage, an optimization driving means based on control points can be constructed according to the physical characteristics of the intelligent body actuator. First, the control point area is defined, and the optimal position of the control point under the target expression is solved by inverse kinematics, and accurate actuator motion instructions are generated to make the motion state of the control point area close to the target epidermal motion. By increasing the density of control points, the refinement and naturalness of expression restoration can be further improved without changing the algorithms in the perception and mapping stages.

[0216] This method not only improves the accuracy of expression imitation, but also ensures the physical executableness of the generated instructions by explicitly modeling the physical constraints of the actuator, avoiding the execution failure problem caused by ignoring physical constraints in traditional methods. In addition, the independence of control point optimization allows the execution stage to flexibly adapt to different types of robot facial structures, providing technical support for the scalability and modular design of the system.

[0217] In summary, in the expression driving stage, the physical constraints of the robot actuators are explicitly modeled and the expression driving parameters are optimized to ensure that the generated control instructions are both physically executable and meet the requirements of high precision and real-time performance.

[0218] Based on the above-mentioned intelligent agent expression imitation execution optimization method, the present invention also provides an intelligent agent expression imitation execution optimization device. Figure 4B The device is described in detail.

[0219] Figure 4B The structural block diagram of the execution optimization device for intelligent agent expression imitation according to embodiment 3 of the present invention is schematically shown.

[0220] like Figure 4B As shown, the execution optimization device 400 for agent expression imitation of this embodiment 3 includes a target task generation module 410, a task optimization module 420 and an imitation execution module 430.

[0221] The target task generation module 410 is used to generate an inverse kinematics target optimization task based on the control point simulation information of the agent expression mesh and the target skin vertex information. In one embodiment, the target task generation module 410 can be used to perform the operation S401 described above, which will not be repeated here.

[0222] The task optimization module 420 is used to perform optimization processing on the inverse kinematics target optimization task to obtain the optimal joint angle parameters corresponding to the control point simulation information. In one embodiment, the task optimization module 420 can be used to perform the operation S402 described above, which will not be repeated here.

[0223] The imitation execution module 430 is used to optimize the execution of the imitation of the intelligent agent's expression according to the optimal joint angle parameters. In one embodiment, the imitation execution module 430 can be used to execute the operation S403 described above, which will not be described in detail here.

[0224] According to an embodiment of the present invention, any multiple modules of the target task generation module 410, the task optimization module 420 and the imitation execution module 430 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present invention, at least one of the target task generation module 410, the task optimization module 420 and the imitation execution module 430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application-specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware and firmware or in any appropriate combination of any of them. Alternatively, at least one of the target task generation module 410, the task optimization module 420 and the imitation execution module 430 can be at least partially implemented as a computer program module, and when the computer program module is run, the corresponding function can be executed.

[0225] Combining the above-mentioned Examples 1-3, it can be seen that the existing intelligent agent expression imitation technology has significant technical limitations in the three key stages of perception, mapping and execution. These problems directly limit the realization of high-precision expression imitation, especially in the expression migration across entities and physical drive implementation. Among them, the perception stage faces the problems of sparse expression representation and coupling between expression and face shape; the mapping stage has difficulty in solving the difficulty of expression migration across topological structures, and the expression semantics is distorted or even lost during the migration; the execution stage has the problem of physically unexecutable generated instructions. These limitations limit the precision and robustness of expression imitation.

[0226] Therefore, in order to overcome the technical bottlenecks existing in the three key stages of perception, mapping and execution of existing expression imitation technologies, the present invention provides a set of brand-new expression imitation technical solutions, covering the three links of perception optimization, mapping optimization and execution optimization, respectively, aiming to achieve more accurate and robust expression perception and migration, while ensuring that the generated control instructions are physically executable, laying a technical foundation for the development of highly bionic expression robots. Among them, through the dense expression representation in the perception stage, the cross-topology semantic consistency migration in the mapping stage, and the high-precision physical drive optimization in the execution stage, the core problems of sparse expression description, distorted semantic migration and unfeasible execution in traditional expression imitation methods are systematically solved. The following is the core design of the new technology solution and its correspondence with the existing problems.

[0227] In the perception stage, the three-dimensional deformable model (3DMM) is mainly used to replace the traditional sparse key point representation method, and the facial motion is decomposed into three independent subspaces of posture, face shape and expression to generate a standardized and dense expression motion representation. That is, dense expression motion is extracted from the input face image to generate a high-precision expression representation under the standardized average face. This method not only improves the refinement of expression description, but also eliminates the coupling problem between expression semantics and individual face shape through standardized average face topology, and achieves consistency of expression semantics among different individuals. For details, see Example 1.

[0228] In the mapping stage, a set of Blendshape semantic spaces based on the standardized average face is provided as an intermediate bridge between the human face and the robot face, which solves the problem of differences in topological structure and geometric characteristics between the two. The dense expression motion generated in the perception stage is mapped to the low-dimensional Blendshape coefficient space in real time, and the distortion-free migration of expression from the human face to the robot face is achieved through the semantically consistent Blendshape basis. In other words, through the semantically consistent Blendshape basis, the dense expression motion is mapped to the robot facial topology to achieve cross-topological expression migration. This method not only ensures the consistency of expression semantics, but also greatly improves the efficiency and robustness of the mapping. See Example 2 for details.

[0229] In the execution phase, through high-precision physical drive optimization based on control points, inverse kinematics is used to solve the optimal position of the control points under the target expression, and accurate actuator motion instructions are generated. That is, the mapping results are converted into physically executable robot skin motion instructions, and accurate expression imitation is achieved through inverse kinematics and physical constraint optimization. This method explicitly models the physical constraints of the actuator to ensure the physical executableness of the generated instructions, see Example 3 for details.

[0230] The process provided by Examples 1-3 of the present invention optimizes layer by layer from perception to execution, ensuring the precision, semantic consistency and physical executability of expression imitation, and providing a complete solution for the realization of a highly bionic expression robot. Through systematic design and optimization, the process can achieve full-link expression imitation from original facial input to accurate reproduction of robot expressions.

[0231] In short, embodiments 1-3 of the present invention focus on a real-time imitation system of a highly biomimetic expression robot based on two-dimensional video, aiming to achieve accurate mapping and physical drive to a highly biomimetic expression robot by accurately capturing the dynamic features of human facial expressions. The technical framework covers three links: perception, migration and execution, breaking through the technical bottlenecks of traditional methods in subtle expression capture, cross-modal expression migration from human facial expressions to robot facial expressions, and physical drive. This imitation system has shown broad prospects in application scenarios. For example, in a remote embodied interactive system, an operator can remotely control a highly biomimetic expression robot through video input to achieve embodied emotional interaction across space, which is suitable for scenarios that require emotional expression, such as remote medical care and psychological counseling. In terms of basic research on social emotional computing, this technology provides solid technical support for autonomous emotional expression of robots and human-machine emotional interaction, and promotes the further development of social emotional robot research. Overall, through the innovative expression perception, mapping and execution technology optimization framework, the key technical problems in the expression migration process are systematically solved, and the real-time and accurate drive of a highly biomimetic expression robot is successfully achieved. This technology not only provides reliable technical support for remote emotional interaction, but also lays an important foundation for the development of autonomous emotional interaction robots in the future, and has extremely high commercial application value.

[0232] Figure 6 A block diagram schematically shows a perception optimization method, a mapping optimization method and an electronic device for executing the optimization method suitable for implementing the imitation of the expression of an intelligent agent according to embodiments 1-3 of the present invention.

[0233] The above-mentioned electronic device provided by an embodiment of the present invention includes one or more processors and a memory, and the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned perception optimization method, mapping optimization method and execution optimization method of intelligent agent expression imitation.

[0234] like Figure 6 As shown, the electronic device 600 according to an embodiment of the present invention includes a processor 601, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 602 or a program loaded from a storage part 608 to a random access memory (RAM) 603. The processor 601 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 601 may also include an onboard memory for caching purposes. The processor 601 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present invention.

[0235] In RAM 603, various programs and data required for the operation of electronic device 600 are stored. Processor 601, ROM 602 and RAM 603 are connected to each other via bus 604. Processor 601 performs various operations of the method flow according to the embodiment of the present invention by executing the program in ROM 602 and / or RAM 603. It should be noted that the program can also be stored in one or more memories other than ROM 602 and RAM 603. Processor 601 can also perform various operations of the method flow according to the embodiment of the present invention by executing the program stored in the one or more memories.

[0236] According to an embodiment of the present invention, the electronic device 600 may further include an input / output (I / O) interface 605, which is also connected to the bus 604. The electronic device 600 may further include one or more of the following components connected to the I / O interface 605: an input portion 606 including a keyboard, a mouse, etc.; an output portion 607 including a cathode ray tube (CRT), a liquid crystal display (LCD), etc., and a speaker, etc.; a storage portion 608 including a hard disk, etc.; and a communication portion 609 including a network interface card such as a LAN card, a modem, etc. The communication portion 609 performs communication processing via a network such as the Internet. A drive 610 is also connected to the I / O interface 605 as needed. A removable medium 611, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed on the drive 610 as needed, so that a computer program read therefrom is installed into the storage portion 608 as needed.

[0237] The present invention also provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned perception optimization method, mapping optimization method and execution optimization method for imitating the expression of an intelligent agent.

[0238] The computer-readable storage medium may be included in the device / apparatus / system described in the above embodiment; or it may exist independently without being assembled into the device / apparatus / system. The above computer-readable storage medium carries one or more programs. When the above one or more programs are executed, the perception optimization method, mapping optimization method and execution optimization method of the above intelligent agent expression imitation according to embodiments 1-3 of the present invention are implemented.

[0239] According to an embodiment of the present invention, the computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, it may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present invention, the computer-readable storage medium may be any tangible medium containing or storing a program, which may be used by or in combination with an instruction execution system, an apparatus or a device. For example, according to an embodiment of the present invention, the computer-readable storage medium may include the ROM 602 and / or RAM 603 described above and / or one or more memories other than ROM 602 and RAM 603.

[0240] An embodiment of the present invention also includes a computer program product, which includes a computer program, which, when executed by a processor, implements the above-mentioned perception optimization method, mapping optimization method and execution optimization method of intelligent agent expression imitation.

[0241] The computer program includes program codes for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program codes are used to enable the computer system to implement the perception optimization method, mapping optimization method and execution optimization method for the above-mentioned intelligent agent expression imitation provided in Embodiments 1-3 of the present invention.

[0242] The computer program executes the above functions defined in the system / device of the embodiment of the present invention when it is executed by the processor 601. According to the embodiment of the present invention, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0243] In one embodiment, the computer program may rely on tangible storage media such as optical storage devices, magnetic storage devices, etc. In another embodiment, the computer program may also be transmitted and distributed in the form of signals on a network medium, and downloaded and installed through the communication part 609, and / or installed from a removable medium 611. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the above.

[0244] In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 609, and / or installed from the removable medium 611. When the computer program is executed by the processor 601, the above functions defined in the system of the embodiment of the present invention are performed. According to the embodiment of the present invention, the system, device, means, module, unit, etc. described above can be implemented by a computer program module.

[0245] According to an embodiment of the present invention, the program code for executing the computer program provided by the embodiment of the present invention can be written in any combination of one or more programming languages. Specifically, these computing programs can be implemented using high-level process and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, Java, C++, python, "C" language or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on the remote computing device, or entirely on the remote computing device or server. In the case of a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (e.g., using an Internet service provider to connect through the Internet).

[0246] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present invention. In this regard, each box in the flow chart or block diagram can represent a module, a program segment, or a part of a code, and the above-mentioned module, program segment, or a part of a code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order from the order marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flow chart, and the combination of the boxes in the block diagram or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0247] In addition, all actions of acquiring information, signals or data in the present invention are carried out in compliance with the corresponding data protection laws, regulations and policies of the country where they are located, and with the authorization given by the owner of the corresponding device.

[0248] It will be appreciated by those skilled in the art that the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways, even if such combinations and / or combinations are not explicitly described in the present invention. In particular, the features described in the various embodiments and / or claims of the present invention may be combined and / or combined in various ways without departing from the spirit and teachings of the present invention. All of these combinations and / or combinations fall within the scope of the present invention.

[0249] The embodiments of the present invention are described above. However, these embodiments are only for the purpose of illustration, and are not intended to limit the scope of the present invention. Although each embodiment is described above, it does not mean that the measures in each embodiment cannot be used in combination. The scope of the present invention is defined by the attached claims and their equivalents. Without departing from the scope of the present invention, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present invention.

Claims

1. A method for optimizing the execution of an intelligent agent's facial expression imitation, characterized in that: include: Generate inverse kinematics target optimization tasks based on control point simulation information and target surface vertex information of the agent's expression mesh; Performing optimization processing on the inverse kinematics target optimization task to obtain optimal joint angle parameters corresponding to the control point simulation information; as well as The execution optimization of the intelligent agent's facial expression imitation is performed according to the optimal joint angle parameters.

2. The method according to claim 1, characterized in that Before the control point simulation information and target surface vertex information based on the intelligent body expression grid generate the inverse kinematics target optimization task, it also includes: Generate an agent expression mesh corresponding to the agent according to the mixed shape basis of the agent's face; A distance weight matrix is ​​generated based on the control point simulation information and target surface vertex information of the agent expression mesh.

3. The method according to claim 2, characterized in that Before generating a distance weight matrix based on the control point simulation information and target epidermal vertex information of the agent expression grid, the method further includes: Generate the control point simulation information according to the preset positive kinematics optimization task of the agent expression grid and the joint angle parameter vector corresponding to the simulation control point; The target epidermal vertex information is generated according to the deformation weight matrix of the epidermal mesh vertices of the agent expression mesh and the control point simulation information.

4. The method according to claim 3, characterized in that In the task of generating inverse kinematics target optimization based on the control point simulation information and target surface vertex information of the intelligent body expression grid, it includes: Expanding the distance weight matrix into a weight augmentation matrix; Generate control point driven skin mesh vertices according to the control point simulation information and the weight amplification matrix; The inverse kinematics target optimization task is generated according to the control point driven skin mesh vertices and target skin vertex information.

5. The method according to claim 1, characterized in that: In performing optimization processing on the inverse kinematics target optimization task to obtain the optimal joint angle parameters corresponding to the control point simulation information, the method includes: The inverse kinematics target optimization task is subjected to minimization target optimization iterative processing according to the joint angle constraint conditions to generate the optimal joint angle parameters.

6. The method according to claim 1, characterized in that In the execution optimization of the agent expression imitation according to the optimal joint angle parameters, it includes: Acquire control point driving information of the agent's facial expression imitation through control point simulation information and the optimal joint angle parameters; The driving of the intelligent agent's expression imitation is executed according to the control point driving information to complete the execution optimization.

7. An execution optimization device for imitating facial expressions of an intelligent agent, characterized in that: include: The target task generation module is used to generate the inverse kinematics target optimization task based on the control point simulation information of the intelligent agent expression mesh and the target skin vertex information; A task optimization module, used to perform optimization processing on the inverse kinematics target optimization task to obtain the optimal joint angle parameters corresponding to the control point simulation information; as well as The imitation execution module is used to optimize the execution of the intelligent agent's expression imitation according to the optimal joint angle parameters.

8. An electronic device comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors execute the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images