Facial expression migration method and device, electronic equipment and storage medium

Abstract

The application provides an expression migration method, device, equipment and storage medium, and relates to the field of computer vision of artificial intelligence. The application obtains a topological alignment first mesh model and a topological alignment second mesh model of a target character by performing topological alignment on an original character and the target character; determines a point pair in mutual nearest neighbor relationship in the first mesh model and the second mesh model; determines a first anchor point of a first expression to which a first neutral face model of the original character is deformed; acquires a second anchor point of a second expression to which a second neutral face model of the target character is deformed according to the point pair in mutual nearest neighbor relationship and the first anchor point; and determines vertex coordinates of the second expression by using an expression migration algorithm with the first anchor point and the second anchor point as constraints. The application can adjust a deformation area based on a neutral face when performing expression migration between characters with different topologies, and ensure that the overall expression model after migration remains stable in relative position.

Description

Technical Field

[0001] This application relates to the field of computer vision in artificial intelligence, and more specifically, to methods, apparatus, devices, and storage media for facial expression transfer. Background Technology

[0002] Blendshape (BS) facial expression models are a technique used to describe the deformation of human facial models. This allows artists and animators to intuitively control facial shapes, and it has applications in many fields such as film, animation, 3D digital human and game production. In typical film and television animation, game, and 3D digital human production, multiple characters may exist, but the basic expression definitions can be reused. Deformation transfer (Detrans) technology is mainly used to quickly transfer the deformed expression base BS of the original character relative to the neutral face mesh to the neutral face of the target character, obtaining an approximate deformed expression base BS of the target character.

[0003] In related technologies, after obtaining the triangular facet correspondence between the original character mesh and the target character mesh, anchor points are determined by calculating the vertex displacements of the original character's facial expression base (BS) and the neutral face. Then, when determining the vertex coordinates of the target character's facial expression base (BS) using the principle of minimizing deformation transfer of triangular facets, this anchor point constraint is added to reduce the offset of the transferred facial expression base (BS). However, this scheme assumes that the original and target characters use the same topology for facial expression transfer, in which case the facet correspondence between the original and target characters is direct. This scheme cannot meet the facial expression transfer requirements between characters with different topologies, thus limiting its application scenarios. Summary of the Invention

[0004] This application provides a method, apparatus, device, and storage medium for facial expression transfer, which enables the adjustment of the deformation area based on a neutral face when transferring facial expressions between characters with different topologies, ensuring that the overall relative position of the transferred facial expression model remains stable.

[0005] Firstly, a method for expression transfer is provided, including:

[0006] Obtain the first neutral face model of the original character and the first expression corresponding to the first neutral face model, as well as the second neutral face model of the target character; wherein, the original character is a first topological structure, and the target character is a second topological structure;

[0007] The original character and the target character are topologically aligned to obtain a first mesh model and a second mesh model of the target character, wherein the first mesh model is the first topological structure and the second mesh model is the second topological structure;

[0008] In the first and second grid models, determine the nearest neighbor pairs of points;

[0009] Based on the vertex displacement between the first neutral face model and the first expression, determine the first anchor point where the first neutral face model is deformed and migrated to the first expression;

[0010] From the nearest neighbor pairs, obtain the nearest neighbor points corresponding to the first anchor point, and use them as the second anchor points for the deformation and migration of the second neutral face model to the second expression.

[0011] Using the first and second anchor points as constraints, and employing the principle of minimizing deformation migration of the triangular facets of the first and second mesh models, the vertex coordinates of the second expression are determined.

[0012] Secondly, an apparatus for expression transfer is provided, comprising:

[0013] The acquisition unit is used to acquire a first neutral face model of the original character and a first expression corresponding to the first neutral face model, as well as a second neutral face model of the target character; wherein the original character is a first topological structure and the target character is a second topological structure;

[0014] An alignment unit is used to perform topological alignment between the original character and the target character to obtain a first mesh model and a second mesh model of the topological alignment of the target character, wherein the first mesh model is the first topological structure and the second mesh model is the second topological structure;

[0015] A determining unit is used to determine the nearest neighbor pairs of points in the first grid model and the second grid model;

[0016] The determining unit is further configured to determine the first anchor point of the first neutral face model to be deformed and migrated to the first expression based on the vertex displacement between the first neutral face model and the first expression;

[0017] The determining unit is further configured to obtain the nearest neighbor point corresponding to the first anchor point from the nearest neighbor point pairs, and use it as the second anchor point for the deformation and migration of the second neutral face model to the second expression.

[0018] An optimization unit is used to determine the vertex coordinates of the second expression by taking the first anchor point and the second anchor point as constraints and using the principle of minimizing deformation migration of the triangular facets of the first mesh model and the second mesh model.

[0019] Thirdly, this application provides an electronic device, comprising:

[0020] Processor, adapted to implement computer instructions; and,

[0021] A memory that stores computer instructions adapted for loading by a processor and executing the method described in the first aspect above.

[0022] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer instructions that, when read and executed by a processor of a computer device, cause the computer device to perform the method described in the first aspect.

[0023] Fifthly, embodiments of this application provide a computer program product or computer program that includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the first aspect.

[0024] Based on the above technical solutions, the embodiments of this application can add anchor point constraints of the original character and the target character to the expression transfer algorithm between characters with different topologies. This is beneficial for adjusting the deformation area of ​​the transferred expression based on the neutral face, without causing the overall coordinates of the transferred expression to shift relative to the neutral face, thus ensuring that the overall position of the transferred expression model remains stable. Attached Figure Description

[0025] Figure 1 This is a schematic diagram illustrating an application scenario according to an embodiment of this application;

[0026] Figure 2 This is a schematic flowchart illustrating a method for facial expression transfer according to an embodiment of this application;

[0027] Figure 3 This is a schematic flowchart illustrating another expression transfer method according to an embodiment of this application;

[0028] Figure 4 This is a schematic flowchart illustrating another expression transfer method according to an embodiment of this application;

[0029] Figure 5 This is a schematic flowchart illustrating another expression transfer method according to an embodiment of this application;

[0030] Figure 6 This is a diagram illustrating the effect of overlaying the target character's facial expression and neutral face after migration.

[0031] Figure 7This is a schematic block diagram of the facial expression transfer device provided in the embodiments of this application;

[0032] Figure 8 This is a schematic block diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0033] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0034] It should be understood that in the embodiments of this application, "B corresponding to A" means that B is associated with A. In one implementation, B can be determined based on A. However, it should also be understood that determining B based on A does not mean determining B solely based on A; B can also be determined based on A and / or other information.

[0035] In the description of this application, unless otherwise stated, "at least one" means one or more, and "multiple" means two or more. Additionally, "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can mean: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0036] It should also be understood that the descriptions of "first", "second", etc. appearing in the embodiments of this application are only for illustration and to distinguish the objects being described, and there is no order to them. They do not indicate any special limitation on the number of devices in the embodiments of this application, and cannot constitute any limitation on the embodiments of this application.

[0037] It should also be understood that specific features, structures, or characteristics relating to embodiments in the specification are included in at least one embodiment of this application. Furthermore, these specific features, structures, or characteristics may be combined in any suitable manner in one or more embodiments.

[0038] Furthermore, the terms “comprising” and “having”, and any variations thereof, are intended to cover non-exclusive inclusion, such that a process, method, system, product, or server that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products, or devices.

[0039] The solutions provided in this application may involve artificial intelligence technology.

[0040] Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or computers-controlled machines to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce new intelligent machines that can react in a way similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities.

[0041] It should be understood that artificial intelligence (AI) technology is a comprehensive discipline involving a wide range of fields, encompassing both hardware and software technologies. Fundamental AI technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly include computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0042] With the research and advancement of artificial intelligence (AI) technology, AI is being studied and applied in various fields, such as smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, autonomous driving, drones, robots, smart healthcare, and smart customer service. It is believed that with the development of technology, AI will be applied in more fields and play an increasingly important role.

[0043] This application's embodiments may relate to Computer Vision (CV) technology within artificial intelligence. Computer vision is a science that studies how to enable machines to "see." More specifically, it refers to machine vision that uses cameras and computers to replace human eyes in recognizing, monitoring, and measuring targets, and further performs image processing to transform the computer-processed images into those more suitable for human observation or transmission to instruments for detection. As a scientific discipline, computer vision researches related theories and technologies, attempting to establish artificial intelligence systems capable of extracting information from images or multidimensional data. Computer vision technologies typically include image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, 3D object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), and other technologies, as well as common biometric recognition technologies such as facial recognition and fingerprint recognition.

[0044] This application's embodiments may also relate to Machine Learning (ML) in artificial intelligence technology. ML is a multidisciplinary field involving probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory, and many other disciplines. It specifically studies how computers can simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to endow computers with intelligence; its applications span all areas of artificial intelligence. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and instructional learning.

[0045] The solutions provided in this application also relate to video processing technology in the field of network media. Network media differs from traditional audio and video equipment in its operation; it relies on technologies and equipment provided by information technology (IT) equipment developers to transmit, store, and process audio and video signals. Traditional Serial Digital Induction (SDI) transmission lacks true network switching capabilities. Significant work is required to utilize SDI to create some of the network functions provided by Ethernet and Internet Protocol (IP). Therefore, network media technology in the video industry has emerged. Furthermore, video processing technology for network media can include the transmission, storage, and processing of audio and video signals, as well as the audio and video...

[0046] To facilitate understanding of the technical solution provided in this application, the following explanation is provided regarding the content related to facial expression transfer.

[0047] Facial expression transfer: refers to the process of transferring the facial expression of the original character relative to the neutral face mesh onto the neutral face of the target character, thereby obtaining an approximate deformed facial expression of the target character. It should be understood that the neutral face model and the corresponding facial expression base BS involved in the embodiments of this application are both 3D facial models.

[0048] 3D Morphable Models (3DMMs): 3DMMs are a general 3D parametric model of faces that uses a fixed number of points to represent a face. The core idea of ​​3DMMs is to match faces one-to-one in 3D space and obtain a face model by performing orthogonal basis weighted linear summation on multiple faces in a database.

[0049] Each 3D face can be represented by a basis vector space consisting of all faces in a database. Solving for the model of any 3D face is actually equivalent to solving for the coefficients of each basis vector.

[0050] The basic attributes of a human face include shape and texture. Each human face can be represented as a linear superposition of shape vectors and texture vectors.

[0051] Shape Vector: S = (X1, Y1, Z1, X2, Y2, Z2, ..., Yn, Zn)

[0052] Texture Vector: T = (R1, G1, B1, R2, G2, B2, ..., Rn, Bn)

[0053] Where n is the number of face samples in the dataset, Xi, Yi, Zi are the coordinates of the shape vector of the i-th face sample in the dataset, and Ri, Gi, Bi are the coordinates of the texture vector of the i-th face sample in the dataset.

[0054] Any face model can be obtained by weighted combination of m face models in the dataset as follows:

[0055] ;

[0056] ;

[0057] Among them, S model For a 3D human face shape model, a i Let S be the target value of the face shape parameter, i=1…m, where m is the number of face samples in the dataset, and S is the target value of the face shape parameter. i Let i be the shape vector of the i-th face sample in the dataset. T is the mean of the shape vectors of all face samples in the dataset. model For a 3D human face texture model, bi The target value for the face texture parameters, T i Let i be the texture vector of the i-th face sample in the dataset. This is the mean of the texture vectors of all face samples in the dataset.

[0058] Constraints refer to finding an element within a given function that minimizes or maximizes a certain metric. Constraints can also be termed mathematical programming (e.g., linear programming). The function can be called the objective function or cost function. A feasible solution to an objective function that minimizes or maximizes a certain metric is called the optimal solution. In the context of this application, the facial expression transfer algorithm can be used to: solve for the optimal solution under multiple constructed constraints, and use the optimal solution as the vertex coordinates of the target character's facial expression corresponding to the original character's facial expression deformation.

[0059] Blend shape (BS): A technique for deforming a single mesh to achieve a number of predefined shapes and any combination thereof, referred to as a deformable target in Maya / 3ds Max. For example, a single mesh can be a basic shape of a default shape, such as an expressionless human face, i.e., the neutral face model involved in this application. Other shapes of the basic shape are used for blending / deformation, representing different expressions (smiling, frowning, closed eyelids). These other shapes are collectively referred to as blend shapes or deformable targets, i.e., a set of basic expression BS corresponding to the neutral face model involved in this application, also known as expression base BS.

[0060] Topology refers to the layout, structure, and connectivity of points, lines, and surfaces in a polygonal network model. In expression transfer algorithms, the source and target characters have different topologies, meaning their topologies differ in the number of vertices or the composition of connecting edges and surfaces between vertices. Conversely, if the source and target characters have the same number of vertices and the composition of connecting edges and surfaces between vertices, then their topologies are the same.

[0061] Figure 1 This is a schematic diagram of an application scenario involved in an embodiment of this application.

[0062] like Figure 1 As shown, the device includes a computing device 101 and a display device 102. The computing device 101 is used to transfer the mesh deformation of the first expression of the original character relative to its neutral face to the neutral face of the target character using the expression transfer method provided in this application embodiment, thereby obtaining an approximately deformed second expression of the target character. The display device 102 is used to display the second expression of the target character obtained by the computing device 101.

[0063] For example, computing device 101 may be a user device, such as a mobile phone, tablet computer, laptop computer, handheld computer, mobile internet device (MID) or other terminal device with browser installation function.

[0064] For example, computing device 101 can be a server. There can be one or more servers. When there are multiple servers, at least two servers are used to provide different services, and / or at least two servers are used to provide the same service, such as providing the same service in a load-balanced manner. This application embodiment does not limit this.

[0065] The aforementioned servers can be independent physical servers, server clusters or distributed systems composed of multiple physical servers, or cloud servers providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Servers can also become nodes in a blockchain.

[0066] For example, when the computing device 101 has a display function, the display device 102 can be the display in the computing device 101.

[0067] For example, the display device 102 and the computing device 101 are different devices, and the display device 102 is connected to the computing device 101 via a network. The network can be an intranet, the Internet, the Global System for Mobile communication (GSM), Wideband Code Division Multiple Access (WCDMA), 4G network, 5G network, Bluetooth, Wi-Fi, voice network, or other wireless or wired networks.

[0068] It should be noted that the application scenarios of this application include, but are not limited to, the transfer of facial expressions in fields such as film, animation, 3D digital human and game production. For example, the solution provided in this application can be applied to the creation of facial expressions for 3D virtual humans, allowing the definition of basic expressions to be reused among multiple characters, avoiding the need to manually create the same set of basic expressions for each character in 3D software, and reducing the workload of artists.

[0069] In related technologies, when there is a neutral face model and N expression bases (BS) for the original character, and a neutral face model for the target character, it is necessary to transfer the deformation of the expression bases of the original character relative to the neutral face model of the original character to the neutral face model of the target character, thereby obtaining a set of expression bases for the target character corresponding to the deformation of the expression bases of the original character. For example, the topology of the original character is Topology 1, and the topology of the target character is Topology 2.

[0070] A basic Detrans method can obtain the target character's facial expression base (BS) corresponding to the original character's BS deformation through the following three steps:

[0071] The first step is to load the neutral face model files of the original and target characters using the corrstool tool. Then, in the interactive interface, select corresponding point pairs of the two character models. The point pairs should maintain consistent semantic information (e.g., left eye corner to left eye corner). Select multiple point pairs; the number of point pairs is uncontrollable. Output the corresponding point files.

[0072] The second step involves using the corresponding points as constraints and solving the problem through least squares iteration to deform the original character mesh into the target character mesh. Then, the error distance between the centroids of the triangles of the deformed target mesh and the target character mesh is calculated to obtain the correspondence between the triangle faces.

[0073] The third step is to set up a system of equations using the triangle facet pairs in the corresponding file as units, and to use the vertices of the triangle facets of the target character's facial expression basis BS as the parameters to be solved. The goal is to calculate the minimum error before and after deformation for all triangle facet pairs, and then solve the problem using least squares in gradient space.

[0074] The original method requires a subjective selection of the number of corresponding points during the migration of facial expression base frames (BSs) for different topological roles. This can easily lead to instability in the migration results of the target role's BSs, resulting in rework. Secondly, the lack of anchor point constraints means that when migrating the same set of BSs, the migration may result in a shift in the BSs relative to the neutral face, as the calculation is independent of the translation parameters, affecting subsequent applications.

[0075] An improved Detrans method, assuming that expression base frame (BS) transfer is performed on both the original and target characters within the same topology, ensures a direct face-to-face correspondence between them. Therefore, only the third step described above is needed to solve for the vertex coordinates of the triangular facets of the target character's expression base BS. Furthermore, this method determines anchor points by calculating the vertex displacements of the original character's expression base BS and the neutral face, then incorporates anchor point constraints for least-squares solution. This improved method, by adding anchor point constraints, considers the offset variable during expression base BS transfer, preventing overall coordinate shifts after transfer. However, this method only considers expression transfer between characters within the same topology and cannot meet the needs of expression transfer between characters with different topologies, thus limiting its application scenarios.

[0076] In view of this, embodiments of this application provide a method, apparatus, device, and storage medium for facial expression transfer, which can adjust the deformation area based on a neutral face when transferring facial expressions between characters with different topologies, ensuring that the overall relative position of the transferred facial expression model remains stable.

[0077] Specifically, in this expression transfer method, the original character and the target character are first topologically aligned to obtain a first mesh model and a second mesh model of the target character. Further, nearest neighbor pairs are determined in the first and second mesh models. Then, based on the vertex displacement between the first neutral face model of the original character and the first expression corresponding to the first neutral face model, the first anchor point for the deformation transfer of the first neutral face model to the first tag is determined. From the aforementioned nearest neighbor pairs, the nearest neighbor points corresponding to the first anchor point are obtained as the second anchor point for the deformation transfer of the second neutral face model of the target character to the second expression. Finally, using the first and second anchor points as constraints, the vertex coordinates of the second expression are determined by minimizing the deformation transfer using the triangle facets of the first and second mesh models.

[0078] The above technical solution enables the addition of anchor point constraints between the original character and the target character in the facial expression transfer algorithm between characters with different topologies. This facilitates the adjustment of the deformation area of ​​the transferred facial expression based on the neutral face, without causing the overall coordinates of the transferred facial expression to shift relative to the neutral face, thus ensuring the stability of the overall relative position of the transferred facial expression model.

[0079] The technical solutions provided in the embodiments of this application will now be described with reference to the accompanying drawings.

[0080] Figure 2A schematic flowchart of an expression transfer method 200 according to an embodiment of this application is shown. This expression transfer method 200 can be executed by any electronic device with data processing capabilities; for example, the electronic device can be implemented as a server, or, for example, as a... Figure 1 The computing device 101 in the application is not limited in this respect.

[0081] like Figure 2 As shown, the expression transfer method 200 may include steps 210 to 260.

[0082] 210. Obtain the first neutral face model of the original character and the first expression corresponding to the first neutral face model, as well as the second neutral face model of the target character; wherein the original character is the first topological structure and the target character is the second topological structure.

[0083] The objective of this application embodiment is to transfer the first expression of the original character to the target character based on the existing first neutral face model and first expression of the original character (topology 1) and the second neutral face model of the target character (topology 2), thereby obtaining a second expression of the target character corresponding to the first expression of the original character, and then performing expression-driven operation on the target character based on the second expression.

[0084] The first neutral face model of the original character can be the basic shape of the default shape of the original character, such as the expressionless face of the original character. The second neutral face model of the target character can be the basic shape of the default shape of the target character, such as the expressionless face of the target character.

[0085] In some embodiments, the first expression may include the expression base (BS) of the original character, or an expression obtained by linearly weighting a combination of at least two expression bases (BS) of the original character. The first expression of the original character, the first neutral face model, and the expression base (BS) corresponding to the first neutral face model have the same topological structure as the original character.

[0086] For example, during the creation of a 3D digital human, a set of basic facial expressions (B) can be created based on the Blendshape method. i (i = 1,2,…,N), also called the standard facial expression base (BS). These facial expression models have the same topological structure. Different facial expressions (B) i This can be seen as a deformation of a mesh based on a neutral face (B0). Furthermore, other complex expressions B can all be derived from this set of basic expressions B. i The result is obtained by linear weighted combination, see formula (1):

[0087] (1)

[0088] In some examples, the first expression can be an expression B obtained by deforming the neutral face B0 of the original character. i Or, for basic expression B i The expression B obtained by linear weighted combination is not limited.

[0089] In some embodiments, a first mouth model of the original character and a second mouth model of the target character can also be obtained. Since a basic expression model typically includes at least one open mouth state, the first mouth model of the original character can be obtained from the basic expression model of the original character. When a basic expression model for the target character does not exist, the second mouth model for the target character can be created separately by an artist.

[0090] 220. Perform topological alignment on the original character and the target character to obtain a first mesh model and a second mesh model of the topological alignment of the target character, wherein the first mesh model is the first topological structure mentioned above, and the second mesh model is the second topological structure mentioned above.

[0091] In some embodiments, as an implementation of topological alignment between the original character and the target character, the first facial expression model of the original character and the second facial expression model of the target character can be topologically aligned to obtain a first mesh model and a second mesh model of the target character. The first facial expression model has the same topology as the original character, forming a first topological structure, and the second facial expression model has the same topology as the target character, forming a second topological structure.

[0092] As one implementation method, the first expression model can be the first mouth model of the original character, and the second expression model can be the second mouth model of the target character. Since the mouth model includes both the outer and inner lip line contour key points, by accurately aligning the inner and outer lip shapes of the original and target characters, the topological alignment of the original and target characters can be achieved more accurately. This helps to make the lip-shape-driven expression base BS transfer results more accurate, thereby making the lip-shape driving of the target character more accurate.

[0093] As an alternative implementation, the first facial expression model can be the first neutral face model of the original character, and the second facial expression model can be the second neutral face model of the target character. For example, when it is inconvenient to provide the mouth-opening model of the target character, the topological alignment of the original character and the target character can be achieved using the first neutral face model of the original character and the second neutral face model of the target character.

[0094] In some embodiments, see Figure 3 The original character and the target character can be topologically aligned through steps 221 to 224 to obtain the first mesh model and the second mesh model of the topological alignment of the target character.

[0095] 221. Select the first corresponding point with the same semantic position on the first expression model of the original character and the second expression model of the target character respectively, and perform rigid transformation on the first expression model and the second expression model according to the first corresponding point.

[0096] Specifically, rigid transformation can roughly adjust the first and second expression models to a roughly aligned position, which helps to make further vertex adjustments easier, and also provides better initial values ​​for the final alignment solution process.

[0097] For example, a first expression model and a second expression model, such as a first mouth model and a second mouth model, can be imported into a topology alignment software (e.g., Wrap 3.4). Then, a first corresponding point can be selected on each of the two expression models at the same semantic location (e.g., at least one of the following: nose tip, corner of eye, corner of mouth, and earlobe). This first corresponding point is then used to perform a rigid transformation on the first and second expression models. Optionally, the first corresponding point can be a small number of points.

[0098] 222, Add a second corresponding point for selecting the lip line region in the first and second expression models after rigid transformation.

[0099] For example, when the first expression model is the first mouth model and the second expression model is the second mouth model, a second corresponding point can be selected from the key points of the outer lip line contour and the key points of the inner lip line contour.

[0100] Optionally, since rigid transformation may affect the position of the first corresponding point in the first expression model and the second expression model, the previously selected first corresponding point can be appropriately adjusted in the first expression model and the second expression model after rigid transformation so that the first corresponding point can correspond accurately.

[0101] 223. Based on the first and second corresponding points mentioned above, perform non-rigid iterative nearest point calculation on the first and second expression models after rigid transformation to obtain the topologically aligned first and second expression models.

[0102] It should be understood that the Non-rigid Iterative Closest Point (NICP) algorithm is based on the Iterative Closest Point (ICP) algorithm. When seeking the matching relationship between corresponding points of the first expression model and the second expression model, it allows non-rigid deformation to occur within the point set of the first expression model.

[0103] 224. The topologically aligned first facial expression model is used as the first mesh model of the target character, and the topologically aligned second facial expression model is used as the second mesh model of the target character.

[0104] In this context, the topologically aligned first and second facial expression models are two mesh models with closely aligned geometric structures for the same character, representing two different topologies. The first facial expression model represents the original character's topology (first topology), while the second facial expression model represents the target character's topology (second topology). Here, assuming the target character is used as a reference, the topologically aligned first facial expression model can be used as the target character's first mesh model, and the topologically aligned second facial expression model can be used as the target character's second mesh model, thus obtaining two topologically aligned mesh models for the target character.

[0105] 230. Determine the nearest neighbor pairs of points in the first and second grid models described above.

[0106] Here, the nearest neighbor pairs in the first and second mesh models can be considered as the corresponding vertices (i.e., vertex correspondence) between the topology of the original character and the target character, or as the vertex proximity relationship between the original character and the target character.

[0107] In some embodiments, see Figure 4 The nearest neighbor pairs can be determined in the first and second grid models through steps 231 to 233.

[0108] 231. Obtain the first nearest neighbor of a point in the first grid model in the second grid model.

[0109] Specifically, we can iterate through all points in the first grid model and find the nearest neighbor of each point in the second grid model in three-dimensional space. For example, we can use the k-nearest neighbors algorithm (KNN) to find the nearest neighbor.

[0110] 232. By reverse lookup, obtain the second nearest neighbor of a point in the second grid model to the first grid model.

[0111] Specifically, the process of obtaining the second nearest neighbor of all points in the first grid model is similar to the process of obtaining the first nearest neighbor of all points in the second grid model. Please refer to the description above, which will not be repeated here.

[0112] 233. Based on the first and second nearest neighbors mentioned above, determine the pairs of points that are nearest neighbors to each other.

[0113] Specifically, the first and second nearest neighbors can be merged, retaining the pairs of points that are each other's nearest neighbors.

[0114] 240. Based on the vertex displacement between the first neutral face model and the first expression, determine the first anchor point for the deformation migration of the first neutral face model to the first expression.

[0115] In some embodiments, a vertex in the first neutral face model whose vertex displacement relative to the first expression is less than a preset value can be used as the first anchor point. It is understood that during the process of the first neutral face model transforming into the first expression, the vertex displacement of the first anchor point is less than the preset value, that is, it can be considered that the position of the first anchor point changes very little or even not at all during the process. Therefore, the first anchor point can also be referred to as the anchor point of the original character.

[0116] As an example, the vertex displacement between the first vertex in the first neutral face model and the second vertex corresponding to the first vertex in the first expression can be determined according to the following formula (2):

[0117] (2)

[0118] The coordinates of the first vertex are ( The coordinates of the second vertex are ( ), d represents the distance between the first vertex and the second vertex.

[0119] In some embodiments, the first anchor point includes a vertex of the back of the head region of a first neutral face model. Specifically, since the vertices of the back of the head region of a typical human model do not move significantly with facial expression deformation, the vertices of the back of the head region are generally considered to be the best candidate vertices for the anchor point.

[0120] 250. From the above pairs of points that are the nearest neighbors, obtain the nearest neighbor points corresponding to the first anchor point, and use them as the second anchor points for the deformation and transfer of the second neutral face model to the second expression.

[0121] Specifically, after obtaining the first anchor point, the nearest neighbor point corresponding to the first anchor point can be obtained based on the nearest neighbor point pairs in the first and second mesh models. This point is the anchor point of the target character corresponding to the anchor point of the original character, that is, the second anchor point of the second neutral face model deformation migration to the second expression.

[0122] It is understandable that the anchor point of the target character, which is the anchor point corresponding to the original character, has very little change in position during the process of the target character's second neutral face shape transforming into the second expression, or even no change at all.

[0123] As one implementation method, the vertex index of the second anchor point can be found from the intersection of the nearest neighbor pairs of the first mesh model and the second mesh model obtained in step 230 and the vertex index of the first anchor point obtained in step 240, and used as the anchor point when the second neutral face model of the target character is migrated to the second expression.

[0124] Therefore, this application embodiment introduces the vertex proximity relationship after topological alignment of the original character and the target character as a priori, and reuses the first anchor point found in the topology of the original character to the topology of the target character to quickly and automatically locate the second anchor point, so as to obtain the anchor point of the target character's second neutral face shape when migrating to the second expression without relying on the semantic information of the face model.

[0125] 260. Using the first and second anchor points as constraints, and employing the principle of minimizing deformation migration of the triangular patches of the first and second mesh models, the vertex coordinates of the second expression are determined.

[0126] Specifically, the triangular facets in the first mesh model correspond to the triangular facets in the second mesh model, forming a set of triangle correspondences. Then, this set of triangle correspondences is traversed, and for each triangle facet pair, the vertex coordinates of the second expression are solved using the least squares method, based on the principle of minimizing deformation transfer and adding the first and second anchor points as constraints.

[0127] As one implementation method, the centroid distance and normal phase deviation of the triangular facets between the two sets of topologically aligned first and second mesh models can be used as threshold judgment conditions to obtain the triangular facets of the second mesh model of the second topological structure corresponding to each triangular facet in the first mesh model of the first topological structure, forming a set of triangular correspondences.

[0128] In some embodiments, see Figure 5 The vertex coordinates of the second expression can be determined by using the principle of minimizing deformation migration of the triangular facets of the first and second mesh models through steps 261 to 263.

[0129] 261. Based on the deformation gradient calculated from the vertex coordinates of the first anchor point triangle before and after the deformation, the deformation gradient of the second anchor point, and the first weight, determine the first constraint condition.

[0130] As an example, the first constraint can be expressed as the following formula (3):

[0131] (3)

[0132] in, These are the vertex coordinates of the second expression after migration, which are to be determined. It is the form after the deformation gradient of the second anchor point of the target character is expanded. This is the deformation gradient matrix of the corresponding second anchor point. The deformation gradient is calculated based on the vertex coordinates of the triangle before and after deformation of the original character's first anchor point. This indicates the first weight, which avoids adjusting the 3D coordinates of the anchor point during the process of solving for the vertex coordinates of the second expression. As an example, when the first weight... When the setting is larger, the adjustment of the 3D coordinates of the second anchor point is smaller during the process of solving the vertex coordinates of the second expression. As a concrete example, the first weight... Values ​​can be retrieved .

[0133] It should be understood that the above formula (3) helps to ensure that the anchor point of the target character in the neutral face model after deformation (i.e., the second expression) remains consistent with its original position on the neutral face. Simultaneously, a first weight is added to both sides of the equation in formula (3) during the overall solution process. Performing a joint solution is equivalent to imposing constraints on these anchor points during the solution process.

[0134] 262. The deformation gradient calculated based on the vertex coordinates of the original character's triangle before and after the transformation is equal to the deformation gradient of the target character, thus determining the second constraint condition. The second constraint condition can also be called the system of equations for the free points.

[0135] As an example, the second constraint can be expressed as the following formula (4):

[0136] (4)

[0137] in, These are the vertex coordinates of the second expression after migration, which are to be determined. It is the form of the second facial expression deformed vertex gradient after unfolding. The gradient matrix of the corresponding second facial expression vertex deformation. It is the deformation gradient calculated based on the vertex coordinates of the original character's triangle before and after deformation.

[0138] In some embodiments, before determining the second constraint, a transformation matrix can be determined based on the first and second anchor points. Then, based on this transformation matrix, the vertex coordinates of the original character in its original coordinate system are transformed to obtain the vertex coordinates of the original character in the target character's coordinate system. After obtaining the vertex coordinates of the original character in the target character's coordinate system, the deformation gradient (i.e., F) calculated from the vertex coordinates of the original character before and after the triangle transformation in the target character's coordinate system is equal to the deformation gradient (i.e., F) of the target character. ), thus determining the second constraint condition mentioned above.

[0139] Specifically, when calculating the deformation gradient F based on the vertex coordinates before and after deformation of the first mesh model corresponding to the original character's topology, if the rotational deviation between the coordinates of the first mesh model and the second mesh model corresponding to the target character's topology is too large, the deviation in the second expression obtained by transferring the deformation of the original character's first expression to the target character will be relatively large. To avoid this situation, a transformation matrix can be obtained by performing Singular Value Decomposition (SVD) on the common point pair formed by the obtained first and second anchor points. This transformation matrix represents the rotational translation between the first and second mesh models. Based on this transformation matrix, the vertex coordinates of the first mesh model (i.e., the original character's coordinate system) are transformed, aligning the vertex coordinates of the first mesh model to the second mesh model (i.e., the target character's coordinate system). This reduces the rotational deviation between the coordinates of the first mesh model and the second mesh model corresponding to the target character's topology, thereby reducing the deviation in the second expression obtained by transferring the deformation of the original character's first expression to the target character.

[0140] 263. Based on the first and second constraints mentioned above, the vertex coordinates of the second expression are determined using the principle of minimizing deformation migration of the triangular facets of the first and second mesh models.

[0141] Specifically, we can iterate through the corresponding sets of the above triangles and obtain the following formula (5) for each triangle facet pair based on the principle of minimizing deformation and migration:

[0142] (5)

[0143] in, These are the vertex coordinates of the second expression after migration, which are to be determined. It is the form of the second facial expression deformed vertex gradient after unfolding. The gradient matrix of the corresponding second facial expression vertex deformation. It is the deformation gradient calculated based on the vertex coordinates of the original character's triangle before and after deformation.

[0144] It should be understood that formula (5) minimizes the difference between the deformation of the original character's first expression before and after migration to the neutral face of the target character and the migration deformation of the original character's neutral face. The process of minimizing formula (5) can be simplified to solving a system of linear equations. For ease of understanding, the embodiments of this application can list the linear system of equations composed of the above formulas (3) and (4), i.e., the following formula (6), for least squares solution.

[0145] (6)

[0146] In addition, in the process of solving least squares, in order to solve the minimization in formula (5), it is equivalent to solving formula (5) in the gradient space, that is, to differentiate formula (5) and the result of the differentiation is equal to 0. After differentiation, the normal equation in the above formula (6) can be obtained, and the solution of global minimization can be achieved.

[0147] In the process of solving the linear system using formula (6), that is, after differentiating formula (5), we can obtain the normal equation shown in formula (7). Finally, we can obtain the value of the unknown parameter x based on the normal equation.

[0148] (7)

[0149] Therefore, this embodiment of the application obtains a first mesh model and a second mesh model of the target character by performing topological alignment on the original character and the target character. Further, it identifies nearest neighbor pairs in the first and second mesh models. Then, based on the vertex displacement between the first neutral face model of the original character and the first expression corresponding to the first neutral face, it determines the first anchor point for the deformation migration of the first neutral face to the first label. From the aforementioned nearest neighbor pairs, it obtains the nearest neighbor point corresponding to the first anchor point, which serves as the second anchor point for the deformation migration of the second neutral face model of the target character to the second expression. Finally, using the first and second anchor points as constraints, it determines the vertex coordinates of the second expression by minimizing the deformation migration using the triangle facets of the first and second mesh models. This embodiment of the application can incorporate anchor point constraints of the original character and the target character into the expression migration algorithm between characters with different topologies. This facilitates adjusting the deformation area of ​​the migrated expression based on the neutral face, without causing an offset in the overall coordinates of the migrated expression relative to the neutral face, ensuring the stability of the overall relative position of the migrated expression model.

[0150] In some embodiments, when it is necessary to transfer a set of expression bases (BSs) of the first neutral face of the original character to the second neutral face of the target character, each expression base of the original character will have a certain number of anchor points relative to the first neutral face. Therefore, the common anchor points of all expression bases relative to the neutral face in the set of expression bases transferred to the target character can be found at once as constraints for solving, thereby avoiding the need to find the anchor points of the target character for each expression base and optimize the algorithm process.

[0151] In some embodiments, after finding the anchor points of the target character, these anchor points can be treated as known parameters and excluded from the minimization process described above. Thus, the unknown parameter to be solved, i.e., the vertex coordinates of the second expression, is the coordinates of these anchor points that have been excluded.

[0152] Figure 6 The diagram illustrates the effect of superimposing the target character's expression and neutral face after migration. Figure (a) shows the effect of superimposing the target character's expression and neutral face according to the original method, and Figure (b) shows the effect of superimposing the target character's expression and neutral face according to the expression migration method of this embodiment. As can be seen from Figure (a), there is an overall coordinate offset between the target character's expression and the neutral face model obtained by the original method. This is because there is no anchor point constraint, which means that in the process of solving the deformation migration minimization, the expression migration only constrains the rotation variable and does not consider the offset variable constraint. This results in an overall coordinate offset between the target character's expression and the neutral face model after migration, and the offset amount will also be different for different expressions obtained by this method. As can be seen from Figure (b), the migrated expression obtained by this embodiment adjusts the deformation area based on the neutral face, and the overall coordinates are not offset relative to the neutral face.

[0153] The specific embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application.

[0154] It should also be understood that, in the various method embodiments of this application, the sequence numbers of the above processes do not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. It should be understood that these sequence numbers can be interchanged where appropriate so that the embodiments of this application described can be implemented in a sequence other than those illustrated or described.

[0155] The method embodiments of this application have been described in detail above. The following description, in conjunction with... Figures 7 to 8 The following describes in detail the device embodiments of this application.

[0156] Figure 7 This is a schematic block diagram of an expression transfer apparatus 700 according to an embodiment of this application. Figure 7 As shown, the facial expression transfer device 700 may include an acquisition unit 710, an alignment unit 720, a determination unit 730, and an optimization unit 740.

[0157] The acquisition unit 710 is used to acquire the first neutral face model of the original character and the first expression corresponding to the first neutral face model, as well as the second neutral face model of the target character; wherein, the original character is a first topological structure and the target character is a second topological structure;

[0158] Alignment unit 720 is used to perform topological alignment between the original role and the target role to obtain a first mesh model and a second mesh model of the topological alignment of the target role, wherein the first mesh model is the first topological structure and the second mesh model is the second topological structure;

[0159] The determining unit 730 is used to determine the nearest neighbor pairs of points in the first grid model and the second grid model;

[0160] The determining unit 730 is further configured to determine the first anchor point of the first neutral face model to be deformed and migrated to the first expression based on the vertex displacement between the first neutral face model and the first expression;

[0161] The determining unit 730 is further configured to obtain the nearest neighbor point corresponding to the first anchor point from the nearest neighbor point pairs, and use it as the second anchor point for the deformation and migration of the second neutral face model to the second expression.

[0162] The optimization unit 740 is used to determine the vertex coordinates of the second expression by taking the first anchor point and the second anchor point as constraints and using the principle of minimizing deformation migration of the triangle patches of the first mesh model and the second mesh model.

[0163] In some embodiments, the optimization unit 740 is specifically used for:

[0164] The first constraint condition is determined based on the deformation gradient calculated from the vertex coordinates of the first anchor point triangle before and after the deformation, the deformation gradient of the second anchor point, and the first weight.

[0165] The deformation gradient calculated based on the vertex coordinates of the original character before and after the triangle transformation is equal to the deformation gradient of the target character, and the second constraint condition is determined.

[0166] Based on the first and second constraints, the vertex coordinates of the second expression are determined using the principle of minimizing deformation migration of the triangular facets of the first and second mesh models.

[0167] In some embodiments, the optimization unit 740 is specifically used for:

[0168] Determine the transformation matrix based on the first anchor point and the second anchor point;

[0169] Based on the transformation matrix, the vertex coordinates of the original character in the original character's coordinate system are transformed to obtain the vertex coordinates of the original character in the target character's coordinate system.

[0170] The second constraint condition is determined by calculating the deformation gradient of the original character's triangle before and after deformation in the target character's coordinate system, which is equal to the deformation gradient of the target character.

[0171] In some embodiments, the alignment unit 720 is specifically used for:

[0172] On the first expression model of the original character and the second expression model of the target character, select the first corresponding point with the same semantic position, and perform rigid transformation on the first expression model and the second expression model according to the first corresponding point;

[0173] In the first and second expression models after rigid transformation, a second corresponding point is added to select the lip line region;

[0174] Based on the first corresponding point and the second corresponding point, non-rigid iterative nearest point calculation is performed on the rigidly transformed first expression model and second expression model to obtain the topologically aligned first expression model and second expression model.

[0175] The first facial expression model with topological alignment is used as the first mesh model, and the second facial expression model with topological alignment is used as the second mesh model.

[0176] In some embodiments, the same semantic location includes at least one of the tip of the nose, the corner of the eye, the corner of the mouth, and the earlobe.

[0177] In some embodiments, the first facial expression model is the first mouth model of the original character, and the second facial expression model is the second mouth model of the target character; or

[0178] The first expression model is the first neutral face model of the original character, and the second expression model is the second neutral face model of the target character.

[0179] In some embodiments, the determining unit 730 is specifically used for:

[0180] The vertex in the first neutral face model whose displacement from the first expression is less than a preset value is used as the first anchor point.

[0181] In some embodiments, the first expression includes the original character's expression base hybrid deformation (BS), or includes an expression obtained by linearly weighting a combination of at least two expression bases (BS) of the original character.

[0182] In some embodiments, the first anchor point includes the vertex of the back of the head region of the first neutral face model.

[0183] It should be understood that the device embodiments and method embodiments can correspond to each other, and similar descriptions can be referred to the method embodiments. To avoid repetition, they will not be repeated here. Specifically, in this embodiment, the expression transfer device 700 can correspond to the corresponding subject that executes the method 200 of the embodiments of this application, and the foregoing and other operations and / or functions of each module in the device 700 are respectively to implement the various methods mentioned above, or the corresponding processes in each method. For the sake of brevity, they will not be repeated here.

[0184] The apparatus and system of this application embodiments have been described above from the perspective of functional modules in conjunction with the accompanying drawings. It should be understood that these functional modules can be implemented in hardware, in software instructions, or in a combination of hardware and software modules. Specifically, the steps of the method embodiments in this application can be completed by integrated logic circuits in the processor's hardware and / or by software instructions. The steps of the methods disclosed in this application embodiments can be directly embodied as being executed by a hardware decoding processor, or by a combination of hardware and software modules in the decoding processor. Optionally, the software module can reside in a mature storage medium in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps in the above method embodiments.

[0185] like Figure 8 This is a schematic block diagram of the electronic device 800 provided in the embodiments of this application.

[0186] like Figure 8 As shown, the electronic device 800 may include:

[0187] The system includes a memory 810 and a processor 820. The memory 810 stores computer programs and transfers the program code to the processor 820. In other words, the processor 820 can retrieve and run the computer program from the memory 810 to implement the facial expression transfer method in the embodiments of this application.

[0188] For example, the processor 820 can be used to execute the steps in the method 200 described above according to the instructions in the computer program.

[0189] In some embodiments of this application, the processor 820 may include, but is not limited to:

[0190] General-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc.

[0191] In some embodiments of this application, the memory 810 includes, but is not limited to:

[0192] Volatile memory and / or non-volatile memory. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced SDRAM (ESDRAM), Synchronous Link DRAM (SLDRAM), and Direct Rambus RAM (DR RAM).

[0193] In some embodiments of this application, the computer program may be divided into one or more modules, which are stored in the memory 810 and executed by the processor 820 to complete the encoding method provided in this application. The one or more modules may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program in the electronic device 800.

[0194] Optional, such as Figure 8 As shown, the electronic device 800 may further include:

[0195] Transceiver 830, which can be connected to processor 820 or memory 810.

[0196] The processor 820 can control the transceiver 830 to communicate with other devices; specifically, it can send information or data to other devices or receive information or data sent by other devices. The transceiver 830 may include a transmitter and a receiver. The transceiver 830 may further include antennas, and the number of antennas may be one or more.

[0197] It should be understood that the various components in the electronic device 800 are connected through a bus system, which includes a data bus, a power bus, a control bus, and a status signal bus.

[0198] According to one aspect of this application, a communication device is provided, including a processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory, causing the encoder to perform the method of the above-described method embodiment.

[0199] According to one aspect of this application, a computer storage medium is provided that stores a computer program thereon, which, when executed by a computer, enables the computer to perform the methods of the above-described method embodiments. Alternatively, embodiments of this application also provide a computer program product containing instructions that, when executed by a computer, cause the computer to perform the methods of the above-described method embodiments.

[0200] According to another aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the method described in the above-described method embodiments.

[0201] In other words, when implemented using software, it can be implemented wholly or partially in the form of a computer program product. This computer program product includes one or more computer instructions. When these computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions can be transmitted from one website, computer, server, or data center to another via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., digital video disc (DVD)), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0202] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0203] In the several embodiments provided in this application, it should be understood that the disclosed devices, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or modules may be electrical, mechanical, or other forms.

[0204] The modules described as separate components may or may not be physically separate. The components shown as modules may or may not be physical modules; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. For example, the functional modules in the various embodiments of this application may be integrated into one processing module, or each module may exist physically separately, or two or more modules may be integrated into one module.

[0205] The above are merely specific embodiments of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for facial expression transfer, characterized in that, include: Obtain the first neutral face model of the original character and the first expression corresponding to the first neutral face model, as well as the second neutral face model of the target character; wherein, the original character is a first topological structure, and the target character is a second topological structure; The original character and the target character are topologically aligned to obtain a first mesh model and a second mesh model of the target character, wherein the first mesh model is the first topological structure and the second mesh model is the second topological structure; In the first and second grid models, determine the nearest neighbor pairs of points; Based on the vertex displacement between the first neutral face model and the first expression, determine the first anchor point where the first neutral face model is deformed and migrated to the first expression; From the nearest neighbor pairs, obtain the nearest neighbor points corresponding to the first anchor point, and use them as the second anchor points for the deformation and migration of the second neutral face model to the second expression. Based on the deformation gradient calculated from the vertex coordinates of the first anchor point triangle before and after deformation, and the deformation gradient of the second anchor point, the constraint conditions are determined. Using the principle of minimizing deformation migration of the triangle patches of the first and second mesh models, the vertex coordinates of the second expression are determined.

2. The method according to claim 1, characterized in that, The process of determining constraints based on the deformation gradient calculated from the vertex coordinates of the first anchor point triangle before and after deformation, and the deformation gradient of the second anchor point, and using the principle of minimizing deformation migration of the triangular facets of the first and second mesh models to determine the vertex coordinates of the second expression, includes: The first constraint condition is determined based on the deformation gradient calculated from the vertex coordinates of the first anchor point triangle before and after the deformation, the deformation gradient of the second anchor point, and the first weight. The deformation gradient calculated based on the vertex coordinates of the original character before and after the triangle transformation is equal to the deformation gradient of the target character, thus determining the second constraint condition; Based on the first and second constraints, the vertex coordinates of the second expression are determined using the principle of minimizing deformation migration of the triangular facets of the first and second mesh models.

3. The method according to claim 2, characterized in that, The second constraint condition is determined by calculating the deformation gradient based on the vertex coordinates of the original character before and after the triangle transformation, which equals the deformation gradient of the target character. This constraint includes: Determine the transformation matrix based on the first anchor point and the second anchor point; Based on the transformation matrix, the vertex coordinates of the original character in the original character's coordinate system are transformed to obtain the vertex coordinates of the original character in the target character's coordinate system. The second constraint condition is determined by calculating the deformation gradient of the original character's triangle before and after deformation in the target character's coordinate system, which is equal to the deformation gradient of the target character.

4. The method according to claim 1, characterized in that, The step of performing topological alignment on the original role and the target role to obtain a first mesh model and a second mesh model of the topological alignment of the target role includes: On the first expression model of the original character and the second expression model of the target character, select the first corresponding point with the same semantic position, and perform rigid transformation on the first expression model and the second expression model according to the first corresponding point; In the first and second expression models after rigid transformation, a second corresponding point is added to select the lip line region; Based on the first corresponding point and the second corresponding point, non-rigid iterative nearest point calculation is performed on the rigidly transformed first expression model and second expression model to obtain the topologically aligned first expression model and second expression model. The first facial expression model with topological alignment is used as the first mesh model, and the second facial expression model with topological alignment is used as the second mesh model.

5. The method according to claim 4, characterized in that, The same semantic location includes at least one of the following: the tip of the nose, the corner of the eye, the corner of the mouth, and the earlobe.

6. The method according to claim 4, characterized in that, The first facial expression model is the first mouth model of the original character, and the second facial expression model is the second mouth model of the target character; or The first expression model is the first neutral face model of the original character, and the second expression model is the second neutral face model of the target character.

7. The method according to claim 1, characterized in that, The step of determining the first anchor point for the deformation and migration of the first neutral face model to the first expression based on the vertex displacement between the first neutral face model and the first expression includes: The vertex in the first neutral face model whose displacement from the first expression is less than a preset value is used as the first anchor point.

8. The method according to claim 1, characterized in that, The first expression includes the original character's expression base hybrid deformation (BS), or includes an expression obtained by linearly weighting a combination of at least two expression bases (BS) of the original character.

9. The method according to claim 1, characterized in that, The first anchor point includes the vertex of the back of the head region of the first neutral face model.

10. A device for facial expression transfer, characterized in that, include: The acquisition unit is used to acquire a first neutral face model of the original character and a first expression corresponding to the first neutral face model, as well as a second neutral face model of the target character; wherein the original character is a first topological structure and the target character is a second topological structure; An alignment unit is used to perform topological alignment between the original character and the target character to obtain a first mesh model and a second mesh model of the topological alignment of the target character, wherein the first mesh model is the first topological structure and the second mesh model is the second topological structure; A determining unit is used to determine the nearest neighbor pairs of points in the first grid model and the second grid model; The determining unit is further configured to determine the first anchor point of the first neutral face model to be deformed and migrated to the first expression based on the vertex displacement between the first neutral face model and the first expression; The determining unit is further configured to obtain the nearest neighbor point corresponding to the first anchor point from the nearest neighbor point pairs, and use it as the second anchor point for the deformation and migration of the second neutral face model to the second expression. The optimization unit is used to determine the constraint conditions based on the deformation gradient calculated from the vertex coordinates of the first anchor point triangle before and after deformation and the deformation gradient of the second anchor point, and to determine the vertex coordinates of the second expression using the principle of minimizing deformation migration of the triangle facets of the first mesh model and the second mesh model.

11. An electronic device, characterized in that, The method includes a processor and a memory, wherein the memory stores instructions, and when the processor executes the instructions, it causes the processor to perform the method according to any one of claims 1-9.

12. A computer storage medium, characterized in that, Used to store a computer program, the computer program including a method for performing any one of claims 1-9.

13. A computer program product, characterized in that, It includes computer program code that, when executed by an electronic device, causes the electronic device to perform the method of any one of claims 1-9.

Images