Method, device and equipment for acquiring edge difference information and storage medium

Abstract

The application discloses a method and device for obtaining edge difference information, equipment and storage medium, belonging to the technical field of artificial intelligence. The method comprises the following steps: obtaining a first virtual image corresponding to a reference object in an original image; obtaining a projection object corresponding to the first virtual image; determining a first edge key point based on edge information of the projection object, the first edge key point being a key point in the first virtual image for indicating an edge of the first virtual image; and obtaining edge difference information based on the first edge key point and a second edge key point. In this process, the first edge key point for indicating the edge of the virtual image is determined based on the edge information of the projection object corresponding to the virtual image, and the matching degree with the actual edge of the virtual image is high. The obtained edge difference information can accurately indicate the difference between the edge of the virtual image and the edge of the object in the image, and the reliability and accuracy of the obtained edge difference information are good and high.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and storage medium for obtaining edge difference information. Background Art

[0002] With the development of artificial intelligence (AI) technology, the use of virtual avatars that match specific objects in images (such as faces and objects) is becoming increasingly common. To ensure that the edges of the virtual avatar are similar to those of the object in the image, it is often necessary to obtain edge difference information. This edge difference information indicates the difference between the edges of the virtual avatar and the edges of the object in the image.

[0003] In the related art, key points with certain semantics in the virtual image are pre-defined as key points for indicating the edges of the virtual image, and then edge difference information is obtained based on the key points with certain semantics in the virtual image and the key points for indicating the edges of objects in the image.

[0004] In the above manner, the key points in the virtual image used as the basis for obtaining edge difference information are key points with fixed semantics in the virtual image. The degree of matching between the key points with fixed semantics in the virtual image and the actual edges of the virtual image is low. The edge difference information obtained using the key points with fixed semantics is difficult to accurately indicate the difference between the edge of the virtual image and the edge of the object in the image. The reliability of the obtained edge difference information is poor and the accuracy is not high. Summary of the Invention

[0005] The embodiments of the present application provide a method, apparatus, device, and storage medium for obtaining edge difference information, which can be used to improve the reliability and accuracy of the obtained edge difference information. The technical solution is as follows:

[0006] In one aspect, an embodiment of the present application provides a method for obtaining edge difference information, the method comprising:

[0007] Acquire a first virtual image corresponding to a reference object in the original image;

[0008] Obtaining a projection object corresponding to the first virtual image;

[0009] determining a first edge key point based on edge information of the projection object, where the first edge key point is a key point in the first virtual image used to indicate an edge of the first virtual image;

[0010] Based on the first edge key point and the second edge key point, edge difference information is obtained, where the second edge key point is a key point in the original image used to indicate the edge of the reference object, and the edge difference information is used to indicate the difference between the edge of the first virtual image and the edge of the reference object.

[0011] In another aspect, a device for obtaining edge difference information is provided, the device comprising:

[0012] A first acquiring unit, configured to acquire a first virtual image corresponding to a reference object in an original image;

[0013] a second acquiring unit, configured to acquire a projection object corresponding to the first virtual image;

[0014] a first determining unit, configured to determine a first edge key point based on edge information of the projection object, wherein the first edge key point is a key point in the first virtual image used to indicate an edge of the first virtual image;

[0015] The third acquisition unit is used to obtain edge difference information based on the first edge key point and the second edge key point, where the second edge key point is a key point in the original image used to indicate the edge of the reference object, and the edge difference information is used to indicate the difference between the edge of the first virtual image and the edge of the reference object.

[0016] In one possible implementation, the number of the first edge key points is at least one, and the number of the second edge key points is at least one; the third acquisition unit is used to obtain, for any second edge key point, a difference index corresponding to any second edge key point based on the position information of the any second edge key point and the position information of the projection points of each first edge key point; and obtain the edge difference information based on the difference index corresponding to each second edge key point.

[0017] In one possible implementation, the third acquisition unit is further used to obtain the position distance between any second edge key point and the projection point of each first edge key point based on the position information of any second edge key point and the position information of the projection point of each first edge key point; determine the position distance that meets the reference condition among the position distances between any second edge key point and the projection point of each first edge key point; and obtain the difference index corresponding to any second edge key point based on the position distance that meets the reference condition.

[0018] In one possible implementation, the first determination unit is used to perform edge detection on the projection object in the projection image to obtain edge information of the projection object; based on the edge information of the projection object, obtain position information of edge pixel points in the projection image; and determine the first edge key point based on the position information of edge pixel points in the projection image.

[0019] In one possible implementation, the number of edge pixel points in the projected image is at least one, and the first determination unit is further used to determine at least one candidate key point in the first virtual image; for any candidate key point among the at least one candidate key point, based on the position information of each edge pixel point in the projected image, obtain a measurement index corresponding to the any candidate key point; in response to the measurement index corresponding to the any candidate key point being not greater than a reference threshold, take the any candidate key point as a first edge key point.

[0020] In one possible implementation, the first determination unit is further used to obtain the position distances between the projection point of any candidate key point and each edge pixel point based on the position information of the projection point of any candidate key point and the position information of each edge pixel point in the projected image; and use the minimum distance between the projection point of any candidate key point and each edge pixel point as the measurement indicator corresponding to any candidate key point.

[0021] In a possible implementation, the second acquisition unit is configured to acquire projection parameters corresponding to the first virtual image; and project the first virtual image based on the projection parameters corresponding to the first virtual image to obtain the projection object.

[0022] In one possible implementation, the first virtual image is generated based on target generation parameters, and the apparatus further includes:

[0023] A fourth acquisition unit is used to obtain updated generation parameters corresponding to the target generation parameters based on the edge difference information, and the updated generation parameters are used to generate a second virtual image corresponding to the reference object in the original image, and the degree of matching between the edge of the second virtual image and the edge of the reference object is higher than the degree of matching between the edge of the first virtual image and the edge of the reference object.

[0024] In one possible implementation, the target generation parameters are predicted by a first neural network model based on the original image, and the fourth acquisition unit is further used to update the model parameters of the first neural network model using the edge difference information to obtain a second neural network model; the original image is input into the second neural network model, the generation parameters predicted by the second neural network model are obtained, and the generation parameters predicted by the second neural network model are used as the updated generation parameters.

[0025] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement any of the above-mentioned methods for obtaining edge difference information.

[0026] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement any of the above methods for obtaining edge difference information.

[0027] In another aspect, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the aforementioned methods for obtaining edge difference information.

[0028] The technical solutions provided by the embodiments of the present application bring at least the following beneficial effects:

[0029] In the embodiment of the present application, the first edge keypoint used to indicate the edge of the virtual image is determined based on the edge information of the projected object corresponding to the virtual image, and is not a predefined keypoint with fixed semantics. The edge information of the projected object matches the actual edge of the virtual image. The first edge keypoint determined based on the edge information of the projected object has a high degree of match with the actual edge of the virtual image. The edge difference information obtained using the first edge keypoint can more accurately indicate the difference between the edge of the virtual image and the edge of the object in the image. The obtained edge difference information is reliable and accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0031] Figure 1 is a schematic diagram of an implementation environment of a method for obtaining edge difference information provided in an embodiment of the present application;

[0032] Figure 2 This is a flow chart of a method for obtaining edge difference information provided by an embodiment of the present application;

[0033] Figure 3 is a schematic diagram of a projection image including a projection object corresponding to a first virtual image provided by an embodiment of the present application;

[0034] Figure 4 is a schematic diagram of an edge of a projection object in a projection image provided by an embodiment of the present application;

[0035] Figure 5 is a schematic diagram of an image obtained by projecting each first edge key point provided in an embodiment of the present application;

[0036] Figure 6 This is a flowchart of a process for obtaining edge difference information based on a first edge key point and a second edge key point provided by an embodiment of the present application;

[0037] Figure 7 This is a schematic diagram of an edge line formed by the face edge key points and the projection points of each first edge key point in an original image provided by an embodiment of the present application;

[0038] Figure 8 is a schematic diagram of a process for iteratively updating generation parameters and projection parameters using edge difference information provided by an embodiment of the present application;

[0039] Figure 9 is a schematic diagram of a process for updating model parameters of a neural network model using edge difference information provided by an embodiment of the present application;

[0040] Figure 10 86 key points in a face image provided by an embodiment of the present application;

[0041] Figure 11 is a schematic diagram of a device for obtaining edge difference information provided by an embodiment of the present application;

[0042] Figure 12is a schematic diagram of a device for obtaining edge difference information provided by an embodiment of the present application;

[0043] Figure 13 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0044] Figure 14 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0046] In order to facilitate understanding of the technical process of the embodiments of the present application, the terms involved in the embodiments of the present application are explained below.

[0047] 3DMM (Three Dimensional Morphable Model): A general-purpose parametric model of 3D faces, representing a face using a fixed number of points. The core concept of 3DMM is that faces can be matched one-to-one in 3D space and can be derived by weighted linear addition of multiple orthogonal bases of faces.

[0048] For example, a three-dimensional face can be represented by two feature vectors: a shape vector S and a texture vector W:

[0049]

[0050]

[0051] in, Represents the elements that make up the shape vector S; This shows that S is in three-dimensional space; Represents the individual elements that make up the texture vector W; It indicates that W is in three-dimensional space; T represents the transpose of the vector; and n is an integer not less than 1.

[0052] The core idea of ​​3DMM is that any new face can be generated by a linear combination of the feature vectors of the basic faces. Based on this core idea, the shape vector S' and texture vector W' of any new face are obtained based on the following formula 1:

[0053] (Formula 1)

[0054] in, The average vector of the shape vectors representing the basic faces; Indicates the l ( lis an integer not less than 1) shape vectors of basic faces; Indicates the l The weight coefficient corresponding to the shape vector of the basic face; m (m is an integer not less than 1) represents the number of basic faces; The average vector of the texture vectors representing the base face; Indicates the l Texture vectors of the base face; Indicates the l The weight coefficient corresponding to the texture vector of the basic face.

[0055] In an exemplary embodiment, the method for obtaining edge difference information provided in the embodiment of the present application can be applied to the field of artificial intelligence technology. Next, artificial intelligence technology is introduced.

[0056] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also involves studying the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0057] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0058] Computer vision (CV) technology is the study of how machines can "see." Specifically, it refers to the use of cameras and computers to replace the human eye in identifying and measuring objects, and further image processing to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, virtual avatars, 3D (three-dimensional) technology, virtual reality, augmented reality, and map construction, as well as common biometric recognition technologies.

[0059] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0060] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, automatic driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0061] Figure 1 A schematic diagram of an implementation environment of the method for obtaining edge difference information provided in an embodiment of the present application is shown. Exemplarily, the implementation environment includes: a terminal 11 and a server 12.

[0062] The method for obtaining edge difference information provided in the embodiment of the present application can be executed by the terminal 11, can be executed by the server 12, or can be jointly executed by the terminal 11 and the server 12, and the embodiment of the present application does not limit this. In the case where the method for obtaining edge difference information provided in the embodiment of the present application is jointly executed by the terminal 11 and the server 12, the server 12 undertakes the primary computing work and the terminal 11 undertakes the secondary computing work; or, the server 12 undertakes the secondary computing work and the terminal 11 undertakes the primary computing work; or, the server 12 and the terminal 11 adopt a distributed computing architecture to perform collaborative computing.

[0063] In one possible implementation, terminal 11 can be any electronic product capable of human-computer interaction with a user through one or more methods, such as a keyboard, touchpad, touch screen, remote control, voice interaction, or handwriting device, such as a PC (Personal Computer), mobile phone, smartphone, PDA (Personal Digital Assistant), wearable device, Pocket PC (Personal Computer), tablet computer, smart car computer, smart TV, smart speaker, etc. Server 12 can be a single server, a server cluster consisting of multiple servers, or a cloud computing service center. Terminal 11 establishes a communication connection with server 12 via a wired or wireless network.

[0064] Those skilled in the art should understand that the above-mentioned terminal 11 and server 12 are only examples. Other existing or future terminals or servers that are applicable to this application should also be included in the scope of protection of this application and are included here by reference.

[0065] Based on the above Figure 1 In the implementation environment shown, the embodiment of the present application provides a method for obtaining edge difference information. Taking the method executed by a computer device as an example, the computer device can be a terminal or a server. Figure 2 As shown, the method for obtaining edge difference information provided in the embodiment of the present application includes the following steps 201 to 204:

[0066] In step 201, a first virtual image corresponding to a reference object in an original image is obtained.

[0067] The first virtual image corresponding to the reference object is used to perform a virtual representation of the reference object. The reference object is an object included in the original image. The embodiment of the present application does not limit the type of the reference object. Exemplarily, the type of the reference object is a human face, an animal face, or an object. The embodiment of the present application is described using the type of the reference object as an example. When the type of the reference object is a human face, the first virtual image corresponding to the reference object refers to a human face virtual image, which is used to perform a virtual representation of the human face. In an exemplary embodiment, the virtual image mentioned in the embodiment of the present application refers to a three-dimensional virtual image, so that the reference object can be represented three-dimensionally by using the virtual image.

[0068] The original image is used to provide the supervision information required to obtain edge difference information, and the edges of the reference object in the original image are used to constrain the edges of the virtual image corresponding to the reference object. For example, taking the type of reference object as a face, the original image refers to a face image. The embodiment of the present application does not limit the source of the original image. For example, the original image may refer to an image obtained by photographing the entity of the reference object using an image acquisition device. For example, the image acquisition device includes but is not limited to a camera, a video camera, etc. For example, the original image may refer to an image including the reference object captured from a video shot of the entity of the reference object. The embodiment of the present application does not limit the type of the original image. For example, the original image refers to an RGB (Red Green Blue) image, or the original image refers to a grayscale image.

[0069] The purpose of an embodiment of the present application is to obtain edge difference information for indicating the difference between the edge of the first virtual image corresponding to the reference object and the edge of the reference object in the original image. Therefore, before obtaining the edge difference information, it is necessary to first obtain the first virtual image corresponding to the reference object in the original image.

[0070] In one possible implementation, obtaining the first virtual image corresponding to the reference object in the original image involves extracting any virtual image from a library of virtual images corresponding to the reference object in the original image as the first virtual image. The library of virtual images corresponding to the reference object in the original image is configured to store at least one virtual image pre-generated based on the reference object in the original image.

[0071] In another possible implementation, the first virtual image corresponding to the reference object in the original image is obtained by generating the first virtual image corresponding to the reference object in the original image based on the target generation parameters. In other words, the first virtual image is generated based on the target generation parameters.

[0072] Target generation parameters refer to parameters required to generate the first virtual image. Exemplarily, the target generation parameters include but are not limited to target shape parameters and target texture parameters. The target shape parameters are used to define the shape of the first virtual image, and the target texture parameters are used to define the texture of the first virtual image.

[0073] In one possible implementation, the target generation parameters refer to parameters required by the virtual image generation model. In this case, the process of generating a first virtual image corresponding to a reference object in the original image based on the target generation parameters is as follows: applying the target generation parameters to the virtual image generation model; and generating the first virtual image using the virtual image generation model with the target generation parameters. The virtual image generation model refers to a model used to generate a virtual image. Depending on the type of reference object, the type of virtual image generation model may vary. This embodiment of the present application does not limit the type of virtual image generation model, as long as it can generate a virtual image corresponding to the reference object in the original image.

[0074] For example, when the reference object is a face, the avatar generation model is a 3DMM model. For example, when the avatar generation model is a 3DMM model, the target generation parameters are 3DMM coefficients, which include, but are not limited to, 3DMM shape coefficients and 3DMM texture coefficients.

[0075] In one possible implementation, when the first virtual image is generated based on the target generation parameters, the target generation parameters need to be acquired before acquiring the first virtual image. The embodiment of the present application does not limit the method for acquiring the target generation parameters.

[0076] Exemplarily, the target generation parameters refer to the default generation parameters or the generation parameters obtained after iteratively updating the default generation parameters at least once. The default generation parameters refer to the generation parameters specified by the staff or computer equipment for generating the virtual image. Exemplarily, the specification method is random initialization, or specification based on experience or application scenarios, etc. Exemplarily, the default generation parameters are related to the type of reference object, and different types of reference objects correspond to different default generation parameters. Exemplarily, the default generation parameters include but are not limited to default shape parameters and default texture parameters. In this case, the purpose of obtaining edge difference information is to directly update the target generation parameters using the edge difference information.

[0077] Exemplarily, the target generation parameters are obtained by inputting the original image into the first neural network model and using the generation parameters predicted by the first neural network model as the target generation parameters. In other words, the target generation parameters are predicted by the first neural network model based on the original image. In this case, the purpose of obtaining edge difference information is to use this edge difference information to train the first neural network model and thereby improve the reliability of the generation parameters predicted by the neural network model.

[0078] The first neural network model refers to a neural network model that exists when the first virtual image needs to be obtained. The first neural network model may refer to an initial neural network model, or may refer to a neural network model obtained after updating the model parameters of the initial neural network model at least once, and this embodiment of the present application does not limit this. The neural network model in the embodiment of the present application is used to predict the generation parameters for generating the virtual image corresponding to the object in the image based on the input image. After the original image is input into the first neural network model, the first neural network model can output the predicted target generation parameters. The embodiment of the present application does not limit the type of neural network model, as long as the generation parameters can be predicted based on the input image. Exemplarily, the type of the neural network model is a convolutional neural network model.

[0079] In step 202, a projection object corresponding to the first virtual image is obtained.

[0080] After obtaining the first virtual image, the projection object corresponding to the first virtual image is further obtained. Exemplarily, the first virtual image refers to a three-dimensional virtual image, and the projection object corresponding to the first virtual image refers to a planar object obtained after projecting the first virtual image. In one possible implementation, projecting the first virtual image refers to projecting the complete first virtual image, or refers to projecting a specified part of the first virtual image, etc., which is not limited in the embodiments of the present application. The specified part is set based on experience or flexibly adjusted according to the application scenario. Exemplarily, for the case where the type of reference object is a human face, the first virtual image is a human face virtual image, the specified part of the human face virtual image refers to the face of the human face virtual image, or the specified part of the human face virtual image refers to other parts of the human face virtual image except hair, ears and eyes.

[0081] In one possible implementation, the projection object corresponding to the first virtual image may be acquired and stored in advance. In this case, the projection object corresponding to the first virtual image is acquired by extracting the projection object corresponding to the first virtual image from the storage.

[0082] In another possible implementation, the method for obtaining the projection object corresponding to the first virtual image is: obtaining projection parameters corresponding to the first virtual image; and projecting the first virtual image based on the projection parameters corresponding to the first virtual image to obtain the projection object.

[0083] The projection parameters corresponding to the first virtual image are used to project the first virtual image. In an exemplary embodiment, when the first virtual image is generated based on the target generation parameters, the projection parameters corresponding to the first virtual image are parameters that exist simultaneously with the target generation parameters.

[0084] For example, when the target generation parameters are default generation parameters, the projection parameters corresponding to the first avatar are the default projection parameters; when the target generation parameters are generation parameters obtained by iteratively updating the default generation parameters at least once, the projection parameters corresponding to the first avatar are projection parameters obtained by iteratively updating the default projection parameters at least once. The default generation parameters are generation parameters specified by a staff member or computer device for generating an avatar, and the default projection parameters are projection parameters specified by a staff member or computer device for projecting an avatar generated based on the default generation parameters.

[0085] For example, when the target generation parameters are predicted by the first neural network model based on the original image, the projection parameters corresponding to the first virtual image are obtained by inputting the original image into the first neural network model and using the first projection parameters predicted by the first neural network model as the projection parameters corresponding to the first virtual image. After the original image is input into the first neural network model, the first neural network model can not only predict the target generation parameters but also predict the projection parameters corresponding to the first virtual image.

[0086] The projection parameters corresponding to the first virtual image are used to provide guidance for the process of projecting the first virtual image. Based on the projection parameters corresponding to the first virtual image, it is possible to determine how to project the first virtual image, thereby obtaining a projection object. Exemplarily, the projection parameters corresponding to the first virtual image include posture parameters corresponding to the first virtual image and lighting parameters corresponding to the first virtual image. The posture parameters corresponding to the first virtual image are used to provide guidance for the posture in which the first virtual image should be projected, and the lighting parameters corresponding to the first virtual image are used to provide guidance for the lighting conditions under which the first virtual image should be projected. In this case, based on the projection parameters corresponding to the first virtual image, the first virtual image in a target posture can be projected using a target lighting condition. The target lighting condition is the lighting condition indicated by the lighting parameters corresponding to the first virtual image, and the target posture is the posture indicated by the posture parameters corresponding to the first virtual image.

[0087] Exemplarily, the target pose is a pose that matches the pose of the reference object in the original image. The possible poses of the reference object in the original image are related to the type of the reference object, which is not limited in the embodiments of the present application. Exemplarily, if the type of the reference object is a face, the possible poses of the reference object in the original image include, but are not limited to, frontal, profile, raised, lowered, etc.

[0088] In an exemplary embodiment, an image including a projection object is used as a projection image, that is, the projection object is in the projection image. For example, taking the example of projecting a first virtual image as an example, that is, projecting a designated part of the first virtual image, assuming that the type of the reference object is a human face, and the designated part of the first virtual image is other parts of the human face virtual image except hair, ears, and eyes, the projection image including the projection object corresponding to the first virtual image is as follows: Figure 3 As shown. Figure 3 In the figure, the white area indicates the area covered by the projection object, and the black area indicates the area not covered by the projection object.

[0089] In an exemplary embodiment, the pixel values ​​of the projected pixels in the projected image are different from the pixel values ​​of the non-projected pixels, so as to intuitively distinguish the projected pixels from the non-projected pixels in the projected image by using different pixel values. The projected pixels in the projected image refer to the pixels in the area covered by the projection object, and the non-projected pixels in the projected image refer to the pixels in the area not covered by the projection object. The embodiment of the present application does not limit the setting of the pixel values ​​of the projected pixels in the projected image and the pixel values ​​of the non-projected pixels, as long as it can ensure that the pixel values ​​of the projected pixels in the projected image are different from the pixel values ​​of the non-projected pixels. Exemplarily, the pixel value of the projected pixels in the projected image is 1, and the pixel value of the non-projected pixels in the projected image is 0. In this case, the projected image is a binary image.

[0090] In step 203 , a first edge key point is determined based on edge information of the projection object. The first edge key point is a key point in the first virtual image that is used to indicate an edge of the first virtual image.

[0091] After acquiring the projection object, edge information of the projection object is acquired. The edge information of the projection object is used to indicate the edge of the projection object. It should be noted that if the acquisition effect of the first virtual image is good, the projection object is relatively similar to the reference object in the original image; if the acquisition effect of the first virtual image is poor, the projection object is significantly different from the reference object in the original image. This embodiment of the application does not limit the acquisition effect of the first virtual image. Regardless of the acquisition effect of the first virtual image, the edge information of the projection object can be acquired.

[0092] In one possible implementation, the projection object is in the projection image, and the edge information of the projection object is obtained by performing edge detection on the projection object in the projection image to obtain the edge information of the projection object. By performing edge detection on the projection object in the projection image, the edge of the projection object in the projection image can be detected, and the edge information of the projection object can be obtained. For example, Figure 3 The edge detection of the projection object in the projection image shown in FIG. Figure 4 As shown. In one possible implementation, the edge of the projected object in the projected image is obtained by calculating the gradient of each pixel in the projected image. Exemplarily, the edge detection process is a process of detecting using an edge detection operator. The embodiment of the present application does not limit the type of edge detection operator.

[0093] Exemplarily, the edge detection operator may be a first-order differential edge detection operator or a second-order differential edge detection operator. First-order differential edge detection operators include, but are not limited to, the Roberts edge detection operator, the Prewitt edge detection operator, the Sobel edge detection operator, and the Canny edge detection operator. Second-order differential edge detection operators include, but are not limited to, the Laplacian edge detection operator and the Marr-Hildreth edge detection operator.

[0094] After obtaining the edge information of the projection object, a first edge key point is determined based on the edge information of the projection object. The first edge key point is a key point in the first virtual image that indicates the edge of the first virtual image. In the embodiment of the present application, the first edge key point is determined based on the edge information of the projection object corresponding to the first virtual image, and has a high degree of match with the actual edge of the first virtual image. Compared to the method of directly pre-defining key points with certain semantics to indicate the edge of the virtual image in the related art, the first edge key point obtained in the embodiment of the present application is more flexible, which is conducive to improving the accuracy and reliability of the subsequently obtained edge difference information.

[0095] In one possible implementation, the process of determining the first edge key point based on the edge information of the projection object is: based on the edge information of the projection object, obtaining the position information of the edge pixel points in the projection image; based on the position information of the edge pixel points in the projection image, determining the first edge key point.

[0096] Edge pixels in a projected image are pixels located on the edge of a projected object in the projected image. Since the edge information of the projected object indicates the edge of the projected object in the projected image, the edge of the projected object in the projected image can be determined based on the edge information of the projected object, and the pixels located on the edge of the projected object in the projected image are considered edge pixels in the projected image. The position information of each pixel in the projected image is known. Once the edge pixels in the projected image are determined, the position information of the edge pixels in the projected image can be obtained.

[0097] The position information of any pixel in the projected image is used to indicate the position coordinates of the pixel. The embodiments of the present application do not limit the representation of the position information of any pixel. Exemplarily, the position information of any pixel is directly the position coordinates of the pixel, or the position information of any pixel includes the index of the pixel and the position coordinates pointed to by the index of the pixel. Exemplarily, the position coordinates of any pixel in the projected image are plane coordinates.

[0098] After obtaining the position information of the edge pixel points in the projected image, the first edge key point is determined based on the position information of the edge pixel points in the projected image. In one possible implementation, the first edge key point is determined from the various candidate key points in the first virtual image. In this case, before determining the first edge key point, it is necessary to determine at least one candidate key point in the first virtual image. The at least one candidate key point in the first virtual image may refer to all the key points in the first virtual image, or may refer to some of the key points in the first virtual image, which is not limited in the embodiment of the present application. For example, the key points in the first virtual image are key points with different semantics obtained by detecting the first virtual image. The embodiment of the present application does not limit the number of key points in the first virtual image and the corresponding semantics, which can be set based on experience or flexibly adjusted based on the application scenario.

[0099] In one possible implementation, the number of edge pixel points in the projected image is at least one, and the process of determining the first edge key point based on the position information of the edge pixel points in the projected image is a process of judging whether each candidate key point in the first virtual image is the first edge key point based on the position information of each edge pixel point in the projected image.

[0100] Exemplarily, based on the position information of each edge pixel point in the projected image, the method for determining whether any candidate key point among at least one candidate key point is a first edge key point is as follows: based on the position information of each edge pixel point in the projected image, obtaining a measurement index corresponding to any candidate key point; in response to the measurement index corresponding to any candidate key point being not greater than a reference threshold, treating any candidate key point as a first edge key point.

[0101] The measurement index corresponding to any candidate key point is an index for measuring whether the candidate key point is a first edge key point. In one possible implementation, the process of obtaining the measurement index corresponding to any candidate key point based on the position information of each edge pixel point in the projected image is as follows: based on the position information of the projection point of any candidate key point and the position information of each edge pixel point in the projected image, obtaining the position distances between the projection point of any candidate key point and each edge pixel point; and using the minimum distance between the projection point of any candidate key point and each edge pixel point as the measurement index corresponding to any candidate key point.

[0102] The positional distance between the projection point of any candidate key point and any edge pixel point is obtained based on the position information of the projection point of the any candidate key point and the position information of the any edge pixel point. In one possible implementation, the method for obtaining the positional distance between the projection point of any candidate key point and any edge pixel point based on the position information of the projection point of any candidate key point and the position information of any edge pixel point is as follows: based on the position information of the projection point of any candidate key point, determining the position coordinates of the projection point of any candidate key point; based on the position information of any edge pixel point, determining the position coordinates of any edge pixel point; and based on the position coordinates of the projection point of any candidate key point and the position coordinates of any edge pixel point, obtaining the positional distance between the projection point of any candidate key point and any edge pixel point.

[0103] The position information of the projection point of any candidate key point is used to indicate the position coordinates of the projection point of any candidate key point. Based on the position information of the projection point of any candidate key point, the position coordinates of the projection point of any candidate key point can be determined. The position information of any edge pixel point is used to indicate the position coordinates of any edge pixel point. Based on the position information of any edge pixel point, the position coordinates of any edge pixel point can be determined. Exemplarily, the position coordinates of the projection point of any candidate key point and the position coordinates of any edge pixel point are both plane coordinates.

[0104] The embodiment of the present application does not limit the method for obtaining the positional distance between the projection point of any candidate key point and any edge pixel point based on the position coordinates of the projection point of any candidate key point and the position coordinates of any edge pixel point. Exemplarily, the Euclidean distance between the projection point of any candidate key point and any edge pixel point is obtained based on the position coordinates of the projection point of any candidate key point and the position coordinates of any edge pixel point.

[0105] In an embodiment of the present application, it is necessary to obtain the positional distances between the projection point of any candidate key point and each edge pixel point, that is, the number of positional distances obtained is the same as the number of edge pixel points. After obtaining the positional distances between the projection point of any candidate key point and each edge pixel point, the minimum distance among the positional distances between the projection point of any candidate key point and each edge pixel point is used as the measurement index corresponding to any candidate key point. Exemplarily, the set of each edge pixel point is denoted as Pe, and the process of obtaining the measurement index corresponding to any candidate key point is to compare the position coordinates of the projection point of any candidate key point with the position coordinates of each edge pixel point in Pe, and record the shortest distance for any candidate key point.

[0106] The minimum distance between the projection point of any candidate keypoint and each edge pixel is the distance between the projection point of the candidate keypoint and the nearest edge pixel. In other words, the metric corresponding to any candidate keypoint is the distance between the projection point of the candidate keypoint and the nearest edge pixel. If the metric corresponding to any candidate keypoint is small, it means that the distance between the projection point of the candidate keypoint and the nearest edge pixel is small, which means that the candidate keypoint is very likely to be a first edge keypoint.

[0107] After obtaining the measurement index corresponding to any candidate key point, the measurement index corresponding to any candidate key point is compared with the reference threshold. If the measurement index corresponding to any candidate key point is not greater than the reference threshold, it means that any candidate key point is very likely to be on the edge of the first virtual image. At this time, any candidate key point is regarded as a first edge key point. If the measurement index corresponding to any candidate key point is less than the reference threshold, it means that any candidate key point is very likely not on the edge of the first virtual image. At this time, it is considered that any candidate key point is not a first edge key point. The reference threshold is set according to experience or flexibly adjusted according to the application scenario, and the embodiments of the present application do not limit this.

[0108] According to the aforementioned method of determining whether any candidate key point in the first virtual image is a first edge key point based on the position information of the edge pixels in the projected image, it is possible to determine whether each candidate key point in the first virtual image is a first edge key point based on the position information of the edge pixels in the projected image, thereby achieving the process of determining the first edge key point from each candidate key point in the first virtual image. It should be noted that the number of first edge key points may be one or more, and this is not limited in this embodiment of the present application.

[0109] In an exemplary embodiment, after determining the first edge key point, the set of the first edge key point is recorded as Pd. For example, the projection image of the projection object corresponding to the first virtual image is recorded as Figure 3 As an example, the image shown in FIG. 1 is projected based on the edge information of the projection object to determine each first edge key point in Pd, and the following can be obtained: Figure 5 The image shown. Figure 5 In the image shown, the projected area is a white area (exemplarily, the pixel value of the pixel points in the white area is 1), and the unprojected area is a black area (exemplarily, the pixel value of the pixel points in the black area is 0). Figure 5 The white edges in the image shown are consistent with Figure 4 The white edges in the image shown are highly similar, which indicates that Pd is a set of candidate key points corresponding to the edges of the projected object in the projected image.

[0110] In step 204, edge difference information is obtained based on the first edge key point and the second edge key point. The second edge key point is a key point in the original image used to indicate the edge of the reference object. The edge difference information is used to indicate the difference between the edge of the first virtual image and the edge of the reference object.

[0111] The first edge keypoints are keypoints in the first avatar that indicate the edges of the first avatar, and the second edge keypoints are keypoints in the original image that indicate the edges of the reference object. Based on the first and second edge keypoints, edge difference information indicating the difference between the edges of the first avatar and the edges of the reference object can be obtained.

[0112] According to steps 201 to 203, the first edge key point can be determined. Before obtaining the edge difference information, the second edge key point needs to be obtained. The second edge key point is the key point in the original image that indicates the edge of the reference object. In one possible implementation, the second edge key point is obtained by performing edge detection on the original image for the reference object and using the detected key point as the second edge key point. The purpose of edge detection for the reference object is to detect the edge of the reference object. The process of edge detection for different types of objects may be different. The embodiment of the present application does not limit this. For example, taking the type of reference object as a face as an example, the edge detection for the reference object is face edge detection. By performing face edge detection, the face edge key points in the image can be detected. For example, edge detection for the reference object can be implemented using an edge detection operator.

[0113] After determining the first edge key point and the second edge key point, edge difference information can be obtained based on the first edge key point and the second edge key point. In one possible implementation, the number of the first edge key point is at least one, and the number of the second edge key point is at least one. The number of the first edge key point and the number of the second edge key point may be the same or different, and this embodiment of the application does not limit this. For example, see Figure 6 Based on the first edge key point and the second edge key point, the process of obtaining edge difference information includes the following steps 2041 and 2042:

[0114] Step 2041: For any second edge key point, based on the position information of any second edge key point and the position information of the projection points of each first edge key point, obtain a difference index corresponding to any second edge key point.

[0115] Edge difference information is obtained based on the difference indicators corresponding to each second edge keypoint. Therefore, before obtaining the edge difference information, it is necessary to obtain the difference indicators corresponding to each second edge keypoint. The difference indicator corresponding to any second edge keypoint is used to indicate the difference between the edge of the first virtual image and the edge of the reference object in the original image from the perspective of that second edge keypoint.

[0116] The difference index corresponding to any second edge key point is obtained based on the position information of the any second edge key point and the position information of the projection points of each first edge key point. Before obtaining the difference index corresponding to any second edge key point, it is necessary to obtain the position information of the any second edge key point and the position information of the projection points of each first edge key point. The second edge key point refers to a key point in the original image, and the position information of any second edge key point can be extracted from the position information of each key point in the original image. The position information of any second edge key point is used to indicate the position coordinates of any second edge key point.

[0117] Each first edge key point is a key point in the first virtual image. Position information of the projection points of each key point in the first virtual image can be extracted from the position information of the projection points of each key point in the first virtual image. For example, the projection points of each key point in the first virtual image are obtained by projecting each key point in the first virtual image according to projection parameters corresponding to the first virtual image. After the projection points of each key point are obtained, the position information of the projection points of each key point can be obtained.

[0118] In one possible implementation, based on the position information of any second edge key point and the position information of the projection points of each first edge key point, a process of obtaining a difference index corresponding to any second edge key point includes the following steps 2041a to 2041c:

[0119] Step 2041a: Based on the position information of any second edge key point and the position information of the projection points of each first edge key point, obtain the position distance between any second edge key point and the projection points of each first edge key point.

[0120] The positional distance between any second edge key point and the projection point of any first edge key point is obtained based on the position information of any second edge key point and the position information of the projection point of any first edge key point. Exemplarily, the positional distance between any second edge key point and the projection point of any first edge key point is obtained by: determining the position coordinates of any second edge key point based on the position information of any second edge key point; determining the position coordinates of the projection point of any first edge key point based on the position information of the projection point of any first edge key point; and obtaining the positional distance between any second edge key point and the projection point of any first edge key point based on the position coordinates of any second edge key point and the position coordinates of the projection point of any first edge key point.

[0121] In an exemplary embodiment, based on the position coordinates of any second edge key point and the position coordinates of the projection point of any first edge key point, the method for obtaining the position distance between any second edge key point and the projection point of any first edge key point is: based on the position coordinates of any second edge key point and the position coordinates of the projection point of any first edge key point, obtaining the Euclidean distance between any second edge key point and the projection point of any first edge key point.

[0122] The positional distances between any second edge key point and the projected points of each first edge key point can be obtained by referring to the method for obtaining the positional distances between any second edge key point and the projected points of any first edge key point, and then step 2041b is performed. The number of positional distances between any second edge key point and the projected points of each first edge key point is the same as the number of each first edge key point.

[0123] Step 2041b: Determine the position distance that meets the reference condition among the position distances between any second edge key point and the projection points of each first edge key point.

[0124] The position distance that meets the reference condition refers to the position distance required for obtaining the difference index corresponding to any second edge key point. The reference condition is set according to experience or flexibly adjusted according to the application scenario, and the embodiment of the present application does not limit this. Exemplarily, the position distance that meets the reference condition refers to the first K (K is an integer not less than 1) smallest position distances among the position distances between any second edge key point and the projection points of each first edge key point. The value of K is set according to experience or flexibly adjusted according to the application scenario. Exemplarily, the value of K is 1. In this case, the position distance that meets the reference condition refers to the minimum distance among the position distances between any second edge key point and the projection points of each first edge key point. It should be noted that the number of position distances that meet the reference condition may be one or more, and the embodiment of the present application does not limit this.

[0125] Step 2041c: Based on the position distance that meets the reference condition, obtain the difference index corresponding to any second edge key point.

[0126] After determining a position distance that satisfies the reference condition, a difference index corresponding to any second edge key point is obtained based on the position distance that satisfies the reference condition. In an exemplary embodiment, the number of position distances that satisfies the reference condition may be one. In this case, the difference index corresponding to any second edge key point is obtained based on the position distance that satisfies the reference condition by using the one position distance that satisfies the reference condition as the difference index corresponding to the second edge key point.

[0127] In an exemplary embodiment, the number of position distances that meet the reference condition may be multiple. In this case, based on the position distance that meets the reference condition, the method for obtaining the difference index corresponding to any second edge key point is: taking the average distance of multiple position distances that meet the reference condition as the difference index corresponding to any second edge key point.

[0128] It should be noted that the above-described method of obtaining a difference index corresponding to any second edge key point based on a position distance that meets the reference condition is merely an example, and the embodiments of the present application are not limited thereto. For example, a weighted average distance of position distances that meet the reference condition can also be used as the difference index corresponding to any second edge key point.

[0129] Step 2042: Obtain edge difference information based on the difference indicators corresponding to each second edge key point.

[0130] Step 2041 above describes the process of obtaining a difference index corresponding to any second edge keypoint from the perspective of any second edge keypoint. Based on the implementation described in step 2041, the difference index corresponding to each second edge keypoint can be obtained. After obtaining the difference index corresponding to each second edge keypoint, edge difference information is obtained based on the difference index corresponding to each second edge keypoint.

[0131] In one possible implementation, edge difference information is obtained based on the difference indicators corresponding to each second edge keypoint by performing a weighted summation of the difference indicators corresponding to each second edge keypoint to obtain the edge difference information. It should be noted that the embodiments of this application do not limit the weights of the difference indicators corresponding to each second edge keypoint in the weighted summation process; these weights may be set based on experience or determined based on the corresponding relationship between edge keypoints and weights.

[0132] In an exemplary embodiment, based on the difference indicators corresponding to the second edge key points, the edge difference information may be obtained by taking the sum of the difference indicators corresponding to the second edge key points as the edge difference information.

[0133] Exemplarily, the edge difference information is obtained based on the position information of each second edge key point and the position information of the projection point of each first edge key point. Taking the type of the reference object as a face and the original image as an example, the second edge key point is the face edge key point in the original image. The face edge key point detection method can be used to determine the face edge key point in the original image. For example, the original image is as follows: Figure 7 As shown, the face edge key points in the original image are as follows Figure 7 As shown by the black dots in .

[0134] The projection points of each first edge key point are obtained by projecting each first edge key point. Assuming that each first edge key point is projected on the original image, the projection points of each first edge key point can constitute Figure 7 The black dashed line in the figure shows the edge line. The process of obtaining the difference index corresponding to any second edge key point (any black dot) can be understood as: finding a point on the black dashed line that satisfies a specified condition for any black dot, and then obtaining the difference index corresponding to the black dot based on the positional distance between the black dot and the point that satisfies the specified condition. The point that satisfies the specified condition refers to a point whose positional distance from the black dot meets a reference condition.

[0135] In an exemplary embodiment, after obtaining edge difference information, subsequent operations are performed using the edge difference information. It should be noted that the subsequent operations performed using the edge difference information can be flexibly adjusted according to the application scenario, and this embodiment of the application does not limit this.

[0136] Exemplarily, in the case where the first virtual image is generated based on the target generation parameters, the subsequent operations performed using the edge difference information are: based on the edge difference information, obtaining updated generation parameters corresponding to the target generation parameters, and the updated generation parameters are used to generate a second virtual image corresponding to the reference object in the original image, and the degree of matching between the edge of the second virtual image and the edge of the reference object is higher than the degree of matching between the edge of the first virtual image and the edge of the reference object.

[0137] That is, after obtaining the edge difference information, the edge difference information is used to obtain updated generation parameters that can generate a virtual avatar whose edges more closely match those of the reference object. In an exemplary embodiment, after obtaining the updated generation parameters, a second virtual avatar is generated based on the updated generation parameters. Because the updated generation parameters are obtained based on the edge difference information, the degree of match between the edges of the second virtual avatar and the reference object is higher than the degree of match between the edges of the first virtual avatar and the reference object. Exemplarily, edges are used to constrain appearance; the higher the degree of match between the edges of the generated virtual avatar and the reference object, the more similar the appearance of the generated virtual avatar is to that of the reference object.

[0138] In one possible implementation, based on the edge difference information, a method for obtaining the updated generation parameter corresponding to the target generation parameter varies depending on the method for obtaining the target generation parameter.

[0139] For example, when the target generation parameters are default generation parameters or are generated by iteratively updating the default generation parameters at least once, the edge difference information can be considered a loss function used to update the target generation parameters. Based on the edge difference information, the updated generation parameters corresponding to the target generation parameters are obtained by directly updating the target generation parameters using the edge difference information to obtain the updated generation parameters. This approach eliminates the need for a neural network model to obtain the updated generation parameters, resulting in a more efficient acquisition of the updated generation parameters.

[0140] In one possible implementation, the target generation parameters are directly updated using the edge difference information by obtaining an update gradient of the target generation parameters based on the edge difference information, and directly updating the target generation parameters using the update gradient of the target generation parameters.

[0141] In the exemplary embodiment, the target generation parameters are directly updated using edge difference information. After obtaining the updated generation parameters, it is determined whether the updated generation parameters meet the update termination conditions. If the updated generation parameters meet the update termination conditions, it means that there is no need to continue updating the updated generation parameters. In this case, the second virtual image generated based on the updated generation parameters is considered to be a virtual image with better virtualization quality corresponding to the reference object. If the updated generation parameters do not meet the update termination conditions, it means that the updated generation parameters need to be continued to be updated until generation parameters that meet the update termination conditions are obtained. The virtual image generated based on the generation parameters that meet the update termination conditions is used as the virtual image with better virtualization quality corresponding to the reference object. The process of continuing to update the updated generation parameters refers to the process of updating the target generation parameters and will not be repeated here.

[0142] The update termination condition is set based on experience or flexibly adjusted according to the application scenario, and is not limited in this embodiment of the present application. For example, the update generation parameters satisfying the update termination condition include, but are not limited to, any of the following situations: the number of iterative updates to the default generation parameters has reached a first threshold when obtaining the updated generation parameters; or the edge difference information used to obtain the updated generation parameters has converged. If the updated generation parameters meet any of the above conditions, the updated generation parameters are considered to meet the update termination condition.

[0143] For example, in the case where the generation parameters also include projection parameters, the generation parameters are iteratively updated using the edge difference information while the projection parameters are also iteratively updated. For example, the process of iteratively updating the generation parameters and projection parameters using the edge difference information is as follows: Figure 8As shown. The generation parameters and projection parameters that need to be updated are obtained. In the initial state, the generation parameters are default generation parameters, and the projection parameters are default projection parameters. Based on the projection parameters, the virtual image generated based on the generation parameters is projected to obtain a projection object. Based on the projected image including the projection object and the original image, edge difference information is obtained. The generation parameters and projection parameters are directly updated using the edge difference information. For example, the generation parameters are 3DMM coefficients, and the projection parameters include posture coefficients and illumination coefficients.

[0144] For example, when the target generation parameters are predicted by a first neural network model based on the original image, the edge difference information can be considered a loss function used to update the model parameters of the first neural network model. Based on the edge difference information, the updated generation parameters corresponding to the target generation parameters are obtained by: updating the model parameters of the first neural network model using the edge difference information to obtain a second neural network model; inputting the original image into the second neural network model to obtain the generation parameters predicted by the second neural network model; and using the generation parameters predicted by the second neural network model as the updated generation parameters.

[0145] In an exemplary embodiment, the method for updating the model parameters of the first neural network model using the edge difference information is: based on the edge difference information, the update gradient of the model parameters of the first neural network model is obtained, and the model parameters of the first neural network model are updated using the update gradient of the model parameters of the first neural network model.

[0146] After updating the model parameters of the first neural network model using the edge difference information, a second neural network model is obtained. After obtaining the second neural network model, the original image is input into the second neural network model, and the generation parameters predicted by the second neural network model are used as updated generation parameters.

[0147] In one possible implementation, the process of inputting the original image into the second neural network model and obtaining the generation parameters predicted by the second neural network model occurs when the second neural network model does not meet the training termination conditions. That is, in response to the second neural network model not meeting the training termination conditions, the original image is input into the second neural network model and the generation parameters predicted by the second neural network model are obtained.

[0148] In an exemplary embodiment, if the second neural network model meets the training termination condition, there is no need to obtain updated generation parameters, and the second neural network model is directly used as the trained target neural network model.

[0149] The training termination condition is set based on experience or flexibly adjusted according to the application scenario, and the embodiments of the present application do not limit this. For example, the second neural network model meets the training termination condition, including but not limited to any of the following situations: when obtaining the second neural network model, the number of iterative updates of the model parameters of the initial neural network model has reached a second number threshold; the edge difference information based on which the second neural network model is obtained converges. If the second neural network model meets any of the above situations, it is considered that the second neural network model meets the training termination condition. The second number threshold can be the same as the first number threshold, or it can be different from the first number threshold, and the embodiments of the present application do not limit this.

[0150] In an exemplary embodiment, if the second neural network model does not meet the training termination condition, after obtaining the second neural network model, it is necessary to continue updating the model parameters of the second neural network model until a trained target neural network model that meets the training termination condition is obtained. The process of continuing to update the model parameters of the second neural network model refers to the process of updating the model parameters of the first neural network model and is not further described here.

[0151] The target neural network model is trained based on edge difference information indicating the difference between the edges of the virtual image and the edges of the object in the input image. Therefore, the target neural network model can be used to predict generation parameters that match the input image, wherein the generation parameters that match the input image are used to generate a virtual image whose edges match the edges of the object in the input image. Exemplarily, the edges are used to constrain the shape, where the shape refers to the external shape. For example, when the reference object is a human face, the edges of the reference object or the edges of the virtual image corresponding to the reference object are used to constrain the face shape.

[0152] In one possible implementation, after obtaining the target neural network model, the method further includes: inputting the target image into the target neural network model to obtain the generation parameters predicted by the target neural network model; and generating a virtual image corresponding to the object in the target image based on the generation parameters predicted by the target neural network model. At this time, the edges of the virtual image corresponding to the object in the target image obtained have a high degree of match with the edges of the object in the target image. Since the edges are used to constrain the shape, the shape of the virtual image corresponding to the object in the target image obtained has a high degree of similarity with the shape of the object in the target image. Exemplarily, the object in the target image is of the same type as the reference object in the original image used in the process of training the target neural network model to ensure the prediction effect of the target neural network model.

[0153] For example, in the case where the neural network model not only predicts the generation parameters but also predicts the projection parameters, the process of updating the model parameters of the neural network model using the edge difference information is as follows: Figure 9 As shown. The original image is input into a neural network model to obtain generation parameters and projection parameters predicted by the neural network model; a virtual image generated based on the generation parameters is projected based on the projection parameters to obtain a projected object; edge difference information is obtained based on the projected image including the projected object and the original image; and the model parameters of the neural network model are updated using the edge difference information. For example, the generation parameters are 3DMM coefficients, and the projection parameters include pose coefficients and illumination coefficients.

[0154] It should be noted that the embodiments of the present application take the use of edge difference information to update the model parameters of the neural network model or the use of edge difference information to directly iteratively update the default generation parameters as an example. In an exemplary embodiment, in the process of updating the model parameters of the neural network model or directly iteratively updating the default generation parameters using edge difference information, in addition to using edge difference information for indicating differences in edges, difference information for indicating differences in other aspects can also be used.

[0155] Exemplarily, other aspects include but are not limited to color feature aspects, overall feature aspects, etc. In an exemplary embodiment, in the process of obtaining overall feature difference information for indicating the difference between the virtual image and the original image in terms of overall features, the difference information between the projection points of each predefined key point in the virtual image and the corresponding key point in the original image is obtained, and the obtained difference information is converted into overall feature difference information. Exemplarily, the key points in the original image are obtained by detecting the original image for a reference object. Exemplarily, taking the case where the type of reference object is a face and the original image is a face image, the key points in the face image are obtained by detecting the face for the face image. For example, by performing face detection on the face image, it is possible to detect the following: Figure 10 The 86 key points in the face image shown.

[0156] The embodiment of the present application provides a method for obtaining edge difference information. For example, the method provided by the embodiment of the present application can be applied in a scenario where a 3D face model is reconstructed based on 3DMM, such as a scenario where the face of a character in a 3D game / 3D film and television work is reconstructed based on 3DMM, thereby constraining the edge of the face. The edge difference information obtained by the embodiment of the present application can effectively constrain the edge of the face in the process of reconstructing a 3D face model based on 3DMM, thereby improving the degree of similarity between the face shape of the 3D reconstruction result and the original image. Under the method provided by the embodiment of the present application, regardless of the input posture of the face photo, the edge of the face can be effectively constrained in the process of reconstructing a 3D face model based on 3DMM. On the one hand, it can reduce the requirements for the input image and improve the robustness of the 3D face reconstruction. On the other hand, it can also effectively improve the degree of similarity between the face shape of the reconstructed 3D face model and the face shape of the face in the original image.

[0157] In the embodiment of the present application, the first edge keypoint used to indicate the edge of the virtual image is determined based on the edge information of the projected object corresponding to the virtual image, and is not a predefined keypoint with fixed semantics. The edge information of the projected object matches the actual edge of the virtual image. The first edge keypoint determined based on the edge information of the projected object has a high degree of match with the actual edge of the virtual image. The edge difference information obtained using the first edge keypoint can more accurately indicate the difference between the edge of the virtual image and the edge of the object in the image. The obtained edge difference information is reliable and accurate.

[0158] See also Figure 11 , an embodiment of the present application provides a device for obtaining edge difference information, the device comprising:

[0159] A first acquiring unit 1101 is configured to acquire a first virtual image corresponding to a reference object in an original image;

[0160] The second acquiring unit 1102 is configured to acquire a projection object corresponding to the first virtual image;

[0161] A first determining unit 1103 is configured to determine a first edge key point based on edge information of the projection object, where the first edge key point is a key point in the first virtual image used to indicate an edge of the first virtual image;

[0162] The third acquisition unit 1104 is used to obtain edge difference information based on the first edge key point and the second edge key point, where the second edge key point is a key point in the original image used to indicate the edge of the reference object, and the edge difference information is used to indicate the difference between the edge of the first virtual image and the edge of the reference object.

[0163] In one possible implementation, the number of first edge key points is at least one, and the number of second edge key points is at least one; the third acquisition unit 1104 is used to obtain, for any second edge key point, the difference index corresponding to any second edge key point based on the position information of any second edge key point and the position information of the projection points of each first edge key point; and obtain edge difference information based on the difference index corresponding to each second edge key point.

[0164] In one possible implementation, the third acquisition unit 1104 is also used to obtain the position distance between any second edge key point and the projection point of each first edge key point based on the position information of any second edge key point and the position information of the projection point of each first edge key point; determine the position distance that meets the reference condition among the position distances between any second edge key point and the projection point of each first edge key point; and obtain the difference index corresponding to any second edge key point based on the position distance that meets the reference condition.

[0165] In one possible implementation, the first determination unit 1103 is used to perform edge detection on the projection object in the projection image to obtain edge information of the projection object; based on the edge information of the projection object, obtain the position information of the edge pixel points in the projection image; based on the position information of the edge pixel points in the projection image, determine the first edge key point.

[0166] In one possible implementation, the number of edge pixel points in the projected image is at least one, and the first determination unit 1103 is further used to determine at least one candidate key point in the first virtual image; for any candidate key point among the at least one candidate key point, based on the position information of each edge pixel point in the projected image, obtain a measurement index corresponding to any candidate key point; in response to the measurement index corresponding to any candidate key point being not greater than a reference threshold, take any candidate key point as a first edge key point.

[0167] In one possible implementation, the first determination unit 1103 is also used to obtain the position distances between the projection point of any candidate key point and each edge pixel point based on the position information of the projection point of any candidate key point and the position information of each edge pixel point in the projected image; and use the minimum distance between the position distances between the projection point of any candidate key point and each edge pixel point as the measurement indicator corresponding to any candidate key point.

[0168] In a possible implementation, the second acquiring unit 1102 is configured to acquire projection parameters corresponding to the first virtual image; and project the first virtual image based on the projection parameters corresponding to the first virtual image to obtain a projection object.

[0169] In one possible implementation, the first virtual image is generated based on target generation parameters, see Figure 12 , the device further comprises:

[0170] The fourth acquisition unit 1105 is used to obtain updated generation parameters corresponding to the target generation parameters based on edge difference information. The updated generation parameters are used to generate a second virtual image corresponding to the reference object in the original image. The degree of matching between the edge of the second virtual image and the edge of the reference object is higher than the degree of matching between the edge of the first virtual image and the edge of the reference object.

[0171] In one possible implementation, the target generation parameters are obtained by predicting the original image by the first neural network model. The fourth acquisition unit 1105 is also used to update the model parameters of the first neural network model using edge difference information to obtain a second neural network model; the original image is input into the second neural network model, the generation parameters predicted by the second neural network model are obtained, and the generation parameters predicted by the second neural network model are used as updated generation parameters.

[0172] In the embodiment of the present application, the first edge keypoint used to indicate the edge of the virtual image is determined based on the edge information of the projected object corresponding to the virtual image, and is not a predefined keypoint with fixed semantics. The edge information of the projected object matches the actual edge of the virtual image. The first edge keypoint determined based on the edge information of the projected object has a high degree of match with the actual edge of the virtual image. The edge difference information obtained using the first edge keypoint can more accurately indicate the difference between the edge of the virtual image and the edge of the object in the image. The obtained edge difference information is reliable and accurate.

[0173] It should be noted that the apparatus provided in the above embodiments is merely illustrated by the division of the above functional modules when implementing its functions. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the apparatus and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0174] In an exemplary embodiment, a computer device is also provided. The computer device includes a processor and a memory, wherein the memory stores at least one computer program. The at least one computer program is loaded and executed by one or more processors to implement any of the aforementioned methods for obtaining edge difference information. The computer device can be a terminal or a server. The following describes the structures of the terminal and the server, respectively.

[0175] Figure 13This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may be a smartphone, tablet computer, laptop computer, or desktop computer. The terminal may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0176] Typically, the terminal includes: a processor 1301 and a memory 1302 .

[0177] Processor 1301 may include one or more processing cores, such as a quad-core processor or an octa-core processor. Processor 1301 may be implemented in hardware using at least one of the following: a DSP (Digital Signal Processing), an FPGA (Field-Programmable Gate Array), or a PLA (Programmable Logic Array). Processor 1301 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), processes data while awake, while the coprocessor is a low-power processor that processes data while in standby mode. In some embodiments, processor 1301 may integrate a graphics processing unit (GPU), which is responsible for rendering and drawing content displayed on the display. In some embodiments, processor 1301 may also include an AI processor for handling computational operations related to machine learning.

[0178] The memory 1302 may include one or more computer-readable storage media, which may be non-transitory. The memory 1302 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1302 is used to store at least one instruction, which is executed by the processor 1301 to implement the method for obtaining edge difference information provided in the method embodiment of the present application.

[0179] In some embodiments, the terminal may optionally include a peripheral device interface 1303 and at least one peripheral device. The processor 1301, memory 1302, and peripheral device interface 1303 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1303 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1304, a display screen 1305, a camera assembly 1306, an audio circuit 1307, and a power supply 1309.

[0180] The peripheral device interface 1303 can be used to connect at least one I / O (Input / Output)-related peripheral device to the processor 1301 and the memory 1302. The radio frequency circuit 1304 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The display screen 1305 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When the display screen 1305 is a touch screen display, the display screen 1305 also has the ability to collect touch signals on or above the surface of the display screen 1305. The camera assembly 1306 is used to collect images or videos. Optionally, the camera assembly 1306 includes a front camera and a rear camera.

[0181] Audio circuit 1307 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals that are input into processor 1301 for processing, or input into RF circuit 1304 for voice communication. The speaker is used to convert electrical signals from processor 1301 or RF circuit 1304 into sound waves. The speaker can be a traditional thin-film speaker or a piezoelectric ceramic speaker. Power supply 1309 is used to power various components in the terminal. Power supply 1309 can be AC ​​power, DC power, disposable batteries, or rechargeable batteries.

[0182] In some embodiments, the terminal further includes one or more sensors 1310 , including but not limited to: an acceleration sensor 1311 , a gyroscope sensor 1312 , a pressure sensor 1313 , an optical sensor 1315 , and a proximity sensor 1316 .

[0183] The acceleration sensor 1311 can detect the magnitude of acceleration on the three coordinate axes of the coordinate system established by the terminal. The gyroscope sensor 1312 can detect the body direction and rotation angle of the terminal. The gyroscope sensor 1312 can cooperate with the acceleration sensor 1311 to collect the user's 3D actions on the terminal. The pressure sensor 1313 can be set on the side frame of the terminal and / or the lower layer of the display screen 1305. When the pressure sensor 1313 is set on the side frame of the terminal, it can detect the user's holding signal of the terminal, and the processor 1301 performs left and right hand recognition or quick operation based on the holding signal collected by the pressure sensor 1313. When the pressure sensor 1313 is set on the lower layer of the display screen 1305, the processor 1301 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1305.

[0184] The optical sensor 1315 is used to collect ambient light intensity. The proximity sensor 1316, also known as a distance sensor, is typically provided on the front panel of the terminal. The proximity sensor 1316 is used to collect the distance between the user and the front of the terminal.

[0185] Those skilled in the art will understand that Figure 13 The structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0186] Figure 14 This is a structural diagram of a server provided in an embodiment of the present application. The server may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1401 and one or more memories 1402, wherein the one or more memories 1402 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1401 to implement the method for obtaining edge difference information provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server may also include other components for implementing device functions, which will not be described here.

[0187] In an exemplary embodiment, a computer-readable storage medium is further provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor of a computer device to implement any of the above methods for obtaining edge difference information.

[0188] In one possible implementation, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, or the like.

[0189] In an exemplary embodiment, a computer program product or computer program is also provided. The computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform any of the aforementioned methods for obtaining edge difference information.

[0190] It should be noted that the terms "first," "second," and the like in the specification and claims of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate so that the embodiments of the application described herein can be implemented in an order other than those illustrated or described herein. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present application. Instead, they are merely examples of devices and methods consistent with certain aspects of the present application as detailed in the appended claims.

[0191] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0192] The above description is merely an exemplary embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A method for obtaining edge difference information, characterized in that: The method comprises: Acquire a first virtual image corresponding to the reference object in the original image; Acquire a projection object corresponding to the first virtual image, where the projection object is in a projection image; Determining a first edge key point based on edge information of the projected object, where the first edge key point is a key point in the first virtual image that indicates an edge of the first virtual image; wherein the first edge key point is a candidate key point in the first virtual image whose corresponding measurement index is not greater than a reference threshold, and the measurement index corresponding to any candidate key point is the minimum distance between the projection point of any candidate key point and each edge pixel point in the projected image; Based on the first edge key point and the second edge key point, edge difference information is obtained, where the second edge key point is a key point in the original image used to indicate the edge of the reference object, and the edge difference information is used to indicate the difference between the edge of the first virtual image and the edge of the reference object.

2. The method according to claim 1, characterized in that The number of the first edge key points is at least one, and the number of the second edge key points is at least one; and obtaining edge difference information based on the first edge key points and the second edge key points includes: For any second edge key point, based on the position information of the any second edge key point and the position information of the projection points of each first edge key point, obtaining a difference index corresponding to the any second edge key point; The edge difference information is obtained based on the difference indicators respectively corresponding to the second edge key points.

3. The method according to claim 2, characterized in that The obtaining, based on the position information of the any second edge key point and the position information of the projection points of each first edge key point, a difference index corresponding to the any second edge key point includes: Based on the position information of any second edge key point and the position information of the projection points of each first edge key point, obtaining the position distance between each second edge key point and the projection point of each first edge key point; Determine, among the position distances between any one of the second edge key points and the projection points of each of the first edge key points, a position distance that satisfies a reference condition; Based on the position distance that meets the reference condition, a difference index corresponding to any second edge key point is obtained.

4. The method according to claim 1, wherein The method further comprises: Performing edge detection on the projection object in the projection image to obtain edge information of the projection object; Based on the edge information of the projection object, obtaining the position information of each edge pixel point; Based on the position information of the projection point of any candidate key point and the position information of each edge pixel point, the position distance between the projection point of any candidate key point and each edge pixel point is obtained.

5. The method according to any one of claims 1 to 4, characterized in that: The obtaining of the projection object corresponding to the first virtual image includes: Obtaining projection parameters corresponding to the first virtual image; The first virtual image is projected based on the projection parameters corresponding to the first virtual image to obtain the projection object.

6. The method according to any one of claims 1 to 4, characterized in that: The first virtual image is generated based on the target generation parameter, and after obtaining edge difference information based on the first edge key point and the second edge key point, the method further includes: Based on the edge difference information, updated generation parameters corresponding to the target generation parameters are obtained, and the updated generation parameters are used to generate a second virtual image corresponding to the reference object in the original image, and the degree of matching between the edge of the second virtual image and the edge of the reference object is higher than the degree of matching between the edge of the first virtual image and the edge of the reference object.

7. The method according to claim 6, characterized in that The target generation parameter is predicted by a first neural network model based on the original image, and obtaining an updated generation parameter corresponding to the target generation parameter based on the edge difference information includes: Using the edge difference information, the model parameters of the first neural network model are updated to obtain a second neural network model; The original image is input into the second neural network model, generation parameters predicted by the second neural network model are obtained, and the generation parameters predicted by the second neural network model are used as the updated generation parameters.

8. A device for obtaining edge difference information, characterized in that: The device comprises: A first acquiring unit, configured to acquire a first virtual image corresponding to a reference object in an original image; a second acquiring unit, configured to acquire a projection object corresponding to the first virtual image, wherein the projection object is in the projection image; a first determining unit, configured to determine a first edge key point based on edge information of the projected object, the first edge key point being a key point in the first virtual image used to indicate an edge of the first virtual image; wherein the first edge key point is a candidate key point in the first virtual image whose corresponding metric is not greater than a reference threshold, and the metric corresponding to any candidate key point is a minimum distance between a projection point of any candidate key point and each edge pixel point in the projected image; The third acquisition unit is used to obtain edge difference information based on the first edge key point and the second edge key point, where the second edge key point is a key point in the original image used to indicate the edge of the reference object, and the edge difference information is used to indicate the difference between the edge of the first virtual image and the edge of the reference object.

9. The device according to claim 8, characterized in that The number of the first edge key points is at least one, and the number of the second edge key points is at least one; the third acquisition unit is used to obtain, for any second edge key point, the difference index corresponding to any second edge key point based on the position information of the any second edge key point and the position information of the projection point of each first edge key point; and obtain the edge difference information based on the difference index corresponding to each second edge key point.

10. The device according to claim 9, characterized in that The third acquiring unit is configured to acquire, based on the position information of the any second edge key point and the position information of the projection points of the first edge key points, the position distances between the any second edge key point and the projection points of the first edge key points; Among the position distances between any second edge key point and the projection points of each first edge key point, determine the position distance that meets the reference condition; and based on the position distance that meets the reference condition, obtain the difference index corresponding to any second edge key point.

11. The device according to claim 8, characterized in that The first determining unit is further configured to perform edge detection on the projection object in the projection image to obtain edge information of the projection object; and obtain position information of each edge pixel point based on the edge information of the projection object; Based on the position information of the projection point of any candidate key point and the position information of each edge pixel point, the position distance between the projection point of any candidate key point and each edge pixel point is obtained.

12. The device according to any one of claims 8 to 11, characterized in that: The second acquisition unit is configured to acquire projection parameters corresponding to the first virtual image; and project the first virtual image based on the projection parameters corresponding to the first virtual image to obtain the projection object.

13. The device according to any one of claims 8 to 11, characterized in that: The first virtual image is generated based on the target generation parameter, and the apparatus further includes: A fourth acquisition unit is used to obtain updated generation parameters corresponding to the target generation parameters based on the edge difference information, and the updated generation parameters are used to generate a second virtual image corresponding to the reference object in the original image, and the degree of matching between the edge of the second virtual image and the edge of the reference object is higher than the degree of matching between the edge of the first virtual image and the edge of the reference object.

14. The device according to claim 13, characterized in that The target generation parameters are predicted by the first neural network model based on the original image. The fourth acquisition unit is used to update the model parameters of the first neural network model using the edge difference information to obtain a second neural network model; the original image is input into the second neural network model, the generation parameters predicted by the second neural network model are obtained, and the generation parameters predicted by the second neural network model are used as the updated generation parameters.

15. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the method for obtaining edge difference information according to any one of claims 1 to 7.

16. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the method for obtaining edge difference information according to any one of claims 1 to 7.

17. A computer program product, characterized in that The computer program product includes computer instructions, which are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the method for obtaining edge difference information as described in any one of claims 1 to 7.

Images