Image processing method and device, computer equipment and storage medium

Abstract

The embodiment of the invention discloses an image processing method and device, computer equipment and a storage medium. According to the scheme, the first perception feature of the to-be-recognized face image and the second perception feature of each virtual character image in the preset image set are acquired, and then the difference information between the first perception feature and each second perception feature is determined; and determining at least one target virtual character image similar to the eye sight line of the to-be-recognized face image from a preset image set based on the difference information, and determining predicted eye sight line information of the to-be-recognized face image according to the eye sight line information of the at least one target virtual character image. When the expression of the face image to be recognized is multiplexed to other virtual characters, the predicted eye sight line information can be directly applied to eye adjustment of other virtual characters. Therefore, the expression multiplexing efficiency between the face image and the virtual character can be improved.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] Eye gaze direction estimation is part of eye tracking technology, which measures eye movement and estimates gaze direction or the location of the eye's gaze point. This technology has numerous applications in healthcare, assisted driving, gaming, and virtual reality. In gaming and animation production, transferring facial expressions from facial capture videos or actors in standard film and television productions to virtual characters requires calculating the corresponding eye rotation angles to drive the character's eye movements.

[0003] Related work on eye angle calculation primarily focuses on calculating the eye gaze angle in the camera coordinate system. This is related not only to the state of the eye (eye rotation position) but also to the head posture. Large-scale datasets exist to ensure the accuracy and generalizability of the prediction results. However, because the virtual character's eye rotation angle is based on the head coordinate system, this rotation angle cannot be directly used to drive the virtual character. Summary of the Invention

[0004] The embodiments of the present application provide an image processing method, apparatus, computer device, and storage medium, which can improve the efficiency of expression reuse between facial images and virtual characters.

[0005] The present invention provides an image processing method, including:

[0006] Obtaining a first perceptual feature of the face image to be recognized and a second perceptual feature of each virtual character image in the preset image set;

[0007] determining difference information between the first perceptual feature and each second perceptual feature;

[0008] Determining at least one target virtual character image having an eye sight line similar to that of the face image to be recognized from the preset image set based on the difference information;

[0009] The predicted eye sight line information of the face image to be recognized is determined based on the eye sight line information of the at least one target virtual character image.

[0010] Accordingly, an embodiment of the present application further provides an image processing device, comprising:

[0011] an acquisition unit, configured to acquire a first perceptual feature of the face image to be recognized and a second perceptual feature of each virtual character image in the preset image set;

[0012] a first determining unit, configured to determine difference information between the first perceptual feature and each second perceptual feature;

[0013] a second determining unit, configured to determine, from the preset image set based on the difference information, at least one target virtual character image having an eye sight line similar to that of the face image to be recognized;

[0014] The third determining unit is configured to determine the predicted eye sight line information of the face image to be recognized based on the eye sight line information of the at least one target virtual character image.

[0015] Correspondingly, an embodiment of the present application also provides a computer device, including a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor executes any image processing method provided in any embodiment of the present application.

[0016] Correspondingly, an embodiment of the present application further provides a storage medium, which stores a plurality of instructions suitable for loading by a processor to execute the above image processing method.

[0017] The embodiment of the present application obtains the first perceptual feature of the face image to be identified and the second perceptual feature of each virtual character image in the preset image set, and then determines the difference information between the first perceptual feature and each second perceptual feature, and determines at least one target virtual character image from the preset image set that is similar to the eye sight line of the face image to be identified based on the difference information, and then determines the predicted eye sight line information of the face image to be identified based on the eye sight line information of at least one target virtual character image. Through the perceptual features of the face to be identified and the virtual character, the virtual character image with similar eye sight line as the face image to be identified in terms of overall facial expression is screened out, and then the predicted eye sight line information obtained based on the eye sight line information of the virtual character image is the eye sight line information on the overall facial expression. When the expression of the face image to be identified is reused on other virtual characters, the predicted eye sight line information can be directly applied to the eye adjustment of other virtual characters. In this way, the efficiency of expression reuse between face images and virtual characters can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 A flowchart of an image processing method provided in an embodiment of the present application.

[0020] Figure 2 A schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application.

[0021] Figure 3 A schematic diagram of an application scenario of another image processing method provided in an embodiment of the present application.

[0022] Figure 4 A schematic diagram of an application scenario of another image processing method provided in an embodiment of the present application.

[0023] Figure 5 This is a structural block diagram of an image processing device provided in an embodiment of the present application.

[0024] Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0025] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making any creative efforts shall fall within the scope of protection of this application.

[0026] The embodiments of the present application provide an information recommendation method, apparatus, storage medium and computer equipment. Specifically, the information recommendation method of the embodiments of the present application can be executed by a computer device, wherein the computer device can be a terminal or server device. The terminal can be a terminal device such as a smart phone, a tablet computer, a laptop computer, a touch screen, a personal computer (PC, Personal Computer), a personal digital assistant (Personal Digital Assistant, PDA), etc. The server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network, content distribution network), and big data and artificial intelligence platforms.

[0027] For example, the computer device can be a server, which can obtain the first perceptual feature of the facial image to be identified, and the second perceptual feature of each virtual character image in a preset image set; determine the difference information between the first perceptual feature and each second perceptual feature; determine at least one target virtual character image from the preset image set that has an eye sight similar to that of the facial image to be identified based on the difference information; and determine the predicted eye sight information of the facial image to be identified based on the eye sight information of at least one target virtual character image.

[0028] Based on the above problems, the embodiments of the present application provide a first image processing method, apparatus, computer device and storage medium, which can improve the efficiency of expression reuse between facial images and virtual characters.

[0029] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0030] An embodiment of the present application provides an image processing method, which can be executed by a terminal or a server. The embodiment of the present application takes the image processing method executed by a server as an example for explanation.

[0031] See also Figure 1 , Figure 1 This is a flow chart of an image processing method provided in an embodiment of the present application. The specific flow of the image processing method can be as follows:

[0032] 101. Obtain a first perceptual feature of a face image to be recognized and a second perceptual feature of each virtual character image in a preset image set.

[0033] In the embodiments of the present application, the facial image to be recognized may be a facial image whose eye gaze needs to be identified. The eye gaze may include, for example, the angle of eye movement. The facial image to be recognized may be a real facial image or a virtual facial image. The facial image to be recognized may be obtained by photographing a facial image or by obtaining it from a facial image library.

[0034] The preset image set includes multiple virtual character images, which can be images of virtual characters drawn using modeling software. The virtual character images include the facial area of ​​the virtual character. The virtual character can be used in game scenes or virtual reality scenes, etc.

[0035] For example, see Figure 2 , Figure 2 A schematic diagram of an application scenario of an image processing method provided in an embodiment of the present application. Figure 2 An image of an avatar is shown.

[0036] In an embodiment of the present application, the multiple virtual character images in the preset image set may be virtual character images with different eye sights. For example, the multiple virtual character images may be virtual character images with different eye rotation angles.

[0037] Since the virtual character images in the preset image set are drawn using modeling software, the eye sight information of each virtual character image in the preset image set can be obtained based on the data during the drawing process.

[0038] For example, for a virtual character A in a preset image set, when drawing it through modeling software, you can input an eye rotation angle, such as 50 degrees, and then the eye rotation angle of the drawn virtual character A will be 50 degrees, and then the eye rotation angle of 50 degrees can be saved as the eye sight information of the virtual character A.

[0039] In an embodiment of the present application, after obtaining the face image to be identified and the preset image set, the face image to be identified and the virtual character image in the preset image set can be preprocessed.

[0040] Among them, image preprocessing can include various processing methods, such as face detection, cropping, scaling, etc.

[0041] Specifically, face detection is performed on the face image to be identified, the face area in the face image to be identified is determined, and then the face image to be identified is cropped according to the face area, and the area outside the face in the image is cropped to remove the part of the image that is not related to the face, which can improve the subsequent estimation and detection of eye gaze.

[0042] Similarly, for each virtual character image in the preset image set, face detection can be performed separately to determine the face area in each virtual character image, and then the virtual character image can be cropped according to the face area, and the area outside the face in the image is cropped to remove the parts of the image that are not related to the face, which can improve the subsequent estimation and detection of eye gaze.

[0043] In some embodiments, to accurately extract perceptual features from an image, the step of "obtaining a first perceptual feature of the face image to be recognized and a second perceptual feature of each virtual character image in a preset image set" may include the following operations:

[0044] Input the face image to be recognized and the images of each virtual character in the preset image set into the target feature extraction model;

[0045] The target feature extraction model is used to extract features of the face image to be identified and each virtual character image, respectively, to obtain the first perception feature and each second perception feature.

[0046] In an embodiment of the present application, a target feature extraction model can be used to extract perceptual features of a face in an image. The target feature extraction model can include a feature extractor, and the feature extractor can be used to extract perceptual features of a face image.

[0047] The perceptual feature may be a feature of the eye sight of a face. Specifically, a feature extractor extracts data of the eye sight of a face in an image, encodes the extracted data, and the processed code can be used as the perceptual feature of the image.

[0048] Specifically, the facial image to be identified and the images of each virtual character can be input into the target feature extraction model respectively, and the feature extractor in the target feature extraction model can be used to perform feature extraction on the facial image to be identified to obtain the first perceptual feature of the facial image to be identified; and feature extraction can be performed on each virtual character image to obtain the second perceptual feature of each virtual character image.

[0049] In some embodiments, before the step of “inputting the face image to be recognized and each virtual character image in the preset image set into the target feature extraction model”, the method may further include the following steps:

[0050] Collecting multiple sample face images and multiple sample virtual character images;

[0051] Determine a triplet sample based on a plurality of sample face images and a plurality of sample virtual character images;

[0052] Based on triple samples, a target feature extraction model is constructed.

[0053] Among them, the sample facial images can be collected by photographing facial images or obtaining them from an existing facial image database; the sample virtual character images can be drawn by modeling software (such as Maya), and the corresponding virtual character images can be rendered by inputting the eye rotation angle value, which can be used as sample virtual character images.

[0054] Among them, the collected multiple sample face images and multiple sample virtual character images are all images that can distinguish the angle of eye rotation, so as to facilitate subsequent accurate eye gaze detection.

[0055] In some embodiments, the step of “determining triplet samples based on multiple sample face images and multiple sample virtual character images” may include the following operations:

[0056] Selecting an image from a plurality of sample face images or a plurality of sample virtual character images to obtain a target sample image;

[0057] Select an image with an eye sight line similar to the target sample image from multiple sample face images or multiple sample virtual character images to obtain a positive sample image;

[0058] Selecting an image whose eye sight is dissimilar to the target sample image from a plurality of sample face images or a plurality of sample virtual character images to obtain a negative sample image;

[0059] Based on the target sample image, the positive sample image and the negative sample image, a triplet sample is obtained.

[0060] In an embodiment of the present application, one image may be selected from a plurality of sample human face images or a plurality of sample virtual character images as a target sample image. For example, one image may be selected from a plurality of sample human face images as a target sample image, or one image may be selected from a plurality of sample virtual character images as a target sample image.

[0061] Then, an image with a similar eye gaze to the target sample image can be selected from the multiple sample face images or multiple sample avatar images as a positive sample image. For example, an image with a similar eye gaze to the target sample image can be selected from the multiple sample face images as a positive sample image, or an image with a similar eye gaze to the target sample image can be selected from the multiple sample avatar images as a positive sample image.

[0062] Finally, an image with a dissimilar eye gaze to the target sample image can be selected from multiple sample facial images or multiple sample avatar images as a negative sample image. For example, an image with a dissimilar eye gaze to the target sample image can be selected from multiple sample facial images as a negative sample image, or an image with a dissimilar eye gaze to the target sample image can be selected from multiple sample avatar images as a negative sample image. A triplet sample is obtained based on the target sample image, the positive image, and the negative image.

[0063] The triplet sample includes at least one sample virtual character image and at least one sample human face image. For example, the triplet sample may include one sample virtual character image and two sample human face images; or the triplet sample may include two sample virtual character images and one sample human face image.

[0064] Specifically, through the above-mentioned method of constructing triple samples, multiple triple samples can be obtained according to multiple sample face images and multiple sample virtual character images.

[0065] In some embodiments, in order to ensure the accuracy of the constructed triple samples, the triple samples can be further annotated and screened manually. That is, for each triple sample, multiple people annotate two images with similar eye gazes and images with dissimilar eye gazes in the triple sample. When the proportion of consistent results of the person annotations is greater than a preset proportion (such as 80%), the triple sample can be determined to be a valid sample and can be retained; and when the proportion of consistent results of the person annotations is not greater than a preset proportion (such as 80%), the triple sample can be determined to be an invalid sample and can be eliminated. Through further manual screening, multiple valid triple samples are obtained.

[0066] In some embodiments, to improve the accuracy of feature extraction, the step of "building a target feature extraction model based on triple samples" may include the following operations:

[0067] Input the target sample image, positive sample image and negative sample image in the triplet sample into the preset feature extraction model;

[0068] Extracting features of the target sample image, the positive sample image, and the negative sample image respectively through a preset feature extraction model to obtain a first sample perception feature of the target sample image, a second sample perception feature of the positive sample image, and a third sample perception feature of the negative sample image;

[0069] A preset feature extraction model is trained based on the first sample perception feature, the second sample perception feature, and the third sample perception feature to obtain a target feature extraction model.

[0070] Among them, the preset feature extraction model can be used to extract image features. The preset feature extraction model can include a feature extractor, and the feature extractor can be a convolutional neural network (CNN), which can be used to extract image features.

[0071] In an embodiment of the present application, in order to eliminate irrelevant information in the image, restore useful real information, enhance the detectability of relevant information and simplify the data to the maximum extent, thereby improving the reliability of feature extraction, image segmentation, matching and recognition, before the target sample image, positive sample image and negative sample image in the triplet sample are input into the preset feature extraction model, each image in the triplet sample can be preprocessed separately.

[0072] Among them, preprocessing can include face detection, image cropping, etc.

[0073] Specifically, face detection is performed on the target sample image, positive sample image and negative sample image in the triplet sample respectively, the face area in each sample image is determined, and then each sample image is cropped according to the face area to obtain a processed sample image.

[0074] Among them, extracting the features of the target sample image, the positive sample image and the negative sample image respectively through the preset feature extraction model can include: extracting the features of the preprocessed target sample image, the preprocessed positive sample image and the preprocessed negative sample image respectively through the preset feature extraction model.

[0075] For example, the target sample image, the positive sample image and the negative sample image are input into a preset feature extraction model, and the eye features of the face in the target sample image are extracted by the feature extractor in the preset feature extraction model to obtain the first sample perception feature; the eye features of the face in the positive sample image are extracted to obtain the second sample perception feature; and the eye features of the face in the negative sample image are extracted to obtain the third sample perception feature.

[0076] In some embodiments, the step of “training a preset feature extraction model based on the first sample perception feature, the second sample perception feature, and the third sample perception feature to obtain a target feature extraction model” may include the following operations:

[0077] calculating a first difference between the first sample perception feature and the second sample perception feature;

[0078] calculating a second difference between the first sample perception feature and the third sample perception feature;

[0079] Based on the first difference, the second difference and the preset loss function, the model parameters of the preset feature extraction model are adjusted until the preset feature extraction model converges to obtain the target feature extraction model.

[0080] Calculating the first difference between the first sample perception feature and the second sample perception feature may include calculating a distance between the first sample perception feature and the second sample perception feature in a feature space to obtain the first difference. The first difference may represent a first difference between the target sample image and the positive sample image.

[0081] Calculating the second difference between the first sample perception feature and the third sample perception feature may include calculating a distance between the first sample perception feature and the third sample perception feature in a feature space to obtain a second difference. The second difference may represent a second difference between the target sample image and the negative sample image.

[0082] Among them, the preset loss function can be a triplet loss function, which can be used to calculate the loss of the model.

[0083] In the embodiment of the present application, the triple loss function can be as follows:

[0084] L TRI (A,P,N)=max(‖f(A)-f(P)‖ 2 -‖f(A)-f(N)‖ 2 +α, 0)

[0085] Among them, A refers to the target sample image; P refers to the positive sample image, N refers to the negative sample image; L TRI (A, P, N) is the loss value calculated by the triplet loss function; f(A) refers to the first sample perception feature of the target sample image, f(P) refers to the second sample perception feature of the positive sample image, and f(N) refers to the third sample perception feature of the negative sample image; α is a preset hyperparameter, whose function is to judge whether the difference between the distance between f(A) and f(P) and the distance between f(A) and f(N) is greater than or equal to α. If it is greater than or equal to α, it means that the distance between f(A) and f(P) has been optimized to be close enough, and the distance between f(A) and f(N) has been optimized to be far enough.

[0086] In an embodiment of the present application, the triplet loss function enables the feature extractor to learn an expressive representation space that maps similar eye features closer and different eye features farther away.

[0087] For example, see Figure 3 , Figure 3 Schematic diagram of an application scenario of another image processing method provided by an embodiment of the present application. For each extracted perceptual feature (f(A), f(P), f(N)), the difference between f(A) and f(P) is calculated to obtain a first difference between A and P, and the difference between f(A) and f(N) is calculated to obtain a second difference between A and N. At this time, the first difference is greater than the second difference. Then, supervised learning is performed using the above-mentioned triplet loss function. The difference between f(A) and f(P) after supervised learning is calculated to be less than the difference between f(A) and f(N), which accurately expresses that the similarity of the eye features of A and P is high.

[0088] In some embodiments, see Figure 4 , Figure 4 A schematic diagram of an application scenario of another image processing method provided in an embodiment of the present application. Figure 4 The process of training the preset feature extraction model to obtain the target feature extraction model in this solution can be as follows:

[0089] First, a constructed triplet sample is obtained, and the target sample image, positive sample image, and negative sample image in the triplet sample are input into the feature extractor of the preset feature extraction model. The feature extractor extracts the eye features of the target sample image to obtain the first perceptual feature f(A); extracts the eye features of the positive sample image to obtain the second perceptual feature f(P); and extracts the eye features of the negative sample image to obtain the third perceptual feature f(P).

[0090] Then, the triplet loss function L is used TRI Calculations are performed based on the first perception feature f(A), the second perception feature f(P), and the third perception feature f(P) to bring features of facial images with similar eye rotation angles closer together in the representation space and to move dissimilar features further apart. This adjusts the model parameters of the preset feature extraction model to obtain a target feature extraction model.

[0091] 102. Determine difference information between the first perceptual feature and each second perceptual feature.

[0092] The difference information refers to the difference between different features, and can be used to represent the difference between images corresponding to different features.

[0093] In some embodiments, the step of “determining difference information between the first perceptual feature and each second perceptual feature” may include the following operations:

[0094] The distance between the first perception feature and each second perception feature is calculated to obtain difference information.

[0095] Calculating the distance between the first perceptual feature and each second perceptual feature may include calculating the Euclidean distance between the first perceptual feature and the second perceptual feature.

[0096] Among them, Euclidean distance is the "normal" (i.e., straight-line) distance between two points in Euclidean space, and is usually used to calculate the distance between two data points. In feature analysis, Euclidean distance can be used to calculate the distance between two samples in feature space. Specifically, given two samples x and y, their Euclidean distance in feature space is defined as follows:

[0097] d(x,y)=sqrt(sum((xi-yi)^2));

[0098] Among them, xi and yi are the values ​​of x and y on the i-th feature respectively.

[0099] For example, in the embodiment of the present application, the calculation formula for calculating the distance between the first perceptual feature and the second perceptual feature may be: sqrt(sum((f1-f2)^2)), where f1 is the first perceptual feature and f2 is the second perceptual feature. The distance between the first perceptual feature and each second perceptual feature can then be calculated as the difference information between the first perceptual feature and each second perceptual feature.

[0100] 103. Determine, from a preset image set based on the difference information, at least one target virtual character image having an eye sight line similar to that of the face image to be recognized.

[0101] Determining the target virtual character image having an eye sight line similar to that of the face image to be recognized may include determining the target virtual character image having an eye rotation angle similar to that of the face image to be recognized.

[0102] In some embodiments, to improve the accuracy of predicting the eye gaze of the facial image to be recognized, the step of "determining, from a preset image set based on the difference information, at least one target virtual character image having an eye gaze similar to that of the facial image to be recognized" may include the following operations:

[0103] Filtering a preset number of difference information from all difference information in descending order to obtain a preset number of target difference information;

[0104] The virtual character images to which the second perception features corresponding to the target difference information belong are acquired from the preset image set to obtain a preset number of target virtual character images.

[0105] In an embodiment of the present application, top-k virtual character images whose perceptual features have the smallest difference from the perceptual features of the face image to be identified can be selected from the preset image combination.

[0106] Specifically, after determining the difference between the face image to be identified and each virtual character image, the virtual character images in the preset image set can be sorted from small to large according to the difference to obtain a sorted virtual character image sequence; then the first k virtual character images can be selected from the virtual character image sequence to obtain the target virtual character image.

[0107] The selected target virtual character image is an image with an eyeball rotation angle similar to that of the face image to be recognized.

[0108] 104. Determine predicted eye sight information of the face image to be recognized based on the eye sight information of at least one target virtual character image.

[0109] In some embodiments, the step of “determining predicted eye gaze information of a face image to be recognized based on eye gaze information of at least one target virtual character image” may include the following operations:

[0110] Obtaining eye gaze information of each target virtual character image;

[0111] A weighted summation process is performed based on the eyeball rotation angle of each target virtual character image to obtain the predicted eyeball rotation angle of the face image to be recognized.

[0112] The eye sight information at least includes the eye rotation angle of the target virtual character.

[0113] In the embodiment of the present application, since the virtual character images in the preset image set are drawn by modeling software and the eye rotation angle is set during drawing, each virtual character image in the preset image set contains the actual eye rotation angle.

[0114] In some embodiments, to further improve the accuracy of the prediction results, the step of "performing a weighted summation process based on the eye movement angles of each target virtual character image to obtain the predicted eye movement angles of the face image to be recognized" may include the following operations:

[0115] determining a weight corresponding to each target virtual character image based on difference information corresponding to the second perception feature of each target virtual character image;

[0116] performing weighted processing on the eyeball rotation angles of the target virtual character images according to the weights corresponding to the target virtual character images to obtain weighted eyeball rotation angles;

[0117] The weighted eye rotation angles are summed to obtain the predicted eye rotation angle of the face image to be recognized.

[0118] Among them, determining the weight corresponding to each target virtual character image based on the difference information corresponding to the second perceptual feature of each target virtual character image can include: calculating the weight value corresponding to each target virtual character image based on the difference information between the second perceptual feature of each target virtual character image and the first perceptual feature of the face image to be identified.

[0119] The target virtual character image with smaller difference information may have a corresponding larger weight value; the target virtual character image with larger difference information may have a corresponding smaller weight value.

[0120] Furthermore, after determining the weight value corresponding to each target virtual character, the eye rotation angle of each target virtual character can be multiplied by the corresponding weight value to obtain the weighted eye rotation angle; then the weighted eye rotation angles are added together to obtain the predicted eye rotation angle of the face image to be identified.

[0121] The calculation formula for predicting the eye rotation angle can be as follows:

[0122]

[0123] Among them, P refers to the predicted eye movement angle of the face image to be recognized; input refers to the face image to be recognized; image i Refers to the i-th target virtual character image among k target virtual character images; dis(input,image i ) refers to the distance between the perceptual features of the face image to be recognized and the i-th target virtual character image; p i Refers to the eye movement angle of the i-th target virtual character image; K represents the preset number.

[0124] in, It can be used to calculate the weight corresponding to each target virtual character image.

[0125] After the predicted eye movement angle of the face image to be recognized is calculated, the predicted eye movement angle can be used as the eye sight estimation result of the face image to be recognized.

[0126] In some embodiments, the predicted eye gaze estimation results for a facial image to be recognized can be applied to virtual characters in animations. For example, the eye gaze angles of a virtual character in an animation can be set based on the predicted eye gaze angles of the facial image to be recognized, thereby controlling the virtual character to mimic the facial expressions and movements of a real person. Alternatively, the eye gaze estimation results can be applied to other scenarios involving virtual characters, which are not specifically described here.

[0127] In some embodiments, the method may further include the following steps:

[0128] Obtaining a virtual character model to be processed and facial expression information of a face image to be recognized;

[0129] The facial expression of the virtual character model to be processed is adjusted according to the facial expression information, and the eye sight of the virtual character model to be processed is adjusted according to the predicted eye sight information to obtain a target virtual character model with an expression similar to that of the face image to be recognized.

[0130] The face image to be recognized may be a captured image of a real face.

[0131] The virtual character model to be processed may be a pre-built object model used in a virtual scene, wherein the virtual scene may include multiple types, such as a game scene, a virtual reality scene, and the like.

[0132] For example, the virtual character model to be processed may be a model of a game character in a game scene, or the virtual character model to be processed may be a virtual character model corresponding to a user in a virtual reality scene, etc.

[0133] The facial expression information refers to the facial expression information of the face in the face image to be recognized, for example, the facial expression information may include frowning, closing the mouth, opening the mouth, etc.

[0134] Among them, the facial expression information of the face image to be identified can be obtained by performing expression recognition on the face image to be identified. For example, the facial expression features of the face image to be identified can be extracted, and then the facial expression features can be recognized to obtain the facial expression information of the face image to be identified.

[0135] Adjusting the facial expression of the virtual character model to be processed according to the facial expression information may include: adjusting facial expression-related parameters of the virtual character model to be processed according to the facial expression information.

[0136] For example, the facial expression information of the face image to be recognized may include: corners of the mouth are raised, and the entire virtual character model to be processed may be adjusted to have the corners of the mouth raised.

[0137] Adjusting the eye sight lines of the virtual character model to be processed according to the predicted eye sight line information may include adjusting eye-related parameters of the virtual character model to be processed according to the predicted eye sight line information.

[0138] For example, the predicted eye sight information may include an eyeball rotation angle of 50 degrees, and the eyeball rotation angle of the entire virtual character model to be processed may be adjusted to 50 degrees.

[0139] By reusing the facial expression information of the face image to be recognized and the predicted eye gaze information onto the virtual character model to be processed, the expression of the real face can be transferred to the virtual character, thereby achieving the authenticity of the expression effect of the virtual character.

[0140] The present application discloses an image processing method comprising: obtaining a first perceptual feature of a facial image to be identified and a second perceptual feature of each virtual character image in a preset image set; determining difference information between the first perceptual feature and each second perceptual feature; determining, based on the difference information, at least one target virtual character image from the preset image set that has an eye gaze similar to that of the facial image to be identified; and determining predicted eye gaze information of the facial image to be identified based on the eye gaze information of the at least one target virtual character image. This method improves the efficiency of expression reuse between facial images and virtual characters.

[0141] To facilitate better implementation of the pop-up window management method provided in the embodiment of the present application, the embodiment of the present application also provides an image processing device based on the above-mentioned image processing method. The meanings of the terms herein are the same as those in the above-mentioned image processing method, and the specific implementation details can be referred to the description in the method embodiment.

[0142] See also Figure 5 , Figure 5 This is a structural block diagram of an image processing device provided in an embodiment of the present application, the device comprising:

[0143] An acquisition unit 301 is configured to acquire a first perceptual feature of a face image to be recognized and a second perceptual feature of each virtual character image in a preset image set;

[0144] A first determining unit 302 is configured to determine difference information between the first perceptual feature and each second perceptual feature;

[0145] A second determining unit 303 is configured to determine, from the preset image set based on the difference information, at least one target virtual character image having an eye sight line similar to that of the face image to be recognized;

[0146] The third determining unit 304 is configured to determine the predicted eye sight line information of the to-be-recognized face image based on the eye sight line information of the at least one target virtual character image.

[0147] In some embodiments, the third determining unit 304 may include:

[0148] A first acquiring subunit is configured to acquire eye sight information of each target virtual character image, wherein the eye sight information at least includes an eyeball rotation angle;

[0149] The first processing sub-unit is configured to perform weighted summation processing based on the eyeball rotation angle of each target virtual character image to obtain a predicted eyeball rotation angle of the face image to be recognized.

[0150] In some embodiments, the first processing subunit may be specifically configured to:

[0151] determining a weight corresponding to each target virtual character image based on difference information corresponding to the second perception feature of each target virtual character image;

[0152] performing weighted processing on the eyeball rotation angles of the target virtual character images according to the weights corresponding to the target virtual character images to obtain weighted eyeball rotation angles;

[0153] The weighted eye rotation angles are summed to obtain the predicted eye rotation angle of the face image to be recognized.

[0154] In some embodiments, the second determining unit 303 may include:

[0155] A screening subunit is configured to screen a preset number of difference information from all the difference information in descending order to obtain the preset number of target difference information;

[0156] The second acquisition subunit is configured to acquire, from the preset image set, virtual character images corresponding to the second perception features of each target difference information, to obtain the preset number of target virtual character images.

[0157] In some embodiments, the first determining unit 302 may include:

[0158] The first calculation subunit is configured to calculate a distance between the first perception feature and each second perception feature to obtain the difference information.

[0159] In some embodiments, the acquiring unit 301 may include:

[0160] A first input subunit is configured to input the face image to be recognized and each virtual character image in the preset image set into a target feature extraction model;

[0161] The first extraction subunit is used to perform feature extraction on the face image to be identified and each virtual character image respectively through the target feature extraction model to obtain the first perception feature and each second perception feature.

[0162] In some embodiments, the apparatus may further comprise:

[0163] An acquisition unit, configured to acquire a plurality of sample human face images and a plurality of sample virtual character images;

[0164] a fourth determining unit, configured to determine a triplet sample based on the plurality of sample face images and the plurality of sample virtual character images;

[0165] A construction unit is used to construct the target feature extraction model based on the triple sample.

[0166] In some embodiments, the fourth determining unit may include:

[0167] A first selection subunit is configured to select an image from the plurality of sample face images or the plurality of sample virtual character images to obtain a target sample image;

[0168] A second selection subunit is configured to select an image having an eye sight line similar to that of the target sample image from the plurality of sample face images or the plurality of sample virtual character images to obtain a positive sample image;

[0169] A third selection subunit is configured to select an image whose eye sight is not similar to the target sample image from the plurality of sample face images or the plurality of sample virtual character images to obtain a negative sample image;

[0170] A determination subunit is configured to obtain the triplet sample based on the target sample image, the positive sample image, and the negative sample image.

[0171] In some embodiments, a building block may include:

[0172] A second input subunit is used to input the target sample image, the positive sample image and the negative sample image in the triplet sample into a preset feature extraction model;

[0173] A second extraction subunit is configured to extract features of the target sample image, the positive sample image, and the negative sample image respectively through the preset feature extraction model to obtain a first sample perception feature of the target sample image, a second sample perception feature of the positive sample image, and a third sample perception feature of the negative sample image;

[0174] A training subunit is used to train the preset feature extraction model based on the first sample perception feature, the second sample perception feature, and the third sample perception feature to obtain the target feature extraction model.

[0175] In some embodiments, the training subunit may be specifically used to:

[0176] Calculating a first difference between the first sample perception feature and the second sample perception feature;

[0177] calculating a second difference between the first sample perception feature and the third sample perception feature;

[0178] Based on the first difference, the second difference and a preset loss function, the model parameters of the preset feature extraction model are adjusted until the preset feature extraction model converges to obtain the target feature extraction model.

[0179] In some embodiments, the apparatus may further comprise:

[0180] A second acquiring unit is used to acquire the virtual character model to be processed and the facial expression information of the face image to be recognized;

[0181] An adjustment unit is used to adjust the facial expression of the virtual character model to be processed according to the facial expression information, and adjust the eye sight of the virtual character model to be processed according to the predicted eye sight information, so as to obtain a target virtual character model with an expression similar to that of the facial image to be recognized.

[0182] The present embodiment discloses an image processing device. An acquisition unit 301 acquires a first perceptual feature of a facial image to be identified and a second perceptual feature of each virtual character image in a preset image set. A first determination unit 302 determines difference information between the first perceptual feature and each second perceptual feature. A second determination unit 303 determines, based on the difference information, at least one target virtual character image from the preset image set that has an eye gaze similar to that of the facial image to be identified. A third determination unit 304 determines predicted eye gaze information of the facial image to be identified based on the eye gaze information of the at least one target virtual character image. This improves the efficiency of expression reuse between facial images and virtual characters.

[0183] Accordingly, the embodiment of the present application also provides a computer device, which may be a server. Figure 6 As shown, Figure 6 A schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 500 includes a processor 501 having one or more processing cores, a memory 502 having one or more computer-readable storage media, and a computer program stored in the memory 502 and executable on the processor. The processor 501 is electrically connected to the memory 502. Those skilled in the art will appreciate that the computer device structure shown in the figure does not constitute a limitation of the computer device, and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0184] The processor 501 is the control center of the computer device 500. It uses various interfaces and lines to connect various parts of the entire computer device 500. By running or loading software programs and / or modules stored in the memory 502 and calling data stored in the memory 502, it executes various functions of the computer device 500 and processes data, thereby monitoring the computer device 500 as a whole.

[0185] In the embodiment of the present application, the processor 501 in the computer device 500 loads instructions corresponding to one or more application processes into the memory 502 according to the following steps, and the processor 501 runs the application stored in the memory 502 to implement various functions:

[0186] Obtain a first perceptual feature of a facial image to be identified, and a second perceptual feature of each virtual character image in a preset image set; determine difference information between the first perceptual feature and each second perceptual feature; based on the difference information, determine at least one target virtual character image from the preset image set that has an eye gaze similar to that of the facial image to be identified; and determine predicted eye gaze information of the facial image to be identified based on the eye gaze information of the at least one target virtual character image.

[0187] The embodiment itself obtains the first perceptual feature of the facial image to be identified, and the second perceptual feature of each virtual character image in the preset image set, and then determines the difference information between the first perceptual feature and each second perceptual feature, and determines at least one target virtual character image from the preset image set that has an eye sight similar to the facial image to be identified based on the difference information, and then determines the predicted eye sight information of the facial image to be identified based on the eye sight information of at least one target virtual character image. Since the predicted eye sight information is determined based on the eye sight information of the virtual character image, it can be directly applied to adjust the eyes of the virtual character later. When the expression of the facial image to be identified is reused on other virtual characters, the predicted eye sight information can be directly applied to the eye adjustment of other virtual characters. In this way, the efficiency of expression reuse between facial images and virtual characters can be improved.

[0188] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0189] Optional, such as Figure 6 As shown, the computer device 500 further includes: a touch screen 503, a radio frequency circuit 504, an audio circuit 505, an input unit 506, and a power supply 507. Among them, the processor 501 is electrically connected to the touch screen 503, the radio frequency circuit 504, the audio circuit 505, the input unit 506, and the power supply 507 respectively. It can be understood by those skilled in the art that Figure 6 The computer device structure shown in the figure does not constitute a limitation to the computer device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0190] The touch display screen 503 can be used for displaying a graphical user interface and receiving the operation instructions generated by the user acting on the graphical user interface. The touch display screen 503 can include a display panel and a touch panel. Among them, the display panel can be used for displaying the information input by the user or the information provided to the user and various graphical user interfaces of the computer device, and these graphical user interfaces can be composed of graphics, text, icons, videos and any combination thereof. Optionally, the display panel can be configured in the form of a liquid crystal display (LCD, Liquid Crystal Display), an organic light emitting diode (OLED, Organic Light-Emitting Diode) and the like. The touch panel can be used for collecting the touch operation of the user thereon or near it (such as the user uses any suitable object or accessory such as a finger, a stylus on the touch panel or near the touch panel), and generates corresponding operation instructions, and the operation instructions execute corresponding programs. Optionally, the touch panel may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the user's touch direction, detects the signal brought by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device, converts it into the touch point coordinates, and then sends it to the processor 501, and can receive the command sent by the processor 501 and execute it. The touch panel can cover the display panel. When the touch panel detects a touch operation on or near it, it is transmitted to the processor 501 to determine the type of touch event, and then the processor 501 provides a corresponding visual output on the display panel according to the type of touch event. In an embodiment of the present application, the touch panel and the display panel can be integrated into the touch display screen 503 to realize input and output functions. However, in some embodiments, the touch panel and the touch panel can be used as two independent components to realize input and output functions. That is, the touch display screen 503 can also be used as part of the input unit 506 to realize the input function.

[0191] The radio frequency circuit 504 may be used to transmit and receive radio frequency signals, so as to establish wireless communication with a network device or other computer device through wireless communication, and to transmit and receive signals between the network device or other computer device.

[0192] Audio circuit 505 can be used to provide an audio interface between the user and the computer device through a speaker and microphone. Audio circuit 505 can convert received audio data into electrical signals and transmit them to the speaker, which then converts them into sound signals for output. Conversely, the microphone converts collected sound signals into electrical signals, which are then received by audio circuit 505 and converted into audio data. The audio data is then output to processor 501 for processing, then transmitted via RF circuit 504 to, for example, another computer device, or to memory 502 for further processing. Audio circuit 505 may also include an earphone jack to allow communication between external headphones and the computer device.

[0193] The input unit 506 may be configured to receive input digital, character information, or user feature information (such as fingerprint, iris, or facial information), and to generate keyboard, mouse, joystick, optical, or trackball signal inputs related to user settings and function control.

[0194] Power supply 507 is used to supply power to various components of computer device 500. Optionally, power supply 507 can be logically connected to processor 501 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. Power supply 507 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0195] although Figure 6 Not shown in the figure, the computer device 500 may also include a camera, a sensor, a wireless fidelity module, a Bluetooth module, etc., which will not be described in detail here.

[0196] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0197] As can be seen from the above, the computer device provided in this embodiment can obtain the first perceptual feature of the facial image to be identified, and the second perceptual feature of each virtual character image in the preset image set; determine the difference information between the first perceptual feature and each second perceptual feature; based on the difference information, determine at least one target virtual character image from the preset image set that has an eye sight similar to that of the facial image to be identified; and determine the predicted eye sight information of the facial image to be identified based on the eye sight information of at least one target virtual character image.

[0198] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0199] To this end, an embodiment of the present application provides a computer-readable storage medium storing a plurality of computer programs, which can be loaded by a processor to execute the steps of any of the image processing methods provided in the embodiments of the present application. For example, the computer program can execute the following steps:

[0200] Obtaining a first perceptual feature of the face image to be recognized and a second perceptual feature of each virtual character image in the preset image set;

[0201] determining difference information between the first perceptual feature and each second perceptual feature;

[0202] Determining at least one target virtual character image having an eye sight line similar to that of the face image to be recognized from a preset image set based on the difference information;

[0203] Predicted eye sight information of a face image to be recognized is determined based on eye sight information of at least one target virtual character image.

[0204] The embodiment itself obtains the first perceptual feature of the facial image to be identified, and the second perceptual feature of each virtual character image in the preset image set, and then determines the difference information between the first perceptual feature and each second perceptual feature, and determines at least one target virtual character image from the preset image set that has an eye sight similar to the facial image to be identified based on the difference information, and then determines the predicted eye sight information of the facial image to be identified based on the eye sight information of at least one target virtual character image. Since the predicted eye sight information is determined based on the eye sight information of the virtual character image, it can be directly applied to adjust the eyes of the virtual character later. When the expression of the facial image to be identified is reused on other virtual characters, the predicted eye sight information can be directly applied to the eye adjustment of other virtual characters. In this way, the efficiency of expression reuse between facial images and virtual characters can be improved.

[0205] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0206] The storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0207] Since the computer program stored in the storage medium can execute the steps of any image processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any image processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0208] The above is a detailed introduction to an image processing method, device, storage medium and computer equipment provided in the embodiments of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, based on the ideas of the present application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Obtaining a first perceptual feature of the face image to be recognized and a second perceptual feature of each virtual character image in the preset image set; determining difference information between the first perceptual feature and each second perceptual feature; Determining at least one target virtual character image having an eye sight line similar to that of the face image to be recognized from the preset image set based on the difference information; The predicted eye sight line information of the face image to be recognized is determined based on the eye sight line information of the at least one target virtual character image.

2. The method according to claim 1, characterized in that The determining, based on the eye sight line information of the at least one target virtual character image, predicted eye sight line information of the face image to be recognized includes: Acquire eye sight information of each target virtual character image, wherein the eye sight information at least includes an eyeball rotation angle; A weighted summation process is performed based on the eyeball rotation angle of each target virtual character image to obtain a predicted eyeball rotation angle of the face image to be recognized.

3. The method according to claim 2, characterized in that The weighted summing process based on the eyeball rotation angle of each target virtual character image to obtain the predicted eyeball rotation angle of the face image to be recognized includes: determining a weight corresponding to each target virtual character image based on difference information corresponding to the second perception feature of each target virtual character image; performing weighted processing on the eyeball rotation angles of the target virtual character images according to the weights corresponding to the target virtual character images to obtain weighted eyeball rotation angles; The weighted eye rotation angles are summed to obtain the predicted eye rotation angle of the face image to be recognized.

4. The method according to claim 1, wherein The determining, from the preset image set based on the difference information, at least one target virtual character image having an eye sight line similar to that of the face image to be recognized comprises: Filtering a preset number of difference information from all difference information in descending order to obtain the preset number of target difference information; The virtual character images to which the second perception features corresponding to the target difference information belong are acquired from the preset image set to obtain the preset number of target virtual character images.

5. The method according to claim 1, wherein The determining of difference information between the first perceptual feature and each second perceptual feature includes: The distance between the first perceptual feature and each second perceptual feature is calculated to obtain the difference information.

6. The method according to claim 1, characterized in that The step of obtaining the first perceptual feature of the face image to be recognized and the second perceptual feature of each virtual character image in the preset image set includes: Inputting the face image to be recognized and each virtual character image in the preset image set into a target feature extraction model; The target feature extraction model is used to extract features of the face image to be identified and each virtual character image, respectively, to obtain the first perception feature and each second perception feature.

7. The method according to claim 6, characterized in that Before inputting the face image to be recognized and each virtual character image in the preset image set into the target feature extraction model, the method further includes: Collecting multiple sample face images and multiple sample virtual character images; Determine a triplet sample based on the multiple sample face images and the multiple sample virtual character images; Based on the triplet samples, the target feature extraction model is constructed.

8. The method according to claim 7, characterized in that The determining of triplet samples according to the plurality of sample face images and the plurality of sample virtual character images comprises: Selecting an image from the plurality of sample face images or the plurality of sample virtual character images to obtain a target sample image; Selecting an image from the multiple sample face images or the multiple sample virtual character images, which has an eye sight line similar to the target sample image, to obtain a positive sample image; Selecting an image from the plurality of sample face images or the plurality of sample virtual character images, the image having an eye sight line that is not similar to the target sample image, to obtain a negative sample image; The triplet sample is obtained based on the target sample image, the positive sample image, and the negative sample image.

9. The method according to claim 8, characterized in that The step of constructing the target feature extraction model based on the triplet sample includes: Inputting the target sample image, the positive sample image and the negative sample image in the triplet sample into a preset feature extraction model; Extracting features of the target sample image, the positive sample image, and the negative sample image respectively through the preset feature extraction model to obtain a first sample perception feature of the target sample image, a second sample perception feature of the positive sample image, and a third sample perception feature of the negative sample image; The preset feature extraction model is trained based on the first sample perception feature, the second sample perception feature, and the third sample perception feature to obtain the target feature extraction model.

10. The method according to claim 9, characterized in that The step of training the preset feature extraction model based on the first sample perception feature, the second sample perception feature, and the third sample perception feature to obtain the target feature extraction model includes: Calculating a first difference between the first sample perception feature and the second sample perception feature; calculating a second difference between the first sample perception feature and the third sample perception feature; Based on the first difference, the second difference and a preset loss function, the model parameters of the preset feature extraction model are adjusted until the preset feature extraction model converges to obtain the target feature extraction model.

11. The method according to claim 1, wherein The method further comprises: Acquire a virtual character model to be processed and facial expression information of the face image to be recognized; The facial expression of the virtual character model to be processed is adjusted according to the facial expression information, and the eye sight of the virtual character model to be processed is adjusted according to the predicted eye sight information to obtain a target virtual character model with an expression similar to that of the facial image to be recognized.

12. An image processing device, characterized in that: The device comprises: an acquisition unit, configured to acquire a first perceptual feature of the face image to be recognized and a second perceptual feature of each virtual character image in the preset image set; a first determining unit, configured to determine difference information between the first perceptual feature and each second perceptual feature; a second determining unit, configured to determine, from the preset image set based on the difference information, at least one target virtual character image having an eye sight line similar to that of the face image to be recognized; The third determining unit is configured to determine the predicted eye sight line information of the face image to be recognized based on the eye sight line information of the at least one target virtual character image.

13. A computer device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, wherein: When the processor executes the program, the image processing method according to any one of claims 1 to 11 is implemented.

14. A storage medium, characterized in that The storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the image processing method according to any one of claims 1 to 11.

Images