Image Processing Method, Apparatus, Device, Storage Medium, and Computer Program Product

Abstract

The present application discloses an image processing method, apparatus, device, storage medium, and computer program product. The method includes: obtaining a to-be-processed image of a reference object; classifying the to-be-processed image to obtain the matching probabilities of each candidate pose with the to-be-processed image; based on the matching probability of the first pose with the to-be-processed image, performing weighted fusion on the pose characterization parameters corresponding to the first pose to obtain fused pose characterization parameters, and taking the pose characterized by the fused pose characterization parameters as the pose of the reference object corresponding to the to-be-processed image, where the first pose is a candidate pose whose matching probability with the to-be-processed image satisfies a first condition. The pose characterized by the fused pose characterization parameters can be regarded as the pose obtained by fusing one or more candidate poses according to the matching probabilities. In this way, the accuracy of the pose of the reference object corresponding to the to-be-processed image determined is relatively high, and the image processing quality is relatively good.

Description

Technical Field

[0001] Embodiments of the present application relate to the field of computer technologies, and in particular, to an image processing method, apparatus, device, storage medium, and computer program product. Background Art

[0002] With the development of computer technologies, there are more and more application scenarios for image processing. One application scenario is: processing an image of a reference object to determine the pose of the reference object corresponding to the image, where the reference object is an object with a variable pose.

[0003] In related technologies, among multiple candidate poses of a reference object, a candidate pose with the highest matching degree with the image is determined, and then this candidate pose is used as the pose of the reference object corresponding to the image. The pose of the reference object corresponding to the image determined by this image processing method is limited to this one candidate pose. Since the setting of the candidate poses of the reference object is usually relatively rough, the accuracy of the pose of the reference object corresponding to the image determined based on this image processing method is relatively low, and the image processing quality is not good. Summary of the Invention

[0004] Embodiments of the present application provide an image processing method, apparatus, device, storage medium, and computer program product, which can be used to improve the accuracy of the pose of the reference object corresponding to the image determined, and further improve the image processing quality. The technical solution is as follows:

[0005] On the one hand, embodiments of the present application provide an image processing method, and the method includes:

[0006] Obtain a to-be-processed image of a reference object;

[0007] Classify the to-be-processed image to obtain the matching probabilities between each candidate pose of the reference object and the to-be-processed image;

[0008] Based on the matching probability between a first pose and the to-be-processed image, perform weighted fusion on the pose characterization parameters corresponding to the first pose to obtain a fused pose characterization parameter, and use the pose characterized by the fused pose characterization parameter as the pose of the reference object corresponding to the to-be-processed image, where the first pose is a candidate pose whose matching probability with the to-be-processed image satisfies a first condition.

[0009] On the other hand, an image processing apparatus is provided, and the apparatus includes:

[0010] A first acquisition unit, configured to obtain a to-be-processed image of a reference object;

[0011] A classification unit for classifying the to-be-processed image to obtain the matching probabilities of each candidate pose of the reference object with the to-be-processed image respectively;

[0012] A second acquisition unit for weighted fusion of the pose characterization parameters corresponding to the first pose based on the matching probability between the first pose and the to-be-processed image to obtain fused pose characterization parameters, and taking the pose characterized by the fused pose characterization parameters as the pose of the reference object corresponding to the to-be-processed image, where the first pose is a candidate pose whose matching probability with the to-be-processed image satisfies a first condition.

[0013] In a possible implementation manner, the second acquisition unit is further configured to acquire a target reference object virtual model that matches the fused pose characterization parameters, and take the pose of the target reference object virtual model as the pose characterized by the fused pose characterization parameters.

[0014] In a possible implementation manner, the second acquisition unit is further configured to generate the target reference object virtual model based on the fused pose characterization parameters.

[0015] In a possible implementation manner, the fused pose characterization parameters are used to indicate variable parameters of the reference pose characterization parameters that match a reference object virtual model, and the reference object virtual model changes as the reference pose characterization parameters are adjusted; the second acquisition unit is further configured to adjust the reference pose characterization parameters according to the fused pose characterization parameters, and take the virtual model obtained after the reference object virtual model changes as the reference pose characterization parameters are adjusted as the target reference object virtual model.

[0016] In a possible implementation manner, the classification unit is configured to acquire an image classification model, where the image classification model is trained based on sample images of the reference object and classification labels corresponding to the sample images, and the classification labels corresponding to the sample images are used to indicate the standard poses corresponding to the sample images, and the standard pose is one of the candidate poses; call the image classification model to classify the to-be-processed image to obtain the matching probabilities of each candidate pose with the to-be-processed image respectively.

[0017] In a possible implementation manner, the image classification model includes a convolutional model, an attention model, and a fully connected model connected in sequence. The classification unit is configured to call the convolutional model to perform feature extraction on the to-be-processed image to obtain first image features; call the attention model to perform feature extraction on the first image features to obtain second image features; call the fully connected model to classify the second image features to obtain the matching probabilities of each candidate pose with the to-be-processed image respectively.

[0018] In a possible implementation, the convolutional model includes at least one convolutional sub-model connected in sequence. Any convolutional sub-model includes a convolutional layer, a pooling layer, and an activation layer connected in sequence. The classification unit is configured to call the convolutional layer, the pooling layer, and the activation layer in the first convolutional sub-model in the convolutional model to perform feature extraction on the image to be processed, so as to obtain the activation features output by the first convolutional sub-model; starting from the second convolutional sub-model in the convolutional model, call the convolutional layer, the pooling layer, and the activation layer in the next convolutional sub-model to perform feature extraction on the activation features output by the previous convolutional sub-model, so as to obtain the activation features output by the next convolutional sub-model, until the activation features output by the last convolutional sub-model are obtained, and use the activation features output by the last convolutional sub-model as the first image features.

[0019] In a possible implementation, the first acquisition unit is configured to acquire an original image, where the original image includes a sub-image of the reference object; determine the area where the sub-image of the reference object is located in the original image; and acquire the image to be processed based on the area where the sub-image of the reference object is located in the original image.

[0020] In a possible implementation, the reference object is a part of the first object, and the device further includes:

[0021] An adjustment unit, configured to acquire a virtual model of the first object, where the virtual model of the first object includes a sub-model of the reference object; and use the pose of the reference object corresponding to the image to be processed to adjust the pose of the sub-model of the reference object.

[0022] In a possible implementation, the reference object is a mouth, and the device further includes:

[0023] An identification unit, configured to identify lip language content that matches the pose of the mouth corresponding to the image to be processed.

[0024] On the other hand, a computer device is provided. The computer device includes a processor and a memory. At least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the computer device implements any one of the above-mentioned image processing methods.

[0025] On the other hand, a computer-readable storage medium is further provided. At least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above-mentioned image processing methods.

[0026] On the other hand, a computer program product is also provided. The computer program product includes a computer program or computer instructions, which are loaded and executed by a processor to enable a computer to implement any one of the above-mentioned image processing methods.

[0027] The technical solutions provided by the embodiments of the present application at least bring the following beneficial effects:

[0028] In the technical solution provided by the embodiment of the present application, the posture characterized by the fused posture characterization parameter is used as the posture of the reference object corresponding to the image to be processed. Among them, the fused posture characterization parameter is obtained by fusing the posture characterization parameters corresponding to the first posture in each candidate posture according to the matching probability between the first posture and the image to be processed. The posture characterized by the fused posture characterization parameter can be regarded as the posture obtained by fusing one or more candidate postures according to the matching probability. The posture of the reference object corresponding to the image to be processed determined in this way is less affected by the rough setting of the candidate posture, has higher accuracy, and better image processing quality. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description of the embodiments. Obviously, the drawings in the following description are only some embodiments of the present application. For those of ordinary skill in the art, without creative efforts, other drawings can be obtained according to these drawings.

[0030] Figure 1 is a schematic diagram of the implementation environment of an image processing method provided by an embodiment of the present application;

[0031] Figure 2 is a flowchart of an image processing method provided by an embodiment of the present application;

[0032] Figure 3 is a schematic diagram of a candidate mouth posture provided by an embodiment of the present application;

[0033] Figure 4 is a schematic diagram of the structure of an image classification model provided by an embodiment of the present application;

[0034] Figure 5 is a schematic diagram of the process of determining the posture of the reference object corresponding to the image to be processed provided by an embodiment of the present application;

[0035] Figure 6 is a schematic diagram of an image processing device provided by an embodiment of the present application;

[0036] Figure 7 is a schematic diagram of an image processing device provided by an embodiment of the present application;

[0037] Figure 8 It is a schematic structural diagram of a server provided by an embodiment of the present application;

[0038] Figure 9 It is a schematic structural diagram of a terminal provided by an embodiment of the present application. Detailed implementation manners

[0039] To make the objectives, technical solutions, and advantages of the present application clearer, the following will further describe in detail the implementation manners of the present application with reference to the accompanying drawings.

[0040] In an exemplary embodiment, the image processing method provided by an embodiment of the present application can be applied to the field of artificial intelligence technology. Next, the artificial intelligence technology will be introduced.

[0041] Artificial Intelligence (AI) is to use a digital computer or a machine controlled by a digital computer to simulate, extend, and expand human intelligence, and is a theory, method, technology, and application system that can perceive the environment, acquire knowledge, and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science. Artificial intelligence attempts to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence also studies the design principles and implementation methods of various intelligent machines, enabling the machines to have the functions of perception, reasoning, and decision-making.

[0042] Artificial intelligence technology is an interdisciplinary subject, involving a wide range of fields, including both hardware-level technologies and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. The software technologies of artificial intelligence mainly include several major directions such as computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning, autonomous driving, and intelligent transportation. The image processing method provided by an embodiment of the present application involves computer vision technology and machine learning technology.

[0043] Computer Vision (CV) technology is a science that studies how to enable machines to "see". More specifically, it refers to using cameras and computers to replace human eyes for machine vision such as object recognition, tracking, and measurement, and further performing graphic processing to make the computer-processed images more suitable for human eyes to observe or be transmitted to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, attempting to establish artificial intelligence systems that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (Three Dimensional) technology, virtual reality, augmented reality, simultaneous localization and mapping, autonomous driving, intelligent transportation, etc. technologies, and also includes common biometric recognition technologies such as face recognition and fingerprint recognition.

[0044] Machine Learning (ML) is an interdisciplinary field that involves multiple disciplines such as probability theory, statistics, approximation theory, convex analysis, and algorithm complexity theory. It specifically studies how computers simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize the existing knowledge structure to continuously improve their own performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent, and its applications cover all fields of artificial intelligence. Machine learning and deep learning usually include technologies such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning from demonstration.

[0045] With the research and progress of artificial intelligence technology, artificial intelligence technology has been studied and applied in multiple fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, driverless, autonomous driving, drones, robots, intelligent healthcare, intelligent customer service, vehicle networking, autonomous driving, intelligent transportation, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0046] In an exemplary embodiment, the image processing method provided in the embodiments of the present application is implemented in a blockchain system. The to-be-processed image involved in the image processing method provided in the embodiments of the present application, the respective candidate poses of the reference object, the matching probabilities with the to-be-processed image, the fused pose characterization parameters, etc. are all stored on the blockchain in the blockchain system for each node device in the blockchain system to use, so as to ensure the security and reliability of the data.

[0047] Figure 1The figure shows a schematic diagram of the implementation environment of the image processing method provided by an embodiment of the present application. The implementation environment includes: a terminal 11 and a server 12.

[0048] The image processing method provided by the embodiment of the present application can be executed by the terminal 11, or can be executed by the server 12, or can also be jointly executed by the terminal 11 and the server 12. The embodiment of the present application does not limit this. For the case where the image processing method provided by the embodiment of the present application is jointly executed by the terminal 11 and the server 12, the server 12 undertakes the main computing work, and the terminal 11 undertakes the secondary computing work; or, the server 12 undertakes the secondary computing work, and the terminal 11 undertakes the main computing work; or, a distributed computing architecture is adopted between the server 12 and the terminal 11 for collaborative computing.

[0049] In a possible implementation manner, the terminal 11 can be any electronic product that can perform human-computer interaction with a user in one or more ways such as a keyboard, a touchpad, a touch screen, a remote control, voice interaction, or a handwriting device. For example, a PC (Personal Computer), a mobile phone, a smart phone, a PDA (Personal Digital Assistant), a wearable device, a PPC (Pocket PC), a tablet computer, a smart car machine, a smart TV, a smart speaker, a vehicle-mounted terminal, etc. The server 12 can be a single server, or a server cluster composed of multiple servers, or a cloud computing service center. The terminal 11 and the server 12 establish a communication connection through a wired or wireless network.

[0050] Those skilled in the art should understand that the above-mentioned terminal 11 and server 12 are only examples. Other existing or future possible terminals or servers that can be applied to the present application should also be included within the protection scope of the present application and are hereby incorporated herein by reference.

[0051] Based on the above Figure 1 shown implementation environment, the embodiment of the present application provides an image processing method. The image processing method is executed by a computer device, and the computer device can be the server 12 or the terminal 11. The embodiment of the present application does not limit this. As Figure 2 shown, the image processing method provided by the embodiment of the present application includes the following steps 201 to step 203.

[0052] In step 201, an image to be processed of a reference object is acquired.

[0053] The image to be processed of the reference object is the image that needs to be processed corresponding to the reference object. Exemplarily, the reference object is an object with variable posture. The embodiments of the present application do not limit the type of the reference object. Exemplarily, the reference object refers to the mouth, or the reference object refers to the eyes, or the reference object refers to an active limb, etc. Under different types of reference objects, the postures of the reference object are different. Exemplarily, if the reference object refers to the mouth, the postures of the reference object include but are not limited to closing the mouth, opening the mouth, etc.; if the reference object refers to the eyes, the postures of the reference object include but are not limited to opening the eyes, closing the eyes, etc.; if the reference object refers to an active limb, the postures of the reference object include but are not limited to limb extension, limb bending, etc.

[0054] In an exemplary embodiment, by collecting an image of the reference object, an original image including a sub-image of the reference object can be obtained. In an exemplary embodiment, after obtaining the original image, the original image can be stored in an image library for subsequent direct extraction and use.

[0055] In an exemplary embodiment, the image to be processed of the reference object is obtained based on the original image. Before obtaining the image to be processed, the original image needs to be obtained first. The embodiments of the present application do not limit the way of obtaining the original image. Exemplarily, the ways for a computer device to obtain the original image include but are not limited to: the computer device extracts the original image from the image library for storing the original image; an image acquisition device having a communication connection with the computer device sends the original image obtained by collecting an image of the reference object to the computer device; the computer device obtains the original image uploaded manually, etc.

[0056] After obtaining the original image, the image to be processed is obtained based on the original image. The embodiments of the present application do not limit the way of obtaining the image to be processed based on the original image. In an exemplary embodiment, the way of obtaining the image to be processed based on the original image is: directly using the original image as the image to be processed. In an exemplary embodiment, the way of obtaining the image to be processed based on the original image is: determining the area where the sub-image of the reference object is located in the original image; based on the area where the sub-image of the reference object is located in the original image, obtaining the image to be processed.

[0057] The sub-image of the reference object refers to the partial image in the original image used to present the reference object. The embodiments of the present application do not limit the method for determining the area where the sub-image of the reference object is located in the original image. Exemplarily, the method for determining the area where the sub-image of the reference object is located in the original image is: calling an image segmentation model to segment the original image, and segmenting out the area where the sub-image of the reference object in the original image is located. The image segmentation model can refer to a U-Net (U-shaped network) model, an FCN (Fully Convolutional Networks) model, a DSN (Deeply-Supervised Networks) model, etc.

[0058] Exemplarily, the method for determining the area where the sub-image of the reference object is located in the original image is: performing key point detection on the original image, and taking the area where the key points belonging to the reference object among the detected key points are located as the area where the sub-image of the reference object is located in the original image. The process of performing key point detection on the original image refers to the process of detecting key points belonging to certain specific objects in the original image, where the key points belonging to a specific object are determined according to the object characteristics of the specific object and can locate the position of the specific object. Since the reference object belongs to a specific object, key points belonging to the reference object can be detected. Exemplarily, the algorithm for performing key point detection on the original image can be: an ASM (Active Shape Model) algorithm, an AAM (Active Appearance Models) algorithm, a CPR (Cascaded Pose Regression) algorithm, etc.

[0059] Exemplarily, taking the reference object as the mouth as an example, the original image refers to a face image including the sub-image of the mouth. By performing key point detection on the face image, key points belonging to each part (such as the mouth, eyes, nose, etc.) can be detected, and then the key points belonging to the mouth are extracted from the key points belonging to each part, and the area where the key points belonging to the mouth are located is taken as the area where the sub-image of the mouth is located in the face image.

[0060] After determining the area where the sub-image of the reference object is located in the original image, a processed image is obtained based on the area where the sub-image of the reference object is located in the original image. In an exemplary embodiment, the method for obtaining a processed image based on the area where the sub-image of the reference object is located in the original image is: cropping out the image including the area where the sub-image of the reference object is located from the original image, and obtaining a processed image based on the cropped image.

[0061] In an exemplary embodiment, the image to be processed can be an image of any size, and the cropped image can be directly used as the image to be processed. In an exemplary embodiment, the image to be processed is an image of a reference size, and the cropped image can be resized (Resized) to obtain an image to be processed of the reference size. The reference size is set based on experience or flexibly adjusted according to the actual application scenario, and this embodiment of the application does not limit this. For example, the reference size is 64×64 (pixels).

[0062] The embodiments of this application do not limit the resolution of the image to be processed. A higher resolution image can more effectively analyze the pose of the reference object corresponding to the image to be processed; a lower resolution image can reduce the performance overhead of the image processing process. For example, the size of the image to be processed can be flexibly adjusted based on the actual application scenario, and different sizes of images can be used in different application scenarios.

[0063] In step 202, the image to be processed is classified to obtain the matching probability between each candidate posture of the reference object and the image to be processed.

[0064] After acquiring the image to be processed, the image is classified to obtain the matching probabilities of each candidate pose of the reference object with the image to be processed. For example, the matching probabilities of each candidate pose of the reference object with the image to be processed constitute the classification result corresponding to the image to be processed. The greater the matching probability of any candidate pose with the image to be processed, the more likely the pose of the reference object corresponding to the image to be processed is that candidate pose.

[0065] A candidate pose refers to a pose that a pre-defined reference object may have. The embodiments of the present application do not limit the method for identifying a candidate pose. For example, a candidate pose may be identified by its name; or, each candidate pose may have a number, and the candidate pose may be identified by its number; or, a virtual model of a reference object having a pose that is a candidate pose may be identified.

[0066] The types and number of candidate gestures are related to the type of reference object, and can be set based on experience or flexibly adjusted according to the application scenario. This is not limited in the embodiments of the present application. In an exemplary embodiment, taking the reference object as the mouth as an example, the candidate gestures (or candidate mouth gestures) include but are not limited to the following 12 gestures: closed mouth, open mouth, slightly open mouth, wide open mouth, round mouth, hissing mouth, pouting mouth, slightly pouting mouth, upper lip covering lower lip, lower lip covering upper lip, sticking out tongue, and pursing lips. For example, the mouth gestures can also be called lip shapes.

[0067] For example, closing the mouth refers to a posture in which the lips are closed and the mouth is relaxed, such as Figure 3as shown in (1) of [the figure]; opening the mouth means that the upper lip remains basically unchanged, the lower lip opens downward by a medium amplitude, and the amplitude of opening the mouth is greater than the width of the lower lip and less than the sum of the widths of the upper and lower lips, such as Figure 3 as shown in (2) of [the figure]; slightly opening the mouth means that the lips are slightly opened, and the gap between the lips can be seen, and the amplitude of opening the mouth is less than the width of the lower lip, such as Figure 3 as shown in (3) of [the figure]; widely opening the mouth means that the mouth is opened to a relatively large extent, and both sides are slightly opened outward, and the amplitude of opening the mouth is greater than the sum of the widths of the upper and lower lips, such as Figure 3 as shown in (4) of [the figure].

[0068] A circular mouth means that the mouth is opened into a circular shape, both sides are slightly closed towards the middle, and the upper lip is slightly turned upwards, such as Figure 3 as shown in (5) of [the figure]; compared with widely opening the mouth, the circular mouth has a tendency to close towards the middle on both sides; a hissing mouth means that the corners of the mouth are pulled outwards to both sides, slightly opened, and the length of the mouth is slightly longer than normal, such as Figure 3 as shown in (6) of [the figure]; for example, when saying "one" in Chinese, the mouth has the posture of a hissing mouth; a pouting mouth means that the corners of the mouth are pouted and close towards the middle of the lips, and there is almost no gap or a small gap in the middle of the mouth, and the upper and lower lips are turned outwards, such as Figure 3 as shown in (7) of [the figure]; a slightly pouting mouth means that the corners of the mouth are pouted and close towards the middle of the lips, the mouth is opened to a certain extent, and the upper and lower lips are turned outwards, such as Figure 3 as shown in (8) of [the figure]; compared with a pouting mouth, the degree of the upper and lower lips turning outwards of a slightly pouting mouth is smaller.

[0069] The upper lip covering the lower lip means that the upper lip bites the lower lip, showing the upper lip and hiding the lower lip, such as Figure 3 as shown in (9) of [the figure]; the lower lip covering the upper lip means that the lower lip bites the upper lip, showing the lower lip and hiding the upper lip, such as Figure 3 as shown in (10) of [the figure]; sticking out the tongue means sticking out the tongue in any direction, and the tongue is exposed outside the plane of the lips, such as Figure 3 as shown in (11) of [the figure]; pursing the lips means that the upper and lower lips are pursed towards the inside of the mouth, and the upper and lower lips are hardly visible, such as Figure 3 as shown in (12) of [the figure].

[0070] In a possible implementation manner, the implementation manner of classifying the image to be processed and obtaining the matching probabilities between each candidate posture of the reference object and the image to be processed is as follows: Detect the key points belonging to the reference object in the image to be processed, calculate the posture representation features corresponding to the image to be processed based on the key points, and use the similarity between the posture representation features and the candidate posture representation features corresponding to the candidate postures as the matching probability between the candidate postures and the image to be processed. Based on this method, the matching probabilities between each candidate posture and the image to be processed are obtained.

[0071] Exemplarily, the method for calculating the pose representation features corresponding to the image to be processed based on the key points is set according to the type of the reference object. For example, when the reference object is the mouth, the key points belonging to the reference object include the left mouth corner key point, the right mouth corner key point, the upper lip key point, and the lower lip key point. The distances between the left and right mouth corners, between the upper and lower lips, and between the highest point of the upper lip and the lowest point of the lower lip are calculated based on the key points, and the calculated information is used as the pose representation features corresponding to the image to be processed.

[0072] Exemplarily, the pose representation features corresponding to the image to be processed and the candidate pose representation features corresponding to the candidate pose are of the same type to ensure comparability. Exemplarily, the candidate pose representation features corresponding to the candidate pose correspond one-to-one with the candidate pose. When setting the candidate pose, the candidate pose representation features corresponding to the candidate reference object can also be set.

[0073] In another possible implementation manner, the method for classifying the image to be processed to obtain the matching probabilities of each candidate pose of the reference object with the image to be processed respectively is as follows: obtain an image classification model; call the image classification model to classify the image to be processed to obtain the matching probabilities of each candidate pose with the image to be processed respectively.

[0074] The image classification model is used to classify the image to be processed to obtain the matching probabilities of each candidate pose with the image to be processed respectively. The image classification model is trained based on the sample images of the reference object and the classification labels corresponding to the sample images, where the classification labels corresponding to the sample images are used to indicate the standard poses corresponding to the sample images, and the standard pose is one of the candidate poses. Therefore, the image classification model has relatively accurate classification performance, that is, it can obtain relatively accurate matching probabilities of each candidate pose with the image to be processed respectively.

[0075] The implementation process of calling the image classification model to classify the image to be processed is the internal processing process of the image classification model and is related to the structure of the image classification model. The implementation processes of calling image classification models with different structures to classify the image to be processed may be different. The embodiments of the present application do not limit the structure of the image classification model, as long as it can output the matching probabilities of each candidate pose with the image to be processed respectively. Exemplarily, the image classification model refers to a convolutional neural network model, and the classification of the image to be processed is realized through deep learning.

[0076] In a possible implementation manner, the image classification model includes a convolutional model, an attention model, and a fully connected model connected in sequence. In this case, the process of calling the image classification model to classify the image to be processed to obtain the matching probabilities of each candidate pose with the image to be processed respectively includes the following steps 1 to 3:

[0077] Step 1: Invoke a convolutional model to extract features from the image to be processed, obtaining the first image feature.

[0078] The implementation process of invoking a convolutional model to extract features from the image to be processed is the internal processing process of the convolutional model and is related to the structure of the convolutional model. After inputting the image to be processed into the convolutional model, the convolutional model will output the first image feature. This first image feature can characterize the image to be processed to a certain extent.

[0079] In a possible implementation manner, the convolutional model includes at least one convolutional sub-model connected in sequence. Then, the process of invoking the convolutional model to extract features from the image to be processed is: invoking at least one convolutional sub-model to extract features from the image to be processed. During the process of invoking at least one convolutional sub-model to extract features from the image to be processed, the input of the first convolutional sub-model is the image to be processed. Starting from the second convolutional sub-model, the input of the next convolutional sub-model is the output of the previous convolutional sub-model, and the output of the last convolutional sub-model is the first image feature. The number of convolutional sub-models included in the convolutional model can be set according to experience or flexibly adjusted according to the actual application scenario. The embodiments of the present application do not limit this. Exemplarily, the number of convolutional sub-models included in the convolutional model is 2, or the number of convolutional sub-models included in the convolutional model is 3, etc.

[0080] In an exemplary embodiment, any convolutional sub-model includes a convolutional layer, a pooling layer, and an activation layer connected in sequence. Exemplarily, the size of the convolutional kernel of the convolutional layer is set according to experience or flexibly adjusted according to the application scenario. The embodiments of the present application do not limit this. For example, the size of the convolutional kernel is 3×3. Taking the image to be processed as a two-dimensional image as an example, the convolutional layer with a convolutional kernel size of 3×3 can be denoted as Conv2d 3×3. Exemplarily, the type of the pooling layer can be a max pooling layer (Max pooling Layer, abbreviated as Maxpool), or an average pooling layer (Average pooling Layer, abbreviated as Avgpool). The embodiments of the present application do not limit this. Exemplarily, the activation function adopted by the activation layer is set according to experience or flexibly adjusted according to the actual application scenario. The embodiments of the present application do not limit this. For example, the activation function adopted by the activation layer is a ReLU (Rectified Linear Unit) function, or a Sigmoid (S-shaped) function, etc.

[0081] In an exemplary embodiment, the sizes of the convolutional kernels in the convolutional layers of different convolutional sub-models may be the same or different; the types of pooling layers in different convolutional sub-models may be the same or different; the activation functions used in the activation layers of different convolutional sub-models may be the same or different.

[0082] In an exemplary embodiment, for the case where a convolutional model includes at least one convolutional sub-model connected in sequence, and any convolutional sub-model includes a convolutional layer, a pooling layer, and an activation layer connected in sequence, the process of calling the convolutional model to perform feature extraction on the image to be processed and obtaining the first image is as follows: Call the convolutional layer, pooling layer, and activation layer in the first convolutional sub-model of the convolutional model to perform feature extraction on the image to be processed, and obtain the activation features output by the first convolutional sub-model; Starting from the second convolutional sub-model in the convolutional model, call the convolutional layer, pooling layer, and activation layer in the next convolutional sub-model to perform feature extraction on the activation features output by the previous convolutional sub-model, and obtain the activation features output by the next convolutional sub-model, until the activation features output by the last convolutional sub-model are obtained, and use the activation features output by the last convolutional sub-model as the first image features.

[0083] In an exemplary embodiment, the process principle of calling the convolutional layer, pooling layer, and activation layer in each convolutional sub-model to perform feature extraction on the input and obtaining the output features is the same. Taking the process of calling the convolutional layer, pooling layer, and activation layer in the first convolutional sub-model of the convolutional model to perform feature extraction on the image to be processed and obtaining the activation features output by the first convolutional sub-model as an example for illustration.

[0084] In an exemplary embodiment, the process of calling the convolutional layer, pooling layer, and activation layer in the first convolutional sub-model of the convolutional model to perform feature extraction on the image to be processed and obtaining the activation features output by the first convolutional sub-model is as follows: Call the convolutional layer in the first convolutional sub-model of the convolutional model to perform convolution on the image to be processed, and obtain the convolutional features output by the convolutional layer in the first convolutional sub-model; Call the pooling layer in the first convolutional sub-model to perform pooling on the convolutional features output by the convolutional layer in the first convolutional sub-model, and obtain the pooling features output by the pooling layer in the first convolutional sub-model; Call the activation layer in the first convolutional sub-model to activate the pooling features output by the pooling layer in the first convolutional sub-model, and obtain the activation features output by the first convolutional sub-model.

[0085] It should be noted that the description that any convolutional sub-model includes a convolutional layer, a pooling layer, and an activation layer connected in sequence above is only an exemplary description, and the embodiments of the present application are not limited thereto. Exemplarily, any convolutional sub-model may also include a convolutional layer, an activation layer, and a pooling layer connected in sequence.

[0086] In an exemplary embodiment, since the activation function adopted by the activation layer is usually a monotonically increasing function and does not change the maximum-minimum sorting of the input, the arrangement order of the activation layer and the pooling layer is interchangeable. Since the pooling layer downsamples the input and can effectively reduce the amount of data of the input passing through the activation layer, the method of passing through the pooling layer first and then the activation layer can reduce the computational amount. For example, if the pooling size of the pooling layer is 2×2, the method of passing through the pooling layer first and then the activation layer can reduce the computational amount by 3 / 4 when passing through the activation layer.

[0087] Step 2: Invoke the attention model to extract features from the first image feature to obtain a second image feature.

[0088] After obtaining the first image feature, input the first image feature into the attention model. The attention model extracts features from the first image feature, and the feature output after the attention model extracts features from the first image feature is used as the second image feature. The second image feature can represent the image to be processed more deeply than the first image feature.

[0089] Exemplarily, the attention model is a model designed based on the attention mechanism and can improve the attention to local features. The type of the attention model is not limited in the embodiments of the present application. Exemplarily, the attention model refers to the SPP (Spatial Pyramid Pooling) model, and this SPP model can extract feature information from multiple scales. Exemplarily, the attention model can also refer to Transformer (Transformer), etc. The embodiments of the present application do not limit this. The implementation process of invoking the attention model to extract features from the first image feature is the internal processing process of the attention model and is related to the type of the attention model. The embodiments of the present application do not limit this.

[0090] Step 3: Invoke the fully connected model to classify the second image feature to obtain the matching probabilities of each candidate pose and the image to be processed respectively.

[0091] After obtaining the second image feature, input the second image feature into the fully connected model. The fully connected model classifies the second image feature and outputs the matching probabilities of each candidate pose and the image to be processed respectively.

[0092] The implementation process of classifying the second image feature by invoking the fully-connected model is the internal processing process of the fully-connected model, which is related to the structure of the fully-connected model. The embodiments of the present application do not limit this. In an exemplary embodiment, the fully-connected model includes at least one fully-connected layer and a logistic regression layer connected in sequence. Among them, the fully-connected layer is used to extract features, and the logistic regression layer is used to output multiple matching probabilities. Exemplarily, the logistic regression layer is a softmax (an activation function) function layer. The embodiments of the present application do not limit the number of fully-connected layers included in the fully-connected model. For example, the number of fully-connected layers included in the fully-connected model is 2 or 3. In an exemplary embodiment, in at least one fully-connected layer, a dropout module may also be included. This dropout module is used during the model training process and not used during the model inference process. This dropout module is used to temporarily discard the neural network units in the fully-connected layer from the fully-connected layer with a certain probability during the training process to avoid overfitting.

[0093] Exemplarily, for the case where the fully-connected model includes at least one fully-connected layer and a logistic regression layer connected in sequence, the method of invoking the fully-connected model to classify the second image feature and obtaining the matching probabilities of each candidate pose with the image to be processed is as follows: Invoke the first fully-connected layer to process the second image feature to obtain the fully-connected feature output by the first fully-connected layer; starting from the second fully-connected layer, invoke the next fully-connected layer to process the fully-connected feature output by the previous fully-connected layer to obtain the fully-connected feature output by the next fully-connected layer until the fully-connected feature output by the last fully-connected layer is obtained; invoke the logistic regression layer to process the fully-connected feature output by the last fully-connected layer to obtain the matching probabilities of each candidate pose with the image to be processed.

[0094] In an exemplary embodiment, during the process of invoking the first fully-connected layer to process the second image feature, first Flatten the second image feature into a one-dimensional feature, and then invoke the first fully-connected layer to process the second image feature. Exemplarily, Flatten is used to one-dimensionalize the multi-dimensional input. The implementation method of Flatten is to stretch the high-dimensional array along the x-axis or y-axis to become a one-dimensional array (i.e., a one-dimensional feature).

[0095] Exemplarily, the structure of the image classification model is as Figure 4As shown, the image classification model includes three sequentially connected convolutional sub-models, an attention model (SPP), two fully connected layers (FC1 and FC2), and a logistic regression layer (softmax). Among them, the fully connected layer FC1 is connected with a dropout module used during training. Each convolutional sub-model includes a convolutional layer (Conv2d3×3) with a convolutional kernel size of 3×3, a max pooling layer (Maxpool), and an activation layer (RELU).

[0096] It should be noted that the above is only an exemplary description of obtaining the matching probabilities of each candidate pose and the image to be processed by calling the image classification model in the case where the image classification model includes a sequentially connected convolutional model, an attention model, and a fully connected model. The embodiments of the present application are not limited thereto. In the exemplary embodiment, the image classification model may not include an attention model. That is to say, the image classification model includes a sequentially connected convolutional model and a fully connected model. In this case, after calling the convolutional model to perform feature extraction on the image to be processed and obtaining the first image feature, the fully connected model can be called to classify the first image feature to obtain the matching probabilities of each candidate pose and the image to be processed. In the exemplary embodiment, the structure of the image classification model is a relatively concise structure to reduce the amount of calculation, so that the image classification model can be deployed and used on a mobile terminal.

[0097] In the exemplary embodiment, obtaining the image classification model in the embodiments of the present application may refer to extracting a pre-trained and stored image classification model, or may refer to obtaining an image classification model through real-time training. The embodiments of the present application do not limit this. In either case, an image classification model needs to be trained.

[0098] In the exemplary embodiment, the method for training the image classification model is as follows: obtaining a sample image of a reference object and a classification label corresponding to the sample image; calling an initial classification model to classify the sample image to obtain a sample classification result, where the sample classification result includes the matching probabilities of each candidate pose and the sample image; obtaining a loss function between the sample classification result and the classification label corresponding to the sample image; and using the loss function to train the initial classification model to obtain the image classification model. Among them, the classification label corresponding to the sample image is used to indicate the standard pose corresponding to the sample image, and the standard pose is one of the candidate poses.

[0099] In an exemplary embodiment, the computer device can obtain a sample image of a reference object by extracting a partial image from an image library for storing original images as the sample image; or by performing an image enhancement operation on the extracted partial image to obtain the sample image. The sample image obtained after performing the image enhancement operation can improve the generalization ability of the classification model. Exemplarily, the image enhancement operation includes, but is not limited to, image flipping, image rotation, image noise addition, etc.

[0100] In an exemplary embodiment, the number of sample images is multiple. Among the multiple sample images, there are both sample images including the front image of the reference object and sample images including the side image of the reference object, so as to improve the usage range of the classification model, so that the classification model can accurately classify various types of images to be processed.

[0101] The classification label corresponding to the sample image is used to indicate the standard pose corresponding to the sample image, and the standard pose is one of the candidate poses. Exemplarily, the standard pose corresponding to the sample image is specified manually according to experience.

[0102] Exemplarily, the loss function between the sample classification result and the classification label corresponding to the sample image is used to reflect the difference between the sample classification result and the classification label corresponding to the sample image. The embodiments of the present application do not limit the manner of obtaining the loss function between the sample classification result and the classification label corresponding to the sample image. For example, obtaining the cross-entropy loss function between the sample classification result and the classification label corresponding to the sample image, or obtaining the mean square error loss function between the sample classification result and the classification label corresponding to the sample image, etc.

[0103] The process of training the initial classification model using the loss function is an iterative process. Each time the classification model is trained using the loss function, it is judged whether the training termination condition is satisfied. If the training termination condition is not satisfied, the loss function is re-obtained and training is performed again. If the training termination condition is satisfied, the classification model obtained when the training termination condition is satisfied is used as the image classification model. Exemplarily, satisfying the training termination condition includes, but is not limited to, the number of training times reaching the number threshold, the loss function converging, the loss function being less than the loss function threshold, etc.

[0104] In an exemplary embodiment, after obtaining the image classification model, the classification performance of the image classification model is tested. The test verification result shows that the top3 accuracy rate of the image classification model (that is, the proportion that the standard pose corresponding to the test image is among the top 3 candidate poses with the highest matching probability with the test image) reaches 98.1%, and the top5 accuracy rate (that is, the proportion that the standard pose corresponding to the test image is among the top 5 candidate poses with the highest matching probability with the test image) reaches 99.7%. This shows that the classification accuracy of the image classification model is relatively high.

[0105] In step 203, based on the matching probability between the first pose and the image to be processed, the pose representation parameters corresponding to the first pose are weighted and fused to obtain fused pose representation parameters, and the pose represented by the fused pose representation parameters is used as the pose of the reference object corresponding to the image to be processed. The first pose is a candidate pose whose matching probability with the image to be processed satisfies a first condition.

[0106] After obtaining the matching probabilities between each candidate pose and the image to be processed, the candidate pose whose matching probability with the image to be processed satisfies the first condition is used as the first pose. The satisfaction of the first condition is set according to experience or flexibly adjusted according to the actual application scenario, and the embodiments of the present application do not limit this. Exemplarily, if it is considered that the matching probabilities between each candidate pose and the image to be processed all satisfy the first condition, then each candidate pose is a first pose, and the number of first poses is the same as the total number of all candidate poses.

[0107] Exemplarily, that the matching probability with the image to be processed satisfies the first condition means that the matching probability with the image to be processed is among the top K (K is an integer not less than 1) largest matching probabilities among the matching probabilities between each candidate pose and the image to be processed. In this case, the number of first poses is K. In this case, the candidate pose whose matching probability with the image to be processed satisfies the first condition refers to the candidate pose with a relatively high matching probability with the image to be processed, so as to improve the accuracy of the pose of the reference object corresponding to the image to be processed determined.

[0108] After determining the first pose, based on the first pose and the matching probability between the first pose and the image to be processed, the pose of the reference object corresponding to the image to be processed is determined. The pose of the reference object corresponding to the image to be processed refers to the pose of the reference object recognized from the image to be processed. In the embodiments of the present application, the process of determining the pose of the reference object corresponding to the image to be processed based on the first pose and the matching probability between the first pose and the image to be processed is as follows: Based on the matching probability between the first pose and the image to be processed, the pose representation parameters corresponding to the first pose are weighted and fused to obtain fused pose representation parameters, and the pose represented by the fused pose representation parameters is used as the pose of the reference object corresponding to the image to be processed.

[0109] The pose characterization parameters corresponding to the first pose are used to uniquely represent the first pose. In an exemplary embodiment, the pose characterization parameters corresponding to a pose include one or more sub-parameters, and different sub-parameters are used to define the poses of different angles of the reference object. The poses of each different angle together constitute the pose of a reference object. Exemplarily, the number of sub-parameters included in the pose characterization parameters and the angles of the poses defined by each sub-parameter are determined according to the type of the reference object, and the embodiments of the present application do not limit this. For example, if the reference object is the mouth, the pose characterization parameters corresponding to a pose include, but are not limited to, sub-parameters for defining the distance between the upper lip and the lower lip, sub-parameters for defining the distance between the left corner of the mouth and the right corner of the mouth, sub-parameters for defining the degree of upward curling of the upper lip, sub-parameters for defining the degree of downward curling of the lower lip, sub-parameters for defining the direction and degree of tongue protrusion, etc.

[0110] There is a one-to-one correspondence between the pose and the pose characterization parameters. Different poses correspond to different pose characterization parameters. For the case where the number of the first poses is multiple, it is necessary to separately obtain the pose characterization parameters corresponding to each first pose.

[0111] In an exemplary embodiment, the pose characterization parameters corresponding to the candidate poses are preset and stored corresponding to the candidate poses. After determining the first pose from the candidate poses, the pose characterization parameters corresponding to the first pose can be extracted from the storage.

[0112] After obtaining the pose characterization parameters corresponding to the first pose, the matching probability between the first pose and the image to be processed is used as the weight of the pose characterization parameters corresponding to the first pose, and then the pose characterization parameters corresponding to the first reference pose are weighted and fused. The pose characterization parameters obtained after the weighted fusion are used as the fused pose characterization parameters. The fused pose characterization parameters can uniquely represent the pose of a reference object. After obtaining the fused pose characterization parameters, the pose represented by the fused pose characterization parameters is used as the pose of the reference object corresponding to the image to be processed. Since the fused pose characterization parameters are obtained by weighted fusion of the pose characterization parameters corresponding to the first pose, the pose represented by the fused pose characterization parameters can be regarded as the fused pose of the first pose, with higher sensitivity, thereby ensuring the accuracy of the pose of the reference object corresponding to the image to be processed determined.

[0113] In a possible implementation, before using the pose represented by the fused pose representation parameter as the pose of the reference object corresponding to the image to be processed, it is necessary to first determine the pose represented by the fused pose representation parameter. The method for determining the pose represented by the fused pose representation parameter is as follows: Obtain a target reference object virtual model that matches the fused pose representation parameter, and use the pose of the target reference object virtual model as the pose represented by the fused pose representation parameter. Exemplarily, the reference object virtual model refers to a virtual three-dimensional solid model of the reference object, and the pose of the target reference object virtual model is the pose represented by the fused pose representation parameter.

[0114] In an exemplary embodiment, the fused pose representation parameter refers to the generation parameter of a reference object virtual model. In this case, the method for obtaining a target reference object virtual model that matches the fused pose representation parameter is as follows: Based on the fused pose representation parameter, generate a target reference object virtual model. Exemplarily, input the fused pose representation parameter into a virtual model generator, and the virtual model generator can generate a target reference object virtual model. The virtual model generator can be designed by designers, and the embodiments of this application do not limit this.

[0115] In an exemplary embodiment, the fused pose representation parameter is used to indicate the variable parameter of the reference pose representation parameter that matches the reference object virtual model of the benchmark. The reference object virtual model of the benchmark changes as the reference pose representation parameter is adjusted. In this case, the method for obtaining a target reference object virtual model that matches the fused pose representation parameter is as follows: Adjust the reference pose representation parameter according to the fused pose representation parameter, and use the virtual model obtained after the reference object virtual model of the benchmark changes as the target reference object virtual model as the reference pose representation parameter is adjusted.

[0116] Since the fused pose representation parameter is used to indicate the variable parameter of the reference pose representation parameter, the reference pose representation parameter can be adjusted according to the fused pose representation parameter. Since the reference object virtual model of the benchmark changes as the reference pose representation parameter is adjusted, after adjusting the reference pose representation parameter according to the fused pose representation parameter, a changed reference object virtual model can be obtained, and this changed reference object virtual model is the target reference object virtual model.

[0117] In an exemplary embodiment, the process of determining the pose of the reference object corresponding to the image to be processed is as Figure 5As shown, the original image is obtained, key point detection is performed on the original image, and after the key point detection, the image to be processed is obtained through preprocessing methods such as cropping and resizing. The image classification model is called to classify the image to be processed, and the matching probabilities of each candidate pose with the image to be processed are obtained; the first pose is determined based on the matching probabilities of each candidate pose with the image to be processed, and then the pose representation parameters corresponding to the first pose are weighted and fused according to the matching probability of the first pose with the image to be processed to obtain the fused pose representation parameters, and the pose represented by the fused pose representation parameters is used as the pose of the reference object corresponding to the image to be processed.

[0118] In a possible implementation manner, the reference object is a part of the first object. After using the pose represented by the fused pose representation parameters as the pose of the reference object corresponding to the image to be processed, it further includes: obtaining the virtual model of the first object, where the virtual model of the first object includes the sub-model of the reference object; using the pose of the reference object corresponding to the image to be processed to adjust the pose of the sub-model of the reference object. The first object refers to the object including the reference object. For example, if the reference object is the mouth, the first object refers to the human face. The virtual model of the first object is a virtual three-dimensional model of the first object. Since the first object includes the reference object, the virtual model of the first object includes the sub-model of the reference object.

[0119] Adjusting the pose of the sub-model of the reference object according to the pose of the reference object corresponding to the image to be processed can make the pose of the sub-model of the reference object be the pose of the reference object corresponding to the image to be processed. In an exemplary embodiment, the sub-model of the reference object in the virtual model of the first object has a default pose. In this case, the process of using the pose of the reference object corresponding to the image to be processed to adjust the pose of the sub-model of the reference object may refer to the process of directly driving the pose of the sub-model of the reference object to be the pose of the reference object corresponding to the image to be processed. In an exemplary embodiment, the sub-model of the reference object in the virtual model of the first object has a pose determined by other means. In this case, the process of using the pose of the reference object corresponding to the image to be processed to adjust the pose of the sub-model of the reference object may refer to the process of correcting the pose determined by other means so that the sub-model of the reference object has the pose of the reference object corresponding to the image to be processed.

[0120] In an exemplary embodiment, the reference object is the mouth. According to the posture of the mouth, some lip language content can be recognized. In this case, after taking the posture represented by the fused posture characterization parameter as the posture of the reference object corresponding to the image to be processed, the following steps are further included: recognizing the lip language content that matches the posture of the mouth corresponding to the image to be processed. In an exemplary embodiment, different lip language contents correspond to different postures of the mouth. By recognizing the posture of the mouth corresponding to the image to be processed, the lip language content that matches the posture of the mouth corresponding to the image to be processed can be analyzed.

[0121] Exemplarily, the reference object is the mouth. The process of recognizing the posture of the reference object can be regarded as a lip shape recognition process. With the continuous development of face recognition related technologies, the application scenarios are also constantly expanding. The understanding of people is no longer limited to live detection, identity authentication, etc. The local features of people are also further refined, such as the postures of the facial features, expressions, voices, lip languages, body movements, etc. Among them, lip shape recognition can assist or interpret the analysis of speech, lip language text, and expression states, etc., and has a wide range of application scenarios. At the same time, it can further promote the lip shape expression driving of virtual anchors, real-time speech / text understanding and generation, etc.

[0122] The embodiment of the present application realizes lip shape classification based on a convolutional neural network model, normalizes the output to a standardized matching probability, and outputs the matching probability of each category while specifying the category. By weighted combination of lip shape categories, various lip shape results (which can be called composite lip shape categories) are expressed, and the expression of lip shape recognition results is more abundant, which can effectively directly drive or indirectly assist in secondary correction of the lip posture of the reconstructed virtual three-dimensional image.

[0123] In an exemplary embodiment, the embodiment of the present application can be applied to virtual anchor lip shape expression driving, real-time speech / text understanding and generation, etc. In the virtual anchor lip shape driving scenario, the technology of the 3D FaceMorphable Model (3DMM) is usually adopted. This technology captures the expression of real people and performs real-time three-dimensional mesh driving on the three-dimensional model of the virtual cartoon to show the same lip shape and expression. However, this technology strongly depends on the accuracy of the face key point detection algorithm, which often leads to insensitive performance or lack of certain expressions. On the other hand, face key point detection is based on two-dimensional information and lacks three-dimensional output information, which will lead to incorrect expression results in many side face angles. After using the embodiment of the application as an assistance, the problems of insensitive expression and lack can be avoided, and at the same time, the side face can be effectively corrected.

[0124] In the image processing method provided by an embodiment of the present application, the posture represented by the fused posture representation parameter is used as the posture of the reference object corresponding to the image to be processed. The fused posture representation parameter is obtained by fusing the posture representation parameters corresponding to the first posture in each candidate posture according to the matching probability between the first posture and the image to be processed. The posture represented by the fused posture representation parameter can be regarded as the posture obtained by fusing one or more candidate postures according to the matching probability. In this way, the posture of the reference object corresponding to the image to be processed determined is less affected by the rough setting of the candidate postures, has higher accuracy, and better image processing quality.

[0125] Next, the implementation process of the image processing method provided by an embodiment of the present application in the application scenario where the reference object is the mouth is introduced. Exemplarily, when the reference object is the mouth, the image to be processed of the mouth can be referred to as a mouth image. In this case, the image processing method is as follows: Obtain a mouth image; classify the mouth image to obtain the matching probability between each candidate posture of the mouth and the mouth image; based on the matching probability between the first posture and the mouth image, perform weighted fusion on the posture representation parameters corresponding to the first posture to obtain a fused posture representation parameter, and use the posture represented by the fused posture representation parameter as the posture of the mouth corresponding to the mouth image. The first posture is a candidate posture whose matching probability with the mouth image satisfies a first condition.

[0126] See Figure 6 , an embodiment of the present application provides an image processing apparatus, and the apparatus includes:

[0127] A first acquisition unit 601, configured to acquire an image to be processed of a reference object;

[0128] A classification unit 602, configured to classify the image to be processed to obtain the matching probability between each candidate posture of the reference object and the image to be processed;

[0129] A second acquisition unit 603, configured to perform weighted fusion on the posture representation parameters corresponding to the first posture based on the matching probability between the first posture and the image to be processed to obtain a fused posture representation parameter, and use the posture represented by the fused posture representation parameter as the posture of the reference object corresponding to the image to be processed. The first posture is a candidate posture whose matching probability with the image to be processed satisfies a first condition.

[0130] In a possible implementation manner, the second acquisition unit 603 is further configured to acquire a target reference object virtual model that matches the fused posture representation parameter, and use the posture of the target reference object virtual model as the posture represented by the fused posture representation parameter.

[0131] In a possible implementation manner, the second acquisition unit 603 is further configured to generate a target reference object virtual model based on the fused posture representation parameter.

[0132] In one possible implementation, the fused posture representation parameters are used to indicate variable parameters of the baseline posture representation parameters that match the baseline reference object virtual model, and the baseline reference object virtual model changes as the baseline posture representation parameters are adjusted; the second acquisition unit 603 is also used to adjust the baseline posture representation parameters according to the fused posture representation parameters, and use the virtual model obtained after the baseline reference object virtual model changes with the adjustment of the baseline posture representation parameters as the target reference object virtual model.

[0133] In one possible implementation, the classification unit 602 is used to obtain an image classification model, which is trained based on a sample image of a reference object and a classification label corresponding to the sample image. The classification label corresponding to the sample image is used to indicate a standard posture corresponding to the sample image, and the standard posture is one of the candidate postures. The image classification model is called to classify the image to be processed to obtain the matching probability of each candidate posture with the image to be processed.

[0134] In one possible implementation, the image classification model includes a convolutional model, an attention model, and a fully connected model connected in sequence, and the classification unit 602 is used to call the convolutional model to perform feature extraction on the image to be processed to obtain a first image feature; call the attention model to perform feature extraction on the first image feature to obtain a second image feature; call the fully connected model to classify the second image feature to obtain the matching probability of each candidate posture with the image to be processed.

[0135] In one possible implementation, the convolution model includes at least one convolution sub-model connected in sequence, and any convolution sub-model includes a convolution layer, a pooling layer, and an activation layer connected in sequence. The classification unit 602 is used to call the convolution layer, the pooling layer, and the activation layer in the first convolution sub-model in the convolution model to perform feature extraction on the processed image, and obtain the activation feature output by the first convolution sub-model; starting from the second convolution sub-model in the convolution model, the convolution layer, the pooling layer, and the activation layer in the next convolution sub-model are called to perform feature extraction on the activation feature output by the previous convolution sub-model, and obtain the activation feature output by the next convolution sub-model, until the activation feature output by the last convolution sub-model is obtained, and the activation feature output by the last convolution sub-model is used as the first image feature.

[0136] In one possible implementation, the first acquisition unit 601 is used to acquire an original image, which includes a sub-image of a reference object; determine the area where the sub-image of the reference object is located in the original image; and acquire an image to be processed based on the area where the sub-image of the reference object is located in the original image.

[0137] In one possible implementation, the reference object is a part of the first object, seeFigure 7 , the apparatus further includes:

[0138] An adjustment unit 604, configured to obtain a virtual model of a first object, where the virtual model of the first object includes a sub-model of a reference object; and perform pose adjustment on the sub-model of the reference object by using the pose of the reference object corresponding to the image to be processed.

[0139] In a possible implementation manner, the reference object is a mouth. Refer to Figure 7 , the apparatus further includes:

[0140] An identification unit 605, configured to identify lip language content that matches the pose of the mouth corresponding to the image to be processed.

[0141] In the image processing apparatus provided in the embodiment of the present application, the pose characterized by the fused pose characterization parameter is used as the pose of the reference object corresponding to the image to be processed, where the fused pose characterization parameter is obtained by fusing the pose characterization parameters corresponding to the first pose in each candidate pose according to the matching probability between the first pose and the image to be processed. The pose characterized by the fused pose characterization parameter can be regarded as the pose obtained by fusing one or more candidate poses according to the matching probability. The pose of the reference object corresponding to the image to be processed determined in this way is less affected by the rough setting of the candidate pose, has higher accuracy, and better image processing quality.

[0142] It should be noted that when the apparatus provided in the above embodiment implements its functions, only the above-mentioned division of each functional unit is used for illustration. In actual applications, the above functions can be allocated to different functional units according to needs, that is, the internal structure of the device is divided into different functional units to complete all or part of the functions described above. In addition, the apparatus provided in the above embodiment and the method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be elaborated here.

[0143] In an exemplary embodiment, a computer device is further provided. The computer device includes a processor and a memory, and at least one computer program is stored in the memory. The at least one computer program is loaded and executed by one or more processors so that the computer device implements any one of the above image processing methods. The computer device may be a server or a terminal, and the embodiment of the present application does not limit this. Next, the structures of the server and the terminal will be introduced respectively.

[0144] Figure 8It is a schematic structural diagram of a server provided by an embodiment of the present application. The server may vary greatly due to different configurations or performances, and may include one or more processors (Central Processing Units, CPUs) 801 and one or more memories 802. Among them, at least one computer program is stored in the one or more memories 802, and the at least one computer program is loaded and executed by the one or more processors 801, so that the server implements the image processing methods provided by the above various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input / output interface for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0145] Figure 9 It is a schematic structural diagram of a terminal provided by an embodiment of the present application. Exemplarily, the terminal may be: a PC, a mobile phone, a smart phone, a PDA, a wearable device, a PPC, a tablet computer, a smart car machine, a smart TV, a smart speaker, a vehicle-mounted terminal, etc. The terminal may also be referred to by other names such as a user equipment, a portable terminal, a laptop terminal, a desktop terminal, etc.

[0146] Generally, the terminal includes: a processor 901 and a memory 902.

[0147] The processor 901 may include one or more processing cores. The processor 901 may also include a main processor and a coprocessor. The main processor is a processor for processing data in the wake state, also known as the CPU (Central Processing Unit, central processor); the coprocessor is a low-power processor for processing data in the standby state. In some embodiments, the processor 901 may be integrated with a GPU, and the GPU is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 901 may also include an AI (Artificial Intelligence) processor, and the AI processor is used to process computational operations related to machine learning.

[0148] The memory 902 may include one or more computer-readable storage media, and the computer-readable storage media may be non-transitory. The memory 902 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash storage devices. In some embodiments, the non-transitory computer-readable storage media in the memory 902 is used to store at least one instruction, and the at least one instruction is used to be executed by the processor 901, so that the terminal implements the image processing method provided by the method embodiments in the present application.

[0149] In some embodiments, the terminal may further optionally include: a peripheral device interface 903 and at least one peripheral device. The processor 901, the memory 902, and the peripheral device interface 903 may be connected through a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 903 through a bus, signal lines, or a circuit board. Specifically, the peripheral devices include at least one of: a radio frequency circuit 904, a display screen 905, a camera assembly 906, an audio circuit 907, a positioning component 908, and a power supply 909.

[0150] The peripheral device interface 903 can be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 901 and the memory 902. The radio frequency circuit 904 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 904 communicates with a communication network and other communication devices through electromagnetic signals. The display screen 905 is used to display a UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. The camera assembly 906 is used to capture images or videos.

[0151] The audio circuit 907 may include a microphone and a speaker. The microphone is used to collect sound waves of the user and the environment, and convert the sound waves into electrical signals for input to the processor 901 for processing, or input to the radio frequency circuit 904 to enable voice communication. The speaker is used to convert electrical signals from the processor 901 or the radio frequency circuit 904 into sound waves. The positioning component 908 is used to locate the current geographical location of the terminal to enable navigation or LBS (Location Based Service). The power supply 909 is used to supply power to each component in the terminal. The power supply 909 may be alternating current, direct current, a disposable battery, or a rechargeable battery.

[0152] In some embodiments, the terminal further includes one or more sensors 910. The one or more sensors 910 include, but are not limited to: an acceleration sensor 911, a gyroscope sensor 912, a pressure sensor 913, a fingerprint sensor 914, an optical sensor 915, and a proximity sensor 916.

[0153] The acceleration sensor 911 can detect the magnitudes of accelerations on the three coordinate axes of the coordinate system established by the terminal. The gyroscope sensor 912 can detect the body orientation and rotation angle of the terminal. The gyroscope sensor 912 can cooperate with the acceleration sensor 911 to collect the 3D actions of the user on the terminal. The pressure sensor 913 can be disposed on the side frame of the terminal and / or the lower layer of the display screen 905. When the pressure sensor 913 is disposed on the side frame of the terminal, it can detect the holding signal of the user on the terminal, and the processor 901 can perform left / right hand recognition or shortcut operations according to the holding signal collected by the pressure sensor 913. When the pressure sensor 913 is disposed on the lower layer of the display screen 905, the processor 901 can control the operable controls on the UI interface according to the pressure operation of the user on the display screen 905.

[0154] The fingerprint sensor 914 is used to collect the fingerprint of the user. The processor 901 can identify the user's identity according to the fingerprint collected by the fingerprint sensor 914, or the fingerprint sensor 914 can identify the user's identity according to the collected fingerprint. The optical sensor 915 is used to collect the ambient light intensity. The proximity sensor 916, also known as the distance sensor, is usually disposed on the front panel of the terminal. The proximity sensor 916 is used to collect the distance between the user and the front of the terminal.

[0155] Those skilled in the art can understand that Figure 9 the structure shown in does not limit the terminal, and it may include more or fewer components than shown in the figure, or combine certain components, or adopt different component arrangements.

[0156] In an exemplary embodiment, a computer-readable storage medium is further provided. At least one computer program is stored in the computer-readable storage medium. The at least one computer program is loaded and executed by a processor of a computer device to enable the computer to implement any one of the above image processing methods.

[0157] In a possible implementation manner, the above computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, and an optical data storage device, etc.

[0158] In an exemplary embodiment, a computer program product is further provided. The computer program product includes a computer program or computer instructions. The computer program or computer instructions are loaded and executed by a processor to enable the computer to implement any one of the above image processing methods.

[0159] It should be noted that the terms "first", "second", etc. in this application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. The embodiments described in the above exemplary embodiments do not represent all embodiments consistent with the present application. On the contrary, they are only examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0160] It should be understood that the term "plurality" mentioned herein refers to two or more. "And / or" describes the association relationship of associated objects and indicates that there can be three relationships. For example, A and / or B can represent three situations: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally represents an "or" relationship between the associated objects before and after.

[0161] The above are only exemplary embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that, The method includes: Obtaining a to-be-processed image of a reference object; Classifying the to-be-processed image to obtain the matching probabilities of the respective candidate poses of the reference object with the to-be-processed image; Based on the matching probability of a first pose with the to-be-processed image, performing weighted fusion on the pose characterization parameters corresponding to the first pose to obtain fused pose characterization parameters, and taking the pose characterized by the fused pose characterization parameters as the pose of the reference object corresponding to the to-be-processed image, where the first pose is a candidate pose whose matching probability with the to-be-processed image satisfies a first condition.

2. The method according to claim 1, wherein Before taking the pose characterized by the fused pose characterization parameters as the pose of the reference object corresponding to the to-be-processed image, the method further includes: Obtaining a target reference object virtual model that matches the fused pose characterization parameters, and taking the pose of the target reference object virtual model as the pose characterized by the fused pose characterization parameters.

3. The method according to claim 2, characterized in that The obtaining of the target reference object virtual model that matches the fused pose characterization parameters includes: Generating the target reference object virtual model based on the fused pose characterization parameters.

4. The method according to claim 2, wherein The fused pose characterization parameters are used to indicate variable parameters of reference pose characterization parameters that match a reference object virtual model, and the reference object virtual model changes as the reference pose characterization parameters are adjusted; The obtaining of the target reference object virtual model that matches the fused pose characterization parameters includes: Adjusting the reference pose characterization parameters according to the fused pose characterization parameters, and taking the virtual model obtained by changing the reference object virtual model as the reference pose characterization parameters are adjusted as the target reference object virtual model.

5. The method according to any one of claims 1-4, characterized in that, The classifying of the to-be-processed image to obtain the matching probabilities of the respective candidate poses of the reference object with the to-be-processed image includes: Obtaining an image classification model, where the image classification model is trained based on sample images of the reference object and classification labels corresponding to the sample images, and the classification labels corresponding to the sample images are used to indicate standard poses corresponding to the sample images, and the standard pose is one of the candidate poses; Invoking the image classification model to classify the to-be-processed image to obtain the matching probabilities of the respective candidate poses with the to-be-processed image.

6. The method according to claim 5, wherein The image classification model includes a convolutional model, an attention model, and a fully connected model that are connected in sequence. The invoking of the image classification model to classify the to-be-processed image to obtain the matching probabilities of the respective candidate poses with the to-be-processed image includes: Invoking the convolutional model to perform feature extraction on the to-be-processed image to obtain first image features; Invoking the attention model to perform feature extraction on the first image features to obtain second image features; Invoking the fully connected model to classify the second image features to obtain the matching probabilities of the respective candidate poses with the to-be-processed image.

7. The method according to claim 6, wherein The convolutional model includes at least one convolutional sub-model connected in sequence. Any convolutional sub-model includes a convolutional layer, a pooling layer, and an activation layer connected in sequence. The step of invoking the convolutional model to perform feature extraction on the image to be processed to obtain first image features includes: Invoking the convolutional layer, pooling layer, and activation layer in the first convolutional sub-model of the convolutional model to perform feature extraction on the image to be processed, and obtaining the activation features output by the first convolutional sub-model; Starting from the second convolutional sub-model in the convolutional model, invoking the convolutional layer, pooling layer, and activation layer in the next convolutional sub-model to perform feature extraction on the activation features output by the previous convolutional sub-model, and obtaining the activation features output by the next convolutional sub-model, until the activation features output by the last convolutional sub-model are obtained. The activation features output by the last convolutional sub-model are used as the first image features.

8. The method according to any one of claims 1-4, 6-7, characterized in that, The step of obtaining the image to be processed of the reference object includes: Obtaining an original image, where the original image includes a sub-image of the reference object; Determining the region where the sub-image of the reference object is located in the original image; Based on the region where the sub-image of the reference object is located in the original image, obtaining the image to be processed.

9. The method according to any one of claims 1-4, 6-7, characterized in that The reference object is a part of the first object. After using the pose represented by the fused pose characterization parameter as the pose of the reference object corresponding to the image to be processed, the method further includes: Obtaining a virtual model of the first object, where the virtual model of the first object includes a sub-model of the reference object; Adjusting the pose of the sub-model of the reference object by using the pose of the reference object corresponding to the image to be processed.

10. According to the method described in any one of claims 1-4, 6-7, characterized in that, The reference object is the mouth. After using the pose represented by the fused pose characterization parameter as the pose of the reference object corresponding to the image to be processed, the method further includes: Identifying lip language content that matches the pose of the mouth corresponding to the image to be processed.

11. An image processing apparatus, characterized in that, The device includes: A first acquisition unit, configured to acquire an image to be processed of a reference object; A classification unit, configured to classify the image to be processed, and obtain the matching probabilities between each candidate pose of the reference object and the image to be processed; A second acquisition unit, configured to perform weighted fusion on the pose characterization parameters corresponding to the first pose based on the matching probability between the first pose and the image to be processed, obtain a fused pose characterization parameter, and use the pose represented by the fused pose characterization parameter as the pose of the reference object corresponding to the image to be processed, where the first pose is a candidate pose whose matching probability with the image to be processed satisfies a first condition.

12. A computer device, characterized in that, The computer device includes a processor and a memory. At least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the computer device implements the image processing method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, At least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements the image processing method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program or computer instructions, which are loaded and executed by a processor to enable a computer to implement the image processing method according to any one of claims 1 to 10.

Images