Image processing method and device, computer device and storage medium

Abstract

This application provides an image processing method, apparatus, computer device, and storage medium, belonging to the field of artificial intelligence technology. The method includes: mapping a first image into three intermediate images, the first image including a human body, the three intermediate images being of different scales and used to represent image features of the first image; fusing the three intermediate images to obtain a second image; and mapping the second image into a target image, the target image being labeled with different parts of the human body. This solution, by mapping the first image into three images of different scales and then fusing them, reduces structural complexity, computational load, and inference time compared to the method in complex neural networks that fuse intermediate images at each scale, thus enabling deployment on mobile terminals.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to an image processing method, apparatus, computer device, and storage medium. Background Technology

[0002] Human body segmentation is a technique that segments the human body in an image or video into multiple semantically consistent regions, such as segmenting the human body into the head, hands, and legs. Currently, human body segmentation technology typically uses deep neural networks to predict pixels belonging to the same semantic region in an image, thereby achieving relatively accurate human body segmentation results.

[0003] However, the neural network structure used in the above scheme is complex, computationally intensive, and has a long inference time, which makes it difficult to deploy on mobile terminals. Summary of the Invention

[0004] This application provides an image processing method, apparatus, computer device, and storage medium. Compared to the method of fusing intermediate images at each scale in complex neural networks, it reduces structural complexity, computational load, and inference time, thereby enabling deployment on mobile terminals. The technical solution is as follows:

[0005] On the one hand, an image processing method is provided, the method comprising:

[0006] The first image is mapped to three intermediate images, the first image including a human body, and the three intermediate images are of different scales and are used to represent the image features of the first image;

[0007] The three intermediate images are fused to obtain the second image;

[0008] The second image is mapped to a target image, which is labeled with different parts of the human body.

[0009] On the other hand, an image processing apparatus is provided, the apparatus comprising:

[0010] The first mapping module is used to map a first image into three intermediate images, wherein the first image includes a human body and the three intermediate images are of different scales.

[0011] An image fusion module is used to fuse the three intermediate images to obtain a second image;

[0012] The second mapping module is used to map the second image into a target image, wherein the target image is labeled with different parts of the human body.

[0013] In some embodiments, the first mapping module is used to convolve the first image to obtain a first intermediate image; to convolve the first intermediate image to obtain a second intermediate image; and to perform channel feature enhancement and semantic feature enhancement on the second intermediate image to obtain a third intermediate image, wherein the channel feature enhancement is used to enhance the importance of different channel features, and the semantic feature enhancement is used to enhance global semantic information.

[0014] In some embodiments, the image fusion module is used to fuse the result of convolving the third intermediate image with the second intermediate image to obtain a first fused image; and to fuse the result of convolving the first fused image with the first intermediate image to obtain a second fused image, and to use the second fused image as the second image.

[0015] In some embodiments, the second mapping module is used to increase the resolution of the second image to obtain a third image, wherein the resolution of the third image is not higher than the resolution of the first image; and to perform convolution on the third image to obtain the target image.

[0016] In some embodiments, the steps performed by the image processing device are based on a human body analysis model, which is used to perform human body analysis on the input image and output an image labeled with different parts of the human body.

[0017] In some embodiments, the apparatus further includes:

[0018] The preprocessing module is used to preprocess the first labeled image of the sample image to obtain the encoded image of the sample image. The first labeled image is used to indicate different parts of the human body in the sample image, and the encoded image is used to indicate the prediction result of the previous frame image of the sample image.

[0019] The stitching module is used to stitch the sample image and the encoded image together to obtain the input image;

[0020] The training module is used to train the human body parsing model in the i-th iteration based on the input image, using the first labeled image as supervision information, where i is a positive integer.

[0021] In some embodiments, the preprocessing module is used to perform image transformation on the first labeled image of the sample image to obtain a second labeled image; and to encode the second labeled image to obtain the encoded image.

[0022] In some embodiments, the preprocessing module is used to perform at least one of rigid transformation and non-rigid transformation on the first labeled image to obtain the second labeled image.

[0023] In some embodiments, the preprocessing module is used to map the pixels in the second labeled image to a target vector space according to their pixel categories to obtain the encoded image.

[0024] In some embodiments, the stitching module is used to stitch the sample image and the encoded image together along the channel dimension to obtain the input image.

[0025] In some embodiments, the training module is configured to perform human body parsing on the input image based on the human body parsing model of the i-th iteration to obtain a predicted image, the predicted image indicating different parts of the predicted human body; determine a first loss, a second loss, and a third loss based on the first labeled image and the predicted image, the first loss indicating the difference between the first labeled image and the predicted image, the second loss indicating the difference between the first labeled image and the predicted image after pixel weighting, and the third loss indicating the difference between the first labeled image and the predicted image after adding dependency information to the pixels, the dependency information indicating the information contained in the pixels surrounding the pixel; and adjust the model parameters of the human body parsing model of the i-th iteration based on the first loss, the second loss, and the third loss.

[0026] In some embodiments, the training module is configured to determine the class weight of each pixel category based on the number of pixels corresponding to each pixel category in the predicted image, wherein the class weight is inversely correlated with the number of pixels; and determine a weighted cross-entropy loss based on the class weight of each pixel category, and use the weighted cross-entropy loss as the second loss.

[0027] In some embodiments, the training module is configured to: determine a label probability distribution based on the first labeled image; determine a prediction probability distribution based on the prediction image; determine a label probability density function, a prediction probability density function, and a joint probability density function based on the label probability distribution and the prediction probability distribution; determine a cross-entropy loss based on the label probability density function, the prediction probability density function, and the joint probability density function, and use the cross-entropy loss as the third loss.

[0028] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded and executed by the processor to implement the image processing method in the embodiments of this application.

[0029] On the other hand, a computer-readable storage medium is provided, wherein at least one computer program is stored in the computer-readable storage medium, the at least one computer program being loaded and executed by a processor to implement the image processing method as described in the embodiments of this application.

[0030] On the other hand, a computer program product or computer program is provided, which includes computer program code stored in a computer-readable storage medium, wherein a processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the image processing method in various alternative implementations of the above aspects.

[0031] This application provides an image processing scheme that maps a first image into three images of different scales and then fuses them. Compared with the method of fusing intermediate images of each scale in complex neural networks, this reduces structural complexity, computational load and inference time, thus enabling deployment on mobile terminals. Attached Figure Description

[0032] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0033] Figure 1 This is a schematic diagram of the implementation environment of an image processing method provided according to an embodiment of this application;

[0034] Figure 2 This is a flowchart of an image processing method provided according to an embodiment of this application;

[0035] Figure 3 This is a flowchart of another image processing method provided according to an embodiment of this application;

[0036] Figure 4 This is a schematic diagram of an image transformation according to an embodiment of this application;

[0037] Figure 5 This is a schematic diagram of an image stitching method provided according to an embodiment of this application;

[0038] Figure 6 This is a schematic diagram of a compression activation module provided according to an embodiment of this application;

[0039] Figure 7 This is a schematic diagram of a pyramid pooling module provided according to an embodiment of this application;

[0040] Figure 8 This is a schematic diagram of a model structure provided according to an embodiment of this application;

[0041] Figure 9 This is a comparative schematic diagram of video human body analysis results provided according to an embodiment of this application;

[0042] Figure 10 This is a block diagram of an image processing apparatus provided according to an embodiment of this application;

[0043] Figure 11 This is a block diagram of another image processing apparatus provided according to embodiments of this application;

[0044] Figure 12 This is a structural block diagram of a terminal provided according to an embodiment of this application;

[0045] Figure 13 This is a schematic diagram of the structure of a server according to an embodiment of this application. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0047] In this application, the terms "first," "second," etc., are used to distinguish identical or similar items with essentially the same function. It should be understood that there is no logical or temporal dependency between "first," "second," and "nth," nor are there any restrictions on quantity or execution order.

[0048] In this application, the term "at least one" means one or more, and "multiple" means two or more.

[0049] It is understood that the embodiments of this application involve user information, images and other related data. When the embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of related data must comply with the relevant laws, regulations and standards of the relevant countries and regions.

[0050] The following is an explanation of the terms used in this application.

[0051] Human body analysis refers to segmenting a captured human body in an image / video into multiple semantically consistent regions, such as the head, hands, and legs.

[0052] mIOU (Mean Intersection over Union) is a standard metric for semantic segmentation, representing the intersection-union ratio of two sets. In semantic segmentation, these two sets are the ground truth and the predicted segmentation.

[0053] Pixel Accuracy (Pixel Accuracy) is the ratio of the number of correctly classified pixels to the total number of pixels.

[0054] FFM (Feature Fusion Module) is used to fuse features at different scales. In many deep learning applications (such as object detection and image segmentation), fusing features at different scales is an important way to improve performance. Low-level features have higher resolution and contain more location and detail information, but because they undergo fewer convolutions, their semantics are lower and they contain more noise. High-level features have stronger semantic information, but their resolution is very low and their ability to perceive details is poor.

[0055] The image processing method provided in this application can be executed by a computer device. In some embodiments, the computer device is a terminal or a server. The following describes the implementation environment of the image processing method provided in this application, using a computer device as an example. Figure 1 This is a schematic diagram illustrating the implementation environment of an image processing method according to an embodiment of this application. See also... Figure 1 The implementation environment includes terminal 101 and server 102.

[0056] Terminal 101 and server 102 can be connected directly or indirectly via wired or wireless communication, and this application does not impose any restrictions on this.

[0057] In some embodiments, terminal 101 may be a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smartwatch, etc., but is not limited thereto. Terminal 101 has image processing applications installed and running, such as photo album applications, camera applications, and social networking applications. Those skilled in the art will understand that the number of terminals 101 may be more or less. For example, there may be one terminal, or dozens or hundreds of terminals, or even more. This application does not limit the number or type of terminals in its embodiments.

[0058] In some embodiments, server 102 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server 102 is used to provide background services for applications that support image processing. In some embodiments, server 102 undertakes the main computing work, and terminal 101 undertakes the secondary computing work; or, server 102 undertakes the secondary computing work, and terminal 101 undertakes the main computing work; or, server 102 and terminal 101 collaborate on computing using a distributed computing architecture.

[0059] In this embodiment, the terminal 101 can obtain a human body analysis model from a server. This model is used to analyze the input image and output images labeled with different parts of the human body. Then, based on the human body analysis model deployed on the terminal, a first image is input into the model, which maps the first image into three intermediate images of different scales. These three intermediate images are then fused using the model to obtain a second image. This second image is then mapped to a target image using the same model, and finally, the terminal displays the target image output by the human body analysis model.

[0060] Figure 2 This is a flowchart of an image processing method provided according to an embodiment of this application, such as... Figure 2 As shown in the illustration, this application embodiment uses execution by a terminal as an example. The method includes the following steps:

[0061] 201. The terminal maps the first image into three intermediate images. The first image includes a human body. The three intermediate images have different scales and are used to represent the image features of the first image.

[0062] In this embodiment of the application, the terminal is Figure 1 Terminal 101 in the system. The first image is the image to be processed. The terminal can perform convolution processing on the first image based on multiple convolutional layers, mapping the first image into three intermediate images of different scales. In other words, the three intermediate images have the same source, but the convolution processing methods are different, resulting in the three intermediate images having different scales. These three intermediate images are feature images of the first image, capable of representing the image features of the first image.

[0063] 202. The terminal fuses the three intermediate images to obtain the second image.

[0064] In this embodiment of the application, the terminal can sequentially fuse the above three intermediate images to obtain a fused second image. The second image contains features of the above three intermediate images at different scales and can be used to improve the performance of human body analysis.

[0065] 203. The terminal maps the second image to a target image, which is labeled with different parts of the human body.

[0066] In this embodiment, the terminal can perform convolution processing on the second image based on multiple convolutional layers, and map the second image to the target image, thereby realizing human body analysis of the human body in the first image.

[0067] This application provides an image processing scheme that maps a first image into three images of different scales and then fuses them. Compared with the method of fusing intermediate images of each scale in complex neural networks, this reduces structural complexity, computational load and inference time, thus enabling deployment on mobile terminals.

[0068] Figure 2 The main flow of the image processing scheme provided in this application embodiment is illustrated by way of example. The following description further illustrates this image processing scheme based on an application scenario. In this application scenario, the image processing scheme is implemented based on a human body analysis model. The following embodiments will describe the process of training the human body analysis model and using the human body analysis model. Figure 3 This is a flowchart of another image processing method provided according to an embodiment of this application, see [link / reference]. Figure 3 As shown in the embodiment of this application, the method is described using a terminal, specifically a mobile terminal. The image processing method includes the following steps:

[0069] 301. The terminal preprocesses the first labeled image of the sample image to obtain the encoded image of the sample image. The first labeled image is used to indicate different parts of the human body in the sample image, and the encoded image is used to indicate the prediction result of the previous frame image of the sample image.

[0070] In this embodiment, the sample image includes a sample human body. This sample image can be an image from a publicly available sample dataset or an image uploaded by a user after obtaining full user authorization. This embodiment does not restrict the source of the sample image. In the first labeled image of the sample image, different parts of the sample human body, such as hair, face, torso, and arms, are labeled with different colors. The terminal can preprocess the first labeled image so that the preprocessed coded image can indicate the prediction result of the previous frame image. It should be noted that the coded image can simulate the prediction result of the previous frame image, but it is not a prediction result obtained by parsing the human body in the previous frame image.

[0071] In some embodiments, the terminal can preprocess the first labeled image using image transformation and encoding. The step of preprocessing the first labeled image of the sample image to obtain an encoded image of the sample image includes: the terminal performing an image transformation on the first labeled image of the sample image to obtain a second labeled image, and then encoding the second labeled image to obtain an encoded image. By performing an image transformation on the first labeled image to obtain the second labeled image, changes caused by human movement can be simulated. Encoding the second labeled image allows the encoded image to be used as a prediction result of the previous frame of the sample image and learned by the model. This enables the model to learn the correlation between the prediction results of adjacent image frames, improving the stability of the model output.

[0072] In some embodiments, the terminal can perform a rigid transformation on the first labeled image to obtain a second labeled image, wherein the rigid transformation includes translation, rotation, and scaling. Alternatively, the terminal can perform a non-rigid transformation on the first labeled image to obtain a second labeled image, wherein the non-rigid transformation includes mesh distortion, optical distortion, and elastic transformation. Alternatively, the terminal can perform both rigid and non-rigid transformations on the first labeled image to obtain a second labeled image, wherein the order of the rigid and non-rigid transformations is not limited. By performing at least one of the rigid and non-rigid transformations to different degrees on the first labeled image, image changes caused by human motion can be simulated, thereby enabling the model to learn the correlation between the prediction results of adjacent image frames and improving the stability of the model output.

[0073] For example, Figure 4 This is a schematic diagram of an image transformation according to an embodiment of this application. See also... Figure 4 As shown, the first labeled image has a black background, and the human body's hair, face, torso skin (neck area), upper clothing, arms, and hands are labeled in different colors other than black. After performing rigid and non-rigid transformations on the first labeled image, a second labeled image is obtained. The colors in the second labeled image are the same as those in the first labeled image, but the size and shape are different.

[0074] In some embodiments, the terminal can map the pixels in the second labeled image to a target vector space according to their pixel categories to obtain an encoded image. The target vector space can be a one-dimensional space or a multi-dimensional space. By encoding the second labeled image, the encoded image can simulate the prediction result of the previous frame, thereby simulating image changes caused by human motion. This allows the model to learn the correlation between the prediction results of adjacent image frames, improving the stability of the model output.

[0075] For example, the pixels in the labeled image have 15 pixel categories, and the values ​​for each category are represented by 15 positive integers from 0 to 14: {0: background; 1: hat; 2: hair; 3: face; 4: sunglasses; 5: torso skin; 6: top; 7: dress; 8: pants; 9: skirt; 10: arm; 11: hand; 12: leg; 13: foot; 14: socks}. The terminal encodes the pixels according to their respective categories.

[0076] In some embodiments, the terminal employs a normalization strategy to map pixels in the second labeled image to a range of 0-1 according to their pixel category, based on formula (1). Formula (1) is shown below:

[0077] E i =y i / (C-1) (1);

[0078] Among them, E i y represents the encoded value of the i-th pixel. i represents the value of the pixel category to which the i-th pixel belongs, and C represents the total number of pixel categories.

[0079] In some embodiments, the terminal maps the pixels in the second labeled image to high-dimensional vectors according to their pixel categories based on formula (2), thereby obtaining multi-channel encoding results. Formula (2) is shown below:

[0080] f(y i )=[sin(2 0 πy i ), cos(2 0 πy i ), ..., sin(2 L-1 πy i ), cos(2 L-1 πy i (2);

[0081] Among them, y i f(y) represents the value of the pixel category to which the i-th pixel belongs. i ) represents the high-dimensional vector encoded from the i-th pixel, and L represents f(y) i Half the number of dimensions.

[0082] 302. The terminal concatenates the sample image with the encoded image to obtain the input image.

[0083] In this embodiment, the terminal can concatenate the sample image and the encoded image along the channel dimension to obtain the input image. By concatenating the sample image and the encoded image, the model can learn the correlation between the prediction results of adjacent image frames, improving the stability of the model output. See also Figure 5 As shown, Figure 5 This is a schematic diagram of an image stitching method provided according to an embodiment of this application.

[0084] 303. The terminal uses the first labeled image as supervision information and trains the human body parsing model for the i-th iteration based on the input image, where i is a positive integer.

[0085] In this embodiment, the terminal can employ supervised learning, using the first labeled image as the supervised learning source, to perform multiple rounds of iterative training to obtain a human body analytical model. The following explanation uses the i-th iteration as an example: when the i-th iteration is the first iteration, the human body analytical model obtained in this iteration is the initial model; when the i-th iteration is not the first iteration, the human body analytical model obtained in this iteration is the human body analytical model whose parameters have been adjusted after the (i-1)-th iteration.

[0086] In some embodiments, the step of the terminal training the human body parsing model for the i-th iteration based on the input image using the first labeled image as supervision information includes steps 3031 to 3033.

[0087] 3031. The terminal performs human body analysis on the input image based on the human body analysis model of the i-th iteration to obtain the predicted image.

[0088] The terminal inputs the input image into the human body analysis model of the i-th iteration, and then performs human body analysis on the input image based on the human body analysis model, and outputs a predicted image, which is used to indicate different parts of the predicted human body sample.

[0089] 3032. The terminal determines the first loss, the second loss, and the third loss based on the first labeled image and the predicted image.

[0090] In this embodiment, the first loss is the cross-entropy loss, which is used to indicate the difference between the first labeled image and the predicted image. The calculation method of the first loss is shown in the following formula (3):

[0091]

[0092] Among them, L ce Let y represent the first loss, C represent the total number of pixel categories, and y represent the total number of pixel categories. c p represents the value of pixel category c. c Indicates that the predicted pixel belongs to y c The probability of.

[0093] In this embodiment, the second loss is a weighted cross-entropy loss, which is used to indicate the difference between the first labeled image and the predicted image after pixel weighting. The process by which the terminal determines the second loss based on the first labeled image and the predicted image includes: the terminal determining the class weight of each pixel class based on the number of pixels corresponding to each pixel class in the predicted image; the class weight is inversely correlated with the number of pixels. The terminal determining the weighted cross-entropy loss based on the class weight of each pixel class, and using the weighted cross-entropy loss as the second loss. This second loss is calculated using formula (4), which is an improvement on formula (3). Formula (4) uses the proportion of the number of pixels for inverse weighting, meaning that the class weight of a pixel class with a larger number of pixels is lower, allowing the model to focus on pixel classes with smaller regions. The calculation method of the second loss is shown in formulas (4) and (5) below:

[0094]

[0095] w c =(NN) c ) / N (5);

[0096] Among them, L w This represents the second loss, where C represents the total number of pixel categories, and w c Represents the category weight, y c The value p represents the pixel category. c Indicates that the predicted pixel belongs to y c The probability, where N represents the total number of pixels in the image. c This indicates the number of pixels corresponding to pixel category c.

[0097] In this embodiment, the third loss is a mutual information loss. This third loss is used to indicate the difference between the first labeled image and the predicted image after adding dependency information to pixels. The dependency information is used to indicate the information contained in the pixels surrounding the first labeled image. The process by which the terminal determines the third loss based on the first labeled image and the predicted image includes: the terminal determining the labeled probability distribution based on the first labeled image; the terminal determining the predicted probability distribution based on the predicted image; then, the terminal determining the labeled probability density function, the predicted probability density function, and the joint probability density function based on the labeled probability density function and the predicted probability distribution; finally, the terminal determining the cross-entropy loss based on the labeled probability density function, the predicted probability density function, and the joint probability density function, and using this cross-entropy loss as the third loss. This third loss is calculated based on the mutual information loss function. Unlike the cross-entropy loss function, which is calculated pixel-by-pixel, the mutual information loss function originates from a regional mutual information strategy. The basis of the regional mutual information strategy is: if the pixel category of a pixel is "shirt," then the pixel category of the pixels surrounding that pixel is also very likely to be "shirt." Based on the user information strategy in this region, the terminal can represent a pixel using pixels and its surrounding pixels, encode the pixel, and thus represent an image as a distribution of multiple high-dimensional points. By calculating the distance between the labeled probability distribution and the predicted probability distribution, the prediction results output by the model can have better high-order consistency. The calculation method of the third loss is shown in the following formula (6):

[0098]

[0099] Among them, L mu Let f(y, p) represent the third loss, Y represent the labeled probability distribution, y represent the labeled probability, P represent the predicted probability distribution, p represent the predicted probability, f(y, p) represent the joint probability density function, f(y) represent the labeled probability density function, and f(p) represent the predicted probability density function.

[0100] 3033. The terminal adjusts the model parameters of the human body analysis model in the i-th iteration based on the first loss, the second loss, and the third loss.

[0101] In this embodiment, the terminal can use the sum of the first loss, the second loss, and the third loss as the training loss, and adjust the model parameters of the human body parsing model in the i-th iteration based on this training loss. By adjusting the model parameters based on multiple losses during model training, the problem of pixel class imbalance can be solved, thereby improving the accuracy of human body parsing.

[0102] The training loss is calculated as shown in the following formula (7):

[0103] L = L ce +L W +L mu (7);

[0104] Where L represents the training loss, L ce Indicates the first loss, L W Indicates the second loss, L mu This represents the third loss. Formula (7) is the objective function for model training.

[0105] 304. The terminal maps the first image into three intermediate images based on the human body analysis model. The first image includes a human body, and the three intermediate images have different scales to represent the image features of the first image.

[0106] In this embodiment, the human body analysis model is a model trained by the terminal, and the first image is the input image of the human body analysis model. The terminal inputs the first image into the human body analysis model, which then maps the first image into three intermediate images of different scales based on multiple convolutional layers. It should be noted that the human body analysis model can also be a model trained by the server, and the terminal obtains the model from the server.

[0107] In some embodiments, the step of mapping a first image to three intermediate images by the terminal includes: first, the terminal convolves the first image to obtain a first intermediate image; then, the terminal convolves the first intermediate image to obtain a second intermediate image. Finally, the terminal performs channel feature enhancement and semantic feature enhancement on the second intermediate image to obtain a third intermediate image. The channel feature enhancement is used to emphasize the importance of different channel features, and the semantic feature enhancement is used to enhance global semantic information. By performing channel feature enhancement and semantic feature enhancement, the performance of the model can be improved without affecting the model's processing speed.

[0108] Specifically, the terminal can enhance the importance of features in different channels based on the squeeze-and-excitation (SE) module in the convolutional layer. This squeeze-and-excitation module can be placed after the output of any network layer. Figure 6 This is a schematic diagram of a compression activation module according to an embodiment of this application. See also... Figure 6 As shown, input X0 is processed by the compression activation module to obtain X1.

[0109] Specifically, the terminal can enhance global semantic information based on the pyramid pooling module (PPM) in the convolutional layer. The pyramid pooling module can be placed after the output of any network layer. Figure 7 This is a schematic diagram of a pyramid pooling module according to an embodiment of this application. See also... Figure 7As shown, the input feature map is first pooled to obtain four pooling results of different sizes. Then, 1×1 convolution is performed on each pooling result to reduce the feature channels to 1 / 4 of the original. Then, bilinear interpolation upsampling is performed on each feature map obtained in the previous step to obtain four feature maps with the same size as the original feature map. The original feature map and the four feature maps are concatenated to obtain the output of the pyramid pooling module.

[0110] For example, Figure 8 This is a schematic diagram of a model structure provided according to an embodiment of this application. See also... Figure 8As shown, the terminal performs convolution on the first image based on convolutional layers A1 and A2 to obtain a first intermediate image. Convolutional layer A1 includes a Standard Convolution (Conv2D) module with an input size of 256*256*4, 8 channels, a stride of 2, and a kernel width of 3. Convolutional layer A2 includes a Depthwise Separable Convolution (DSConv) module with an input size of 128*128*8, 16 channels, a stride of 2, and a kernel width of 3. Then, the terminal performs convolution on the first intermediate image based on convolutional layer A3 to obtain a second intermediate image. Convolutional layer A3 includes a Depthwise Separable Convolution module with an input size of 64*64*16, 32 channels, a stride of 2, and a kernel width of 3. Then, the terminal sequentially performs channel feature enhancement and semantic feature enhancement on the second intermediate image based on convolutional layers B and C to obtain the third intermediate image. Convolutional layer B consists of three cascaded convolutional modules. The first convolutional module comprises inverted residual bottleneck blocks (bottleneck) and a compressed activation module. The original bottleneck design is: Conv2D + DWConv (Depthwise Convolution) + Conv2D. The first convolutional module adds an activation element (SE) to the bottleneck, resulting in the structure: Conv2D + DWConv + SE + Conv2D. The input size of the first convolutional module in convolutional layer B is 32*32*32, the number of channels is 32, the stride is 2, and the kernel width is 3. The structures of the second and third convolutional modules in convolutional layer B are the same as those of the first convolutional model and will not be described further. The second convolutional module in convolutional layer B has an input size of 16*16*32, 64 channels, a stride of 2, and a kernel width of 3. The third convolutional module in convolutional layer B1 has an input size of 8*8*64, 128 channels, a stride of 1, and a kernel width of 3. Convolutional layer C includes a pyramid pooling module and a standard convolutional module; its input size is 8*8*128, and it has 64 channels.

[0111] 305. The terminal fuses the three intermediate images based on the human body analysis model to obtain the second image.

[0112] In this embodiment, the terminal can fuse the result of convolving the third intermediate image with the second intermediate image to obtain a first fused image, and then fuse the result of convolving the first fused image with the first intermediate image to obtain a second fused image, which is then used as the second image. By fusing images of three different scales sequentially, compared to fusing intermediate images of each scale in complex neural networks, the structural complexity, computational load, and inference time are reduced.

[0113] For example, see still Figure 8 As shown, the terminal performs convolution on the second intermediate image based on convolutional layer D1, and then fuses the output of convolutional layer D1 with the third intermediate image based on fusion layer F1 to obtain the first fused image. Convolutional layer D1 includes a depthwise convolution module with an input size of 32*32*64, 32 channels, a stride of 1, and a kernel width of 1. Fusion layer F1 includes a feature fusion module (FFM) with an input size of 32*32*32, 32 channels, a stride of 1, and a kernel width of 1. The terminal performs convolution on the first intermediate image based on convolutional layer D2, and then fuses the output of convolutional layer D2 with the output of fusion layer F1 based on fusion layer F2 to obtain a second fused image; alternatively, the terminal performs convolution on the output of fusion layer F1 based on another convolutional layer, and then fuses the output of convolutional layer D2 with the output of the other convolutional layer based on fusion layer F2 to obtain a second fused image. Convolutional layer D2 includes a standard convolutional module with an input size of 64*64*32, 32 channels, a stride of 1, and a kernel width of 3. Fusion layer F2 includes a feature fusion module with an input size of 64*64*32, 32 channels, a stride of 1, and a kernel width of 1.

[0114] 306. The terminal maps the second image to a target image based on the human body analysis model. The target image is labeled with different parts of the human body.

[0115] In this embodiment, the terminal can perform convolution processing on the second image based on the convolution layer to increase the resolution of the second image and reduce edge jitter.

[0116] In some embodiments, the terminal can increase the resolution of the second image to obtain a third image, the resolution of which is not higher than that of the first image, and then perform convolution on the third image to obtain the target image.

[0117] For example, see still Figure 8As shown, the terminal performs convolution processing on the second image based on convolutional layer G1 to increase the resolution of the second image, resulting in a third image. The resolution of the third image is half that of the first image. Then, it performs convolution processing on the third image based on convolutional layer G2 to obtain the target image. Convolutional layer G1 includes a depthwise separable convolution module with an input size of 64*64*32, 16 channels, a stride of 1, and a kernel width of 3. Convolutional layer G2 also includes a depthwise separable convolution module with an input size of 128*128*32, 15 channels, a stride of 1, and a kernel width of 3.

[0118] The solution provided in this application reduces structural complexity, computational load, and inference time by mapping the first image into three images of different scales and then fusing them. Compared to the approach in complex neural networks that fuses intermediate images at each scale, this reduces structural complexity, computational load, and inference time, enabling deployment on mobile terminals. Furthermore, by introducing a previous frame masking strategy during model training—that is, transforming and encoding the labeled image to simulate the labeled image of the previous frame—the human body analysis model significantly improves the accuracy and stability of prediction results when performing human body analysis on videos. Additionally, by adjusting model parameters based on multivariate losses during model training, the problem of pixel class imbalance can be addressed, thereby improving the accuracy of human body analysis.

[0119] It should be noted that, in order to verify the effectiveness of the human body parsing model trained in the embodiments of this application, quantitative and qualitative evaluations were also performed on the human body parsing model. The model used for comparison was DFANet (Deep Feature Aggregation for Real-Time Semantic Segmentation, a lightweight network), and the comparison metrics were time consumption and the two quantitative evaluation metrics of average intersection-over-union ratio and pixel accuracy. The test terminal was a low-end mobile phone equipped with Qualcomm 660, and the test data consisted of 5000 test images. The test results are shown in Table 1.

[0120] Table 1

[0121] time consuming Average Intersection over Union (IoU) / Pixel Accuracy DFANet 55ms 57.2 / 90.5 This application proposal 45ms 61.3 / 91.5

[0122] As shown in Table 1, the proposed solution reduces the processing time by 10ms compared to DFANet, and also improves both the average intersection-to-union ratio and pixel accuracy in the two quantitative evaluation metrics.

[0123] In addition, for human body analysis in video scenes, by randomly selecting videos for testing, it can be found that the prediction results output by the model after using the previous frame masking strategy are more accurate and more stable. Figure 9This is a comparative schematic diagram of video human body analysis results provided according to an embodiment of this application. See also... Figure 9 As shown, Figure 9 (1) and (3) in the figure are the prediction results without using the previous frame masking strategy. Figure 9 (2) and (4) in the table are the prediction results using the previous masking strategy.

[0124] Furthermore, the scheme provided in this application uses three types of losses for model training. To verify the advantages and disadvantages of different combinations of loss functions, quantitative evaluations of the loss functions were conducted on the aforementioned test data. The evaluation metrics were the average intersection-over-union ratio (AUC) and pixel accuracy. The combinations of loss functions are: first loss, first loss + second loss, and first loss + second loss + third loss. The meanings and calculation methods of the first loss, second loss, and third loss are described in step 303 above and will not be repeated here. The evaluation results are shown in Table 2.

[0125] Table 2

[0126]

[0127] As shown in Table 2, adding a second and third loss to the first loss can improve the performance of the model.

[0128] Figure 10 This is a block diagram of an image processing apparatus according to an embodiment of this application. The apparatus is used to perform the steps in the image processing method described above. See also... Figure 10 The device includes: a first mapping module 1001, an image fusion module 1002, and a second mapping module 1003.

[0129] The first mapping module 1001 is used to map a first image into three intermediate images, wherein the first image includes a human body and the three intermediate images are of different scales.

[0130] Image fusion module 1002 is used to fuse the three intermediate images to obtain a second image;

[0131] The second mapping module 1003 is used to map the second image into a target image, wherein the target image is labeled with different parts of the human body.

[0132] In some embodiments, the first mapping module 1001 is used to convolve the first image to obtain a first intermediate image; to convolve the first intermediate image to obtain a second intermediate image; and to perform channel feature enhancement and semantic feature enhancement on the second intermediate image to obtain a third intermediate image. The channel feature enhancement is used to enhance the importance of different channel features, and the semantic feature enhancement is used to enhance global semantic information.

[0133] In some embodiments, the image fusion module 1002 is used to fuse the result of convolving the third intermediate image with the second intermediate image to obtain a first fused image; to fuse the result of convolving the first fused image with the first intermediate image to obtain a second fused image, and to use the second fused image as the second image.

[0134] In some embodiments, the second mapping module 1002 is used to increase the resolution of the second image to obtain a third image, the resolution of the third image being no higher than the resolution of the first image; and to perform convolution on the third image to obtain the target image.

[0135] In some embodiments, the steps performed by the image processing device are based on a human body analysis model, which is used to perform human body analysis on the input image and output an image labeled with different parts of the human body.

[0136] In some embodiments, Figure 11 This is a block diagram of another image processing apparatus provided according to an embodiment of this application. See also: Figure 11 As shown, the image processing apparatus also includes:

[0137] The preprocessing module 1004 is used to preprocess the first labeled image of the sample image to obtain the encoded image of the sample image. The first labeled image is used to indicate different parts of the human body in the sample image, and the encoded image is used to indicate the prediction result of the previous frame image of the sample image.

[0138] The stitching module 1005 is used to stitch the sample image and the coded image together to obtain the input image;

[0139] Training module 1006 is used to train the human body parsing model in the i-th iteration based on the input image, using the first labeled image as supervision information, where i is a positive integer.

[0140] In some embodiments, the preprocessing module 1004 is used to perform image transformation on the first labeled image of the sample image to obtain a second labeled image; and to encode the second labeled image to obtain the encoded image.

[0141] In some embodiments, the preprocessing module 1004 is used to perform at least one of rigid transformation and non-rigid transformation on the first labeled image to obtain the second labeled image.

[0142] In some embodiments, the preprocessing module 1004 is used to map the pixels in the second labeled image to the target vector space according to their respective pixel categories to obtain the encoded image.

[0143] In some embodiments, the stitching module 1005 is used to stitch the sample image and the encoded image in the channel dimension to obtain the input image.

[0144] In some embodiments, the training module 1006 is configured to perform human body parsing on the input image based on the human body parsing model of the i-th iteration to obtain a predicted image, the predicted image being used to indicate different parts of the predicted human body in the sample; based on the first labeled image and the predicted image, determine a first loss, a second loss, and a third loss, the first loss being used to indicate the difference between the first labeled image and the predicted image, the second loss being used to indicate the difference between the first labeled image and the predicted image after pixel weighting, and the third loss being used to indicate the difference between the first labeled image and the predicted image after adding dependency information to the pixels, the dependency information being used to indicate the information contained in the pixels surrounding the pixel; and adjust the model parameters of the human body parsing model of the i-th iteration based on the first loss, the second loss, and the third loss.

[0145] In some embodiments, the training module 1006 is configured to determine the class weight of each pixel class based on the number of pixels corresponding to each pixel class in the predicted image, wherein the class weight is inversely correlated with the number of pixels; and to determine a weighted cross-entropy loss based on the class weight of each pixel class, and to use the weighted cross-entropy loss as the second loss.

[0146] In some embodiments, the training module 1006 is configured to determine a label probability distribution based on the first labeled image; determine a prediction probability distribution based on the prediction image; determine a label probability density function, a prediction probability density function, and a joint probability density function based on the label probability distribution and the prediction probability distribution; determine a cross-entropy loss based on the label probability density function, the prediction probability density function, and the joint probability density function, and use the cross-entropy loss as the third loss.

[0147] The device provided in this application reduces structural complexity, computational load, and inference time by mapping a first image into three images of different scales and then fusing them. Compared with the method of fusing intermediate images of each scale in complex neural networks, this reduces structural complexity, computational load, and inference time, thus enabling deployment on mobile terminals.

[0148] It should be noted that the image processing apparatus provided in the above embodiments is only illustrated by the division of the above functional modules. In practical applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the apparatus can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus and the image processing method embodiments provided in the above embodiments belong to the same concept, and their specific implementation process can be found in the method embodiments, which will not be repeated here.

[0149] In this application embodiment, the computer device can be configured as a terminal or a server. When the computer device is configured as a terminal, the terminal acts as the execution subject to implement the technical solution provided in this application embodiment. When the computer device is configured as a server, the server acts as the execution subject to implement the technical solution provided in this application embodiment. Alternatively, the technical solution provided in this application can be implemented through interaction between the terminal and the server. This application embodiment does not limit the implementation of the technical solution in this way.

[0150] When a computer device is configured as a terminal, Figure 12 This is a structural block diagram of a terminal 1200 provided according to an embodiment of this application. The terminal 1200 can be a portable mobile terminal, such as a smartphone, tablet computer, MP3 player (Moving Picture Experts Group Audio Layer III), MP4 player (Moving Picture Experts Group Audio Layer IV), laptop computer, or desktop computer. The terminal 1200 may also be referred to as user equipment, portable terminal, laptop terminal, desktop terminal, or other names.

[0151] Typically, terminal 1200 includes a processor 1201 and a memory 1202.

[0152] Processor 1201 may include one or more processing cores, such as a quad-core processor, an octa-core processor, etc. Processor 1201 may be implemented using at least one hardware form selected from DSP (Digital Signal Processing), FPGA (Field-Programmable Gate Array), and PLA (Programmable Logic Array). Processor 1201 may also include a main processor and a coprocessor. The main processor, also known as a CPU (Central Processing Unit), is used to process data in the wake-up state; the coprocessor is a low-power processor used to process data in the standby state. In some embodiments, processor 1201 may integrate a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the screen. In some embodiments, processor 1201 may also include an AI (Artificial Intelligence) processor, which is used to handle computational operations related to machine learning.

[0153] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include high-speed random access memory and non-volatile memory, such as one or more disk storage devices or flash memory devices. In some embodiments, the non-transitory computer-readable storage media in the memory 1202 are used to store at least one computer program, which is executed by the processor 1201 to implement the image processing method provided in the method embodiments of this application.

[0154] In some embodiments, the terminal 1200 may also optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, memory 1202, and peripheral device interface 1203 can be connected via a bus or signal line. Each peripheral device can be connected to the peripheral device interface 1203 via a bus, signal line, or circuit board. Specifically, the peripheral device includes at least one of the following: a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, a positioning assembly 1208, and a power supply 1209.

[0155] Peripheral device interface 1203 can be used to connect at least one I / O (Input / Output) related peripheral device to processor 1201 and memory 1202. In some embodiments, processor 1201, memory 1202 and peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of processor 1201, memory 1202 and peripheral device interface 1203 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0156] The radio frequency (RF) circuit 1204 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The RF circuit 1204 communicates with communication networks and other communication devices via electromagnetic signals. The RF circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals back into electrical signals. In some embodiments, the RF circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The RF circuit 1204 can communicate with other terminals via at least one wireless communication protocol. This wireless communication protocol includes, but is not limited to: the World Wide Web, metropolitan area networks, intranets, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or WiFi (Wireless Fidelity) networks. In some embodiments, the RF circuit 1204 may also include circuitry related to NFC (Near Field Communication), which is not limited in this application.

[0157] Display screen 1205 is used to display a UI (User Interface). This UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1205 is a touch display screen, it also has the ability to collect touch signals on or above its surface. These touch signals can be input as control signals to processor 1201 for processing. In this case, display screen 1205 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, there may be one display screen 1205, disposed on the front panel of terminal 1200; in other embodiments, there may be at least two display screens, disposed on different surfaces of terminal 1200 or in a folded design; in still other embodiments, display screen 1205 may be a flexible display screen, disposed on a curved or folded surface of terminal 1200. Furthermore, display screen 1205 may also be configured as a non-rectangular, irregular shape, i.e., a non-rectangular screen. The display screen 1205 can be made of materials such as LCD (Liquid Crystal Display) and OLED (Organic Light-Emitting Diode).

[0158] The camera assembly 1206 is used to acquire images or videos. In some embodiments, the camera assembly 1206 includes a front-facing camera and a rear-facing camera. Typically, the front-facing camera is located on the front panel of the terminal, and the rear-facing camera is located on the back of the terminal. In some embodiments, there are at least two rear-facing cameras, which are any one of a main camera, a depth-sensing camera, a wide-angle camera, and a telephoto camera, to achieve background blurring by fusion of the main camera and the depth-sensing camera, panoramic shooting by fusion of the main camera and the wide-angle camera, VR (Virtual Reality) shooting, or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a single-color temperature flash or a dual-color temperature flash. A dual-color temperature flash is a combination of a warm-light flash and a cool-light flash, which can be used for light compensation at different color temperatures.

[0159] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, converting the sound waves into electrical signals that are input to the processor 1201 for processing, or input to the radio frequency circuit 1204 for voice communication. For stereo sound acquisition or noise reduction purposes, multiple microphones may be used, each positioned at a different location on the terminal 1200. The microphone may also be an array microphone or an omnidirectional microphone. The speaker is used to convert electrical signals from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a conventional diaphragm speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can convert electrical signals not only into audible sound waves but also into inaudible sound waves for purposes such as distance measurement. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0160] The positioning component 1208 is used to locate the current geographical location of the terminal 1200 in order to enable navigation or LBS (Location Based Service). The positioning component 1208 can be a positioning component based on the US GPS (Global Positioning System), China's BeiDou system, or Russia's Galileo system.

[0161] Power supply 1209 is used to power the various components in terminal 1200. Power supply 1209 can be AC ​​power, DC power, a disposable battery, or a rechargeable battery. When power supply 1209 includes a rechargeable battery, the rechargeable battery can be a wired rechargeable battery or a wireless rechargeable battery. A wired rechargeable battery is a battery that is charged via a wired line, and a wireless rechargeable battery is a battery that is charged via a wireless coil. The rechargeable battery can also be used to support fast charging technology.

[0162] In some embodiments, the terminal 1200 further includes one or more sensors 1210. The one or more sensors 1210 include, but are not limited to: an accelerometer 1211, a gyroscope 1212, a pressure sensor 1213, a fingerprint sensor 1214, an optical sensor 1215, and a proximity sensor 1216.

[0163] Accelerometer 1211 can detect the magnitude of acceleration along the three coordinate axes of a coordinate system established with terminal 1200. For example, accelerometer 1211 can be used to detect the components of gravitational acceleration along the three coordinate axes. Processor 1201 can control display screen 1205 to display the user interface in either a landscape or portrait view based on the gravitational acceleration signal acquired by accelerometer 1211. Accelerometer 1211 can also be used for games or for acquiring user motion data.

[0164] The gyroscope sensor 1212 can detect the orientation and rotation angle of the terminal 1200. The gyroscope sensor 1212 can work in conjunction with the accelerometer sensor 1211 to collect the user's 3D movements on the terminal 1200. Based on the data collected by the gyroscope sensor 1212, the processor 1201 can perform the following functions: motion sensing (e.g., changing the UI based on the user's tilt), image stabilization during shooting, game control, and inertial navigation.

[0165] The pressure sensor 1213 can be disposed on the side bezel of the terminal 1200 and / or on the lower layer of the display screen 1205. When the pressure sensor 1213 is disposed on the side bezel of the terminal 1200, it can detect the user's grip signal on the terminal 1200, and the processor 1201 can perform left / right hand recognition or quick operation based on the grip signal collected by the pressure sensor 1213. When the pressure sensor 1213 is disposed on the lower layer of the display screen 1205, the processor 1201 can control the operable controls on the UI interface based on the user's pressure operation on the display screen 1205. The operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0166] The fingerprint sensor 1214 is used to collect a user's fingerprint. The processor 1201 identifies the user based on the fingerprint collected by the fingerprint sensor 1214, or vice versa. When the user's identity is identified as trusted, the processor 1201 authorizes the user to perform relevant sensitive operations, including unlocking the screen, viewing encrypted information, downloading software, making payments, and changing settings. The fingerprint sensor 1214 can be located on the front, back, or side of the terminal 1200. When the terminal 1200 has physical buttons or a manufacturer's logo, the fingerprint sensor 1214 can be integrated with the physical buttons or manufacturer's logo.

[0167] The optical sensor 1215 is used to collect ambient light intensity. In one embodiment, the processor 1201 can control the display brightness of the display screen 1205 based on the ambient light intensity collected by the optical sensor 1215. Specifically, when the ambient light intensity is high, the display brightness of the display screen 1205 is increased; when the ambient light intensity is low, the display brightness of the display screen 1205 is decreased. In another embodiment, the processor 1201 can also dynamically adjust the shooting parameters of the camera assembly 1206 based on the ambient light intensity collected by the optical sensor 1215.

[0168] The proximity sensor 1216, also known as a distance sensor, is typically mounted on the front panel of the terminal 1200. The proximity sensor 1216 is used to detect the distance between the user and the front of the terminal 1200. In one embodiment, when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually decreasing, the processor 1201 controls the display screen 1205 to switch from a screen-on state to a screen-off state; when the proximity sensor 1216 detects that the distance between the user and the front of the terminal 1200 is gradually increasing, the processor 1201 controls the display screen 1205 to switch from a screen-off state to a screen-on state.

[0169] Those skilled in the art will understand that Figure 12 The structure shown does not constitute a limitation on terminal 1200 and may include more or fewer components than shown, or combine certain components, or use different component arrangements.

[0170] When a computer device is configured as a server, Figure 13 This is a schematic diagram of a server structure according to an embodiment of this application. The server 1300 can vary considerably due to different configurations or performance. It may include one or more Central Processing Units (CPUs) 1301 and one or more memories 1302. The memory 1302 stores at least one computer program, which is loaded and executed by the processor 1301 to implement the image processing methods provided in the above-described method embodiments. Of course, the server may also have wired or wireless network interfaces, a keyboard, and input / output interfaces for input and output. The server may also include other components for implementing device functions, which will not be elaborated here.

[0171] This application also provides a computer-readable storage medium storing at least one computer program. This computer program is loaded and executed by a processor of a computer device to implement the operations performed by the computer device in the image processing method of the above embodiments. For example, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), magnetic tape, floppy disk, and optical data storage device, etc.

[0172] In some embodiments, the computer program involved in the present application embodiments may be deployed and executed on a computer device, or executed on multiple computer devices located in one location, or executed on multiple computer devices distributed in multiple locations and interconnected through a communication network. Multiple computer devices distributed in multiple locations and interconnected through a communication network may constitute a blockchain system.

[0173] This application also provides a computer program product or computer program, which includes computer program code stored in a computer-readable storage medium. A processor of a computer device reads the computer program code from the computer-readable storage medium and executes the computer program code, causing the computer device to perform the image processing methods provided in the various optional implementations described above.

[0174] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware or by a program instructing related hardware. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0175] The above description is merely an optional embodiment of this application and is not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. An image processing method, characterized in that, The image processing method is based on a human body analysis model, which is used to perform human body analysis on a first image and output an image labeled with different parts of the human body. The method includes: The first image is mapped to three intermediate images, the first image including a human body, and the three intermediate images are of different scales to represent the image features of the first image; The three intermediate images are fused to obtain the second image; The second image is mapped to a target image, the target image being labeled with different parts of the human body; The method further includes: The first labeled image of the sample image is preprocessed to obtain the encoded image of the sample image. The first labeled image is used to indicate different parts of the human body in the sample image, and the encoded image is used to indicate the prediction result of the previous frame image of the sample image. The sample image and the encoded image are concatenated to obtain the input image; Using the first labeled image as supervision information, the human body parsing model for the i-th iteration is trained based on the input image, where i is a positive integer.

2. The method according to claim 1, characterized in that, The process of mapping the first image into three intermediate images includes: Convolve the first image to obtain the first intermediate image; Convolve the first intermediate image to obtain the second intermediate image; The second intermediate image is enhanced with channel features and semantic features to obtain a third intermediate image. The channel feature enhancement is used to strengthen the importance of different channel features, and the semantic feature enhancement is used to strengthen global semantic information.

3. The method according to claim 2, characterized in that, The process of fusing the three intermediate images to obtain the second image includes: The result of convolving the third intermediate image with the second intermediate image is fused to obtain the first fused image; The first fused image is fused with the result of convolving the first intermediate image to obtain a second fused image, which is then used as the second image.

4. The method according to claim 1, characterized in that, The step of mapping the second image to the target image includes: Increase the resolution of the second image to obtain a third image, wherein the resolution of the third image is not higher than the resolution of the first image; The third image is convolved to obtain the target image.

5. The method according to claim 1, characterized in that, The preprocessing of the first labeled image of the sample image to obtain the encoded image of the sample image includes: The first labeled image of the sample image is transformed to obtain the second labeled image; The second labeled image is encoded to obtain the encoded image.

6. The method according to claim 5, characterized in that, The step of performing image transformation on the first labeled image of the sample image to obtain the second labeled image includes: The first labeled image is subjected to at least one of rigid transformation and non-rigid transformation to obtain the second labeled image.

7. The method according to claim 5, characterized in that, The step of encoding the second labeled image to obtain the encoded image includes: The pixels in the second labeled image are mapped to the target vector space according to their pixel categories to obtain the encoded image.

8. The method according to claim 1, characterized in that, The step of concatenating the sample image and the encoded image to obtain the input image includes: The sample image and the encoded image are concatenated along the channel dimension to obtain the input image.

9. The method according to claim 1, characterized in that, The step of training the human body parsing model for the i-th iteration based on the input image, using the first labeled image as supervision information, includes: The input image is analyzed based on the human body analysis model in the i-th iteration to obtain a predicted image, which is used to indicate different parts of the predicted human body sample. Based on the first labeled image and the predicted image, a first loss, a second loss, and a third loss are determined. The first loss is used to indicate the difference between the first labeled image and the predicted image. The second loss is used to indicate the difference between the first labeled image and the predicted image after pixel weighting. The third loss is used to indicate the difference between the first labeled image and the predicted image after adding dependency information to the pixels. The dependency information is used to indicate the information contained in the pixels surrounding the pixel. Based on the first loss, the second loss, and the third loss, the model parameters of the human body analysis model in the i-th iteration are adjusted.

10. The method according to claim 9, characterized in that, The process of determining the second loss based on the first labeled image and the predicted image includes: Based on the number of pixels corresponding to each pixel category in the predicted image, the category weight of each pixel category is determined, and the category weight is inversely correlated with the number of pixels; Based on the category weights of each pixel category, a weighted cross-entropy loss is determined, and this weighted cross-entropy loss is used as the second loss.

11. The method according to claim 9, characterized in that, The process of determining the third loss based on the first labeled image and the predicted image includes: Based on the first labeled image, determine the label probability distribution; Based on the predicted image, determine the predicted probability distribution; Based on the labeled probability distribution and the predicted probability distribution, the labeled probability density function, the predicted probability density function, and the joint probability density function are determined. Based on the labeled probability density function, the predicted probability density function, and the joint probability density function, the cross-entropy loss is determined, and the cross-entropy loss is used as the third loss.

12. An image processing apparatus, characterized in that, The steps performed by the image processing device are based on a human body analysis model, which is used to perform human body analysis on the first image and output an image labeled with different parts of the human body. The device includes: The first mapping module is used to map the first image into three intermediate images, the first image including a human body, the three intermediate images having different scales, and used to represent the image features of the first image; An image fusion module is used to fuse the three intermediate images to obtain a second image; The second mapping module is used to map the second image into a target image, wherein the target image is labeled with different parts of the human body; The device further includes: The preprocessing module is used to preprocess the first labeled image of the sample image to obtain the encoded image of the sample image. The first labeled image is used to indicate different parts of the human body in the sample image, and the encoded image is used to indicate the prediction result of the previous frame image of the sample image. The stitching module is used to stitch the sample image and the encoded image together to obtain the input image; The training module is used to train the human body parsing model in the i-th iteration based on the input image, using the first labeled image as supervision information, where i is a positive integer.

13. The apparatus according to claim 12, characterized in that, The first mapping module is used for: Convolve the first image to obtain the first intermediate image; Convolve the first intermediate image to obtain the second intermediate image; The second intermediate image is enhanced with channel features and semantic features to obtain a third intermediate image. The channel feature enhancement is used to strengthen the importance of different channel features, and the semantic feature enhancement is used to strengthen global semantic information.

14. The apparatus according to claim 13, characterized in that, The image fusion module is used for: The result of convolving the third intermediate image with the second intermediate image is fused to obtain the first fused image; The first fused image is fused with the result of convolving the first intermediate image to obtain a second fused image, which is then used as the second image.

15. The apparatus according to claim 12, characterized in that, The second mapping module is used for: Increase the resolution of the second image to obtain a third image, wherein the resolution of the third image is not higher than the resolution of the first image; The third image is convolved to obtain the target image.

16. The apparatus according to claim 12, characterized in that, The preprocessing module is used for: The first labeled image of the sample image is transformed to obtain the second labeled image; The second labeled image is encoded to obtain the encoded image.

17. The apparatus according to claim 16, characterized in that, The preprocessing module is used for: The first labeled image is subjected to at least one of rigid transformation and non-rigid transformation to obtain the second labeled image.

18. The apparatus according to claim 16, characterized in that, The preprocessing module is used for: The pixels in the second labeled image are mapped to the target vector space according to their pixel categories to obtain the encoded image.

19. The apparatus according to claim 12, characterized in that, The splicing module is used for: The sample image and the encoded image are concatenated along the channel dimension to obtain the input image.

20. The apparatus according to claim 12, characterized in that, The training module is used for: The input image is analyzed based on the human body analysis model in the i-th iteration to obtain a predicted image, which is used to indicate different parts of the predicted human body sample. Based on the first labeled image and the predicted image, a first loss, a second loss, and a third loss are determined. The first loss is used to indicate the difference between the first labeled image and the predicted image. The second loss is used to indicate the difference between the first labeled image and the predicted image after pixel weighting. The third loss is used to indicate the difference between the first labeled image and the predicted image after adding dependency information to the pixels. The dependency information is used to indicate the information contained in the pixels surrounding the pixel. Based on the first loss, the second loss, and the third loss, the model parameters of the human body analysis model in the i-th iteration are adjusted.

21. The apparatus according to claim 20, characterized in that, The training module is used for: Based on the number of pixels corresponding to each pixel category in the predicted image, the category weight of each pixel category is determined, and the category weight is inversely correlated with the number of pixels; Based on the category weights of each pixel category, a weighted cross-entropy loss is determined, and this weighted cross-entropy loss is used as the second loss.

22. The apparatus according to claim 20, characterized in that, The training module is used for: Based on the first labeled image, determine the label probability distribution; Based on the predicted image, determine the predicted probability distribution; Based on the labeled probability distribution and the predicted probability distribution, the labeled probability density function, the predicted probability density function, and the joint probability density function are determined. Based on the labeled probability density function, the predicted probability density function, and the joint probability density function, the cross-entropy loss is determined, and the cross-entropy loss is used as the third loss.

23. A computer device, characterized in that, The computer device includes a processor and a memory, the memory being used to store at least one computer program, the at least one computer program being loaded by the processor and executed as the image processing method according to any one of claims 1 to 11.

24. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store at least one computer program for performing the image processing method according to any one of claims 1 to 11.

25. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the image processing method as described in any one of claims 1 to 11.

Images