Image processing method and system, model training method and system

Abstract

This application discloses an image processing method and system, and a model training method and system. The image processing method includes: receiving a target image, wherein the target image includes a target region; using a region recognition model to recognize the target image, obtaining a target recognition result for the target region, wherein the target recognition result is used to predict the location of the target region, and the region recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; and outputting the target recognition result. This application solves the technical problem in related technologies where the data processing uses large models, resulting in high memory consumption and slow processing speed.

Description

Technical Field

[0001] This application relates to the field of data processing, and more specifically, to an image processing method and system, and a model training method and system. Background Technology

[0002] Currently, online shopping platforms offer users a virtual shoe-trying feature, allowing them to experience the virtual shoe-trying effect directly on their mobile devices. For merchants, this can help reduce return rates, increase conversion rates, and reduce inventory; for users, it can help them purchase more comfortable shoes.

[0003] To implement virtual shoe try-on functionality, the six-dimensional pose of the foot in the image or video must first be estimated. Then, a virtual shoe is drawn based on the pose. Finally, the virtual shoe is occluded based on foot segmentation to achieve a realistic try-on experience. To achieve six-dimensional pose estimation, a model can be used to process the input image or video data to obtain foot keypoints and segmentation. However, these models are often large, resulting in high memory consumption on mobile devices and slow processing speeds, negatively impacting the user experience.

[0004] There is currently no effective solution to the above problems. Summary of the Invention

[0005] This application provides an image processing method and system, and a model training method and system, to at least solve the technical problem in related technologies that the large size of the models used for data processing leads to high memory consumption and slow processing speed.

[0006] According to one aspect of the embodiments of this application, an image processing method is provided, comprising: receiving a target image, wherein the target image includes a target region; using a region recognition model to recognize the target image to obtain a target recognition result of the target region, wherein the target recognition result is used to predict the position of the target region, and the region recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together; and outputting the target recognition result.

[0007] According to another aspect of the embodiments of this application, an image processing method is also provided, including: acquiring a target image, wherein the target image includes a target part; using a part recognition model to recognize the target image to obtain a target recognition result of the target part, wherein the target recognition result is used to predict the position of the target part, and the part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together.

[0008] According to another aspect of the embodiments of this application, a model training method is also provided, comprising: receiving a model training request; obtaining training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; training the multi-task model using the training samples to obtain a body part recognition model; and outputting the body part recognition model.

[0009] According to another aspect of the embodiments of this application, a model training method is also provided, comprising: acquiring training samples, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts; training a multi-task model using the training samples to obtain a body part recognition model, wherein the multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together.

[0010] According to another aspect of the embodiments of this application, a model training method is also provided, comprising: obtaining a model training request by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the model training request; obtaining training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; training the multi-task model using the training samples to obtain a body part recognition model; and outputting the body part recognition model by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the body part recognition model.

[0011] According to another aspect of the embodiments of this application, a computer-readable storage medium is also provided, which includes a stored program, wherein the program controls the device where the computer-readable storage medium is located to execute the above-described image processing method and model training method when it is running.

[0012] According to another aspect of the embodiments of this application, a computer terminal is also provided, including: a memory and a processor, wherein the processor is used to run a program stored in the memory, wherein the program executes the above-described image processing method and model training method when it runs.

[0013] According to another aspect of the embodiments of this application, an image processing system is also provided, including: a processor; and a memory connected to the processor, for providing the processor with instructions to process the following processing steps: receiving a target image, wherein the target image includes: a target region; recognizing the target image using a region recognition model to obtain a target recognition result of the target region, wherein the target recognition result is used to predict the position of the target region, and the region recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together; and outputting the target recognition result.

[0014] According to another aspect of the embodiments of this application, a model training system is also provided, including: a processor; and a memory connected to the processor, configured to provide the processor with instructions for processing the following steps: receiving a model training request; acquiring training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; training the multi-task model using the training samples to obtain a body part recognition model; and outputting the body part recognition model.

[0015] According to another aspect of the embodiments of this application, a clothing matching method is also provided, comprising: receiving a target image and a target clothing, wherein the target image includes a target part; using a part recognition model to recognize the target image to obtain a target recognition result of the target part, wherein the part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; generating a virtual clothing corresponding to the target clothing based on the target recognition result; using the virtual clothing to occlude the target part to generate a matching image; and outputting the matching image.

[0016] In this embodiment, after receiving the target image, a part recognition model can be used to identify the target image, obtain the target part recognition result, and output the target recognition result to the user for viewing, thereby achieving the purpose of key point detection, region segmentation, and classification. It is noteworthy that the part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together, which greatly reduces the number of model parameters. This allows the mobile terminal's operating speed to meet the real-time requirements of image processing, achieving the technical effect of reducing the model's memory footprint on the mobile terminal and improving processing speed. This solves the technical problem in related technologies where large models are used for data processing, resulting in high memory consumption and slow processing speed. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0018] Figure 1 This is a hardware structure block diagram of a computer terminal (or mobile device) for implementing an image processing method and a model training method according to an embodiment of this application.

[0019] Figure 2 This is a flowchart of an image processing method according to an embodiment of this application;

[0020] Figure 3 This is a schematic diagram of an optional interactive interface according to an embodiment of this application;

[0021] Figure 4 This is an optional key point heatmap and segmentation diagram according to an embodiment of this application;

[0022] Figure 5 This is a schematic diagram of an optional part recognition model structure according to an embodiment of this application;

[0023] Figure 6 This is a flowchart of another image processing method according to an embodiment of this application;

[0024] Figure 7 This is a flowchart of a model training method according to an embodiment of this application;

[0025] Figure 8 This is a flowchart of another model training method according to an embodiment of this application;

[0026] Figure 9 This is a flowchart of another model training method according to an embodiment of this application;

[0027] Figure 10 This is a schematic diagram of an image processing apparatus according to an embodiment of this application;

[0028] Figure 11 This is a schematic diagram of another image processing apparatus according to an embodiment of this application;

[0029] Figure 12 This is a schematic diagram of a model training apparatus according to an embodiment of this application;

[0030] Figure 13 This is a schematic diagram of another model training apparatus according to an embodiment of this application;

[0031] Figure 14 This is a schematic diagram of another model training device according to an embodiment of this application;

[0032] Figure 15 This is a flowchart of a clothing matching method according to an embodiment of this application;

[0033] Figure 16 This is a structural block diagram of a computer terminal according to an embodiment of this application. Detailed Implementation

[0034] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort should fall within the scope of protection of the present application.

[0035] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0036] First, some nouns or terms that appear in the description of the embodiments of this application shall be interpreted as follows:

[0037] Six-dimensional pose: Six-dimensional can refer to 6 degrees of freedom, representing 3 degrees of freedom of displacement (i.e., translation) and 3 degrees of freedom of spatial rotation. The combination of the above 6 degrees of freedom can be called pose.

[0038] PnP: Perspective-n-Point, n-point perspective, can be a method for solving the motion of 3D to 2D point pairs, that is, given 2D to 3D point pairs, solve for their pose.

[0039] Multi-task model: This can refer to a model that can perform multiple tasks, such as key point detection, region segmentation, and classification, but it is not limited to these.

[0040] Two-dimensional convolutional layers: can contain two dimensions, height and width, and are often used to process image data.

[0041] Depthwise separable convolution: This refers to using different convolution kernels for different input and output channels. Ordinary convolution operations can be decomposed into two processes: depthwise convolution and pointwise convolution.

[0042] BN: Batch Normalization, can normalize each batch of data and each feature so that each feature has a distribution with a mean of 0 and a variance of 1.

[0043] SE: Squeeze-and-Excitation, can refer to a channel-based attention mechanism module that enhances accuracy by modeling the correlation between feature channels and strengthening important features.

[0044] MSE: mean-square error, can refer to the relationship between a data sequence and the true value.

[0045] MobileNetV3: This can refer to a lightweight model for mobile devices that combines the ideas of three models: depthwise separable convolution, inverse residual structure with linear bottleneck, and lightweight attention model.

[0046] Currently, in the field of virtual shoe fitting, the models used for foot key point detection and segmentation tasks are often large, consume a lot of memory, and have a slow processing speed. Moreover, users encounter a wide variety of scenarios when using the device on mobile devices, but the robustness of these models is low, and they cannot obtain stable results in all of these scenarios, which affects the user experience.

[0047] Furthermore, to achieve six-dimensional pose estimation, the input image or video data can be processed using a model to obtain foot keypoints and segmentation. However, these models are often large, resulting in high memory consumption on mobile devices and slow processing speed, which negatively impacts the user experience.

[0048] To address the aforementioned issues, this application employs a lightweight method to detect key foot points and segment foot regions in images or videos. It also identifies the left and right feet, achieving real-time processing on mobile devices. Furthermore, the training data includes data from various scenarios, shoe styles, angles, and postures, effectively ensuring the richness of the training data.

[0049] Example 1

[0050] According to an embodiment of this application, an image processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0051] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, or similar computing devices. Figure 1 A hardware block diagram of a computer terminal (or mobile device) for implementing image processing methods and model training methods is shown. Figure 1 As shown, the computer terminal 10 (or mobile device 10) may include one or more processors 102 (shown as 102a, 102b, ..., 102n in the figure) 102 (processor 102 may include, but is not limited to, a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 104 for storing data, and a transmission device 106 for communication functions. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of a BUS bus), a network interface, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, computer terminal 10 may also include... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0052] It should be noted that the aforementioned one or more processors 102 and / or other data processing circuitry are generally referred to herein as "data processing circuitry". This data processing circuitry may be embodied, in whole or in part, in software, hardware, firmware, or any other combination thereof. Furthermore, the data processing circuitry may be a single, independent processing module, or may be integrated, in whole or in part, into any other element within the computer terminal 10 (or mobile device). This data processing circuitry serves as a processor control mechanism (e.g., selection of a variable resistor termination path connected to an interface).

[0053] The memory 104 can be used to store software programs and modules of application software, such as the program instructions / data storage device corresponding to the image processing method and model training method in the embodiments of this application. The processor 102 executes various functional applications and data processing by running the software programs and modules stored in the memory 104, thereby realizing the above-mentioned image processing method and model training method. The memory 104 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include memory remotely located relative to the processor 102, and these remote memories can be connected to the computer terminal 10 via a network. Examples of the above-mentioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0054] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of the computer terminal 10. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module, used for wireless communication with the Internet.

[0055] The display can be, for example, a touchscreen liquid crystal display (LCD) that allows the user to interact with the user interface of the computer terminal 10 (or mobile device).

[0056] It should be noted here that, in some optional embodiments, the above... Figure 1 The computer device (or mobile device) shown may include hardware elements (including circuitry), software elements (including computer code stored on a computer-readable medium), or a combination of both hardware and software elements. It should be noted that... Figure 1 This is only one instance of a specific particular instance and is intended to illustrate the types of components that may exist in the aforementioned computer device (or mobile device).

[0057] Under the aforementioned operating environment, this application provides the following: Figure 2 The image processing method shown. Figure 2 This is a flowchart of an image processing method according to an embodiment of this application. Figure 2 As shown, the method may include the following steps:

[0058] Step S202: Receive the target image, wherein the target image includes: the target region.

[0059] The target image in the above steps can be an image taken by the user targeting the target area, or it can be an image frame segmented from a video taken by the user targeting the target area. The target area can be a part of the user's body, such as a hand, foot, face, ear, etc. In this embodiment, the foot is used as an example for explanation. For body parts such as hands and feet, the target image can include only one side (e.g., left hand, right hand, left foot, right foot) or both sides (e.g., left and right hands, left and right feet).

[0060] In one alternative embodiment, an interactive interface can be provided on a mobile terminal (e.g., smartphone, tablet, PDA, laptop, etc.), such as Figure 3 As shown, users can use the mobile terminal's camera to capture images or videos by clicking the "Shoot" button on the interactive interface. The captured image or video frame is the target image mentioned above. Alternatively, users can click the "Upload" button on the interactive interface to view images or videos stored on the mobile terminal and select one. The video frame in the selected image or video is the target image mentioned above.

[0061] Step S204: The target image is identified using the part recognition model to obtain the target recognition result of the target part. The target recognition result is used to predict the position of the target part. The part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together.

[0062] Optionally, the target recognition results mentioned above include: target detection image, target segmentation image, and target classification result. The target detection image contains key point information of the target part, the target segmentation image is an image of the region where the target part is located, and the target classification result is used to characterize the position of the target part relative to the body trunk.

[0063] The target detection image mentioned above can be an image containing detected key points, for example, a key point heatmap of the foot key points of a detection path. Each key point heatmap contains information about only one key point; therefore, the number of key points to be detected can be pre-set according to the detection needs, thus obtaining the same number of key point heatmaps.

[0064] The aforementioned target segmentation image can be an image obtained by segmenting a target image, containing only the target region and excluding other background images.

[0065] The target classification result mentioned above can refer to the side of the target body relative to its location. For example, it could be the left foot, the right foot, or both feet, thus avoiding confusion caused by the left foot being mistaken for the right foot. The target classification result can be represented by two binary digits, where the higher-order binary digit represents the left side and the lower-order binary digit represents the right side. A binary 1 indicates that the target body part was detected, and a binary 0 indicates that the target body part was not detected. For example, if the target image contains only the left foot, the target classification result is 10; if the target image contains only the right foot, the target classification result is 01; and if the target image contains both feet, the target classification result is 10.

[0066] The target recognition results in the above steps are not only used to predict the three-dimensional position of the target part, but also to predict the three-dimensional pose of the target part. Based on the target recognition results, virtual objects can be constructed. For example, for the foot, after obtaining the key points of the foot and the side where the foot is located, six-dimensional pose estimation can be performed, and then a virtual shoe can be drawn based on the pose. Finally, the virtual shoe is occluded based on the foot image to achieve a realistic try-on experience.

[0067] The part recognition model in the above steps can be a pre-trained multi-task model that uses a lightweight method to recognize target images, obtaining multiple target recognition results: detection results, segmentation results, and classification results. In this model, the first layer is a conventional 2D convolutional module, and the rest are depthwise separable convolutional modules, which greatly reduces the number of model parameters. The model size is 1.8M. The specific structure of the conventional 2D convolutional module can be 2D convolution + Batch Normalization (BN) + nonlinear activation function; the depthwise separable convolutional module can be a basic module in MobileNetV3, and its specific structure can be depthwise separable convolution + residual model + Sequence Optimization (SE).

[0068] To train a high-accuracy body part recognition model, a large number of training samples are often required for multiple iterations, resulting in a significant amount of data and computation. To reduce resource consumption on mobile devices, the model can be trained on a server, with only the trained body part recognition model deployed on the mobile device, thus achieving the image processing objective.

[0069] In one optional embodiment, after receiving the target image, the mobile terminal can input the target image into a deployed part recognition model. The output of this model is the target recognition result, which is a combination of key point heatmaps, segmentation, and classification. For example, taking a target image containing both a left and right foot as an example... Figure 4 As shown, we can obtain 5 left foot key point heatmaps, 5 right foot key point heatmaps, as well as left foot segmentation map and right foot segmentation map. The classification result can be 11.

[0070] Step S206: Output the target recognition result.

[0071] In an optional embodiment, the mobile terminal can directly display the target recognition result as shown in the image. Figure 3 In the display area of ​​the interactive interface shown, users can view the identified key point heatmap, segmentation, and classification. Users can adjust the target recognition results, and the adjusted target recognition results perform six-dimensional pose estimation on the target part.

[0072] The technical solution provided by the above embodiments of this application allows for the identification of target images after receiving a target image using a part recognition model. This results in the identification of the target part, which is then output to the user for viewing, achieving key point detection, region segmentation, and classification. It is noteworthy that the part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules, significantly reducing the number of model parameters. This allows the mobile terminal's operating speed to meet the real-time requirements of image processing, achieving the technical effect of reducing the model's memory footprint on the mobile terminal and improving processing speed. This solves the technical problem in related technologies where large models are used for data processing, resulting in high memory consumption and slow processing speed.

[0073] In the above embodiments of this application, the target recognition result of the target part is obtained by using a part recognition model to recognize the target image. This includes: using an encoding network to extract features from the target image to obtain multi-scale features of the target image, wherein the multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; using a decoding network to fuse the multi-scale features to obtain a target detection image and a target segmentation image, wherein the encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure; and using a classifier to perform a classification operation on the smallest scale feature among the multi-scale features to obtain a target classification result, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

[0074] To avoid the problem of poor fusion performance of feature pyramids across multiple scales, in the embodiments of this application, such as Figure 5As shown, the part recognition model can employ an encoder-decoder structure. The encoder network in the above steps can be composed of multiple first network structures of different scales, which can perform convolution-downsampling of the image. The decoder network in the above steps can be composed of multiple second network structures of different scales, which can perform upsampling-convolution of the feature map. Each second network structure can be connected to the previous second network structure and the corresponding first network structure, thereby convolving the feature map output by the first network structure with the upsampled fusion result. The classifier in the above steps can use existing structures; this application does not impose specific limitations on it. The classifier is cascaded after the feature map of the smallest scale to achieve the purpose of distinguishing between the left and right feet.

[0075] In one alternative embodiment, such as Figure 5 As shown, the target image can be input into the encoding network, which performs multiple convolutions and downsampling to obtain features at multiple scales. Then, the features at multiple scales are input into the decoding network, which performs multiple upsampling and convolutions to fuse them into a key point heatmap and a segmentation map. At the same time, the smallest scale features are input into the classifier, which performs classification to determine whether the target part is on the left or right side, and obtains the classification result.

[0076] In the above embodiments of this application, the method may further include the following steps: obtaining training samples, wherein the training samples include: training images and a first recognition result corresponding to the training images, the training images being images collected from multiple different scenes; and using the training samples to train a multi-task model to obtain a part recognition model.

[0077] The different scenarios in the above steps can refer to a variety of scenarios such as different states (e.g., barefoot, wearing socks, wearing shoes), different shoe types, different postures, different perspectives, occlusion, and lighting effects.

[0078] To improve the robustness of the part recognition model, images under different states, shoe types, postures, and angles can be used as training images. To ensure the richness of the training images, approximately 500,000 training images can be used to construct training samples.

[0079] In one optional embodiment, a multi-task model is first constructed according to the structure of the part recognition model. At this time, the network parameters of the multi-task model are preset values ​​and need to be adjusted. Next, images from different scenarios can be acquired as training images, and the corresponding recognition results can be manually labeled to construct training samples. Finally, the multi-task model is trained using the training samples, and the network parameters of the multi-task model are adjusted. The resulting multi-task model is the part recognition model.

[0080] In the above embodiments of this application, training a multi-task model using training samples to obtain a part recognition model includes: using the multi-task model to recognize a training image to obtain a second recognition result of the training image; determining the model loss value of the multi-task model based on the first recognition result and the second recognition result; and training the multi-task model using training samples when the model loss value is greater than a preset value until the model loss value is less than or equal to the preset value to obtain a part recognition model.

[0081] The preset values ​​in the above steps can be loss thresholds set according to the requirements of key point detection, region segmentation and classification, and can be modified according to the actual needs of users. This application does not make specific limitations on this.

[0082] In one optional embodiment, during the training of the multi-task model, a strategy of simultaneous training of multiple tasks can be adopted, and different loss functions can be calculated for different tasks. The final model loss value is then obtained through weighted summation. After each iteration, it is determined whether the model loss function value is less than a loss threshold. If the model loss function value is still greater than the loss threshold, the next iteration continues until the model loss function value is less than the loss threshold, or the number of iterations reaches a threshold, thus determining that training is complete.

[0083] In the above embodiments of this application, determining the model loss value of the multi-task model based on the first recognition result and the second recognition result includes: determining a detection loss value based on the first detection image in the first recognition result and the second detection image in the second recognition result; determining a segmentation loss value based on the first segmentation image in the first recognition result and the second segmentation image in the second recognition result; determining a classification loss value based on the first classification result in the first recognition result and the second classification result in the second recognition result; and obtaining a weighted sum of the detection loss value, the segmentation loss value, and the classification loss value to obtain the model loss value.

[0084] In one optional embodiment, the detection loss can be obtained using the MSE loss function, the segmentation loss can be obtained using the cross-entropy loss function, and the classification loss can be obtained using the cross-loss function. The model loss can then be obtained by weighting the three loss values. The specific calculation formula is as follows:

[0085]

[0086]

[0087]

[0088] L=α*L hm +β*L seg +γ*L c ,

[0089] Among them, L hm To detect the loss value, L seg For the segmentation loss value, L c L is the classification loss value, and L is the model loss value. For the i-th keypoint heatmap predicted by the model (i.e., the second detection image mentioned above), y h，i For the corresponding ground truth heatmap (i.e., the first detection image mentioned above), K is the number of key point heatmaps. For example, the value of K can be 10. For the t-th segmentation image predicted by the model (i.e., the second segmentation image mentioned above), y s，t The corresponding ground truth segmentation image (i.e., the first segmentation image mentioned above) is N, which is the number of segmentation images. The value of N can be determined based on the number of parts contained in the image. For example, the value of N can be 2. For the j-th "classification" result predicted by the model (i.e., the second classification result mentioned above), y c，j The corresponding true value is "classified" (i.e., the first classification result mentioned above), M is the number of "classifications", and the values ​​of M and N are often the same. For example, the value of M can also be 2; α, β, and γ are the weight coefficients of the three losses.

[0090] It should be noted that the training method of the part recognition model used in this embodiment can refer to the model training method in the following embodiments. The specific implementation scheme and application scenarios are the same as those in the following embodiments, and will not be repeated here.

[0091] It should also be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this application is not limited to the described order of actions, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also understand that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily essential to this application.

[0092] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods according to the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0093] Example 2

[0094] According to an embodiment of this application, an image processing method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0095] Figure 6 This is a flowchart of another image processing method according to an embodiment of this application. For example... Figure 6 As shown, the method may include the following steps:

[0096] Step S602: Obtain the target image, wherein the target image includes the target region.

[0097] The target image in the above steps can be an image taken by the user targeting the target area, or it can be an image frame segmented from a video taken by the user targeting the target area. The target area can be a part of the user's body, such as a hand, foot, face, ear, etc. In this embodiment, the foot is used as an example for explanation. For body parts such as hands and feet, the target image can include only one side (e.g., left hand, right hand, left foot, right foot) or both sides (e.g., left and right hands, left and right feet).

[0098] Step S604: The target image is identified using the part recognition model to obtain the target recognition result of the target part. The target recognition result is used to predict the position of the target part. The part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together.

[0099] Optionally, the target recognition results mentioned above include: target detection image, target segmentation image, and target classification result. The target detection image contains key point information of the target part, the target segmentation image is an image of the region where the target part is located, and the target classification result is used to characterize the position of the target part relative to the body trunk.

[0100] The target detection image mentioned above can be an image containing detected key points, for example, a key point heatmap of the foot key points of a detection path. Each key point heatmap contains information about only one key point; therefore, the number of key points to be detected can be pre-set according to the detection needs, thus obtaining the same number of key point heatmaps.

[0101] The aforementioned target segmentation image can be an image obtained by segmenting a target image, containing only the target region and excluding other background images.

[0102] The target classification result mentioned above can refer to the side of the target body relative to its location. For example, it could be the left foot, the right foot, or both feet, thus avoiding confusion caused by the left foot being mistaken for the right foot. The target classification result can be represented by two binary digits, where the higher-order binary digit represents the left side and the lower-order binary digit represents the right side. A binary 1 indicates that the target body part was detected, and a binary 0 indicates that the target body part was not detected. For example, if the target image contains only the left foot, the target classification result is 10; if the target image contains only the right foot, the target classification result is 01; and if the target image contains both feet, the target classification result is 11.

[0103] The part recognition model in the above steps can be a pre-trained multi-task model that uses a lightweight method to recognize target images, obtaining multiple target recognition results: detection results, segmentation results, and classification results. In this model, the first layer is a conventional 2D convolutional module, and the rest are depthwise separable convolutional modules, which greatly reduces the number of model parameters. The model size is 1.8M. The specific structure of the conventional 2D convolutional module can be 2D convolution + Batch Normalization (BN) + nonlinear activation function; the depthwise separable convolutional module can be a basic module in MobileNetV3, and its specific structure can be depthwise separable convolution + residual model + Sequence Optimization (SE).

[0104] In the above embodiments of this application, the target recognition result of the target part is obtained by using a part recognition model to recognize the target image. This includes: using an encoding network to extract features from the target image to obtain multi-scale features of the target image, wherein the multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; using a decoding network to fuse the multi-scale features to obtain a target detection image and a target segmentation image, wherein the encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure; and using a classifier to perform a classification operation on the smallest scale feature among the multi-scale features to obtain a target classification result, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

[0105] In the above embodiments of this application, the method may further include the following steps: obtaining training samples, wherein the training samples include: training images and a first recognition result corresponding to the training images, the training images being images collected from multiple different scenes; and using the training samples to train a multi-task model to obtain a part recognition model.

[0106] The different scenarios in the above steps can refer to a variety of scenarios such as different states (e.g., barefoot, wearing socks, wearing shoes), different shoe types, different postures, different perspectives, occlusion, and lighting effects.

[0107] In the above embodiments of this application, training a multi-task model using training samples to obtain a part recognition model includes: using the multi-task model to recognize a training image to obtain a second recognition result of the training image; determining the model loss value of the multi-task model based on the first recognition result and the second recognition result; and training the multi-task model using training samples when the model loss value is greater than a preset value until the model loss value is less than or equal to the preset value to obtain a part recognition model.

[0108] The preset values ​​in the above steps can be loss thresholds set according to the requirements of key point detection, region segmentation and classification, and can be modified according to the actual needs of users. This application does not make specific limitations on this.

[0109] In the above embodiments of this application, determining the model loss value of the multi-task model based on the first recognition result and the second recognition result includes: determining the detection loss value based on the first detection image and the second detection image; determining the segmentation loss value based on the first segmentation image and the second segmentation image; determining the classification loss value based on the first classification result and the second classification result; and obtaining the weighted sum of the detection loss value, the segmentation loss value and the classification loss value to obtain the model loss value.

[0110] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0111] Example 3

[0112] According to an embodiment of this application, a model training method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0113] Figure 7 This is a flowchart of a model training method according to an embodiment of this application. Figure 7 As shown, the method may include the following steps:

[0114] Step S702: Receive model training request.

[0115] To train a high-accuracy body part recognition model, a large number of training samples are often required for multiple iterations, resulting in a significant amount of data and computation. To reduce resource consumption on mobile devices, the model can be trained on a server, with only the trained body part recognition model deployed on the mobile device, thus achieving the image processing objective.

[0116] The model training request in the above steps can be generated according to the user's model usage needs. For example, in a virtual shoe trying-on scenario, the model training request can be a training request for a foot recognition model.

[0117] In one alternative embodiment, an interactive interface can be provided on the mobile terminal, where the user can input a model training request. The mobile terminal can then send the model training request to the server via a network. Since the model usage requirements are relatively fixed in the virtual shoe-fitting scenario, a selection button can be provided to the user to choose whether to generate a foot recognition model, thereby generating the aforementioned model training request.

[0118] Step S704: Obtain the training samples and multi-task model corresponding to the model training request. The training samples include: training images containing body parts and the first recognition result corresponding to the training images. The first recognition result is used to predict the position of the body parts. The multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together.

[0119] To improve the robustness of the part recognition model, images under different states, shoe types, postures, and angles can be used as training images. To ensure the richness of the training images, approximately 500,000 training images can be used to construct training samples.

[0120] The training images in the above steps can refer to images or video frames segmented from images collected in various scenarios such as different states (e.g., barefoot, wearing socks, wearing shoes), different shoe types, different postures, different perspectives, occlusion, and lighting effects.

[0121] The multi-task model in the above steps can be a model built using lightweight methods. This model can recognize training images and obtain various types of recognition results, namely detection results, segmentation results, and classification results. In this model, the first layer is a conventional two-dimensional convolutional module, and the rest are depthwise separable convolutional modules, which greatly reduces the number of model parameters. The model size is 1.8M. The specific structure of the conventional two-dimensional convolutional module can be two-dimensional convolution + Batch Normalization (BN) + nonlinear activation function; the depthwise separable convolutional module can be a basic module in MobileNetV3, and its specific structure can be depthwise separable convolution + residual model + Sequence Optimization (SE).

[0122] Optionally, the first recognition result mentioned above includes: a first detection image, a first segmentation image, and a first classification result. The first detection image contains key point information of the body part, the first segmentation image is an image of the region where the body part is located, and the first classification result is used to characterize the position of the body part relative to the torso.

[0123] The first detection image mentioned above can be an image containing detected key points, for example, a key point heatmap of the foot key points of the detection path. Each key point heatmap contains information about only one key point; therefore, the number of key points to be detected can be preset according to the detection needs, thus obtaining the same number of key point heatmaps.

[0124] The first segmented image mentioned above can be an image obtained by segmenting the target image, containing only the target part and not other background images.

[0125] The first classification result mentioned above can refer to the side of the target body relative to its location. For example, it could be the left foot, the right foot, or both feet, thus avoiding confusion caused by the left foot being mistaken for the right foot. The target classification result can be represented by two binary digits, where the higher-order binary digit represents the left side and the lower-order binary digit represents the right side. A binary 1 indicates that the target body part was detected, and a binary 0 indicates that the target body part was not detected. For example, if the target image contains only the left foot, the target classification result is 10; if the target image contains only the right foot, the target classification result is 01; and if the target image contains both feet, the target classification result is 11.

[0126] Step S706: Train the multi-task model using training samples to obtain the part recognition model.

[0127] In one optional embodiment, a multi-task model is first constructed according to the structure of the part recognition model. At this time, the network parameters of the multi-task model are preset values ​​and need to be adjusted. Next, images from different scenarios can be acquired as training images, and the corresponding recognition results can be manually labeled to construct training samples. Finally, the multi-task model is trained using the training samples, and the network parameters of the multi-task model are adjusted. The resulting multi-task model is the part recognition model.

[0128] Step S708: Output the part recognition model.

[0129] In one alternative embodiment, since the part recognition model is deployed in the mobile terminal, the server can transmit the part recognition model to the mobile terminal via the network, so that the mobile terminal can use the deployed part recognition model to recognize the image and obtain key point heatmaps, segmented images and classification results.

[0130] The technical solution provided by the above embodiments of this application allows for the acquisition of training samples and a multi-task model corresponding to the model training request after receiving the request. The training samples are then used to train the multi-task model to obtain a part recognition model, which is then output to the mobile terminal, achieving the purpose of model training. It is noteworthy that the multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules, thereby significantly reducing the number of model parameters. This allows the mobile terminal's operating speed to meet the real-time requirements of image processing, achieving the technical effect of reducing the model's memory footprint on the mobile terminal and improving processing speed. This solves the technical problem in related technologies where large models are used for data processing, resulting in high memory consumption and slow processing speed.

[0131] In the above embodiments of this application, training a multi-task model using training samples to obtain a part recognition model includes: using the multi-task model to recognize training images to obtain a second recognition result of the training images, wherein the training images are images collected from multiple different scenes; determining the model loss value of the multi-task model based on the first recognition result and the second recognition result; and training the multi-task model using training samples when the model loss value is greater than a preset value until the model loss value is less than or equal to the preset value to obtain the part recognition model.

[0132] The different scenarios in the above steps can refer to a variety of scenarios such as different states (e.g., barefoot, wearing socks, wearing shoes), different shoe types, different postures, different perspectives, occlusion, and lighting effects.

[0133] To improve the robustness of the part recognition model, images under different states, shoe types, postures, and angles can be used as training images. To ensure the richness of the training images, approximately 500,000 training images can be used to construct training samples.

[0134] The preset values ​​in the above steps can be loss thresholds set according to the requirements of key point detection, region segmentation and classification, and can be modified according to the actual needs of users. This application does not make specific limitations on this.

[0135] In one optional embodiment, during the training of the multi-task model, a strategy of simultaneous training of multiple tasks can be adopted, and different loss functions can be calculated for different tasks. The final model loss value is then obtained through weighted summation. After each iteration, it is determined whether the model loss function value is less than a loss threshold. If the model loss function value is still greater than the loss threshold, the next iteration continues until the model loss function value is less than the loss threshold, or the number of iterations reaches a threshold, thus determining that training is complete.

[0136] In the above embodiments of this application, the second recognition result of the training image is obtained by recognizing the training image using a multi-task model, which includes: extracting features from the training image using an encoding network to obtain multi-scale features of the training image, wherein the multi-scale features include: features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; fusing the multi-scale features using a decoding network to obtain a second detection image and a second segmentation image, wherein the encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure; and classifying the smallest scale feature among the multi-scale features using a classifier to obtain a second classification result, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

[0137] To avoid the problem of poor fusion performance of feature pyramids across multiple scales, the multi-task model in this embodiment can adopt an encoder-decoder structure. The encoder network in the above steps can be composed of multiple first network structures of different scales, which can perform convolution-downsampling of the image. The decoder network in the above steps can be composed of multiple second network structures of different scales, which can perform upsampling-convolution of the feature map. Each second network structure can be connected to the previous second network structure and the corresponding first network structure, thereby convolving the feature map output by the first network structure with the upsampled fusion result. The classifier in the above steps can adopt existing structures, and this application does not specifically limit it. The classifier is cascaded after the feature map of the smallest scale to achieve the purpose of distinguishing between the left and right feet.

[0138] In one alternative embodiment, the training image can be input into an encoding network, which performs multiple convolutions and downsampling to obtain features at multiple scales. These features are then input into a decoding network, which performs multiple upsampling and convolutions to fuse them into a keypoint heatmap and a segmentation map. Simultaneously, the smallest scale features are input into a classifier, which classifies the body part to determine whether it is on the left or right side.

[0139] In the above embodiments of this application, determining the model loss value of the multi-task model based on the first recognition result and the second recognition result includes: determining the detection loss value based on the first detection image and the second detection image; determining the segmentation loss value based on the first segmentation image and the second segmentation image; determining the classification loss value based on the first classification result and the second classification result; and obtaining the weighted sum of the detection loss value, the segmentation loss value and the classification loss value to obtain the model loss value.

[0140] In one optional embodiment, the detection loss can be obtained using the MSE loss function, the segmentation loss can be obtained using the cross-entropy loss function, and the classification loss can be obtained using the cross-loss function. The model loss can then be obtained by weighting the three loss values. The specific calculation formula is as follows:

[0141]

[0142]

[0143]

[0144] L=α*L hm +β*L seg +γ*L c ,

[0145] Among them, L hm To detect the loss value, L seg For the segmentation loss value, L c L is the classification loss value, and L is the model loss value. For the i-th keypoint heatmap predicted by the model (i.e., the second detection image mentioned above), y h，i For the corresponding ground truth heatmap (i.e., the first detection image mentioned above), K is the number of key point heatmaps. For example, the value of K can be 10. For the t-th segmentation image predicted by the model (i.e., the second segmentation image mentioned above), y s，t The corresponding ground truth segmentation image (i.e., the first segmentation image mentioned above) is N, which is the number of segmentation images. The value of N can be determined based on the number of parts contained in the image. For example, the value of N can be 2. For the j-th "classification" result predicted by the model (i.e., the second classification result mentioned above), y c，j The corresponding true value is "classified" (i.e., the first classification result mentioned above), M is the number of "classifications", and the values ​​of M and N are often the same. For example, the value of M can also be 2; α, β, and γ are the weight coefficients of the three losses.

[0146] It should be noted that the application method of the part recognition model trained in this embodiment can refer to the image processing method in the above embodiment. The specific implementation scheme and application scenario are the same as those in the following embodiment, and will not be repeated here.

[0147] Example 4

[0148] According to an embodiment of this application, a model training method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0149] Figure 8 This is a flowchart of another model training method according to an embodiment of this application. For example... Figure 8 As shown, the method may include the following steps:

[0150] Step S802: Obtain training samples, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts.

[0151] The training images in the above steps can refer to images or video frames segmented from images collected in various scenarios such as different states (e.g., barefoot, wearing socks, wearing shoes), different shoe types, different postures, different perspectives, occlusion, and lighting effects.

[0152] Optionally, the first recognition result mentioned above includes: a first detection image, a first segmentation image, and a first classification result. The first detection image contains key point information of the body part, the first segmentation image is an image of the region where the body part is located, and the first classification result is used to characterize the position of the body part relative to the torso.

[0153] The first detection image mentioned above can be an image containing detected key points, for example, a key point heatmap of the foot key points of the detection path. Each key point heatmap contains information about only one key point; therefore, the number of key points to be detected can be preset according to the detection needs, thus obtaining the same number of key point heatmaps.

[0154] The first segmented image mentioned above can be an image obtained by segmenting the target image, containing only the target part and not other background images.

[0155] The first classification result mentioned above can refer to the side of the target body relative to its location. For example, it could be the left foot, the right foot, or both feet, thus avoiding confusion caused by the left foot being mistaken for the right foot. The target classification result can be represented by two binary digits, where the higher-order binary digit represents the left side and the lower-order binary digit represents the right side. A binary 1 indicates that the target body part was detected, and a binary 0 indicates that the target body part was not detected. For example, if the target image contains only the left foot, the target classification result is 10; if the target image contains only the right foot, the target classification result is 01; and if the target image contains both feet, the target classification result is 11.

[0156] Step S804: Train the multi-task model using training samples to obtain a part recognition model. The multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together.

[0157] The multi-task model in the above steps can be a model built using lightweight methods. This model can recognize training images and obtain various types of recognition results, namely detection results, segmentation results, and classification results. In this model, the first layer is a conventional two-dimensional convolutional module, and the rest are depthwise separable convolutional modules, which greatly reduces the number of model parameters. The model size is 1.8M. The specific structure of the conventional two-dimensional convolutional module can be two-dimensional convolution + Batch Normalization (BN) + nonlinear activation function; the depthwise separable convolutional module can be a basic module in MobileNetV3, and its specific structure can be depthwise separable convolution + residual model + Sequence Optimization (SE).

[0158] In the above embodiments of this application, training a multi-task model using training samples to obtain a part recognition model includes: using the multi-task model to recognize training images to obtain a second recognition result of the training images, wherein the training images are images collected from multiple different scenes; determining the model loss value of the multi-task model based on the first recognition result and the second recognition result; and training the multi-task model using training samples when the model loss value is greater than a preset value until the model loss value is less than or equal to the preset value to obtain the part recognition model.

[0159] The different scenarios in the above steps can refer to a variety of scenarios such as different states (e.g., barefoot, wearing socks, wearing shoes), different shoe types, different postures, different perspectives, occlusion, and lighting effects.

[0160] The preset values ​​in the above steps can be loss thresholds set according to the requirements of key point detection, region segmentation and classification, and can be modified according to the actual needs of users. This application does not make specific limitations on this.

[0161] In the above embodiments of this application, the second recognition result of the training image is obtained by recognizing the training image using a multi-task model, which includes: extracting features from the training image using an encoding network to obtain multi-scale features of the training image, wherein the multi-scale features include: features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; fusing the multi-scale features using a decoding network to obtain a second detection image and a second segmentation image, wherein the encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure; and classifying the smallest scale feature among the multi-scale features using a classifier to obtain a second classification result, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

[0162] In the above embodiments of this application, determining the model loss value of the multi-task model based on the first recognition result and the second recognition result includes: determining the detection loss value based on the first detection image and the second detection image; determining the segmentation loss value based on the first segmentation image and the second segmentation image; determining the classification loss value based on the first classification result and the second classification result; and obtaining the weighted sum of the detection loss value, the segmentation loss value and the classification loss value to obtain the model loss value.

[0163] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 3, but are not limited to the schemes provided in Embodiment 3.

[0164] Example 5

[0165] According to an embodiment of this application, a model training method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0166] Figure 9 This is a flowchart of yet another model training method according to an embodiment of this application. For example... Figure 9 As shown, the method may include the following steps:

[0167] Step S902: Obtain the model training request by calling the first interface, wherein the first interface contains a first parameter, and the parameter value of the first parameter is the model training request.

[0168] The first interface in the above steps can be an interface for data interaction between the server and the mobile terminal. The mobile terminal can pass the model training request to the interface function as a parameter of the interface function to achieve the purpose of uploading the model training request to the server.

[0169] The model training request in the above steps can be generated according to the user's model usage needs. For example, in a virtual shoe trying-on scenario, the model training request can be a training request for a foot recognition model.

[0170] Step S904: Obtain the training samples and multi-task model corresponding to the model training request. The training samples include: training images containing body parts and the first recognition result corresponding to the training images. The first recognition result is used to predict the position of the body parts. The multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together.

[0171] The training images in the above steps can refer to images or video frames segmented from images collected in various scenarios such as different states (e.g., barefoot, wearing socks, wearing shoes), different shoe types, different postures, different perspectives, occlusion, and lighting effects.

[0172] The multi-task model in the above steps can be a model built using lightweight methods. This model can recognize training images and obtain various types of recognition results, namely detection results, segmentation results, and classification results. In this model, the first layer is a conventional two-dimensional convolutional module, and the rest are depthwise separable convolutional modules, which greatly reduces the number of model parameters. The model size is 1.8M. The specific structure of the conventional two-dimensional convolutional module can be two-dimensional convolution + Batch Normalization (BN) + nonlinear activation function; the depthwise separable convolutional module can be a basic module in MobileNetV3, and its specific structure can be depthwise separable convolution + residual model + Sequence Optimization (SE).

[0173] Optionally, the first recognition result mentioned above includes: a first detection image, a first segmentation image, and a first classification result. The first detection image contains key point information of the body part, the first segmentation image is an image of the region where the body part is located, and the first classification result is used to characterize the position of the body part relative to the torso.

[0174] The first detection image mentioned above can be an image containing detected key points, for example, a key point heatmap of the foot key points of the detection path. Each key point heatmap contains information about only one key point; therefore, the number of key points to be detected can be preset according to the detection needs, thus obtaining the same number of key point heatmaps.

[0175] The first segmented image mentioned above can be an image obtained by segmenting the target image, containing only the target part and not other background images.

[0176] The first classification result mentioned above can refer to the side of the target body relative to its location. For example, it could be the left foot, the right foot, or both feet, thus avoiding confusion caused by the left foot being mistaken for the right foot. The target classification result can be represented by two binary digits, where the higher-order binary digit represents the left side and the lower-order binary digit represents the right side. A binary 1 indicates that the target body part was detected, and a binary 0 indicates that the target body part was not detected. For example, if the target image contains only the left foot, the target classification result is 10; if the target image contains only the right foot, the target classification result is 01; and if the target image contains both feet, the target classification result is 11.

[0177] Step S906: Train the multi-task model using training samples to obtain the part recognition model.

[0178] Step S908: Output the body part recognition model by calling the second interface, wherein the second interface contains a second parameter, and the parameter value of the second parameter is the body part recognition model.

[0179] The second interface in the above steps can be an interface for data interaction between the server and the mobile terminal. The server can pass the part recognition model into the interface function as a parameter of the interface function to achieve the purpose of sending the part recognition model to the mobile terminal.

[0180] In the above embodiments of this application, training a multi-task model using training samples to obtain a part recognition model includes: using the multi-task model to recognize training images to obtain a second recognition result of the training images, wherein the training images are images collected from multiple different scenes; determining the model loss value of the multi-task model based on the first recognition result and the second recognition result; and training the multi-task model using training samples when the model loss value is greater than a preset value until the model loss value is less than or equal to the preset value to obtain the part recognition model.

[0181] The different scenarios in the above steps can refer to a variety of scenarios such as different states (e.g., barefoot, wearing socks, wearing shoes), different shoe types, different postures, different perspectives, occlusion, and lighting effects.

[0182] The preset values ​​in the above steps can be loss thresholds set according to the requirements of key point detection, region segmentation and classification, and can be modified according to the actual needs of users. This application does not make specific limitations on this.

[0183] In the above embodiments of this application, the second recognition result of the training image is obtained by recognizing the training image using a multi-task model, which includes: extracting features from the training image using an encoding network to obtain multi-scale features of the training image, wherein the multi-scale features include: features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; fusing the multi-scale features using a decoding network to obtain a second detection image and a second segmentation image, wherein the encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure; and classifying the smallest scale feature among the multi-scale features using a classifier to obtain a second classification result, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

[0184] In the above embodiments of this application, determining the model loss value of the multi-task model based on the first recognition result and the second recognition result includes: determining the detection loss value based on the first detection image and the second detection image; determining the segmentation loss value based on the first segmentation image and the second segmentation image; determining the classification loss value based on the first classification result and the second classification result; and obtaining the weighted sum of the detection loss value, the segmentation loss value and the classification loss value to obtain the model loss value.

[0185] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 3, but are not limited to the schemes provided in Embodiment 3.

[0186] Example 6

[0187] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided, such as... Figure 10 As shown, the device 1000 includes: a receiving module 1002, an identification module 1004, and an output module 1006.

[0188] The receiving module 1002 is used to receive the target image, which includes a target part; the recognition module 1004 is used to recognize the target image using a part recognition model to obtain the target recognition result of the target part, wherein the target recognition result is used to predict the position of the target part, and the part recognition model is composed of a two-dimensional convolution module and multiple depth-separable convolution modules connected together; the output module 1006 is used to output the target recognition result.

[0189] It should be noted that the receiving module 1002, the identification module 1004, and the output module 1006 mentioned above correspond to steps S202 to S206 in Embodiment 1. The three modules and their corresponding steps implement the same examples and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0190] In the above embodiments of this application, the identification module 1004 includes: an extraction unit, a fusion unit, and a classification unit.

[0191] The extraction unit uses an encoding network to extract features from the target image, obtaining multi-scale features of the target image. These multi-scale features include features at multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together. The fusion unit uses a decoding network to fuse the multi-scale features, obtaining a target detection image and a target segmentation image. The encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure. The classification unit uses a classifier to perform a classification operation on the smallest scale feature among the multi-scale features, obtaining a target classification result. The classifier is connected to the first network structure corresponding to the smallest scale feature.

[0192] In the above embodiments of this application, the device may further include: an acquisition module and a training module.

[0193] The acquisition module is used to acquire training samples, which include training images and the first recognition result corresponding to the training images. The training images are images collected from multiple different scenes. The training module is used to train the multi-task model using the training samples to obtain the part recognition model.

[0194] In the above embodiments of this application, the training module includes: an identification unit, a determination unit, and a training unit.

[0195] The recognition unit is used to recognize the training image using the multi-task model to obtain the second recognition result of the training image; the determination unit is used to determine the model loss value of the multi-task model based on the first recognition result and the second recognition result; the training unit is used to train the multi-task model using training samples when the model loss value is greater than a preset value, until the model loss value is less than or equal to the preset value, and obtain the part recognition model.

[0196] In the above embodiments of this application, the determining unit is further configured to perform the following steps: determining a detection loss value based on a first detection image and a second detection image; determining a segmentation loss value based on a first segmentation image and a second segmentation image; determining a classification loss value based on a first classification result and a second classification result; and obtaining a weighted sum of the detection loss value, the segmentation loss value, and the classification loss value to obtain a model loss value.

[0197] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0198] Example 7

[0199] According to embodiments of this application, an image processing apparatus for implementing the above-described image processing method is also provided, such as... Figure 11 As shown, the device 1100 includes: an acquisition module 1102 and an identification module 1104.

[0200] The acquisition module 1102 is used to acquire a target image, wherein the target image includes a target part; the recognition module 1104 is used to recognize the target image using a part recognition model to obtain the target recognition result of the target part, wherein the target recognition result is used to predict the position of the target part, and the part recognition model is composed of a two-dimensional convolution module and multiple depth-separable convolution modules connected together.

[0201] It should be noted that the acquisition module 1102 and the identification module 1104 mentioned above correspond to steps S602 to S604 in Embodiment 2. The two modules and the corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 1. It should be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0202] In the above embodiments of this application, the identification module 1104 includes: an extraction unit, a fusion unit, and a classification unit.

[0203] The extraction unit uses an encoding network to extract features from the target image, obtaining multi-scale features of the target image. These multi-scale features include features at multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together. The fusion unit uses a decoding network to fuse the multi-scale features, obtaining a target detection image and a target segmentation image. The encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure. The classification unit uses a classifier to perform a classification operation on the smallest scale feature among the multi-scale features, obtaining a target classification result. The classifier is connected to the first network structure corresponding to the smallest scale feature.

[0204] In the above embodiments of this application, the device may further include a training module.

[0205] The acquisition module is also used to acquire training samples, which include training images and the first recognition result corresponding to the training images. The training images are images collected from multiple different scenes. The training module is used to train the multi-task model using the training samples to obtain the part recognition model.

[0206] In the above embodiments of this application, the training module includes: an identification unit, a determination unit, and a training unit.

[0207] The recognition unit is used to recognize the training image using the multi-task model to obtain the second recognition result of the training image; the determination unit is used to determine the model loss value of the multi-task model based on the first recognition result and the second recognition result; the training unit is used to train the multi-task model using training samples when the model loss value is greater than a preset value, until the model loss value is less than or equal to the preset value, and obtain the part recognition model.

[0208] In the above embodiments of this application, the determining unit is further configured to perform the following steps: determining a detection loss value based on a first detection image and a second detection image; determining a segmentation loss value based on a first segmentation image and a second segmentation image; determining a classification loss value based on a first classification result and a second classification result; and obtaining a weighted sum of the detection loss value, the segmentation loss value, and the classification loss value to obtain a model loss value.

[0209] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0210] Example 8

[0211] According to embodiments of this application, a model training apparatus for implementing the above-described model training method is also provided, such as... Figure 12 As shown, the device 1200 includes: a receiving module 1202, an acquisition module 1204, a training module 1206, and an output module 1208.

[0212] The receiving module 1202 is used to receive model training requests; the acquiring module 1204 is used to acquire training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; the training module 1206 is used to train the multi-task model using the training samples to obtain a body part recognition model; and the output module 1208 is used to output the body part recognition model.

[0213] It should be noted that the receiving module 1202, acquiring module 1204, training module 1206, and output module 1208 mentioned above correspond to steps S702 to S708 in Embodiment 3. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 3. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0214] In the above embodiments of this application, the training module 1206 includes: an identification unit, a determination unit, and a training unit.

[0215] The recognition unit is used to recognize the training images using a multi-task model to obtain a second recognition result of the training images, wherein the training images are images collected from multiple different scenes; the determination unit is used to determine the model loss value of the multi-task model based on the first recognition result and the second recognition result; the training unit is used to train the multi-task model using training samples when the model loss value is greater than a preset value, until the model loss value is less than or equal to the preset value, thereby obtaining a part recognition model.

[0216] In the above embodiments of this application, the identification unit includes: an extraction subunit, a fusion subunit, and a classification subunit.

[0217] The extraction subunit is used to extract features from the training image using an encoding network to obtain multi-scale features of the training image. The multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together. The fusion subunit is used to fuse the multi-scale features using a decoding network to obtain a second detection image and a second segmentation image. The encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure. The classification subunit is used to classify the smallest scale feature among the multi-scale features using a classifier to obtain a second classification result. The classifier is connected to the first network structure corresponding to the smallest scale feature.

[0218] In the above embodiments of this application, the determining unit is further configured to perform the following steps: determining a detection loss value based on a first detection image and a second detection image; determining a segmentation loss value based on a first segmentation image and a second segmentation image; determining a classification loss value based on a first classification result and a second classification result; and obtaining a weighted sum of the detection loss value, the segmentation loss value, and the classification loss value to obtain a model loss value.

[0219] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 3, but are not limited to the schemes provided in Embodiment 3.

[0220] Example 9

[0221] According to embodiments of this application, a model training apparatus for implementing the above-described model training method is also provided, such as... Figure 13 As shown, the device 1300 includes an acquisition module 1302 and a training module 1304.

[0222] The acquisition module 1302 is used to acquire training samples, which include training images containing body parts and a first recognition result corresponding to the training images. The first recognition result is used to predict the position of the body parts. The training module 1304 is used to train the multi-task model using the training samples to obtain a body part recognition model. The multi-task model is composed of a two-dimensional convolution module and multiple depthwise separable convolution modules connected together.

[0223] It should be noted that the acquisition module 1302 and training module 1304 mentioned above correspond to steps S802 to S804 in Embodiment 4. The two modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 4. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0224] In the above embodiments of this application, the training module 1304 includes: an identification unit, a determination unit, and a training unit.

[0225] The recognition unit is used to recognize the training images using a multi-task model to obtain a second recognition result of the training images, wherein the training images are images collected from multiple different scenes; the determination unit is used to determine the model loss value of the multi-task model based on the first recognition result and the second recognition result; the training unit is used to train the multi-task model using training samples when the model loss value is greater than a preset value, until the model loss value is less than or equal to the preset value, thereby obtaining a part recognition model.

[0226] In the above embodiments of this application, the identification unit includes: an extraction subunit, a fusion subunit, and a classification subunit.

[0227] The extraction subunit is used to extract features from the training image using an encoding network to obtain multi-scale features of the training image. The multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together. The fusion subunit is used to fuse the multi-scale features using a decoding network to obtain a second detection image and a second segmentation image. The encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure. The classification subunit is used to classify the smallest scale feature among the multi-scale features using a classifier to obtain a second classification result. The classifier is connected to the first network structure corresponding to the smallest scale feature.

[0228] In the above embodiments of this application, the determining unit is further configured to perform the following steps: determining a detection loss value based on a first detection image and a second detection image; determining a segmentation loss value based on a first segmentation image and a second segmentation image; determining a classification loss value based on a first classification result and a second classification result; and obtaining a weighted sum of the detection loss value, the segmentation loss value, and the classification loss value to obtain a model loss value.

[0229] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 3, but are not limited to the schemes provided in Embodiment 3.

[0230] Example 10

[0231] According to embodiments of this application, a model training apparatus for implementing the above-described model training method is also provided, such as... Figure 14 As shown, the device 1400 includes: a first invocation module 1402, an acquisition module 1404, a training module 1406, and a second invocation module 1408.

[0232] The first calling module 1402 is used to obtain a model training request by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the model training request; the obtaining module 1404 is used to obtain the training samples and multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; the training module 1406 is used to train the multi-task model using the training samples to obtain a body part recognition model; the second calling module 1408 is used to output the body part recognition model by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the body part recognition model.

[0233] It should be noted that the first calling module 1402, the acquisition module 1404, the training module 1406, and the second calling module 1408 mentioned above correspond to steps S902 to S908 in Embodiment 5. The four modules and their corresponding steps implement the same instances and application scenarios, but are not limited to the content disclosed in Embodiment 3. It should also be noted that the above modules, as part of the device, can run in the computer terminal 10 provided in Embodiment 1.

[0234] In the above embodiments of this application, the training module 1406 includes: an identification unit, a determination unit, and a training unit.

[0235] The recognition unit is used to recognize the training images using a multi-task model to obtain a second recognition result of the training images, wherein the training images are images collected from multiple different scenes; the determination unit is used to determine the model loss value of the multi-task model based on the first recognition result and the second recognition result; the training unit is used to train the multi-task model using training samples when the model loss value is greater than a preset value, until the model loss value is less than or equal to the preset value, thereby obtaining a part recognition model.

[0236] In the above embodiments of this application, the identification unit includes: an extraction subunit, a fusion subunit, and a classification subunit.

[0237] The extraction subunit is used to extract features from the training image using an encoding network to obtain multi-scale features of the training image. The multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together. The fusion subunit is used to fuse the multi-scale features using a decoding network to obtain a second detection image and a second segmentation image. The encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure. The classification subunit is used to classify the smallest scale feature among the multi-scale features using a classifier to obtain a second classification result. The classifier is connected to the first network structure corresponding to the smallest scale feature.

[0238] In the above embodiments of this application, the determining unit is further configured to perform the following steps: determining a detection loss value based on a first detection image and a second detection image; determining a segmentation loss value based on a first segmentation image and a second segmentation image; determining a classification loss value based on a first classification result and a second classification result; and obtaining a weighted sum of the detection loss value, the segmentation loss value, and the classification loss value to obtain a model loss value.

[0239] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 3, but are not limited to the schemes provided in Embodiment 3.

[0240] Example 11

[0241] According to an embodiment of this application, an image processing system is also provided, comprising:

[0242] Processor; and

[0243] The memory, connected to the processor, is used to provide the processor with instructions to perform the following processing steps: receiving a target image, wherein the target image includes a target region; recognizing the target image using a region recognition model to obtain a target recognition result for the target region, wherein the target recognition result is used to predict the position of the target region, and the region recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; and outputting the target recognition result.

[0244] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 1, but are not limited to the schemes provided in Embodiment 1.

[0245] Example 12

[0246] According to an embodiment of this application, a model training system is also provided, comprising:

[0247] Processor; and

[0248] The memory, connected to the processor, is used to provide the processor with instructions to perform the following processing steps: receiving a model training request; obtaining training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; training the multi-task model using the training samples to obtain a body part recognition model; and outputting the body part recognition model.

[0249] It should be noted that the preferred implementation schemes involved in the above embodiments of this application are the same as the schemes, application scenarios and implementation processes provided in Embodiment 3, but are not limited to the schemes provided in Embodiment 3.

[0250] Example 13

[0251] According to an embodiment of this application, a clothing matching method is also provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0252] Figure 15This is a flowchart of a clothing matching method according to an embodiment of this application. Figure 15 As shown, the method may include the following steps:

[0253] Step S1502: Receive the target image and the target clothing, wherein the target image includes the target body part.

[0254] The target image in the above steps can be an image taken by the user targeting the target area, or it can be an image frame segmented from a video taken by the user targeting the target area. The target area can be a part of the user's body, such as a hand, foot, face, ear, etc. In this embodiment, the foot is used as an example for explanation. For body parts such as hands and feet, the target image can include only one side (e.g., left hand, right hand, left foot, right foot) or both sides (e.g., left and right hands, left and right feet).

[0255] The target clothing in the above steps can be clothing selected by the user on an e-commerce shopping platform. In order to see the fitting effect, the user can choose the virtual fitting function to simulate the fitting. For different virtual fitting functions, the types of clothing are different. For example, in the virtual shoe fitting function, the target clothing can be shoes of different types and styles.

[0256] Step S1504: The target image is identified using the part recognition model to obtain the target recognition result of the target part. The part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together.

[0257] Optionally, the target recognition results mentioned above include: target detection image, target segmentation image, and target classification result. The target detection image contains key point information of the target part, the target segmentation image is an image of the region where the target part is located, and the target classification result is used to characterize the position of the target part relative to the body trunk.

[0258] The target detection image mentioned above can be an image containing detected key points, for example, a key point heatmap of the foot key points of a detection path. Each key point heatmap contains information about only one key point; therefore, the number of key points to be detected can be pre-set according to the detection needs, thus obtaining the same number of key point heatmaps.

[0259] The aforementioned target segmentation image can be an image obtained by segmenting a target image, containing only the target region and excluding other background images.

[0260] The target classification result mentioned above can refer to the side of the target body relative to its location. For example, it could be the left foot, the right foot, or both feet, thus avoiding confusion caused by the left foot being mistaken for the right foot. The target classification result can be represented by two binary digits, where the higher-order binary digit represents the left side and the lower-order binary digit represents the right side. A binary 1 indicates that the target body part was detected, and a binary 0 indicates that the target body part was not detected. For example, if the target image contains only the left foot, the target classification result is 10; if the target image contains only the right foot, the target classification result is 01; and if the target image contains both feet, the target classification result is 11.

[0261] The part recognition model in the above steps can be a pre-trained multi-task model that uses a lightweight method to recognize target images, obtaining multiple target recognition results: detection results, segmentation results, and classification results. In this model, the first layer is a conventional 2D convolutional module, and the rest are depthwise separable convolutional modules, which greatly reduces the number of model parameters. The model size is 1.8M. The specific structure of the conventional 2D convolutional module can be 2D convolution + Batch Normalization (BN) + nonlinear activation function; the depthwise separable convolutional module can be a basic module in MobileNetV3, and its specific structure can be depthwise separable convolution + residual model + Sequence Optimization (SE).

[0262] Step S1506: Based on the target recognition results, generate virtual clothing corresponding to the target clothing.

[0263] In one alternative embodiment, after identifying the key point heatmap and classification of the target part, six-dimensional pose estimation can be performed based on the key point heatmap and classification, and then virtual clothing can be drawn according to the pose and the target clothing selected by the user. For example, for the foot, after obtaining the foot key points and the side where the foot is located, six-dimensional pose estimation can be performed, and then combined with the style of the shoes selected by the user, virtual shoes can be drawn according to the pose.

[0264] Step S1508: Use virtual clothing to occlude the target area and generate a matching image.

[0265] In one optional embodiment, after generating the virtual clothing, the target area can be occluded based on the segmentation results using the virtual clothing to obtain a try-on image, i.e., the matching image mentioned above. For example, for the feet, after drawing the virtual shoes, the virtual shoes can be occluded based on the foot image to achieve a realistic try-on experience.

[0266] Step S1510: Output the matching image.

[0267] The technical solution provided by the above embodiments of this application allows for the identification of the target image and target clothing after receiving the target image and target clothing. A part recognition model can be used to identify the target image, obtaining the target recognition result of the target part. Based on the target recognition result, a virtual garment corresponding to the target clothing is generated. Furthermore, the virtual garment is used to occlude the target part, generating a matching image to achieve the purpose of virtual clothing try-on. It is noteworthy that the part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together, thereby greatly reducing the number of model parameters. This allows the mobile terminal's operating speed to meet the real-time requirements of image processing, achieving the technical effect of reducing the model's memory footprint on the mobile terminal and improving processing speed. This solves the technical problem in related technologies where large models are used for data processing, resulting in high memory consumption and slow processing speed.

[0268] It should be noted that the application method of the part recognition model trained in this embodiment can refer to the image processing method in the above embodiment. The specific implementation scheme and application scenario are the same as those in the following embodiment, and will not be repeated here.

[0269] Example 14

[0270] Embodiments of this application may provide a computer terminal, which may be any computer terminal device in a group of computer terminals. Optionally, in this embodiment, the aforementioned computer terminal may also be replaced by a mobile terminal or other terminal device.

[0271] Optionally, in this embodiment, the computer terminal may be located in at least one of a plurality of network devices in a computer network.

[0272] In this embodiment, the computer terminal described above can execute the program code for the following steps in the image processing method: receiving a target image, wherein the target image includes a target part; using a part recognition model to recognize the target image and obtain a target recognition result for the target part, wherein the target recognition result is used to predict the position of the target part, and the part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together; and outputting the target recognition result.

[0273] Optionally, Figure 16 This is a structural block diagram of a computer terminal according to an embodiment of this application. Figure 16 As shown, the computer terminal A may include one or more (only one is shown in the figure) processors 1602 and memory 1604.

[0274] The memory can be used to store software programs and modules, such as the program instructions / modules corresponding to the image processing method and apparatus, model training method and apparatus, and clothing matching method in the embodiments of this application. The processor executes various functional applications and data processing by running the software programs and modules stored in the memory, thereby realizing the aforementioned image processing method, model training method, and clothing matching method. The memory may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory may further include memory remotely located relative to the processor, and these remote memories can be connected to terminal A via a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0275] The processor can call the information and application program stored in the memory through the transmission device to perform the following steps: receiving a target image, wherein the target image includes a target part; using a part recognition model to recognize the target image and obtain the target recognition result of the target part, wherein the target recognition result is used to predict the position of the target part, and the part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together; and outputting the target recognition result.

[0276] Optionally, the processor may also execute program code for the following steps: extracting features from the target image using an encoding network to obtain multi-scale features of the target image, wherein the multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; fusing the multi-scale features using a decoding network to obtain a target detection image and a target segmentation image, wherein the encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure; and performing a classification operation on the smallest scale feature among the multi-scale features using a classifier to obtain a target classification result, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

[0277] Optionally, the processor may also execute program code for the following steps: acquiring training samples, wherein the training samples include: training images and the first recognition result corresponding to the training images, the training images being images collected from multiple different scenes; and training a multi-task model using the training samples to obtain a part recognition model.

[0278] Optionally, the processor may also execute program code for the following steps: using a multi-task model to recognize the training image and obtain a second recognition result of the training image; based on the first recognition result and the second recognition result, determining the model loss value of the multi-task model; if the model loss value is greater than a preset value, training the multi-task model using training samples until the model loss value is less than or equal to the preset value, thereby obtaining a part recognition model.

[0279] Optionally, the processor may also execute program code for the following steps: determining a detection loss value based on a first detection image and a second detection image; determining a segmentation loss value based on a first segmentation image and a second segmentation image; determining a classification loss value based on a first classification result and a second classification result; and obtaining a weighted sum of the detection loss value, the segmentation loss value, and the classification loss value to obtain the model loss value.

[0280] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: acquire a target image, wherein the target image includes a target part; use a part recognition model to recognize the target image and obtain the target recognition result of the target part, wherein the target recognition result is used to predict the position of the target part, and the part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together.

[0281] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: receiving a model training request; obtaining training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; training the multi-task model using the training samples to obtain a body part recognition model; and outputting the body part recognition model.

[0282] The processor can call the information and application stored in the memory through the transmission device to perform the following steps: acquiring training samples, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts; training a multi-task model using the training samples to obtain a body part recognition model, wherein the multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together.

[0283] The processor can invoke information and application programs stored in the memory via a transmission device to perform the following steps: obtaining a model training request by calling a first interface, wherein the first interface includes a first parameter, the value of which is the model training request; obtaining training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; training the multi-task model using the training samples to obtain a body part recognition model; and outputting the body part recognition model by calling a second interface, wherein the second interface includes a second parameter, the value of which is the body part recognition model.

[0284] The processor can access information and applications stored in the memory via a transmission device to perform the following steps: receiving a target image and a target garment, wherein the target image includes a target body part; recognizing the target image using a body part recognition model to obtain a target recognition result for the target body part, wherein the body part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; generating a virtual garment corresponding to the target garment based on the target recognition result; using the virtual garment to occlude the target body part to generate a matching image; and outputting the matching image.

[0285] This application provides an image processing solution. A part recognition model, composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules, is used to identify the target image and obtain the target part recognition result. This significantly reduces the number of model parameters, allowing the mobile terminal's operating speed to meet the real-time requirements of image processing. It achieves the technical effect of reducing the model's memory footprint on the mobile terminal and improving processing speed, thereby solving the technical problem in related technologies where large models are used for data processing, resulting in high memory consumption and slow processing speed.

[0286] Those skilled in the art will understand that Figure 16 The structure shown is for illustrative purposes only. The computer terminal can also be a smartphone (such as an Android phone, an iOS phone, etc.), a tablet computer, a PDA, a mobile Internet device (MID), a PAD, and other terminal devices. Figure 16 This does not limit the structure of the aforementioned electronic device. For example, computer terminal A may also include components that are more... Figure 16 The more or fewer components shown (such as network interfaces, display devices, etc.), or having the same Figure 16 The different configurations shown.

[0287] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by a program instructing the hardware related to the terminal device. The program can be stored in a computer-readable storage medium, which may include: flash drive, read-only memory (ROM), random access memory (RAM), disk or optical disk, etc.

[0288] Example 14

[0289] Embodiments of this application also provide a storage medium. Optionally, in this embodiment, the storage medium can be used to store the program code executed by the image processing method, model training method, and clothing matching method provided in the above embodiments.

[0290] Optionally, in this embodiment, the storage medium may be located in any computer terminal in a group of computer terminals in a computer network, or in any mobile terminal in a group of mobile terminals.

[0291] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving a target image, wherein the target image includes a target part; using a part recognition model to recognize the target image and obtain a target recognition result for the target part, wherein the target recognition result is used to predict the position of the target part, and the part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together; and outputting the target recognition result.

[0292] Optionally, the aforementioned storage medium is further configured to store program code for performing the following steps: extracting features from the target image using an encoding network to obtain multi-scale features of the target image, wherein the multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; fusing the multi-scale features using a decoding network to obtain a target detection image and a target segmentation image, wherein the encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure; and performing a classification operation on the smallest scale feature among the multi-scale features using a classifier to obtain a target classification result, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

[0293] Optionally, the storage medium is further configured to store program code for performing the following steps: acquiring training samples, wherein the training samples include: training images and a first recognition result corresponding to the training images, the training images being images collected from multiple different scenes; and training a multi-task model using the training samples to obtain a part recognition model.

[0294] Optionally, the storage medium is further configured to store program code for performing the following steps: using a multi-task model to recognize the training image to obtain a second recognition result of the training image; based on the first recognition result and the second recognition result, determining the model loss value of the multi-task model; if the model loss value is greater than a preset value, training the multi-task model using training samples until the model loss value is less than or equal to the preset value, thereby obtaining a part recognition model.

[0295] Optionally, the storage medium is further configured to store program code for performing the following steps: determining a detection loss value based on a first detection image and a second detection image; determining a segmentation loss value based on a first segmentation image and a second segmentation image; determining a classification loss value based on a first classification result and a second classification result; and obtaining a weighted sum of the detection loss value, the segmentation loss value, and the classification loss value to obtain a model loss value.

[0296] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: acquiring a target image, wherein the target image includes a target part; using a part recognition model to recognize the target image and obtain a target recognition result for the target part, wherein the target recognition result is used to predict the position of the target part, and the part recognition model is composed of a two-dimensional convolutional module and multiple depth-separable convolutional modules connected together.

[0297] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving a model training request; obtaining training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; training the multi-task model using the training samples to obtain a body part recognition model; and outputting the body part recognition model.

[0298] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining training samples, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts; training a multi-task model using the training samples to obtain a body part recognition model, wherein the multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together.

[0299] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: obtaining a model training request by calling a first interface, wherein the first interface includes a first parameter, and the parameter value of the first parameter is the model training request; obtaining training samples and a multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; training the multi-task model using the training samples to obtain a body part recognition model; and outputting the body part recognition model by calling a second interface, wherein the second interface includes a second parameter, and the parameter value of the second parameter is the body part recognition model.

[0300] Optionally, in this embodiment, the storage medium is configured to store program code for performing the following steps: receiving a target image and a target garment, wherein the target image includes a target part; using a part recognition model to recognize the target image and obtain a target recognition result for the target part, wherein the part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together; generating a virtual garment corresponding to the target garment based on the target recognition result; using the virtual garment to occlude the target part and generate a matching image; and outputting the matching image.

[0301] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0302] In the above embodiments of this application, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0303] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling, direct coupling, or communication connection may be through some interfaces; the indirect coupling or communication connection between units or modules may be electrical or other forms.

[0304] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0305] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0306] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as a USB flash drive, read-only memory (ROM), random access memory (RAM), portable hard drive, magnetic disk, or optical disk.

[0307] The above description is only a preferred embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. An image processing method, comprising: Receive a target image, wherein the target image includes: a target region; The target image is identified using a part recognition model to obtain the target part recognition result. The target recognition result is used to predict the position of the target part. The part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together. The part recognition model is a multi-task model trained using training samples. The training samples include: training images and a first recognition result corresponding to the training images. The training images are images collected from multiple different scenes. Output the target recognition result.

2. The method according to claim 1, wherein, The target recognition result includes: target detection image, target segmentation image and target classification result. The target detection image contains key point information of the target part. The target segmentation image is an image of the region where the target part is located. The target classification result is used to characterize the position of the target part relative to the body trunk.

3. The method according to claim 2, wherein, The target image is identified using a part recognition model, and the target part recognition result is obtained as follows: The target image is feature extracted using an encoding network to obtain multi-scale features of the target image, wherein the multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; The multi-scale features are fused using a decoding network to obtain the target detection image and the target segmentation image. The encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure. The target classification result is obtained by classifying the smallest scale feature among the multi-scale features using a classifier, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

4. The method according to any one of claims 1 to 3, wherein, The method further includes: Obtain training samples; The multi-task model is trained using the training samples to obtain the part recognition model.

5. The method according to claim 4, wherein, The multi-task model is trained using the training samples to obtain the part recognition model, which includes: The training image is identified using the multi-task model to obtain a second recognition result for the training image; Based on the first identification result and the second identification result, the model loss value of the multi-task model is determined; If the model loss value is greater than a preset value, the multi-task model is trained using the training samples until the model loss value is less than or equal to the preset value, thus obtaining the part recognition model.

6. The method according to claim 5, wherein, Based on the first identification result and the second identification result, the model loss value of the multi-task model is determined as follows: Based on the first detection image in the first recognition result and the second detection image in the second recognition result, a detection loss value is determined; Based on the first segmented image in the first recognition result and the second segmented image in the second recognition result, a segmentation loss value is determined; Based on the first classification result in the first identification result and the second classification result in the second identification result, the classification loss value is determined; The model loss value is obtained by weighting the detection loss value, the segmentation loss value, and the classification loss value.

7. An image processing method, comprising: Acquire a target image, wherein the target image includes: a target region; The target image is identified using a part recognition model to obtain the target part recognition result. The target recognition result is used to predict the position of the target part. The part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together. The part recognition model is a multi-task model trained using training samples. The training samples include: training images and the first recognition result corresponding to the training images. The training images are images collected from multiple different scenes.

8. The method according to claim 7, wherein, The target recognition result includes: target detection image, target segmentation image and target classification result. The target detection image contains key point information of the target part. The target segmentation image is an image of the region where the target part is located. The target classification result is used to characterize the position of the target part relative to the body trunk.

9. The method according to claim 8, wherein, The target image is identified using a part recognition model, and the target part recognition result is obtained as follows: The target image is feature extracted using an encoding network to obtain multi-scale features of the target image, wherein the multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together; The multi-scale features are fused using a decoding network to obtain the target detection image and the target segmentation image. The encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure. The target classification result is obtained by classifying the smallest scale feature among the multi-scale features using a classifier, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

10. A model training method, comprising: Receive model training requests; Obtain the training samples and multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together, and the training images being images collected from multiple different scenes; The multi-task model is trained using the training samples to obtain a part recognition model; Output the location recognition model.

11. The method according to claim 10, wherein, The first recognition result includes: a first detection image, a first segmentation image, and a first classification result. The first detection image contains key point information of the body part, the first segmentation image is an image of the region where the body part is located, and the first classification result is used to characterize the position of the body part relative to the torso.

12. The method according to claim 11, wherein, The multi-task model is trained using the training samples to obtain a part recognition model, including: The training image is identified using the multi-task model to obtain a second recognition result for the training image; Based on the first identification result and the second identification result, the model loss value of the multi-task model is determined; If the model loss value is greater than a preset value, the multi-task model is trained using the training samples until the model loss value is less than or equal to the preset value, thus obtaining the part recognition model.

13. The method according to claim 12, wherein, The second recognition result of the training image obtained by using the multi-task model to recognize the training image includes: The training image is used to extract features using an encoding network to obtain multi-scale features of the training image, wherein the multi-scale features include features of multiple different scales, and the encoding network is composed of multiple first network structures of different scales connected together. The multi-scale features are fused using a decoding network to obtain a second detection image and a second segmentation image. The encoding network is composed of multiple second network structures of different scales connected together, and each second network structure is connected to a corresponding first network structure. A classifier is used to classify the smallest scale feature among the multi-scale features to obtain a second classification result, wherein the classifier is connected to the first network structure corresponding to the smallest scale feature.

14. The method according to claim 13, wherein, Based on the first identification result and the second identification result, the model loss value of the multi-task model is determined as follows: Based on the first detected image and the second detected image, a detection loss value is determined; Based on the first segmented image and the second segmented image, determine the segmentation loss value; Based on the first classification result and the second classification result, determine the classification loss value; The model loss value is obtained by weighting the detection loss value, the segmentation loss value, and the classification loss value.

15. A model training method, comprising: Acquire training samples, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, and the training images being images collected from multiple different scenes; The multi-task model is trained using the training samples to obtain a part recognition model, wherein the multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together.

16. The method according to claim 15, wherein, The first recognition result includes: a first detection image, a first segmentation image, and a first classification result. The first detection image contains key point information of the body part, the first segmentation image is an image of the region where the body part is located, and the first classification result is used to characterize the position of the body part relative to the torso.

17. A model training method, comprising: A model training request is obtained by calling a first interface, wherein the first interface contains a first parameter, and the value of the first parameter is the model training request; Obtain the training samples and multi-task model corresponding to the model training request, wherein the training samples include: training images containing body parts, and a first recognition result corresponding to the training images, the first recognition result being used to predict the position of the body parts, the multi-task model being composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together, and the training images being images collected from multiple different scenes; The multi-task model is trained using the training samples to obtain a part recognition model; The location recognition model is output by calling the second interface, wherein the second interface contains a second parameter, and the parameter value of the second parameter is the location recognition model.

18. A computer-readable storage medium comprising a stored program, wherein, When the program is running, it controls the device containing the computer-readable storage medium to perform the image processing method according to any one of claims 1 to 9, or the model training method according to any one of claims 10 to 17.

19. A computer terminal, comprising: A memory and a processor, the processor being configured to run a program stored in the memory, wherein the program, when running, performs the image processing method according to any one of claims 1 to 9, or the model training method according to any one of claims 10 to 17.

20. An image processing system, comprising: processor; as well as A memory, connected to the processor, is used to provide the processor with instructions to perform the following processing steps: receiving a target image, wherein the target image includes a target part; recognizing the target image using a part recognition model to obtain a target recognition result for the target part, wherein the target recognition result is used to predict the position of the target part, the part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together, the part recognition model is a multi-task model trained using training samples, the training samples include: training images and a first recognition result corresponding to the training images, the training images being images collected from multiple different scenes; and outputting the target recognition result.

21. A model training system, comprising: processor; as well as A memory, connected to the processor, is used to provide the processor with instructions to perform the following processing steps: receiving a model training request; Obtain the training samples and multi-task model corresponding to the model training request. The training samples include: training images containing body parts and a first recognition result corresponding to the training images. The training images are images collected from multiple different scenes. The first recognition result is used to predict the position of the body parts. The multi-task model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together. Train the multi-task model using the training samples to obtain a body part recognition model. Output the body part recognition model.

22. A clothing matching method, comprising: Receive a target image and a target garment, wherein the target image includes: a target body part; The target image is identified using a part recognition model to obtain the target recognition result of the target part. The part recognition model is composed of a two-dimensional convolutional module and multiple depthwise separable convolutional modules connected together. The part recognition model is a multi-task model trained using training samples. The training samples include: training images and the first recognition result corresponding to the training images. The training images are images collected from multiple different scenes. Based on the target recognition results, a virtual outfit corresponding to the target outfit is generated; The target area is occluded using the virtual clothing to generate a matching image; Output the matched image.

Images