Model training method and device, image processing method and device, electronic equipment, storage medium and program product

Abstract

The invention provides a model training method and device, an image processing method and device, electronic equipment, a storage medium and a program product. The method comprises the following steps: based on images of a plurality of first parts of a first object in a first image, performing image reconstruction through a model to obtain images of a plurality of second parts; combining the images of the plurality of second parts to obtain a second image; segmenting a third image to obtain images of a plurality of third parts of a third object, the third image being a reference image; and training the model based on the difference between the first image and the second image and the difference between the images of the plurality of third parts and the images of the plurality of second parts. According to the invention, the training burden of the model can be reduced, and the image processing effect of the trained model can be improved.

Description

Technical Field

[0001] The present application relates to the field of deep learning technology, and in particular to a model training method, image processing method, device, electronic device, storage medium and program product. Background Art

[0002] There is a lack of data sets for processing human body weight loss in related technologies, and the cost of labeling data is too high. Optical flow algorithms are also used, but this method itself requires labels. Because the amount of training set data is small, the trained model has no obvious weight loss effect, or is easily distorted. It also requires a lot of computing resources, especially when processing high-resolution images, which can easily lead to long running times and high hardware requirements. Summary of the invention

[0003] The embodiments of the present application provide a model training method, an image processing method, an apparatus, an electronic device, a storage medium and a program product, which can reduce the training burden of the model and improve the image processing effect of the trained model.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] The present application embodiment provides a model training method, including:

[0006] Based on the images of the plurality of first parts of the first object in the first image, reconstruct the images by using the model to obtain the plurality of images of the second parts;

[0007] Merging the multiple images of the second part to obtain a second image;

[0008] Segmenting the third image to obtain images of a plurality of third parts of the third object, wherein the third image is a reference image;

[0009] The model is trained based on the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

[0010] The present application provides a model training device, including:

[0011] A reconstruction module, configured to reconstruct the images based on the images of the plurality of first parts of the first object in the first image by using a model to obtain the images of the plurality of second parts;

[0012] A merging module, used for merging the plurality of images of the second part to obtain a second image;

[0013] a segmentation module, configured to segment the third image to obtain images of a plurality of third parts of the third object, wherein the third image is a reference image;

[0014] A training module is used to train the model based on the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

[0015] The present application provides an image processing method, including:

[0016] Acquire an image to be processed including an object to be processed;

[0017] The image to be processed is reconstructed by means of a model to obtain a reconstructed image including a reconstructed object, wherein the model is obtained by training according to the model training method provided in an embodiment of the present application.

[0018] The present application provides an image processing device, including:

[0019] An acquisition module, used for acquiring an image to be processed including an object to be processed;

[0020] An image reconstruction module is used to reconstruct the image to be processed through a model to obtain a reconstructed image including a reconstructed object, wherein the model is trained according to the model training method provided in an embodiment of the present application.

[0021] An embodiment of the present application provides an electronic device, the electronic device comprising:

[0022] A memory for storing computer executable instructions or computer programs;

[0023] The processor is used to implement the model training method provided in the embodiment of the present application, or the image processing method provided in the embodiment of the present application when executing the computer executable instructions or computer program stored in the memory.

[0024] An embodiment of the present application provides a computer-readable storage medium storing a computer program or computer-executable instructions, which are used to implement the model training method provided by the embodiment of the present application, or the image processing method provided by the embodiment of the present application when executed by a processor.

[0025] An embodiment of the present application provides a computer program product, including a computer program or computer executable instructions. When the computer program or computer executable instructions are executed by a processor, the model training method provided by the embodiment of the present application, or the image processing method provided by the embodiment of the present application is implemented.

[0026] The embodiments of the present application have the following beneficial effects:

[0027] First, the model is used to reconstruct the images of multiple first parts of the first object in the first image to obtain multiple images of the second parts, so that the model can process the images of multiple first parts in a targeted manner, reduce the model's attention to unnecessary content, and thus reduce the model's training burden, and then merge the multiple second parts images to obtain the second image, and segment the third image as the reference image to obtain multiple third parts images of the third object, and finally based on the difference between the first image and the second image, and the difference between the multiple third parts images and the multiple second parts images, the model is trained, and the model is supervised by the multiple third parts images of the third object in the reference image. The model can be better fit to the reference image. In other words, through this application, the model's training burden can be effectively reduced, and the image processing effect of the trained model can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] Figure 1 It is a schematic diagram of the architecture of the model training system provided in the embodiment of the present application;

[0029] Figure 2A is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application;

[0030] Figure 2B is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application;

[0031] Figure 3 It is a first flow chart of the model training method provided in the embodiment of the present application;

[0032] Figure 4 It is a second flow chart of the model training method provided in the embodiment of the present application;

[0033] Figure 5 It is a third flow chart of the model training method provided in the embodiment of the present application;

[0034] Figure 6 It is a fourth flow chart of the model training method provided in the embodiment of the present application;

[0035] Figure 7 It is a fifth flow chart of the model training method provided in the embodiment of the present application;

[0036] Figure 8 It is a sixth flow chart of the model training method provided in the embodiment of the present application;

[0037] Fig. 9 is a flowchart of an image processing method provided in an embodiment of the present application;

[0038] Fig.10It is a process schematic diagram of the digital human algorithm provided in the embodiment of the present application;

[0039] Fig.11 is a schematic diagram of the process of training a slimming network provided in an embodiment of the present application;

[0040] Fig.12 It is a schematic diagram of the process of training a classification network provided in an embodiment of the present application.

[0041] It should be pointed out that the above-mentioned "first" and "second" are only used to distinguish different solutions, and do not represent the degree of superiority or inferiority of the solutions or the priority in the implementation process. DETAILED DESCRIPTION

[0042] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of this application.

[0043] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0044] In the following description, the terms "first\second\third" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second\third" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0045] In the embodiments of the present application, the term "module" or "unit" refers to a computer program or a part of a computer program with a predetermined function, and works together with other related parts to achieve a predetermined goal, and can be implemented in whole or in part by using software, hardware (such as processing circuits or memories) or a combination thereof. Similarly, a processor (or multiple processors or memories) can be used to implement one or more modules or units. In addition, each module or unit can be part of an overall module or unit that includes the function of the module or unit.

[0046] Unless otherwise defined, all technical and scientific terms used in the embodiments of the present application have the same meanings as those commonly understood by those skilled in the art. The terms used in the embodiments of the present application are only for the purpose of describing the embodiments of the present application and are not intended to limit the present application.

[0047] The relevant data collection and processing in the embodiments of this application should be strictly in accordance with the requirements of relevant laws and regulations when applied in examples, obtain the informed consent or separate consent of the personal information subject, and carry out subsequent data use and processing within the scope of authorization of laws and regulations and the personal information subject.

[0048] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0049] 1) Discriminator network: It is the discriminator in the generative adversarial network and a key component of the network structure. It is used to distinguish whether the input data comes from the real data distribution or the data distribution generated by the generator. The output discrimination result represents the probability that the received data sample comes from the real data distribution. For example, the discriminator network can identify the probability that the second image is a real image.

[0050] 2) Classification network: The classification network is trained using multiple third parts segmented from the reference image. The classification network can classify the objects in the image and the multiple parts segmented from the objects, and can obtain the corresponding reconstruction level. The reconstruction level represents the score of the image meeting the training objectives.

[0051] In related technologies, algorithms for human body slimming usually input the entire image, which is a heavy burden for the slimming network. The network not only needs to learn how to slim down, but also needs to determine which part should be slimmed down. This requires a lot of computing resources, and the training effect is not obvious.

[0052] The embodiments of the present application provide a model training method, an image processing method, an apparatus, an electronic device, a storage medium and a program product, which can reduce the training burden of the model and improve the image processing effect of the trained model. The exemplary application of the electronic device provided by the embodiments of the present application is described below. The device provided by the embodiments of the present application can be implemented as various types of terminals such as laptops, tablet computers, desktop computers, set-top boxes, smart phones, smart speakers, smart watches, smart TVs, and vehicle-mounted terminals, and can also be implemented as a server.

[0053] For example, see Figure 1 , Figure 1 This is an architectural diagram of the model training system provided in an embodiment of the present application. In order to support an application that can reduce the training burden of the model and improve the image processing effect of the trained model, in the system 100, the terminal 400 is connected to the server 200 through the network 300. The network 300 can be a wide area network or a local area network, or a combination of the two.

[0054] The server 200 is used to reconstruct images through a model based on multiple images of first parts of the first object in the first image to obtain multiple images of second parts; merge the multiple images of the second parts to obtain a second image; segment the third image to obtain multiple images of third parts of the third object, where the third image is a reference image; and train the model based on the difference between the first image and the second image, and the difference between the images of the multiple third parts and the multiple images of the second parts.

[0055] The server 200 is also used to send the trained model to the terminal 400, so that the terminal 400 can perform image processing based on the trained model.

[0056] The terminal 400 is also used to obtain an image to be processed including an object to be processed; the image to be processed is reconstructed using the trained model sent by the server 200 to obtain a reconstructed image including a reconstructed object (eg, an object after weight loss).

[0057] Of course, the trained model can also be deployed in the server 200. After the terminal 400 obtains the image to be processed including the object to be processed, the image to be processed can be sent to the server 200 through the network 300, so that the server 200 can reconstruct the image by calling the trained model to obtain a reconstructed image including the reconstructed object. Then the server 200 can return the reconstructed image to the terminal 400 through the network 300.

[0058] In some embodiments, the server 200 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. The terminal and the server may be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0059] See also Figure 2A , Figure 2A is a schematic diagram of the structure of an electronic device 500 provided in an embodiment of the present application, taking the electronic device 500 as a server as an example, Figure 2AThe electronic device 500 shown includes: at least one processor 510, a memory 540, and at least one network interface 520. The various components in the electronic device 500 are coupled together via a bus system 530. It is understood that the bus system 530 is used to achieve connection and communication between these components. In addition to the data bus, the bus system 530 also includes a power bus, a control bus, and a status signal bus. However, for the sake of clarity, the bus system 530 is not described in detail. Figure 2A Various buses are labeled as bus system 530 .

[0060] The processor 510 can be an integrated circuit chip with signal processing capabilities, such as a general-purpose processor, a digital signal processor (DSP), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc., where the general-purpose processor can be a microprocessor or any conventional processor, etc.

[0061] The memory 540 may be removable, non-removable, or a combination thereof. Exemplary hardware devices include solid-state memory, hard drives, optical drives, etc. The memory 540 may optionally include one or more storage devices that are physically remote from the processor 510.

[0062] The memory 540 includes a volatile memory or a non-volatile memory, and may also include both volatile and non-volatile memories. The non-volatile memory may be a read-only memory (ROM), and the volatile memory may be a random access memory (RAM). The memory 540 described in the embodiments of the present application is intended to include any suitable type of memory.

[0063] In some embodiments, memory 540 is capable of storing data to support various operations, examples of which include programs, modules, and data structures, or a subset or superset thereof, as exemplarily described below.

[0064] The operating system 541 includes system programs for processing various basic system services and performing hardware-related tasks, such as a framework layer, a core library layer, a driver layer, etc., which are used to implement various basic businesses and process hardware-based tasks.

[0065] The network communication module 542 is used to reach other electronic devices via one or more (wired or wireless) network interfaces 520. Exemplary network interfaces 520 include: Bluetooth, Wireless LAN (WiFi), and Universal Serial Bus (USB).

[0066] In some embodiments, the model training device provided in the embodiments of the present application can be implemented in a software manner. Figure 2A The model training device 543 stored in the memory 540 is shown, which can be software in the form of a program and a plug-in, etc., including the following software modules: a reconstruction module 5431, a merging module 5432, a segmentation module 5433 and a training module 5434. These modules are logical, so they can be arbitrarily combined or further split according to the functions implemented. The functions of each module will be explained below.

[0067] Next, the electronic device for implementing the image processing provided by the embodiment of the present application will be described. Figure 2B , Figure 2B 6 is a schematic diagram of the structure of an electronic device 600 provided in an embodiment of the present application, taking the electronic device 600 as a terminal as an example, Figure 2B The electronic device 600 shown includes: at least one processor 610, a memory 650, at least one network interface 620, and a user interface 630. The various components in the electronic device 600 are coupled together via a bus system 640. The user interface 630 includes: an output device 631, an input device 632. The memory 650 includes: an operating system 651, a network communication module 652, a presentation module 653, and an input processing module 654. It should be noted that Figure 2B The function of the structure in Figure 2A The functions of the structures in are similar.

[0068] In some embodiments, the image processing device provided in the embodiments of the present application can be implemented in a software manner. Figure 2B An image processing device 655 stored in a memory 650 is shown, which may be software in the form of a program or a plug-in, and includes the following software modules: an acquisition module 6551 and an image reconstruction module 6552. These modules are logical, and thus may be arbitrarily combined or further split according to the functions implemented. The functions of each module will be described below.

[0069] In other embodiments, the model training device provided in the embodiments of the present application can be implemented in hardware. As an example, the model training device provided in the embodiments of the present application can be a processor in the form of a hardware decoding processor, which is programmed to execute the model training method provided in the embodiments of the present application. For example, the processor in the form of a hardware decoding processor can adopt one or more application-specific integrated circuits (Application Specific Integrated Circuit, ASIC), digital signal processors (Digital Signal Processor, DSP), programmable logic devices (Programmable Logic Device, PLD), complex programmable logic devices (Complex Programmable Logic Device, CPLD), field programmable gate arrays (Field-Programmable Gate Array, FPGA) or other electronic components.

[0070] Below, an exemplary application when an electronic device is implemented as a server will be described, and the model training method provided in an embodiment of the present application will be described.

[0071] See also Figure 3 , Figure 3 This is a first flow chart of the model training method provided in the embodiment of the present application, which will be combined with Figure 3 Steps 101 to 104 are shown for explanation.

[0072] In step 101, based on the images of a plurality of first parts of a first object in a first image, image reconstruction is performed using a model to obtain a plurality of images of a second part.

[0073] Here, the first image refers to a sample image used to train the model.

[0074] In some examples, the model can be a model for slimming down a person in an image, a model for beautifying the face of a person in an image, or a model for migrating the clothing style of a person in an image; the first image is a sample image for training the model; the first object included in the first image can be a real person, an animated character, or a virtual character, etc., or it can be a real animal, an animated animal, or a virtual animal, etc., and the embodiments of the present application are not limited to this.

[0075] In some embodiments, see Figure 4 , Figure 4 is a second flow chart of the model training method provided in an embodiment of the present application, Figure 3 The illustrated step 101 can be implemented by following the steps 1011 to 1012, which are described in detail below.

[0076] In step 1011, the first image is segmented to obtain images of a plurality of first parts of the first object.

[0077] In some examples, a segmentation algorithm can be used to segment the external contour of the first object included in the first image. For example, the model can be a model for slimming down a person in the image, and the first object can be the person. The Matting Anything algorithm can be used to segment the external contour of the person in the image to obtain images corresponding to the torso, arms, and legs (i.e., multiple first parts); the model can be a model for beautifying the face of the person in the image, and the first object can be the entire face of the person. An edge detection algorithm can be used to detect edge information in the image, and segmentation can be performed after locating the contours of the facial features to obtain images corresponding to the eyes, nose, and mouth (i.e., multiple first parts). The embodiments of the present application do not limit the specific segmentation algorithm and can be selected according to actual conditions.

[0078] In step 1012, based on the images of the plurality of first parts, image reconstruction is performed using a model to obtain a plurality of images of the second part.

[0079] In some embodiments, see Figure 5 , Figure 5 is a third flow chart of the model training method provided in the embodiment of the present application, Figure 4 The illustrated step 1012 may be implemented by following steps 10121 to 10124, which are described in detail below.

[0080] In the embodiment of the present application, the model can be used to perform steps 10121 to 10124:

[0081] In step 10121, feature extraction processing is performed on the images of the multiple first parts respectively to obtain multiple first part features.

[0082] In some examples, the images corresponding to the multiple first parts can be respectively sent to the feature extraction network for feature extraction processing to obtain multiple first part features corresponding to the multiple first parts. The feature extraction network can be a mobile network V3 (Mobilenetv3), a convolutional neural network (CNN), a residual neural network (Residual Network, ResNet) series, etc., and the embodiments of the present application are not limited to this.

[0083] For example, the obtained torso, arms, legs and other body parts can be sent to Mobilenetv3 for feature extraction processing, and the torso features, arm features and leg features corresponding to the torso, arms and legs can be obtained respectively; the eyes, nose, mouth and other parts can be sent to CNN for feature extraction processing, and the eye features, nose features and mouth features corresponding to the eyes, nose and mouth can be obtained respectively.

[0084] In step 10122, downsampling processing is performed on the multiple first part features to obtain a downsampling result.

[0085] In some embodiments, the model may include a sampling network, and the sampling network includes a downsampling layer. The above-mentioned "downsampling multiple first part features to obtain downsampling results" can be achieved through the following technical solutions: splicing multiple first part features to obtain splicing features; downsampling the splicing features through the downsampling layer to obtain downsampling results.

[0086] In some examples, the obtained multiple first part features may be sequentially spliced ​​to obtain spliced ​​features, and then the spliced ​​features may be downsampled using a downsampling layer of a sampling network to obtain a downsampling result.

[0087] For example, the sampling network can be a network layer used to change the spatial dimension of the input data, which usually includes a downsampling layer and an upsampling layer. The downsampling layer can be a maximum pooling layer (reducing the spatial dimension of the data by selecting the maximum value in each local area while retaining important features), an average pooling layer (similar to maximum pooling, but reducing the spatial dimension of the data by calculating the average value of each local area), etc. If the sampling network is a U-type network (Unet), the downsampling layer can be an encoder of Unet. For example, for the obtained torso features, arm features, and leg features, the torso features, arm features, and leg features can be connected in order to obtain the spliced ​​features, and the spliced ​​features can be sent to the encoder of Unet for downsampling to obtain the downsampling results.

[0088] In step 10123, the down-sampling result is up-sampled to obtain an up-sampling result.

[0089] In some embodiments, the model may include a sampling network, and the sampling network includes an upsampling layer. The above-mentioned "upsampling the downsampling result to obtain the upsampling result" can be achieved through the following technical solution: upsampling the downsampling result through the upsampling layer to obtain the upsampling result.

[0090] In some examples, the downsampling result can be sent to the upsampling layer for upsampling to obtain the upsampling result. The upsampling network layer may include: transposed convolution (increasing the size of the output feature map by using a convolution kernel with a step size less than 1 and padding), nearest neighbor upsampling (increasing the size of the image by copying pixel values, which is simple and fast but may cause jagged effects on the edges of the image), bilinear interpolation upsampling (estimating the value of the new pixel by bilinear interpolation to achieve smooth upsampling), etc. If the sampling network is a U-type network (Unet), the upsampling layer can be the decoder of Unet. For example, the downsampling results obtained by downsampling the trunk features, arm features, and leg features can be sent to the decoder of Unet to obtain the upsampling results.

[0091] In step 10124, convolution processing is performed on the up-sampling result to obtain multiple images of the second part.

[0092] In some embodiments, the above-mentioned "performing convolution processing on the upsampling result to obtain multiple images of the second part" can be achieved through the following technical solutions: performing smoothing processing on the upsampling result to obtain a smoothed upsampling result; performing a first convolution processing on the smoothed upsampling result to obtain a first convolution result; performing a second convolution processing on the first convolution result to obtain multiple images of the second part.

[0093] In some examples, the upsampled image may have artifacts such as aliasing or aliasing, so it is usually necessary to smooth the upsampled result to improve the image quality. Commonly used smoothing methods include: bilinear interpolation (determine the value of the new pixel by calculating the weighted average of the four nearest pixels around it), Lanczos interpolation (use a special filter to weight the surrounding pixels to obtain a clearer and smoother image), convolution layer (use convolution layer for smoothing after upsampling), etc. Performing a first convolution process on the smoothed upsampled result can obtain the features corresponding to each first part (i.e., the first convolution result), and performing a second convolution process on the first convolution result to obtain multiple second parts of the second object reconstructed from the first object, and the multiple second parts correspond one-to-one to the multiple first parts.

[0094] For example, the upsampling results corresponding to the torso features, arm features, and leg features can first be smoothed by a convolutional layer conv3*3, then passed through a conv1*1*9 convolutional layer to obtain the texture features of the torso, arms, and legs, and finally passed through a conv1*1 convolutional layer to output the three body parts after slimming.

[0095] Through steps 10121 to 10124, multiple first part features can be downsampled first to reduce the dimension of the data and reduce the computational complexity. Upsampling enables the model to process detail information and generate high-quality output. By downsampling first and then upsampling, the model can learn features at different resolutions, which helps to improve the performance of the model.

[0096] Through step 1011 to step 1012, the model only needs to focus on multiple images of the first part, and the model does not need to process the background parts that do not need to be paid attention to, thereby reducing the data processed by the model, reducing the calculation complexity, reducing the training burden of the model, and helping to improve the performance of the model.

[0097] Continue to see Figure 3 , and the description continues with step 101 above.

[0098] In step 102, a plurality of images of the second part are merged to obtain a second image.

[0099] Here, the second object refers to the first object after image reconstruction, for example, the second object may be the first object after weight loss.

[0100] In some embodiments, the obtained multiple images of the second part can be merged, and the multiple images of the second part can be restored to a second image after image reconstruction, and the second image includes the second object after image reconstruction of the first object.

[0101] In some examples, multiple images of the second part can be fed into the convolution layer, and after convolution processing, a second image including the second object can be obtained. For example, images of the torso, arms, and legs after weight loss can be fed into the conv3*3*3 convolution layer to obtain an image of the complete body after weight loss.

[0102] In some embodiments, a third convolution process may be performed on the first convolution result to obtain a second image including a second object.

[0103] In some examples, a third convolution process can be performed on the features corresponding to each first part (i.e., the first convolution result) to obtain a second image including the second object. For example, the texture features of the torso, arms, and legs (i.e., the first convolution result) are obtained through a conv1*1*9 convolution layer, and then through a conv3*3*3 convolution layer to obtain an image of the complete body after weight loss.

[0104] In step 103, the third image is segmented to obtain images of a plurality of third parts of the third object.

[0105] Here, the third image refers to a reference image, for example, an image obtained by photographing a model.

[0106] In some examples, if the model is a model for slimming down the person in the image, the third image may be a photo of male and female models (i.e., a reference image) or a photo of male and female bodies with standard body shapes (i.e., a reference image); if the model is a model for beautifying the face of the person in the image, the third image may be a facial picture of beautiful male and female stars; if the model is a model for migrating the clothing style of the person in the image, the third image may be a photo of the person wearing clothing of the target style.

[0107] In some examples, the implementation of the above step 103 can refer to step 1011 and will not be repeated here.

[0108] In step 104, the model is trained based on the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

[0109] In some embodiments, when the model includes a discriminant network, see Figure 6 , Figure 6 is a fourth flow chart of the model training method provided in an embodiment of the present application, Figure 3 The illustrated step 104 may also be implemented by the following steps 1041A to 1042A, which are described in detail below.

[0110] In step 1041A, the second image is authenticated by the authentication network to obtain an authentication result.

[0111] The identification result is used to characterize the authenticity of the second image.

[0112] In some examples, when the model includes an identification network, the model can be a generative adversarial network, and the generated second image can be identified by the identification network (i.e., the discriminator network) to obtain an identification result. The identification result can be used to characterize the authenticity of the second image. The identification network can be a basic discriminator, a convolutional neural network discriminator, a conditional generative adversarial network discriminator, etc., which is not limited to the embodiment of the present application. The identification result output by the identification network is a probability value, which indicates the probability that the input image is a real image. This probability value usually ranges from 0 to 1, where: a probability value close to 1: indicates that the identification network believes that the input image is very likely to be a real image; a probability value close to 0: indicates that the identification network believes that the input image is very likely to be an image generated by the model.

[0113] For example, the model may be a model of slimming down a person in an image, and the second image may be a slimming image of the person after slimming down. The slimming image is identified through an identification network (such as a basic discriminator). For example, if the identification result is 0.2, the identification network believes that the slimming image is an image generated by the model and is not real.

[0114] In step 1042A, the model is trained based on the difference between the first image and the second image, the identification result, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

[0115] In some examples, the difference between the first image and the second image can be determined by: Mean Squared Error (MSE), which is used to calculate the average of the sum of the squares of the differences in corresponding pixel values ​​of the two images. The smaller the MSE value, the more similar the two images are; Peak Signal-to-Noise Ratio (PSNR), which is calculated based on MSE and takes into account the dynamic range of the image. The larger the PSNR value, the better the image quality and the smaller the difference; Structural Similarity Index (SSIM), SSIM takes into account the structural information, brightness and contrast of the image. The closer the SSIM value is to 1, the more similar the images are; Mutual Information (MI), mutual information measures the degree of information overlap between two images. The larger the MI value, the more similar the images are. The difference between multiple images of the third part and multiple images of the second part can also be determined in the above manner, which will not be repeated here.

[0116] A first loss function can be designed based on the difference between the first image and the second image, a second loss function can be designed based on the identification result, and a third loss function can be designed based on the difference between multiple images of third parts and multiple images of second parts, and then the model is trained based on the first loss function, the second loss function and the third loss function.

[0117] For example, the expression of the first loss function can be the following formula (1):

[0118] L rec =MSE(x′, x) (1)

[0119] Among them, L rec represents the first loss, MSE represents the mean square error, x′ represents the second image, and x represents the first image.

[0120] The expression of the second loss function can be the following formula (2):

[0121]

[0122] Among them, L adv represents the second loss, D represents the identification network, x′ represents the second image, and N represents the number of training batches (batchsize).

[0123] The expression of the third loss function can be the following formula (3):

[0124] Loss n =BCE(mobile(mat(x″)),mobile(mat(y))) (3)

[0125] Loss n represents the third loss corresponding to the n second part images, mat represents the segmentation of the second image and the third image, x″ represents the generated second part; y represents the third part.

[0126] The first loss function, the second loss function and the third loss function can be directly added to obtain the total loss function, or the first loss function, the second loss function and the third loss function can be weighted to obtain the total loss function, which is not limited in the embodiments of the present application.

[0127] Furthermore, after obtaining the total loss function, the gradient of the total loss function with respect to the model parameters (weights and biases) is calculated. The gradient is a vector that contains the partial derivatives of each parameter with respect to the total loss function, indicating the direction and rate of change of the total loss function in the parameter space. The learning rate is a hyperparameter that controls the amplitude of parameter updates, and determines the step size of the parameter moving in the direction of minimum loss in each iteration. According to the rule of gradient descent, the model parameters are updated according to the gradient and learning rate in each iteration. The specific update rule is that the new parameter is equal to the old parameter minus the learning rate multiplied by the gradient. The update rule indicates that the parameter will move in the direction of gradient descent in each iteration, thereby gradually reducing the value of the total loss function. Then, the steps of calculating the gradient and updating the parameters are repeated until the total loss function reaches a preset threshold, or the loss function value is no longer significantly reduced after the parameter update, indicating that the model has converged to a local minimum of the total loss function. During the training process, the training can be terminated when the performance of the model on the validation set no longer improves, or when the preset number of training rounds is reached.

[0128] Through steps 1041A to 1042A, the model can be trained through the identification result of the second image, the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts. The identification result of the second image can indicate that the model needs to adjust its parameters to generate a more realistic image. Therefore, the model is trained through the identification result of the second image, and the model is stimulated to generate an image closer to the real image, thereby improving the quality and authenticity of the generated second image; the difference between the first image and the second image can indicate that the second image generated by the model is closer to the first image, and guide the model to adjust its parameters so that the second image is more natural as a whole; so as to reduce the difference between the generated image and the standard image, thereby improving the accuracy of the generated image. The difference between the images of the plurality of third parts and the images of the plurality of second parts can capture the subtle differences between the images of the third parts and the images of the second parts, thereby guiding the model to adjust at a fine granularity, which is conducive to improving the quality of the image, and the images of the plurality of third parts serve as a direct supervisory signal, which helps the model to learn the correct features and patterns faster, and improves the image processing effect of the trained model.

[0129] In some embodiments, when the model includes a classification network, the classification network is trained based on multiple images of the third part, see Figure 7 , Figure 7 is a fifth flow chart of the model training method provided in the embodiment of the present application, Figure 3 The illustrated step 104 may also be implemented by the following steps 1041B to 1042B, which are described in detail below.

[0130] In some examples, if the model is a model for slimming down the person in the image, the third image may be a photo of a male or female model (i.e., a reference image) or a photo of a male or female body with a standard body shape, and a segmentation algorithm may be used to segment the body parts in the photo, and corresponding labels may be set using the segmented body parts to serve as training samples for training the classification network; if the model is a model for beautifying the face of the person in the image, the third image may be a facial picture of a beautiful male or female star, and a segmentation algorithm may be used to segment the facial features in the photo, and corresponding labels may be set using the segmented facial features to serve as training samples for training the classification network; if the model is a model for migrating the clothing style of the person in the image, the third image may be a photo of a person wearing clothing of the target style, and a segmentation algorithm may be used to segment the body parts in the photo, and corresponding labels may be set using the segmented body parts to serve as training samples for training the classification network.

[0131] For example, the model is a model for slimming down the characters in the image. The segmentation network is used to segment the body parts (i.e., torso, arms, and legs) in the photos of male and female models (i.e., reference images), and corresponding labels are set. The classification network can classify five categories, three of which are the obesity levels of the torso, arms, and legs, and the labels set are [-5, 5] (the smaller the label value, the thinner); one is the overall beauty of the body, and the label set is [-5, 5] (the smaller the label value, the thinner); one is the overall visual visibility, and the label set is [0, 5] (the larger the label value, the more beautiful). Because the embodiment of the present application is mainly for slimming down, the classification network is mainly used to prevent image distortion caused by excessive slimming, and also to prevent the body from being too thin, as people who are too thin are not beautiful.

[0132] In step 1041B, the second image is classified through a classification network to obtain a classification result.

[0133] The classification result is used to characterize the reconstruction level of the second image and the images of the plurality of second parts.

[0134] In some embodiments, a second image including a second object and images of multiple second parts of the second object are respectively classified through a classification network to obtain classification results. The classification network has different classification results corresponding to the second image including the second object and the images of multiple second parts of the second object, and the classification results represent the reconstruction levels of the second image and the images of the multiple second parts.

[0135] For example, taking the model that is used to slim down the person in the image as an example, the classification network is used to classify the image after slimming, and the obesity degree of the torso, arms, and legs, the overall beauty of the body shape, and the overall visual appearance can be obtained. For example, the classification result of a slimming image is [-5, -4, -5, -3, 0], indicating that the torso, arms, and legs are too thin, the overall body shape is too thin and unsightly, and the overall visual appearance is poor.

[0136] In step 1042B, the model is trained based on the difference between the first image and the second image, the classification result, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

[0137] In some examples, a first loss function can be designed based on the difference between the first image and the second image, a fourth loss function can be designed based on the classification result, and a third loss function can be designed based on the difference between multiple images of third parts and multiple images of second parts, and then the model is trained based on the first loss function, the fourth loss function and the third loss function.

[0138] For example, the expression of the first loss function may be the above formula (1), the expression of the third loss function may be the above formula (3), and the expression of the fourth loss function may be the following formula (4):

[0139] MLoss = mobile(x′) / 2 (4)

[0140] Among them, MLoss represents the fourth loss, mobile represents the classification network, and x′ represents the second image.

[0141] The first loss function, the fourth loss function and the third loss function can be directly added to obtain the total loss function, or the first loss function, the fourth loss function and the third loss function can be weighted to obtain the total loss function, which is not limited in the embodiments of the present application.

[0142] Furthermore, after obtaining the total loss function, the implementation method of training the model can refer to step 1042A, which will not be repeated here.

[0143] Through steps 1041B to 1042B, a classification network can be pre-trained by using a reference image, and the model can be trained using the classification results. The supervised learning method can guide the second image generated by the model to fit the reference image, thereby improving the image processing effect of the trained model; the difference between the first image and the second image can indicate that the second image generated by the model is closer to the first image, and guide the model to adjust its parameters so that the second image is more natural as a whole; so as to reduce the difference between the generated image and the standard image, thereby improving the accuracy of the generated image. The difference between the images of the multiple third parts and the images of the multiple second parts can capture the subtle differences between the images of the third parts and the images of the second parts, thereby guiding the model to make adjustments at a fine granularity, which is conducive to improving the quality of the image, and the images of the multiple third parts serve as a direct supervisory signal, which helps the model to learn the correct features and patterns faster, thereby improving the image processing effect of the trained model.

[0144] In some embodiments, when the model includes a discrimination network and a classification network, the classification network is trained based on multiple images of the third part, see Figure 8 , Figure 8 is a sixth flow chart of the model training method provided in the embodiment of the present application, Figure 3 The illustrated step 104 may also be implemented by the following steps 1041C to 1043C, which are described in detail below.

[0145] In step 1041C, the second image is authenticated by the authentication network to obtain an authentication result.

[0146] The identification result is used to characterize the authenticity of the second image.

[0147] In some examples, the implementation of step 1041C can refer to step 1041A and will not be repeated here.

[0148] In step 1042C, the second image is classified through a classification network to obtain a classification result.

[0149] The classification result is used to characterize the reconstruction level of the second image and the images of the plurality of second parts.

[0150] In some examples, the implementation of step 1042C can refer to step 1041B and will not be repeated here.

[0151] In step 1043C, the model is trained based on the difference between the first image and the second image, the identification result, the classification result, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

[0152] The above-mentioned "training the model based on the difference between the first image and the second image, the identification result, the classification result, and the difference between the images of multiple third parts and the images of multiple second parts" can be achieved through the following technical solutions: determining the first error between the first image and the second image; determining the second error between the image of each third part and the corresponding image of the second part; determining the total error based on the first error, the identification result, the classification result, and the multiple second errors; and training the model based on the total error.

[0153] In some examples, a first loss function can be designed based on the difference between the first image and the second image, a second loss function can be designed based on the identification result, a fourth loss function can be designed based on the classification result, and a third loss function can be designed based on the difference between images of multiple third parts and images of multiple second parts, and then the model is trained based on the first loss function, the second loss function, the fourth loss function and the third loss function.

[0154] For example, the expression of the first loss function can be the above formula (1), the expression of the second loss function can be the above formula (2), the expression of the third loss function can be the above formula (3), and the expression of the fourth loss function can be the above formula (4).

[0155] The first loss function, the second loss function, the fourth loss function and the third loss function can be directly added to obtain the total loss function, or the first loss function, the second loss function, the fourth loss function and the third loss function can be weighted to obtain the total loss function, which is not limited to this embodiment of the present application.

[0156] Furthermore, after obtaining the total loss function, the implementation method of training the model can refer to step 1042A, which will not be repeated here.

[0157] Through steps 1041C to 1043C, the model can be trained through the identification result of the second image, the difference between the first image and the second image, and the difference between the images of multiple third parts and the images of multiple second parts. The identification result of the second image can indicate that the model needs to adjust its parameters to generate a more realistic image. Therefore, the model is trained through the identification result of the second image, and the model is stimulated to generate an image that is closer to the reality, thereby improving the quality and authenticity of the generated second image. A classification network is pre-trained with a reference image, and the model is trained using the classification result. The supervised learning method can guide the second image generated by the model to fit the reference image, thereby improving the image processing effect of the trained model; the difference between the first image and the second image can indicate that the second image generated by the model is closer to the first image, guiding the model to adjust its parameters so that the second image is more natural as a whole; so as to reduce the difference between the generated image and the standard image, thereby improving the accuracy of the generated image. The difference between multiple third part images and multiple second part images can capture subtle differences between the third part images and the second part images, thereby guiding the model to make fine-grained adjustments, which is beneficial to improving the image quality. In addition, multiple third part images serve as a direct supervisory signal, which helps the model learn the correct features and patterns more quickly, thereby improving the image processing effect of the trained model.

[0158] See also Fig. 9 , Fig. 9 is a flowchart of the image processing method provided in the embodiment of the present application, which will be combined with Fig. 9 Steps 201 to 202 are shown for explanation.

[0159] In step 201, an image to be processed including an object to be processed is acquired.

[0160] In some examples, if the model is a weight loss model, the image to be processed may be an image of an obese person who wants to lose weight; if the model is a style transfer model, the image to be processed may be an image of a person who wants to change his clothes to a specified style, etc. The embodiments of the present application are not limited to this.

[0161] In step 202, the image to be processed is reconstructed by using the model to obtain a reconstructed image including a reconstructed object.

[0162] Here, the model may be obtained by training according to the model training method provided in the embodiment of the present application. In addition, the reconstructed image corresponds to the above-mentioned second image, and the reconstructed object corresponds to the above-mentioned second object.

[0163] In some examples, if the model is a slimming model, by reconstructing the image to be processed by the model, a slimming image including a human body after slimming can be obtained; if the model is a style transfer model, by reconstructing the image to be processed by the model, a transfer image including a person wearing clothes in the target style can be obtained.

[0164] Through step 201 to step 202, the model can be used to perform different tasks, thereby improving the flexibility and adaptability of the model.

[0165] Through the embodiments of the present application, the model can be used to reconstruct images of multiple first parts of the first object in the first image to obtain multiple images of the second parts. This allows the model to process the multiple images of the first parts in a targeted manner, reduce the model's attention to unnecessary content, and reduce the training burden of the model. The multiple images of the second parts are then merged to obtain the second image, and the third image used as the reference image is segmented to obtain multiple images of the third parts of the third object. Finally, based on the difference between the first image and the second image, and the difference between the images of the multiple third parts and the images of the multiple second parts, the model is trained. By supervising the model with the images of the multiple third parts of the third object in the reference image, the image reconstructed by the model can be better fitted to the reference image. While reducing the training burden of the model, the image processing effect of the trained model can also be improved.

[0166] The following is an explanation of an exemplary application of the embodiments of the present application in a practical application scenario.

[0167] Here are some practical application scenarios of models: Photo editing software: In photo editing applications, users can use models to adjust the body proportions in the picture to make it look thinner or more in line with the user's aesthetic standards. Movies and entertainment: In the production of movies or TV series, models can be used for special effects, such as changing the body proportions of actors to suit different role requirements. For example, if a particularly fat character is needed, the model can be used to process the actor's body in the video to generate a fat body that meets the requirements. Virtual fitting room: In online shopping websites or mobile applications, virtual fitting rooms can use models to let users see the effects of different clothes on their bodies, thereby helping users make purchasing decisions.

[0168] The following is an example of the application scenario of digital human weight loss. The relevant technology can generate a high-definition virtual digital human video with lip sync and the level of real photo realism through a video of a voice and a person. The embodiment of the present application can be used to slim down the body of a digital human.

[0169] See also Fig.10 , Fig.10It is a process diagram of the digital human algorithm provided in the embodiment of the present application. The voice (Audio) and the base image (the frame in the video) are input, and the lip shape of the corresponding voice is generated through the Neural Voice Puppetry (NVP) algorithm. Then, the corresponding lip shape is pasted back to the base image to generate a complete digital human. Then, the digital human is slimmed down by the slimming algorithm of the embodiment of the present application to obtain a new digital human.

[0170] like Fig.10 As shown, the embodiment of the present application mainly slims down the base image of the digital human, because the digital human is mainly driven for the face and has no effect on the torso. The slimmed-down base image is replaced, and a slimmed-down digital human can be obtained after each drive.

[0171] See also Fig.11 , Fig.11 : is a schematic diagram of the process of training a slimming network provided in an embodiment of the present application. In the embodiment of the present application, Mobilenetv3 is used as the feature extraction backbone network. The convolution in the lower half of Unet represents a 3*3 convolution with a step size of 2, a convolution kernel size of 64, and a Prelu layer after the convolution; the convolution in the upper half represents a 3*3 convolution with a step size of (A step size of 2 means that the step size of each movement is 2, that is, the resolution will be reduced, which means downsampling. A step size of 1 simply represents a convolutional layer for feature extraction, which further extracts the attributes or identity features extracted by the previous layer), the convolution kernel size is 64, and the convolution is followed by a Prelu layer, followed by a deconvolution layer for upsampling; the convolution layer in the lower half represents the downsampling layer, and the convolution layer in the upper half represents the upsampling layer.

[0172] First, the embodiment of the present application uses the Matting Anything algorithm to segment the outer contour of the human body (i.e., the first object) in the original image (i.e., the first image), and obtains the torso, arms, and legs (i.e., the first part) respectively, and concatenates the three as input for Mobilenetv3 to extract features. The embodiment of the present application does not use the whole body of the human body but uses several parts of the body separately as input because the human body is composed of multiple parts, and usually the whole body of the human body is difficult to distinguish from the background. When the network is slimming down, it is impossible to pay effective attention to the part of the human body that needs to be slimmed down. By inputting a separate body part, the difficulty of the network can be effectively reduced, so that all parts of the human body can be effectively slimmed down; then the features extracted by Mobilenetv3 are sent to the Unet network, smoothed by a layer of conv3*3, and then the texture features of each part are obtained by a conv1*1*9 convolution layer, and finally restored to a slimmed down image (i.e., the second image) by a layer of conv3*3*3 convolution; at the same time, the external conv1*1 convolution layer outputs three body parts (i.e., the second part).

[0173] At the same time, the embodiment of the present application sets a body beauty standard, and uses a segmentation network in advance to segment the torso, arms, and legs (i.e., the third part) in the photos of male and female models (i.e., the third image) as labels for separate parts (using a binary image (mask) of the body part as a label). In addition, a beauty evaluation classifier (i.e., a classification network) is pre-trained using Mobilenetv3, see Fig.12 , Fig.12 is a schematic diagram of the process of training a classification network provided in an embodiment of the present application, such as Figure 5 As shown, the classification results can be divided into 5 categories, 3 of which are the fatness of the trunk, arms, and legs, and the labels are set to [-5, 5] (the smaller the label value, the thinner), because the fatness of each part of each picture is different; 1 category is the overall beauty of the body, and the labels are set to [-5, 5] (the smaller the label value, the thinner); 1 category is the overall visual appearance, and the labels are set to [0, 5] (the larger the label value, the more beautiful), which is mainly to prevent image distortion caused by weight loss. The training process also sets labels for the three parts separately ([-5, 5], the smaller the thinner), and the classification network is mainly to prevent image distortion caused by excessive weight loss, and to prevent the body from being too thin, because people who are too thin are not beautiful.

[0174] The loss function used in the embodiment of the present application is as follows:

[0175] Human reconstruction loss L rec The above formula (1) can be used and will not be repeated here.

[0176] The training of the overall slimming network can be done by generating adversarial networks. The generator is the backbone network, and the discriminator can use dcgan to get Ganloss (i.e. L adv ), which can make the obtained image look more comfortable. The expression of Ganloss can adopt the above formula (2), which will not be repeated here.

[0177] In addition, a pre-trained aesthetic evaluation classifier is used to guide network training, which can be defined as MLoss. The aesthetic evaluation classifier contains five parts, namely, the obesity degree of three body parts, the overall aesthetic degree of the body, and the overall visual observability. Here, only two classification values ​​are used, the overall aesthetic degree of the body and the overall visual observability, and the average is taken. The expression of MLoss can adopt the above formula (4), which will not be repeated here.

[0178] At the same time, the loss between the three body parts of the intermediate output (i.e., the second part) and the corresponding body parts of the standard body (i.e., the third part) is calculated, and the expression is the following formula (5):

[0179] Loss1=Loss2=Loss3=BCE(mobile(mat(x″)),mobile(mat(y))) (5)

[0180] Among them, Loss1=Loss2=Loss3 means that the formulas for calculating the three losses are in the same form, but have different meanings, that is, the body parts segmented by mat are different, including torso, arms, and legs; x″ represents the generated torso, arms, and legs; y represents the torso, arms, and legs of the standard body; BCE represents the binary cross entropy loss function; mobile represents the pre-trained classification network; mat represents the MattingAnything network.

[0181] The formula for the final total loss function is:

[0182] Loss = L rec +5MLoss+2Loss1+2Loss2+2Loss3+L adv (6)

[0183] The above formula is the total loss of training. Since body reconstruction is not difficult, the embodiment of the present application mainly aims at making the body thinner and more beautiful based on supervision, so the weight of MLoss is relatively large.

[0184] Through the embodiments of the present application, different body parts can be used as input to better enable the network to achieve targeted slimming. In addition, the embodiments of the present application searched for some photos with standard body shapes as reference images, extracted body parts separately, and then used a pre-trained aesthetic assessment classifier for supervised training, which can better enable the network's slimmed-down body to fit the model's figure.

[0185] The following is a description of an exemplary structure of the model training device 543 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2A As shown, the software modules stored in the model training device 543 of the memory 540 may include:

[0186] The reconstruction module 5431 is used to reconstruct the image through the model based on the images of the multiple first parts of the first object in the first image, so as to obtain the images of the multiple second parts; the merging module 5432 is used to merge the images of the multiple second parts to obtain the second image; the segmentation module 5433 is used to segment the third image to obtain the images of the multiple third parts of the third object, where the third image is the reference image; the training module 5434 is used to train the model based on the difference between the first image and the second image, and the difference between the images of the multiple third parts and the images of the multiple second parts.

[0187] In some embodiments, the model includes an identification network, a training module 5434, and is also used to identify the second image through the identification network to obtain an identification result, wherein the identification result is used to characterize the authenticity of the second image; the model is trained based on the difference between the first image and the second image, the identification result, and the difference between the images of multiple third parts and the images of multiple second parts.

[0188] In some embodiments, the model includes a classification network, which is trained based on multiple images of third parts. The training module 5434 is also used to classify the second image through the classification network to obtain a classification result, wherein the classification result is used to characterize the reconstruction level of the second image and the multiple images of the second parts; the model is trained based on the difference between the first image and the second image, the classification result, and the difference between the images of the multiple third parts and the images of the multiple second parts.

[0189] In some embodiments, the training module 5434 is further used to classify the second image including the second object and the images of multiple second parts of the second object respectively through a classification network to obtain classification results.

[0190] In some embodiments, the model includes an identification network and a classification network, wherein the classification network is trained based on multiple images of the third part, and the training module 5434 is also used to identify the second image through the identification network to obtain an identification result, wherein the identification result is used to characterize the authenticity of the second image; classify the second image through the classification network to obtain a classification result, wherein the classification result is used to characterize the reconstruction level of the second image and multiple images of the second part; and train the model based on the difference between the first image and the second image, the identification result, the classification result, and the difference between the images of the multiple third parts and the images of the multiple second parts.

[0191] In some embodiments, the training module 5434 is also used to determine a first error between the first image and the second image; determine a second error between the image of each third part and the corresponding image of the second part; determine a total error based on the first error, the identification result, the classification result, and multiple second errors; and train the model based on the total error.

[0192] In some embodiments, the reconstruction module 5431 is further used to segment the first image to obtain multiple images of the first part of the first object; based on the multiple images of the first part, the image is reconstructed through the model to obtain multiple images of the second part.

[0193] In some embodiments, the reconstruction module 5431 is also used to perform the following processing through the model: perform feature extraction processing on multiple images of the first part respectively to obtain multiple first part features; perform downsampling processing on the multiple first part features to obtain downsampling results; perform upsampling processing on the downsampling results to obtain upsampling results; perform convolution processing on the upsampling results to obtain multiple images of the second part.

[0194] In some embodiments, the model includes a sampling network, the sampling network includes a downsampling layer and an upsampling layer, and the reconstruction module 5431 is also used to splice multiple first part features to obtain splicing features; downsample the splicing features through the downsampling layer to obtain a downsampling result; and upsample the downsampling results through the upsampling layer to obtain an upsampling result.

[0195] In some embodiments, the reconstruction module 5431 is also used to smooth the upsampling result to obtain a smoothed upsampling result; perform a first convolution process on the smoothed upsampling result to obtain a first convolution result; perform a second convolution process on the first convolution result to obtain multiple images of the second part.

[0196] The following is a description of an exemplary structure of the image processing device 655 provided in the embodiment of the present application implemented as a software module. In some embodiments, Figure 2BAs shown, the software modules stored in the image processing device 655 of the memory 650 may include:

[0197] The acquisition module 6551 is used to acquire the image to be processed including the object to be processed; the image reconstruction module 6552 is used to reconstruct the image to be processed through a model to obtain a reconstructed image including the reconstructed object, wherein the model is trained according to the model training method provided in the embodiment of the present application.

[0198] The embodiment of the present application provides a computer program product, which includes a computer program or a computer executable instruction, and the computer program or the computer executable instruction is stored in a computer-readable storage medium. The processor of the electronic device reads the computer executable instruction from the computer-readable storage medium, and the processor executes the computer executable instruction, so that the electronic device executes the above-mentioned model training method or image processing method of the embodiment of the present application.

[0199] The present application embodiment provides a computer-readable storage medium, in which computer executable instructions or computer programs are stored. When the computer executable instructions or computer programs are executed by a processor, the processor will execute the model training method or image processing method provided in the present application embodiment, for example, Figures 3 to 8 The model training method shown in any of the accompanying drawings, or Fig. 9 The image processing method is shown.

[0200] In some embodiments, the computer-readable storage medium may be a memory such as RAM, ROM, flash memory, magnetic surface memory, optical disk, or CD-ROM; or may be various devices including one or any combination of the above memories.

[0201] In some embodiments, computer executable instructions may be in the form of a program, software, software module, script or code, written in any form of programming language (including compiled or interpreted languages, or declarative or procedural languages), and may be deployed in any form, including as a stand-alone program or as a module, component, subroutine or other unit suitable for use in a computing environment.

[0202] As an example, computer-executable instructions may, but need not, correspond to a file in a file system, may be stored as part of a file that stores other programs or data, such as in one or more scripts in a HyperText Markup Language (HTML) document, in a single file dedicated to the program in question, or in multiple coordinated files (e.g., files storing one or more modules, subroutines, or code portions).

[0203] As an example, computer executable instructions may be deployed to be executed on one electronic device, or on multiple electronic devices located at one site, or on multiple electronic devices distributed at multiple sites and interconnected by a communication network.

[0204] In summary, through the embodiments of the present application, multiple first-part features can be downsampled first to reduce the dimension of the data and the computational complexity. Upsampling enables the model to process detail information and generate high-quality output. By downsampling first and then upsampling, the model can learn features at different resolutions, which helps to improve the performance of the model. The model only needs to focus on the images of multiple first parts, and the model does not need to process the background parts that do not need to be paid attention to, which reduces the data processed by the model, reduces the computational complexity, reduces the training burden of the model, and helps to improve the performance of the model. The model can be identified by the second image The difference between the first image and the second image, as well as the difference between the images of multiple third parts and the images of multiple second parts The identification result of the second image can indicate that the model needs to adjust its parameters to generate a more realistic image. Therefore, the model is trained by the identification result of the second image, and the model is stimulated to generate an image closer to the real image, thereby improving the quality and authenticity of the generated second image. A classification network is pre-trained by the reference image, and the model is trained by the classification result. The supervised learning method can guide the second image generated by the model to fit the reference image, thereby improving the effect of the trained model. The difference between the first image and the second image can indicate that the second image generated by the model is closer to the first image, and guide the model to adjust its parameters so that the second image is more natural as a whole; so as to reduce the difference between the generated image and the standard image, thereby improving the accuracy of the generated image. The difference between the images of the third parts and the images of the second parts can capture the subtle differences between the images of the third parts and the images of the second parts, thereby guiding the model to adjust at a fine granularity, which is conducive to improving the quality of the image. The images of the third parts serve as a direct supervisory signal, which helps the model to learn the correct features and patterns faster, thereby improving the image processing effect of the trained model. The model can be used to perform different tasks, thereby improving the flexibility and adaptability of the model.

[0205] The above is only an embodiment of the present application and is not intended to limit the protection scope of the present application. Any modifications, equivalent substitutions and improvements made within the spirit and scope of the present application are included in the protection scope of the present application.

Claims

1. A model training method, characterized in that: The method comprises: Based on the images of the plurality of first parts of the first object in the first image, reconstruct the images by using the model to obtain the plurality of images of the second parts; Merging the multiple images of the second part to obtain a second image; Segmenting the third image to obtain images of a plurality of third parts of the third object, wherein the third image is a reference image; The model is trained based on the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

2. The method according to claim 1, characterized in that The model includes a discriminant network; The training of the model based on the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts, comprises: Authentication of the second image by the authentication network to obtain an authentication result, wherein the authentication result is used to characterize the authenticity of the second image; The model is trained based on the difference between the first image and the second image, the identification result, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

3. The method according to claim 1, characterized in that The model includes a classification network, and the classification network is trained based on the images of the plurality of third parts; The training of the model based on the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts, comprises: Classifying the second image through the classification network to obtain a classification result, wherein the classification result is used to characterize the reconstruction level of the second image and the images of the plurality of second parts; The model is trained based on the difference between the first image and the second image, the classification result, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

4. The method according to claim 3, characterized in that: The classifying the second image by the classification network to obtain a classification result includes: The second image including the second object and the images of the plurality of second parts of the second object are respectively classified by the classification network to obtain classification results.

5. The method according to claim 1, characterized in that The model includes a discrimination network and a classification network, wherein the classification network is trained based on the images of the plurality of third parts; The training of the model based on the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts, comprises: Authentication of the second image by the authentication network to obtain an authentication result, wherein the authentication result is used to characterize the authenticity of the second image; Classifying the second image through the classification network to obtain a classification result, wherein the classification result is used to characterize the reconstruction level of the second image and the images of the plurality of second parts; The model is trained based on the difference between the first image and the second image, the identification result, the classification result, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

6. The method according to claim 5, characterized in that The training of the model based on the difference between the first image and the second image, the identification result, the classification result, and the difference between the images of the plurality of third parts and the images of the plurality of second parts comprises: determining a first error between the first image and the second image; determining a second error between each of the images of the third portion and the corresponding image of the second portion; determining a total error based on the first error, the identification result, the classification result, and a plurality of the second errors; Based on the total error, the model is trained.

7. The method according to claim 1, characterized in that The method of reconstructing the images of the plurality of first parts of the first object in the first image by using a model to obtain the plurality of images of the second parts includes: Segmenting the first image to obtain images of a plurality of first parts of the first object; Based on the images of the plurality of first parts, image reconstruction is performed using the model to obtain images of the plurality of second parts.

8. The method according to claim 7, characterized in that The step of reconstructing the images based on the multiple images of the first parts by using the model to obtain multiple images of the second parts includes: The following processing is performed by the model: Performing feature extraction processing on the images of the plurality of first parts respectively to obtain a plurality of first part features; Performing downsampling processing on the plurality of first part features to obtain a downsampling result; Performing upsampling processing on the downsampling result to obtain an upsampling result; The up-sampling result is subjected to convolution processing to obtain a plurality of images of the second part.

9. The method according to claim 8, characterized in that The model includes a sampling network, and the sampling network includes a downsampling layer and an upsampling layer; The downsampling process is performed on the plurality of first part features to obtain a downsampling result, including: Splicing the plurality of first part features to obtain a splicing feature; Downsampling the splicing feature through the downsampling layer to obtain a downsampling result; The upsampling process is performed on the downsampling result to obtain the upsampling result, including: The down-sampling result is up-sampled by the up-sampling layer to obtain an up-sampling result.

10. The method according to claim 8, characterized in that The step of performing convolution processing on the up-sampling result to obtain a plurality of images of the second part includes: Smoothing the up-sampling result to obtain a smoothed up-sampling result; Performing a first convolution process on the smoothed up-sampling result to obtain a first convolution result; A second convolution process is performed on the first convolution result to obtain multiple images of the second part.

11. An image processing method, characterized in that: The method comprises: Acquire an image to be processed including an object to be processed; The image to be processed is reconstructed by using a model to obtain a reconstructed image including a reconstructed object, wherein the model is trained according to the method according to any one of claims 1 to 10.

12. A model training device, characterized in that: The device comprises: A reconstruction module, configured to reconstruct the images based on the images of the plurality of first parts of the first object in the first image by using a model to obtain the images of the plurality of second parts; A merging module, used for merging the plurality of images of the second part to obtain a second image; a segmentation module, configured to segment the third image to obtain images of a plurality of third parts of the third object, wherein the third image is a reference image; A training module is used to train the model based on the difference between the first image and the second image, and the difference between the images of the plurality of third parts and the images of the plurality of second parts.

13. An electronic device, characterized in that: The electronic device comprises: A memory for storing computer executable instructions or computer programs; A processor, used to implement the model training method described in any one of claims 1 to 10 or the image processing method described in claim 11 when executing computer executable instructions or computer programs stored in the memory.

14. A computer-readable storage medium storing computer-executable instructions or a computer program, characterized in that: When the computer executable instructions or computer program are executed by a processor, the model training method described in any one of claims 1 to 10 or the image processing method described in claim 11 is implemented.

15. A computer program product comprising computer executable instructions or a computer program, characterized in that When the computer executable instructions or computer program are executed by a processor, the model training method described in any one of claims 1 to 10 or the image processing method described in claim 11 is implemented.

Images