Image processing model training method, image processing method, device and equipment

Abstract

The present application discloses a training method, an image processing method, an apparatus and a device for an image processing model, and belongs to the field of image processing technology. The training method includes: filling the texture features of the defective area in the sample defective image based on the texture features of the effective area in the sample defective image and the rough structural features of the sample defective image to obtain the complete texture features of the sample defective image; further, obtaining the refined structural features of the sample defective image based on the complete texture features and the rough structural features; after obtaining the predicted repaired image of the sample defective image based on the complete texture features and the refined structural features, determining the loss value to realize model training. Since the above method realizes the interactive guidance between texture features and structural features, when performing image repair based on the trained image processing model, the texture and structure of the defective area and the effective area in the repaired image can be kept consistent in terms of fineness, and the image repair effect is better.

Description

Technical Field

[0001] The present application relates to the field of image processing technology, and in particular to an image processing model training method, image processing method, device and equipment. Background Art

[0002] As an image processing technology, image restoration technology aims to repair the defective areas in the image and ensure the consistency of the texture and structure of the defective areas and the valid areas in the restored image as much as possible.

[0003] In related technologies, deep learning methods are usually used to train an image processing model using sample defective images. The trained image processing model is used to extract the structural features of the valid area in the defective image to be repaired. The structural features are then extended from the valid area to the defective area, and the texture features of the defective area are further synthesized to obtain the repaired image.

[0004] However, the above method relies too much on the structural features of the valid area, directly extending the structural features from the valid area to the defective area, and lacks feature reorganization, resulting in inconsistent texture and structure between the defective area and the valid area in the repaired image. The image repair effect is poor and the quality of the repaired image is poor. Summary of the Invention

[0005] The present invention provides an image processing model training method, image processing method, apparatus, and device, which can ensure that the texture and structure of the defective area and the effective area in the repaired image are consistent in terms of fineness. The technical solution is as follows:

[0006] In one aspect, a method for training an image processing model is provided, the method comprising:

[0007] Inputting a sample defect image and a sample mask image into an image processing model, wherein the sample mask image is used to indicate a defect area and a valid area of ​​the sample defect image, and the image processing model includes a texture feature layer and a structure feature layer;

[0008] Extracting a first texture feature of the sample defect image through the texture feature layer, and extracting a first structural feature of the sample defect image through the structural feature layer, wherein the first texture feature indicates a texture feature of a valid area, and the first structural feature indicates a rough structural feature of the sample defect image;

[0009] Processing the first texture feature and the first structural feature through the texture feature layer to obtain a second texture feature of the sample defect image; processing the second texture feature and the first structural feature through the structural feature layer to obtain a second structural feature of the sample defect image, wherein the second texture feature indicates texture features of the valid area and the defect area, and the second structural feature indicates a refined structural feature of the sample defect image;

[0010] Based on the second texture feature and the second structural feature, a predicted restoration image of the sample defective image is obtained, and an image processing model is trained based on the real image of the sample defective image, the predicted restoration image and the sample mask image.

[0011] In another aspect, an image processing method is provided, the method comprising:

[0012] Inputting an image to be repaired into an image processing model, wherein the image includes a valid area and a defective area, and the image processing model includes a texture feature layer and a structural feature layer;

[0013] Extracting a first texture feature of the image through the texture feature layer, and extracting a first structural feature of the image through the structure feature layer, wherein the image includes a valid area and a defective area, the first texture feature indicates a texture feature of the valid area of ​​the image, and the first structural feature indicates a rough structural feature of the image;

[0014] Processing the first texture feature and the first structural feature through the texture feature layer of the image processing model to obtain a second texture feature of the image; processing the second texture feature and the first structural feature through the structural feature layer of the image processing model to obtain a second structural feature of the image, the second texture feature indicating texture features of a valid area and a defective area of ​​the image, and the second structural feature indicating a refined structural feature of the image;

[0015] A restored image of the image is obtained based on the second texture feature and the second structural feature.

[0016] In one possible implementation, extracting a first texture feature of an image through a texture feature layer and extracting a first structural feature of the image through a structural feature layer includes:

[0017] Downsampling the image through the coding units in each texture feature layer to obtain first texture features corresponding to each texture feature layer;

[0018] Downsampling the first texture feature corresponding to the first texture feature layer through the encoding unit in each structural feature layer to obtain the first intermediate structural feature corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0019] The first intermediate structural features corresponding to the first structural feature layer are up-sampled by the decoding units in each structural feature layer to obtain the first structural features corresponding to each structural feature layer, where the first structural feature layer is the bottommost structural feature layer.

[0020] In one possible implementation, processing the first texture feature and the first structural feature through the texture feature layer to obtain the second texture feature of the image includes:

[0021] Upsampling the first structural features corresponding to the second structural feature layer through the decoding units in each texture feature layer to obtain the second intermediate structural features corresponding to each texture feature layer, where the second structural feature layer is the topmost structural feature layer;

[0022] The second intermediate structural features and the first texture features corresponding to each texture feature layer are processed by the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer.

[0023] In one possible implementation, the texture enhancement unit in each texture feature layer processes the second intermediate structural features and the first texture features corresponding to each texture feature layer to obtain the second texture features corresponding to each texture feature layer, including:

[0024] Based on the structural features of each pixel in the second intermediate structural feature corresponding to the target texture feature layer, obtaining an attention matrix corresponding to the target texture feature layer, the attention matrix indicating the affinity between any two pixels in the second intermediate structural feature corresponding to the target texture feature layer, the target texture feature layer being any texture feature layer;

[0025] Based on the attention matrix, the mask image corresponding to the target texture feature layer, and the first texture feature corresponding to the target texture feature layer, the second texture feature corresponding to the target texture feature layer is obtained.

[0026] In one possible implementation, processing the second texture feature and the first structural feature through the structural feature layer to obtain the second structural feature of the image includes:

[0027] Downsampling the second texture features corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain third intermediate structural features corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0028] The third intermediate structural features and the first structural features corresponding to each structural feature layer are processed by the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer.

[0029] In one possible implementation, the feature enhancement unit in each structural feature layer processes the third intermediate structural feature and the first structural feature corresponding to each structural feature layer to obtain the second structural feature corresponding to each structural feature layer, including:

[0030] Obtaining a residual between a third intermediate structural feature and the first structural feature corresponding to a target structural feature layer, where the target structural feature layer is any structural feature layer;

[0031] Compress each feature channel in the residual to obtain the feature value of each feature channel;

[0032] Based on the eigenvalues ​​of each feature channel, the weight of each feature channel is obtained, and the weight of the feature channel indicates the importance of the feature channel to the structural feature;

[0033] Based on the weights of the respective feature channels, each feature channel in the third intermediate structural feature corresponding to the target structural feature layer is transformed to obtain the second structural feature corresponding to the target structural feature layer.

[0034] In another aspect, a training device for an image processing model is provided, the device comprising:

[0035] A sample input module is used to input a sample defect image and a sample mask image into an image processing model, wherein the sample mask image is used to indicate a defect area and a valid area of ​​the sample defect image, and the image processing model includes a texture feature layer and a structural feature layer;

[0036] A first feature extraction module is configured to extract a first texture feature of the sample defect image through a texture feature layer, and to extract a first structural feature of the sample defect image through a structural feature layer, wherein the first texture feature indicates a texture feature of a valid area, and the first structural feature indicates a rough structural feature of the sample defect image;

[0037] a second feature extraction module, configured to process the first texture feature and the first structural feature through a texture feature layer to obtain a second texture feature of the sample defective image, and process the second texture feature and the first structural feature through a structural feature layer to obtain a second structural feature of the sample defective image, wherein the second texture feature indicates texture features of a valid area and a defective area of ​​the image to be repaired, and the second structural feature indicates a refined structural feature of the sample defective image;

[0038] The model training module is used to obtain the predicted repair image of the sample defective image based on the second texture feature and the second structural feature, and train the image processing model based on the real image of the sample defective image, the predicted repair image and the sample mask image.

[0039] In one possible implementation, the first feature extraction module is used to:

[0040] Downsampling the sample defective image through the coding units in each texture feature layer to obtain the first texture features corresponding to each texture feature layer;

[0041] Downsampling the first texture feature corresponding to the first texture feature layer through the encoding unit in each structural feature layer to obtain the first intermediate structural feature corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0042] The first intermediate structural features corresponding to the first structural feature layer are up-sampled by the decoding units in each structural feature layer to obtain the first structural features corresponding to each structural feature layer, where the first structural feature layer is the bottommost structural feature layer.

[0043] In one possible implementation, the second feature extraction module includes:

[0044] A first feature extraction unit is configured to upsample the first structural features corresponding to the second structural feature layer through the decoding units in each texture feature layer to obtain second intermediate structural features corresponding to each texture feature layer, where the second structural feature layer is the topmost structural feature layer;

[0045] The second feature extraction unit is used to process the second intermediate structural features and the first texture features corresponding to each texture feature layer through the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer.

[0046] In one possible implementation, the second feature extraction unit is configured to:

[0047] Based on the structural features of each pixel in the second intermediate structural feature corresponding to the target texture feature layer, obtaining an attention matrix corresponding to the target texture feature layer, the attention matrix indicating the affinity between any two pixels in the second intermediate structural feature corresponding to the target texture feature layer, the target texture feature layer being any texture feature layer;

[0048] Based on the attention matrix, the mask image corresponding to the target texture feature layer, and the first texture feature corresponding to the target texture feature layer, the second texture feature corresponding to the target texture feature layer is obtained.

[0049] In one possible implementation, the second feature extraction module includes:

[0050] a third feature extraction unit, configured to downsample the second texture features corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain third intermediate structural features corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0051] The fourth feature extraction unit is used to process the third intermediate structural features and the first structural features corresponding to each structural feature layer through the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer.

[0052] In one possible implementation, the fourth feature extraction unit is configured to:

[0053] Obtaining a residual between a third intermediate structural feature and the first structural feature corresponding to a target structural feature layer, where the target structural feature layer is any structural feature layer;

[0054] Compress each feature channel in the residual to obtain the feature value of each feature channel;

[0055] Based on the eigenvalues ​​of each feature channel, the weight of each feature channel is obtained, and the weight of the feature channel indicates the importance of the feature channel to the structural feature;

[0056] Based on the weights of the respective feature channels, each feature channel in the third intermediate structural feature corresponding to the target structural feature layer is transformed to obtain the second structural feature corresponding to the target structural feature layer.

[0057] In one possible implementation, the model training module includes:

[0058] a first loss value determining unit, configured to input the predicted restoration image into an image discriminator, obtain a discrimination result of the predicted restoration image, and determine a first loss value of the image processing model based on the discrimination result of the predicted restoration image, the predicted restoration image, and the real image, wherein the discrimination result of the predicted restoration image indicates a probability of restoration failure for each pixel in the predicted restoration image;

[0059] a second loss value determining unit, configured to determine a second loss value of the image processing model based on a distance between a target restoration image and a true image in a feature space, wherein the target restoration image is obtained by merging the predicted restoration image and the sample defective image;

[0060] The model training unit is used to determine a target loss value based on the first loss value and the second loss value, and train the image processing model based on the target loss value until a training end condition is met, thereby obtaining a trained image processing model.

[0061] In one possible implementation, the model training module is further used to:

[0062] Determine the loss value of the image discriminator based on the discrimination result of the predicted restoration image and the sample mask image;

[0063] Based on the loss value of the image discriminator, the image discriminator is trained.

[0064] In one possible implementation, the model training module is further used to:

[0065] Merge the predicted restoration image and the sample defect image to obtain the target restoration image;

[0066] Input the target restoration image into the image enhancement model to obtain the texture features and structural features of the target restoration image;

[0067] Input the real image into the image enhancement model to obtain the texture features and structural features of the real image;

[0068] Based on the texture features and structural features of the target inpainted image and the texture features and structural features of the real image, the distance between the target inpainted image and the real image in the feature space is determined.

[0069] In another aspect, an image processing apparatus is provided, the apparatus comprising:

[0070] An image input module is used to input an image to be repaired into an image processing model, where the image includes a valid area and a defective area, and the image processing model includes a texture feature layer and a structural feature layer;

[0071] a first feature extraction module, configured to extract a first texture feature of the image through a texture feature layer, and extract a first structural feature of the image through a structural feature layer, wherein the image includes a valid area and a defective area, the first texture feature indicates a texture feature of the valid area of ​​the image, and the first structural feature indicates a rough structural feature of the image;

[0072] a second feature extraction module, configured to process the first texture feature and the first structural feature through a texture feature layer of the image processing model to obtain a second texture feature of the image, and process the second texture feature and the first structural feature through a structural feature layer of the image processing model to obtain a second structural feature of the image, wherein the second texture feature indicates texture features of a valid area and a defective area of ​​the image, and the second structural feature indicates a refined structural feature of the image;

[0073] The restoration image acquisition module is used to acquire a restoration image of the image based on the second texture feature and the second structural feature.

[0074] In one possible implementation, the first feature extraction module is used to:

[0075] Downsampling the image through the coding units in each texture feature layer to obtain first texture features corresponding to each texture feature layer;

[0076] Downsampling the first texture feature corresponding to the first texture feature layer through the encoding unit in each structural feature layer to obtain the first intermediate structural feature corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0077] The first intermediate structural features corresponding to the first structural feature layer are up-sampled by the decoding units in each structural feature layer to obtain the first structural features corresponding to each structural feature layer, where the first structural feature layer is the bottommost structural feature layer.

[0078] In one possible implementation, the second feature extraction module includes:

[0079] A first feature extraction unit is configured to upsample the first structural features corresponding to the second structural feature layer through the decoding units in each texture feature layer to obtain second intermediate structural features corresponding to each texture feature layer, where the second structural feature layer is the topmost structural feature layer;

[0080] The second feature extraction unit is used to process the second intermediate structural features and the first texture features corresponding to each texture feature layer through the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer.

[0081] In one possible implementation, the second feature extraction unit includes:

[0082] Based on the structural features of each pixel in the second intermediate structural feature corresponding to the target texture feature layer, obtaining an attention matrix corresponding to the target texture feature layer, the attention matrix indicating the affinity between any two pixels in the second intermediate structural feature corresponding to the target texture feature layer, the target texture feature layer being any texture feature layer;

[0083] Based on the attention matrix, the mask image corresponding to the target texture feature layer, and the first texture feature corresponding to the target texture feature layer, the second texture feature corresponding to the target texture feature layer is obtained.

[0084] In one possible implementation, the second feature extraction module includes:

[0085] a third feature extraction unit, configured to downsample the second texture features corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain third intermediate structural features corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0086] The fourth feature extraction unit is used to process the third intermediate structural features and the first structural features corresponding to each structural feature layer through the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer.

[0087] In one possible implementation, the fourth feature extraction unit is configured to:

[0088] Obtaining a residual between a third intermediate structural feature and the first structural feature corresponding to a target structural feature layer, where the target structural feature layer is any structural feature layer;

[0089] Compress each feature channel in the residual to obtain the feature value of each feature channel;

[0090] Based on the eigenvalues ​​of each feature channel, the weight of each feature channel is obtained, and the weight of the feature channel indicates the importance of the feature channel to the structural feature;

[0091] Based on the weights of the respective feature channels, each feature channel in the third intermediate structural feature corresponding to the target structural feature layer is transformed to obtain the second structural feature corresponding to the target structural feature layer.

[0092] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement a training method for an image processing model such as any one of the above-mentioned image processing models, or any one of the above-mentioned image processing methods.

[0093] On the other hand, a computer-readable storage medium is provided, which stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement a training method for any of the above-mentioned image processing models, or any of the above-mentioned image processing methods.

[0094] On the other hand, a computer program product is provided, which includes at least one computer program, which is loaded and executed by a computer device to implement a training method for an image processing model as described above, or any image processing method as described above.

[0095] In the technical solution provided in the present application, a training method for an image processing model and an image processing method are provided. The training method includes: filling the texture features of the defective area in the sample defective image based on the texture features of the effective area in the sample defective image and the rough structural features of the sample defective image to obtain the complete texture features of the sample defective image; further, obtaining the refined structural features of the sample defective image based on the complete texture features and the rough structural features; after obtaining the predicted repaired image of the sample defective image based on the complete texture features and the refined structural features, determining the loss value to achieve model training. The image processing method performs image repair based on the image processing model trained by the above training method. Since the above training method realizes the interactive guidance between texture features and structural features, when performing image repair based on the trained image processing model, the texture and structure of the defective area and the effective area in the repaired image can be kept consistent in terms of fineness, and the image repair effect is better. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0097] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present application;

[0098] Figure 2 This is a schematic diagram of the model architecture of an image processing model provided in an embodiment of the present application;

[0099] Figure 3 This is a flowchart of a method for training an image processing model provided in an embodiment of the present application;

[0100] Figure 4 is a schematic diagram of a texture enhancement unit provided in an embodiment of the present application;

[0101] Figure 5 is a schematic diagram of a structural reinforcement unit provided in an embodiment of the present application;

[0102] Figure 6 This is a flowchart of an image processing method provided by an embodiment of the present application;

[0103] Figure 7 This is a schematic diagram of the effect of an image processing method provided by an embodiment of the present application;

[0104] Figure 8This is a comparison chart of the processing results of an image processing method provided by an embodiment of the present application and related technologies;

[0105] Figure 9 1 is a schematic diagram of a mask shape test result provided in an embodiment of the present application;

[0106] Figure 10 Schematic diagram of a training device for an image processing model provided in an embodiment of the present application;

[0107] Figure 11 is a structural diagram of an image processing device provided in an embodiment of the present application;

[0108] Figure 12 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0109] Figure 13 This is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0110] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0111] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of methods and systems consistent with certain aspects of the present application, as detailed in the appended claims.

[0112] In this application, the terms "first," "second," and the like are used to distinguish identical or similar items having substantially the same role and function. It should be understood that "first," "second," and "nth" do not have a logical or temporal dependency, nor do they limit the quantity or execution order. It should also be understood that although the following description uses the terms "first," "second," and the like to describe various elements, these elements should not be limited by these terms.

[0113] These terms are only used to distinguish one element from another.For example, without departing from the scope of various examples, a first structural feature can be referred to as a second structural feature, and similarly, a second structural feature can also be referred to as a first structural feature.

[0114] Here, "at least one" means one or more. For example, at least one structural feature can be one structural feature, two structural features, three structural features, or any other integer greater than or equal to one. And "plurality" means two or more. For example, multiple structural features can be two structural features, three structural features, or any other integer greater than or equal to two.

[0115] It should be noted that the information (including but not limited to user device information, user personal information, etc.), data (including but not limited to data used for analysis, storage, display, etc.), and signals involved in this application are all authorized by the user or fully authorized by all parties, and the collection, use, and processing of relevant data must comply with the relevant laws, regulations, and standards of the relevant countries and regions. For example, the sample defect images involved in this application were obtained with full authorization.

[0116] The technical solution provided in this application involves the field of artificial intelligence and can be applied to various scenarios such as image processing, cloud technology, and big data.

[0117] The key technologies involved in this application are introduced below.

[0118] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning.

[0119] Computer vision (CV) technology is the study of how machines can "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing the images to create images more suitable for human observation or transmission to instrumentation. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image processing, image recognition, image semantic understanding, image retrieval, character recognition, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, and simultaneous localization and mapping.

[0120] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge sub-models to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0121] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0122] Next, the implementation environment of the embodiments of the present application is introduced.

[0123] Figure 1 This is a schematic diagram of an implementation environment provided by an embodiment of the present application, see Figure 1 , Figure 1 The illustrated implementation environment includes a terminal 101 and a server 102. The terminal 101 can be connected to the server 102 via a wireless network or a wired network.

[0124] The terminal 101 is a smartphone, tablet computer, laptop computer, desktop computer, smart speaker, smart watch, etc., but is not limited thereto. Illustratively, the terminal 101 is used to execute the image processing method provided in the embodiments of the present application to achieve image restoration. For example, the terminal can install and run a target application, which processes the image to be restored by calling the image processing model to obtain a restored image. For example, the target application can be a social application, a conference application, an image processing application, etc., without limitation to this.

[0125] The server 102 is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud databases, cloud computing, cloud storage, network services, cloud communications, middleware services, domain name services, security services, distribution networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms. Schematically, the server 102 is used to execute the training method of the image processing model provided in the embodiment of the present application, and performs model training based on the information provided by the terminal 101. In other embodiments, the server 102 can be a node in the blockchain system. After the server 102 trains and obtains the image processing model, the server 102 can publish the image processing model to the blockchain system, that is, store it in the blockchain in the form of a block, so that other nodes in the blockchain system can apply the image processing model.

[0126] In some embodiments, the training method of the image processing model and the image processing method provided in the embodiments of the present application can also be jointly executed by the terminal 101 and the server 102. For example, when the training method of the image processing model is jointly executed by the terminal 101 and the server 102, the terminal 101 can send the acquired defective image including the valid area and the defective area and the mask image corresponding to the defective image to the server 102, thereby providing the server 102 with sample defective images and sample mask images for training the image processing model. For another example, when the image processing method is jointly executed by the terminal 101 and the server 102, the terminal 101 can send the acquired image to be repaired to the server 102 and display the repaired image returned by the server 102. Among them, when the terminal 101 and the server 102 jointly execute any of the above methods, the server 102 can undertake the main computing work and the terminal 101 can undertake the secondary computing work; or, the server 102 undertakes the secondary computing work and the terminal 101 undertakes the main computing work; or, the server 102 and the terminal 101 adopt a distributed computing architecture to perform collaborative computing, and the embodiments of the present application do not limit this.

[0127] In some embodiments, the wired or wireless network uses standard communication technologies and / or protocols. The network is typically the Internet, but can also be any network, including but not limited to a local area network (LAN), a metropolitan area network (MAN), a wide area network (WAN), a mobile, wired or wireless network, a dedicated network, or any combination of virtual private networks. In some embodiments, technologies and / or formats including Hypertext Markup Language (HTML), Extensible Markup Language (XML), etc. are used to represent data exchanged over the network. In addition, conventional encryption technologies such as Secure Socket Layer (SSL), Transport Layer Security (TLS), Virtual Private Network (VPN), Internet Protocol Security (IPsec), etc. can be used to encrypt all or some links. In other embodiments, customized and / or dedicated data communication technologies can be used to replace or supplement the above-mentioned data communication technologies.

[0128] The above describes the training method of an image processing model and the implementation environment of the image processing method provided by the embodiment of the present application. The following describes the model architecture of the image processing model provided by the embodiment of the present application. Figure 2 As shown, Figure 2 2 is a schematic diagram of a model architecture of an image processing model provided in an embodiment of the present application. The image processing model includes an input layer 201, a texture feature layer 202, a structural feature layer 203 and an output layer 204.

[0129] The input layer 201 is used to input the defective image (also called the image to be repaired) into the image processing model, the texture feature layer 202 is used to output the texture features of the defective image, the structural feature layer 203 is used to output the structural features of the defective image, and the output layer 204 is used to output the predicted repaired image.

[0130] Schematically, for any texture feature layer 202, the texture feature layer 202 includes an encoding unit, a texture enhancement unit, and a decoding unit. The encoding unit is used to extract texture features of the effective area of ​​the defective image; the texture enhancement unit is used to obtain texture features of the effective area and the defective area of ​​the defective image based on the texture features of the effective area of ​​the defective image and the coarse structural features of the defective image; the decoding unit is used to obtain a predicted repaired image of the defective image based on the texture features and coarse structural features of the effective area and the defective area of ​​the defective image. In some embodiments, the encoding unit includes a convolution layer and an activation function layer, wherein the convolution layer is used to extract the texture features of the effective area of ​​the defective image, and the activation function layer is used to output feature channels of the texture features of the effective area. In some embodiments, the decoding unit includes a convolution layer and an activation function layer, wherein the convolution layer is used to extract the size of the features of the defective image, and the activation function is used to output the corresponding feature channels. It should be noted that the specific processing method of the texture feature layer 202 will be described in detail in subsequent method embodiments and will not be repeated here.

[0131] Schematically, for any structural feature layer 203, the structural feature layer 203 includes an encoding unit, a structure enhancement unit, and a decoding unit. The encoding unit is used to extract the first intermediate structural features of the defective image based on the texture features of the effective area extracted by the encoding unit of the bottommost texture feature layer; the decoding unit is used to obtain the coarse structural features of the defective image based on the first intermediate structural features of the defective image; and the structure enhancement unit obtains the refined structural features of the defective image based on the texture features and coarse structural features of the effective area and the defective area of ​​the defective image. In some embodiments, the encoding unit includes a convolution layer and an activation function layer, wherein the convolution layer is used to extract the first intermediate structural features of the defective image, and the activation function layer is used to output the feature channels of the first intermediate structural features. In some embodiments, the decoding unit includes a convolution layer and an activation function layer, wherein the convolution layer is used to extract the size of the structural features of the defective image, and the activation function is used to output the corresponding feature channels. It should be noted that the specific processing method of the structural feature layer 203 will be described in detail in the subsequent method embodiments and will not be repeated here.

[0132] In some embodiments, as Figure 2 As shown in the figure, the image processing model includes 3 texture feature layers AC and 5 structural feature layers DH. The size of the defective image input to the image processing model is 256×256. The sizes of the features corresponding to the 3 texture feature layers AC are 256×256, 128×128 and 64×64 respectively. The sizes of the features corresponding to the 5 structural feature layers DH are 32×32, 16×16, 8×8, 4×4, 2×2 and 1×1 respectively.

[0133] The above introduces the model architecture of an image processing model provided in an embodiment of the present application. Based on the above model architecture, the following introduces the process of the training method of the image processing model provided in an embodiment of the present application and the process of the image processing method.

[0134] The following describes a process for training an image processing model provided in an embodiment of the present application. Figure 3 This is a flowchart of a training method for an image processing model provided in an embodiment of the present application, which is described by taking the method executed by a server as an example. Figure 3 As shown, the method includes the following steps 301 to 306.

[0135] 301. The server inputs a sample defective image and a sample mask image into an image processing model. The sample mask image is used to indicate a defective area and a valid area of ​​the sample defective image. The image processing model includes a texture feature layer and a structural feature layer.

[0136] Among them, the sample mask image is a binary image, including two types of pixel values ​​0 and 1. The pixels in the sample mask image correspond one-to-one to the pixels in the sample defective image. Schematically, the area where the pixels with a pixel value of "0" in the sample mask image are located indicates the defective area of ​​the sample defective image, that is, the area to be repaired, and the area where the pixels with a pixel value of "1" in the sample mask image are located indicates the valid area of ​​the sample defective image, that is, the area that does not need to be repaired.

[0137] In some embodiments, the real image (i.e., the complete image) is merged with the sample mask image to obtain a sample defective image. The process of merging the real image with the sample mask image is: performing an “AND” operation on the pixel value of each pixel of the real image and the pixel value of the corresponding pixel in the sample mask image to obtain the sample defective image. This process can be expressed by the following formula (1):

[0138] I im =I gt ☉(1-M)+M (1)

[0139] In formula (1), I im represents the sample defective image, I gt Represents the real image, M represents the sample mask image, and ⊙ represents the pixel-level inner product.

[0140] It should be noted that the embodiment of the present application uses an example of inputting a sample defective image and a sample mask image corresponding to the sample defective image to illustrate the training method of the image processing model. Each iteration of the model training can use multiple training samples, and the embodiment of the present application does not limit this.

[0141] 302. The server extracts a first texture feature of the sample defective image through a texture feature layer, and extracts a first structural feature of the sample defective image through a structural feature layer.

[0142] The first texture feature indicates the texture feature of the valid area in the sample defect image, and the first structural feature indicates the rough structural feature of the sample defect image, or in other words, the coarse structural feature of the sample defect image. It should be understood that since the first structural feature is obtained by downsampling the first texture feature, that is, it is obtained based on incomplete texture features, the structure of the defect area indicated by the first structural feature is relatively coarse and has low clarity.

[0143] The following describes in detail the process of extracting the first texture feature and the first structural feature of the sample defect image. The process includes the following steps 3021 to 3023.

[0144] 3021. Downsample the sample defective image through the coding units in each texture feature layer to obtain a first texture feature corresponding to each texture feature layer.

[0145] The server sequentially downsamples the sample defective image through the encoding units in each texture feature layer to obtain the first texture features corresponding to each texture feature layer. Based on the first texture features corresponding to each texture feature layer and the sample mask image, the server obtains the mask image of the first texture feature corresponding to each texture feature layer. The mask image of the first texture feature indicates the defective area in the first texture feature. In this process, the sample defective image is downsampled once, and the number of convolution kernels corresponding to the next layer is doubled to that of the previous layer. The number of feature images obtained is also doubled, and the size of each feature image is reduced to half of the original size.

[0146] refer to Figure 2The image processing model shown includes three sequentially connected texture feature layers AC. For example, the encoding unit in texture feature layer A convolves the sample defective image with a convolution kernel to obtain a feature image, each of which has the same resolution as the sample defective image, where a is a positive integer. Based on the a feature images, a feature channel is output to obtain the first texture feature corresponding to texture feature layer A. Based on the first texture feature corresponding to texture feature layer A and the sample mask image, a mask image of the first texture feature corresponding to texture feature layer A is obtained, and the mask image indicates the defective area in the first texture feature. The encoding unit in texture feature layer B downsamples the first texture feature corresponding to texture feature layer A, reducing the size of the first texture feature corresponding to texture feature layer A to half of its original size, thereby obtaining a reduced feature image. The reduced feature image a is convolved with b convolution kernels to obtain b feature images, each of which has the same resolution as the reduced feature image a, where b is a positive integer and b = 2a. Based on the b feature images, b feature channels are output to obtain the first texture feature corresponding to texture feature layer B. Based on the first texture feature corresponding to texture feature layer B and the sample mask image, a mask image of the first texture feature corresponding to texture feature layer B is obtained. The process of downsampling the first texture feature corresponding to texture feature layer B by texture feature layer C is similar to the above process and will not be repeated here.

[0147] 3022. Downsample the first texture features corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain first intermediate structural features corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer.

[0148] Among them, the server sequentially downsamples the first texture features corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain the first intermediate structural features corresponding to each structural feature layer, and obtains the mask image of the first intermediate structural features corresponding to each structural feature layer based on the first intermediate structural features corresponding to each structural feature layer and the sample mask image.

[0149] Continue to refer Figure 2 The image processing model shown in FIG3 includes five sequentially connected structural feature layers DH, and the first texture feature layer is also the texture feature layer C shown in the figure. The process of sequentially downsampling the first texture feature corresponding to the texture feature layer C by the structural feature layer DH is similar to step 3021 and will not be repeated here.

[0150] 3023. Upsample the first intermediate structural features corresponding to the first structural feature layer through the decoding units in each structural feature layer to obtain the first structural features corresponding to each structural feature layer, where the first structural feature layer is the bottommost structural feature layer.

[0151] The server sequentially upsamples the first intermediate structural features corresponding to the first structural feature layer through the decoding units in each structural feature layer to obtain the first structural features corresponding to each structural feature layer. In this process, the sample defective image is upsampled once, and the number of convolution kernels corresponding to the previous layer is reduced to half that of the next layer. The number of feature images obtained is also reduced to half that of the next layer, and the size of each feature image is doubled.

[0152] Continue to refer Figure 2 The image processing model shown in the figure includes five sequentially connected structural feature layers DH, where the first structural feature layer is the structural feature layer H shown in the figure. For example, the decoding unit in the structural feature layer G upsamples the first intermediate structural feature output by the structural feature layer E to obtain the first structural feature corresponding to the structural feature layer G; the structural feature layer F upsamples the first structural feature output by the structural feature layer G to obtain the first structural feature corresponding to the structural feature layer F; the process of other structural feature layers upsampling the first structural feature corresponding to the next structural feature layer is similar to the above process and will not be repeated here.

[0153] It should be noted that the above step 302 is explained by taking the example of performing convolution on the feature image during the downsampling and upsampling process, and the obtained feature image has the same resolution as the image before the convolution. In other embodiments, the feature image is convolved, and the obtained feature image is smaller than the feature image before the convolution. The embodiments of the present application do not limit the convolution method adopted.

[0154] 303. The server processes the first texture feature and the first structural feature through a texture feature layer to obtain a second texture feature of the sample defective image.

[0155] The second texture feature indicates the texture features of the valid area and the defective area of ​​the sample defective image, that is, the complete texture feature.

[0156] The following describes in detail a process of obtaining a second texture feature by processing the first texture feature and the first structural feature through the texture feature layer. The process includes the following steps 3031 to 3032.

[0157] 3031. Upsample the first structural features corresponding to the second structural feature layer through the decoding units in each texture feature layer to obtain second intermediate structural features corresponding to each texture feature layer. The second structural feature layer is the topmost structural feature layer.

[0158] Continue to refer Figure 2 The image processing model shown in FIG. 1 includes three sequentially connected texture feature layers AC and five sequentially connected structure feature layers DH. Let X be the structure feature corresponding to the second structure feature layer (i.e., structure feature layer D), Y n The first texture feature corresponding to the nth texture feature layer, n is a positive integer. The decoding unit in the nth texture feature layer performs (4-n) upsampling on X to obtain the second intermediate structure feature X corresponding to the nth texture feature layer. n , where X n With Y n have the same resolution.

[0159] In step 3031, the decoding unit in each texture feature layer upsamples the first structural feature corresponding to the second structural feature layer to obtain the second intermediate structural feature corresponding to each texture feature layer. The process is the same as step 3023 in step 302 and will not be repeated here.

[0160] 3032. Process the second intermediate structural features and the first texture features corresponding to each texture feature layer through the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer.

[0161] Reference below Figure 4 , taking any texture feature layer as an example (hereinafter referred to as the target texture feature layer), step 3032 is introduced as follows: Figure 4 As shown, Figure 4 30 is a schematic diagram of a texture enhancement unit provided in an embodiment of the present application, wherein step 3032 includes the following steps 1 and 2.

[0162] Step 1: Based on the structural features of each pixel in the second intermediate structural feature corresponding to the target texture feature layer, an attention matrix corresponding to the target texture feature layer is obtained, wherein the attention matrix indicates the affinity between any two pixels in the second intermediate structural feature corresponding to the target texture feature layer.

[0163] The affinity between pixels refers to the similarity between two pixels. The server obtains the attention matrix corresponding to the target texture feature layer based on the structural features of any two pixels in the second intermediate structural features. This process refers to the following formula (2):

[0164]

[0165] In formula (2), is the affinity between the pixels at the i-th position and the j-th position, is the inner product of the structural features of the pixels at the i-th position and the j-th position; k indicates any pixel; N indicates the number of pixels in the second intermediate structural feature. The server is based on the affinity between any two pixels in the second intermediate structural feature. Determine the attention matrix A corresponding to the nth texture feature layer n .

[0166] Step 2: Based on the attention matrix, the mask image corresponding to the target texture feature layer, and the first texture feature corresponding to the target texture feature layer, obtain the second texture feature corresponding to the target texture feature layer.

[0167] In this step, the server replaces the target area in the second intermediate structural feature with the defective area of ​​the first texture feature corresponding to the target texture feature layer to obtain the replaced first texture feature. The target area refers to the area corresponding to the defective area in the first texture feature. The process is expressed by the following formula (3):

[0168]

[0169] In formula (3), is the first texture feature after replacement, Y n is the first texture feature corresponding to the target texture feature layer, X n is the second intermediate structural feature corresponding to the nth texture feature layer, M n The server converts the first texture feature corresponding to the target texture feature layer into a mask image of the first texture feature corresponding to the target texture feature layer. The second intermediate structural feature X corresponding to the nth texture feature layer n Reshape to N×C size, where N is The number of pixels included in a feature channel, C is The number of feature channels in , that is, and X n Each feature channel in is compressed from H×W size to 1×N size, where N, C, H and W are all positive integers, and H and W are The height and width of the feature channel in the middle. Through this process, the target area in the second intermediate structural feature is replaced with the defective area of ​​the first texture feature corresponding to the target texture feature layer. Since the second intermediate structural feature is obtained through multiple depth sampling, the second intermediate structural feature includes more information about the defective area than the first texture feature, so the first texture feature can be filled to obtain a richer texture feature.

[0170] Furthermore, the second texture feature corresponding to the target texture feature layer is obtained by the following formula (4):

[0171]

[0172] In formula (4), represents the second texture feature corresponding to the target texture feature layer, and F(·) represents the reshaping operation from N×C to H×W×C, that is, converting each feature channel from 1×C size to H×W size, where N, C, H and W are all positive integers, and H and W are respectively The height and width of the feature channel in ,γ is a learnable parameter, represents matrix multiplication, is the first texture feature after replacement, A n is the attention matrix corresponding to the target texture feature layer, A n The dimension is N×N, A n The value at the corresponding position in M n It is the mask image of the first texture feature corresponding to the target texture feature layer.

[0173] In the above embodiment, step 3032 is explained by taking step 1 as an example and then step 2 as an example. In some embodiments, the process of obtaining the replaced first texture feature in step 1 and step 2 in step 3032 is performed synchronously, and the embodiment of the present application does not limit this.

[0174] Through the above steps 1 and 2, an attention matrix is ​​obtained based on the second intermediate structural feature. The first texture feature is enhanced by the attention matrix to obtain the second texture feature. This can establish a long-term dependency between the texture feature and the structural feature in the spatial domain, which is conducive to maintaining consistency in the fineness of the texture and structure of the defective area and the valid area in the repaired image.

[0175] 304. The server processes the second texture feature and the first structural feature through the structural feature layer to obtain a second structural feature of the sample defective image.

[0176] The second structural feature indicates the refined structural feature of the sample defect image, or in other words, the precise structural feature of the sample defect image. It should be understood that since the second structural feature is obtained based on the second texture feature, that is, based on the complete texture feature, the structure of the defect area indicated by the second structural feature is more refined and has higher clarity than that of the first structural feature, which can ensure that the structural features of the defect area and the valid area remain consistent in terms of refinement.

[0177] The following describes in detail the process of obtaining the second structural feature by processing the second texture feature and the first structural feature through the structural feature layer. The process includes the following steps 3041 to 3042.

[0178] 3041. Downsample the second texture features corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain third intermediate structural features corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer.

[0179] This step 3041 is similar to step 3022 in step 302, except that in step 3041, each texture feature layer aggregates the second texture feature corresponding to the previous texture feature layer with the second texture feature corresponding to the current texture feature layer to obtain the aggregated second texture feature. That is, through the encoding unit in each structural feature layer, the second texture feature after the aggregation of the second texture features corresponding to each texture feature layer is down-sampled to obtain the third intermediate structural feature corresponding to each structural feature layer.

[0180] 3042. Process the third intermediate structural features and the first structural features corresponding to each structural feature layer through the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer.

[0181] Reference below Figure 5 , taking any structural feature layer as an example (hereinafter referred to as the target structural feature layer), step 3042 is introduced as follows: Figure 5 As shown, Figure 5 This is a schematic diagram of a structural reinforcement unit provided in an embodiment of the present application. Step 3042 includes the following steps 1 to 4.

[0182] Step 1: Obtain the residual between the third intermediate structural feature and the first structural feature corresponding to the target structural feature layer.

[0183] Step 2: Compress each feature channel in the residual to obtain a feature value of each feature channel. The server compresses each feature channel in the residual from two dimensions to a feature value, which indicates the description information of the feature channel.

[0184] Step 3: Based on the eigenvalues ​​of each feature channel, the weight of each feature channel is obtained. The weight of the feature channel indicates the importance of the feature channel to the structural feature. The server inputs the eigenvalues ​​of each feature channel into three fully connected layers to obtain the weight of each feature channel.

[0185] Step 4: Based on the weights of each feature channel, transform each feature channel in the third intermediate structural feature corresponding to the target structural feature layer to obtain a second structural feature corresponding to the target structural feature layer. The server multiplies the weight of each feature channel by the corresponding feature channel to obtain the second structural feature corresponding to the target structural feature layer.

[0186] The above steps 1 to 4 can be expressed by the following formula (5):

[0187]

[0188] In formula (5), Indicates the second structural feature corresponding to the target structural feature layer, S r Indicates the third intermediate structural feature corresponding to the target structural feature layer, S c Indicates the first structural feature corresponding to the target structural feature layer, |S r -S c | represents the residual between the third intermediate structural feature and the first structural feature corresponding to the target structural feature layer, Indicates compressing the information of the feature channel in the residual from two dimensions to one eigenvalue. represents the multiplication in the feature channel direction, and FC represents the fully connected layer in the structure enhancement unit corresponding to the target structure feature layer.

[0189] Through the above steps, the weights of each feature channel are determined based on the residual between the third intermediate structural feature and the first structural feature. Since the third intermediate structural feature is obtained based on the second texture feature (complete texture feature), it can more finely represent the structural information than the first structural feature. Therefore, the weights of each feature channel in the third intermediate structural feature are enhanced based on the weights of each feature channel. This can strengthen the importance of each feature channel to the structural feature and adaptively re-calibrate the feature channel response of the third intermediate structural feature, thereby bringing better image restoration results. In contrast, the related technology, based on the structural features obtained after downsampling the coding units in each structural feature layer and upsampling the decoding units in each structural feature layer, often results in blurred semantics and unclear structures in the image restoration results. In particular, when the area of ​​the defective area is large, the effect of repairing the image is poor.

[0190] 305. The server obtains a predicted repaired image of the sample defective image based on the second texture feature and the second structural feature.

[0191] Among them, the predicted repair image refers to the completed image output by the image processing model after processing the sample defective image.

[0192] The process by which the server obtains a predicted repair image of a sample defective image based on the second texture feature and the second structural feature includes: upsampling the second structural feature corresponding to the target structural feature layer through the decoding unit in each structural feature layer to obtain the fourth intermediate structural feature corresponding to each structural feature layer, where the target structural feature layer is any structural feature layer; and upsampling the fourth intermediate structural feature corresponding to the topmost structural feature layer based on the corresponding second texture feature by the decoding unit in each texture feature layer to obtain a predicted repair image of the sample defective image.

[0193] Among them, the process of upsampling the second structural features corresponding to the target structural feature layer through the decoding units in each structural feature layer to obtain the fourth intermediate structural features corresponding to each structural feature layer is the same as step 3023 in step 302 and will not be repeated; the process of upsampling the fourth intermediate structural features corresponding to the topmost structural feature layer based on the corresponding second texture features by the decoding units in each texture feature layer is the same as step 3023 in step 302, except that in step 305, the corresponding second texture features and the corresponding fourth intermediate structural features are spliced ​​to fuse the features, and step 3023 in step 302 fuses the features by adding them, and similar steps are not repeated.

[0194] 306. The server trains the image processing model based on the real image of the sample defective image, the predicted repair image, and the sample mask image.

[0195] The server determines a target loss value for the image processing model based on the actual image of the sample defective image, the predicted repaired image, and the sample mask image. Based on the target loss value, the image processing model is trained until the training end condition is met, thereby obtaining a trained image processing model. This process includes steps 3061 to 3063.

[0196] 3061. Input the predicted restoration image into the image discriminator to obtain a discrimination result of the predicted restoration image, and determine a first loss value of the image processing model based on the discrimination result of the predicted restoration image, the predicted restoration image, and the real image.

[0197] The judgment result of the predicted restoration image indicates the probability of restoration failure of each pixel in the predicted restoration image.

[0198] Schematically, the process of determining the first loss value of the image processing model includes: based on an exponential function, converting the probability of failure to repair each pixel in the predicted repair image into the weight of each pixel, wherein the weight of the pixel indicates the importance of the pixel to the predicted repair image; determining the difference between the pixel value of each pixel in the predicted repair image and the pixel value of the corresponding pixel in the real image; based on the weight of each pixel, transforming the difference corresponding to each pixel in the predicted repair image and the real image to obtain the first loss value of the image processing model. The above process can be expressed by the following formula (6):

[0199]

[0200] In formula (6), G is the image processing model, D is the image discriminator, G(I im , M) is the predicted restoration image output by the image processing model, D(G(I im , M)) is the probability of failure to repair each pixel in the predicted repaired image output by the image discriminator, I gt is a real image, ⊙ represents the pixel-level inner product, and x is the base of the exponential function. In some embodiments, the value of x is set to 10, which is not limited.

[0201] 3062. Determine a second loss value of the image processing model based on a distance between a target repair image and a real image in a feature space, wherein the target repair image is obtained by merging the predicted repair image and the sample defective image.

[0202] In some embodiments, the process by which the server determines the distance between the target repaired image and the real image in the feature space includes: merging the predicted repaired image and the sample defective image to obtain the target repaired image; inputting the target repaired image into the image enhancement model to obtain the texture features and structural features of the target repaired image; inputting the real image into the image enhancement model to obtain the texture features and structural features of the real image; and determining the distance between the target repaired image and the real image in the feature space based on the texture features and structural features of the target repaired image and the texture features and structural features of the real image.

[0203] The server merges the predicted restoration image and the sample defective image to obtain the target restoration image, that is, replaces the valid area in the sample defective image with the corresponding area of ​​the predicted restoration image. This process can be expressed by the following formula (7):

[0204] I re =(1-M)☉I im +M☉I pred (7)

[0205] In formula (7), I re Repair the image for the target, I imis the sample defect image, I pred is the predicted restoration image, and M is the sample mask image.

[0206] In some embodiments, the image enhancement model is a VGG16 pre-trained model trained on a dataset. The VGG16 pre-trained model includes multiple convolutional layers. Inputting an image into the VGG16 pre-trained model can obtain the texture features and structural features of the image corresponding to each convolutional layer. The above embodiment is merely an exemplary description of the image enhancement model, and the image enhancement model can be customized according to actual needs, which is not limited in this embodiment of the present application.

[0207] Taking the VGG16 pre-trained model as an example, the process of determining the second loss value of the image processing model based on the distance between the target repaired image and the real image in the feature space can be determined by the following formula (8):

[0208]

[0209] In formula (8), L perc is the second loss value of the image processing model, Φ t (·) is the t-th layer output of the VGG16 pre-trained model, I re Repair the image for the target, I gt is a real image, t indicates a convolutional layer of the VGG16 pre-trained model, and L is the number of convolutional layers in the VGG16 pre-trained model.

[0210] 3063. Based on the first loss value and the second loss value, determine a target loss value of the image processing model, and train the image processing model based on the target loss value until a training end condition is met, thereby obtaining a trained image processing model.

[0211] The server determines the target loss value based on the first loss value and the second loss value. The training strategy for training the image processing model can be expressed by the following formula (9):

[0212]

[0213] In formula (9), λ is a trade-off weight. In some embodiments, the value of λ is set to 0.05. The meanings of other symbols in the formula are the same as those in steps 3061 and 3062 and are not repeated here.

[0214] The process of training an image processing model based on the loss value of the image processing model includes: if the target loss value meets the training end condition, outputting the image processing model; if the target loss value does not meet the training end condition, updating the model parameters of the image processing model, and performing the next iteration based on the updated image processing model until the training end condition is met. In some embodiments, the training end condition is that the loss value of the image processing model is less than the target value, and the target value is the minimum value during the most recent W iterations, where W is a positive integer; in other embodiments, the training end condition is that the difference between the loss value of this iteration and the loss value of the previous iteration is less than or equal to a preset threshold, which can also be understood as the change curve of the loss value tending to be flat. The embodiments of the present application do not limit the specific form of the training end condition.

[0215] Through steps 3061 to 3063 above, the process of training the image processing model is described. In some embodiments, the image discriminator is also trained during the image processing model training process. Illustratively, the image discriminator training process includes: determining a loss value for the image discriminator based on the discrimination result of the predicted inpainted image and the sample mask image; and training the image discriminator based on the loss value for the image discriminator.

[0216] The above process can be expressed by the following formula (10):

[0217]

[0218] In formula (10), L focal (V, M) is the loss value of the image discriminator, V represents the probability of failure to repair each pixel in the predicted repair image output by the image discriminator, M is the sample mask image, α is the weight factor, and the value of α is the mask ratio of the sample defective image, where the mask ratio of the sample defective image refers to the ratio of the number of pixels with a pixel value of "0" in the sample mask image to the total number of pixels in the sample mask image, m indicates a pixel in the predicted repair image, and N is the number of pixels in the predicted repair image. is an adjustable focusing parameter. In some embodiments, The value of is set to 2, which is not limited.

[0219] Based on the loss value of the image discriminator, the training strategy adopted in the process of training the image discriminator is to minimize the loss value of the image discriminator. The training strategy can be expressed by the following formula (11):

[0220]

[0221] In formula (11), D is the image discriminator, G is the image processing model in the embodiment of the present application, G(I im, M) is the predicted restoration image output by the image processing model, D(G(I im , M)) is the probability of failure to repair each pixel in the predicted repair image output by the image discriminator, L focal (D(G(I im , M)), M) is the loss value of the image discriminator.

[0222] Through the above steps 301 to 306, a training method for an image processing model provided in an embodiment of the present application is introduced. In the above scheme, steps 301 to 303 can be understood as the first stage of the image processing model processing the sample defective image. Through this stage, the complete texture features of the sample defective image are obtained; steps 304 to 305 can be understood as the second stage of the image processing model processing the sample defective image. Through this stage, the refined structural features of the sample defective image are obtained. Schematically, the image processing model divides the texture features and structural features of the sample defective image into two information streams. The texture feature stream is processed in the first stage, with the texture features of the defective area as input and the coarse structural features as intermediary to synthesize the texture features of the valid area and the defective area, that is, the complete texture features; the structural feature stream is processed in the second stage, with the coarse structural features as the starting point and the complete texture features as the intermediary to supplement the structural features and obtain the refined structural features. The interactive guidance between the texture features and the structural features improves the ability of the image processing model to repair the texture features and structural features in the complementarity, thereby effectively improving the accuracy of the image processing model.

[0223] In summary, the above scheme fills in the texture features of the defective region in the sample defective image based on the texture features of the valid region and the coarse structural features of the sample defective image, obtaining a complete texture feature. Furthermore, based on the complete texture features and the coarse structural features, a refined structural feature is obtained. Finally, based on the complete texture features and the refined structural features, a predicted inpainted image of the sample defective image is obtained. This achieves interactive guidance between texture and structural features, ensuring that the texture and structure of the defective region and the valid region in the inpainted image remain consistent in detail, resulting in a good image inpainting effect.

[0224] The following describes the process of the image processing method provided in the embodiment of the present application. Figure 6 This is a flowchart of an image processing method provided by an embodiment of the present application, which is described by taking the method executed by a server as an example. Figure 6 As shown, based on the image processing model trained in the above steps 301 to 306, the method includes the following steps 601 to 605.

[0225] 601. Input an image to be repaired into an image processing model, where the image includes a valid area and a defective area, and the image processing model includes a texture feature layer and a structural feature layer.

[0226] 602. The server extracts a first texture feature of the image through the texture feature layer of the image processing model, and extracts a first structural feature of the image through the structural feature layer of the image processing model.

[0227] The image to be repaired includes a valid area and a defective area, the first texture feature indicates a texture feature of the valid area of ​​the image, and the first structural feature indicates a rough structural feature of the image.

[0228] The server extracts the first texture feature of the image through the texture feature layer of the image processing model, and the process of extracting the first structural feature of the image through the structural feature layer of the image processing model includes: downsampling the image through the encoding units in each texture feature layer to obtain the first texture feature corresponding to each texture feature layer; downsampling the first texture feature corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain the first intermediate structural feature corresponding to each structural feature layer, and the first texture feature layer is the bottommost texture feature layer; upsampling the first intermediate structural feature corresponding to the first structural feature layer through the decoding units in each structural feature layer to obtain the first structural feature corresponding to each structural feature layer, and the first structural feature layer is the bottommost structural feature layer.

[0229] The above 602 is similar to step 302 and will not be described in detail.

[0230] 603. The server processes the first texture feature and the first structural feature through a texture feature layer to obtain a second texture feature of the image.

[0231] The second texture feature indicates the texture features of the valid area and the defective area of ​​the image, that is, the complete texture feature.

[0232] The server processes the first texture feature and the first structural feature through the texture feature layer to obtain the second texture feature of the image, including: upsampling the first structural feature corresponding to the second structural feature layer through the decoding unit in each texture feature layer to obtain the second intermediate structural feature corresponding to each texture feature layer, where the second structural feature layer is the topmost structural feature layer; and processing the second intermediate structural feature and the first texture feature corresponding to each texture feature layer through the texture enhancement unit in each texture feature layer to obtain the second texture feature corresponding to each texture feature layer.

[0233] In some embodiments, the second intermediate structural features and first texture features corresponding to each texture feature layer are processed by the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer. The process includes: based on the structural features of each pixel in the second intermediate structural features corresponding to the target texture feature layer, obtaining the attention matrix corresponding to the target texture feature layer, the attention matrix indicating the affinity between any two pixels in the second intermediate structural features corresponding to the target texture feature layer, and the target texture feature layer is any texture feature layer; based on the attention matrix, the mask image corresponding to the target texture feature layer and the first texture features corresponding to the target texture feature layer, obtaining the second texture features corresponding to the target texture feature layer.

[0234] 604. The server processes the second texture feature and the first structural feature through the structural feature layer to obtain a second structural feature of the image.

[0235] The server processes the second texture feature and the first structural feature through the structural feature layer to obtain the second structural feature of the image, including: down-sampling the second texture feature corresponding to the first texture feature layer through the encoding unit in each structural feature layer to obtain the third intermediate structural feature corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer; processing the third intermediate structural feature and the first structural feature corresponding to each structural feature layer through the structure enhancement unit in each structural feature layer to obtain the second structural feature corresponding to each structural feature layer.

[0236] In some embodiments, the third intermediate structural features and the first structural features corresponding to each structural feature layer are processed by the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer. The process includes: obtaining the residual between the third intermediate structural features and the first structural features corresponding to the target structural feature layer, where the target structural feature layer is any structural feature layer; compressing each feature channel in the residual to obtain the feature values ​​of each feature channel; obtaining the weight of each feature channel based on the feature values ​​of each feature channel, the weight of the feature channel indicating the importance of the feature channel to the structural feature; transforming each feature channel in the third intermediate structural feature corresponding to the target structural feature layer based on the weight of each feature channel to obtain the second structural feature corresponding to the target structural feature layer.

[0237] 605. The server obtains a restored image of the image based on the second texture feature and the second structural feature.

[0238] The above steps 603 to 605 are similar to steps 303 to 305 and are not described in detail.

[0239] Through this approach, the texture features of the defective area in the image to be repaired are filled in based on the texture features of the valid area in the image to be repaired and the coarse structural features of the sample defective image, resulting in a complete texture feature. Furthermore, based on the complete texture features and the coarse structural features, a refined structural feature is obtained. Finally, based on the complete texture features and the refined structural features, the inpainted image to be repaired is obtained. This achieves interactive guidance between texture and structural features, ensuring that the texture and structure of the defective area and the valid area in the repaired image remain consistent in detail, resulting in a good image restoration effect.

[0240] It should be noted that the above embodiment takes the image restoration scene as an example to introduce the process of the image processing method in the embodiment of the present application. In addition to the image restoration scene, the image processing method provided by the embodiment of the present application can also be applied to the scene of removing objects in the image to obtain better removal effects, such as Figure 7 As shown, Figure 7 This is a schematic diagram of the effect of an image processing method provided in an embodiment of the present application. The image processing method provided in an embodiment of the present application can be used to remove decorations such as glasses and earrings in an image, magazine border styles in an image, vehicles on the street in an image, and people in the scenery in an image, etc.

[0241] The following qualitative and quantitative analysis of the effects of the image processing model in the embodiments of the present application is conducted.

[0242] First, the results of quantitative analysis of five image restoration models are introduced, among which the five image restoration models are: the image processing model in the embodiment of the present application, the gated convolution (GConv) model, the conditional texture and structure dual generation (CTSDG) model, the mutual encoder-decoder with feature equalization (MEDFE) model and the diverse structure for image inpainting (DSI) model. For the sake of convenience, W-net is used in the following introduction to refer to the image processing model in the embodiment of the present application. As shown in Table 1, Table 1 is the numerical results of the five image restoration models provided by the embodiment of the present application on three public datasets: the wild deep learning face attribute dataset (CelebA-HQ), the Paris street view dataset (ParisSreetView) and the tens of millions of image datasets (Places) for scene recognition.

[0243] Among them, the lower the Learned Perceptual Image Patch Similarity (LPIPS) value, the better, while the higher the Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) values, the better. 1%-10% represents the percentage of the area of ​​the defective region to the total area of ​​the image. As can be seen from Table 1, the image processing model in the embodiment of the present application can achieve better performance in most cases compared to related technologies.

[0244] Table 1

[0245]

[0246]

[0247] The following describes the qualitative analysis results of the above five image restoration models in the embodiment of the present application. Figure 8 As shown, Figure 8 This is a comparison chart of the processing results of an image processing method provided by an embodiment of the present application and related technologies. The image processing method in the embodiment of the present application obtains a powerful ability to reconstruct facial shape and infer the structure of the defective area through rich depth sampling and structure enhancement units. GConv, as a model based on the residual network (ResNet), is not sufficient to process fine textures and structures, and the CTSDG model performs even worse due to the lack of effective structural refinement in depth sampling. In addition, due to the insufficient organization structure between the structural and texture features in the generators of the MEDFE model and the DSI model, they are unable to synthesize reasonable textures on the correct structure. Only the image processing model in the embodiment of the present application effectively utilizes the texture features and structural features of the effective area to synthesize content, such as facial symmetry and similar patterns, and obtains a repaired image with higher authenticity, such as Figure 8 This interactive guidance based on structure and texture is crucial to achieving visually realistic image restoration.

[0248] In order to evaluate the universality of the defect area shape in the mask image, the embodiment of the present application tested the central square mask image without additional training. The results are as follows: Figure 9 As shown, Figure 9 Figure 2 is a schematic diagram of a mask shape test result provided by an embodiment of the present application. Compared to the other four methods, the image processing method in this embodiment produces fewer visual artifacts in the inpainted image and is closer to reality. Visual artifacts refer to pixels in the inpainted image where inpainting failed.

[0249] Figure 10This is a structural diagram of a training device for an image processing model provided in an embodiment of the present application, see Figure 10 The device includes: a sample input module 1001, a first feature extraction module 1002, a second feature extraction module 1003 and a model training module 1004.

[0250] A sample input module 1001 is used to input a sample defect image and a sample mask image into an image processing model, where the sample mask image is used to indicate a defect area and a valid area of ​​the sample defect image. The image processing model includes a texture feature layer and a structural feature layer.

[0251] A first feature extraction module 1002 is configured to extract a first texture feature of the sample defect image through a texture feature layer, and extract a first structural feature of the sample defect image through a structural feature layer, wherein the first texture feature indicates a texture feature of a valid area, and the first structural feature indicates a rough structural feature of the sample defect image;

[0252] A second feature extraction module 1003 is configured to process the first texture feature and the first structural feature through a texture feature layer to obtain a second texture feature of the sample defective image, and process the second texture feature and the first structural feature through a structural feature layer to obtain a second structural feature of the sample defective image, wherein the second texture feature indicates texture features of a valid area and a defective area of ​​the image to be repaired, and the second structural feature indicates a refined structural feature of the sample defective image;

[0253] The model training module 1004 is used to obtain a predicted repaired image of the sample defective image based on the second texture feature and the second structural feature, and train the image processing model based on the real image of the sample defective image, the predicted repaired image and the sample mask image.

[0254] In one possible implementation, the first feature extraction module 1002 is configured to:

[0255] Downsampling the sample defective image through the coding units in each texture feature layer to obtain the first texture features corresponding to each texture feature layer;

[0256] Downsampling the first texture feature corresponding to the first texture feature layer through the encoding unit in each structural feature layer to obtain the first intermediate structural feature corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0257] The first intermediate structural features corresponding to the first structural feature layer are up-sampled by the decoding units in each structural feature layer to obtain the first structural features corresponding to each structural feature layer, where the first structural feature layer is the bottommost structural feature layer.

[0258] In one possible implementation, the second feature extraction module 1003 includes:

[0259] A first feature extraction unit is configured to upsample the first structural features corresponding to the second structural feature layer through the decoding units in each texture feature layer to obtain second intermediate structural features corresponding to each texture feature layer, where the second structural feature layer is the topmost structural feature layer;

[0260] The second feature extraction unit is used to process the second intermediate structural features and the first texture features corresponding to each texture feature layer through the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer.

[0261] In one possible implementation, the second feature extraction unit is configured to:

[0262] Based on the structural features of each pixel in the second intermediate structural feature corresponding to the target texture feature layer, obtaining an attention matrix corresponding to the target texture feature layer, the attention matrix indicating the affinity between any two pixels in the second intermediate structural feature corresponding to the target texture feature layer, the target texture feature layer being any texture feature layer;

[0263] Based on the attention matrix, the mask image corresponding to the target texture feature layer, and the first texture feature corresponding to the target texture feature layer, the second texture feature corresponding to the target texture feature layer is obtained.

[0264] In one possible implementation, the second feature extraction module 1003 includes:

[0265] a third feature extraction unit, configured to downsample the second texture features corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain third intermediate structural features corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0266] The fourth feature extraction unit is used to process the third intermediate structural features and the first structural features corresponding to each structural feature layer through the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer.

[0267] In one possible implementation, the fourth feature extraction unit is configured to:

[0268] Obtaining a residual between a third intermediate structural feature and the first structural feature corresponding to a target structural feature layer, where the target structural feature layer is any structural feature layer;

[0269] Compress each feature channel in the residual to obtain the feature value of each feature channel;

[0270] Based on the eigenvalues ​​of each feature channel, the weight of each feature channel is obtained, and the weight of the feature channel indicates the importance of the feature channel to the structural feature;

[0271] Based on the weights of the respective feature channels, each feature channel in the third intermediate structural feature corresponding to the target structural feature layer is transformed to obtain the second structural feature corresponding to the target structural feature layer.

[0272] In one possible implementation, the model training module 1004 includes:

[0273] a first loss value determining unit, configured to input the predicted restoration image into an image discriminator, obtain a discrimination result of the predicted restoration image, and determine a first loss value of the image processing model based on the discrimination result of the predicted restoration image, the predicted restoration image, and the real image, wherein the discrimination result of the predicted restoration image indicates a probability of restoration failure for each pixel in the predicted restoration image;

[0274] a second loss value determining unit, configured to determine a second loss value of the image processing model based on a distance between a target restoration image and a true image in a feature space, wherein the target restoration image is obtained by merging the predicted restoration image and the sample defective image;

[0275] The model training unit is used to determine a target loss value based on the first loss value and the second loss value, and train the image processing model based on the target loss value until a training end condition is met, thereby obtaining a trained image processing model.

[0276] In one possible implementation, the model training module 1004 is further configured to:

[0277] Determine the loss value of the image discriminator based on the discrimination result of the predicted restoration image and the sample mask image;

[0278] Based on the loss value of the image discriminator, the image discriminator is trained.

[0279] In one possible implementation, the model training module 1004 is further configured to:

[0280] Merge the predicted restoration image and the sample defect image to obtain the target restoration image;

[0281] Input the target restoration image into the image enhancement model to obtain the texture features and structural features of the target restoration image;

[0282] Input the real image into the image enhancement model to obtain the texture features and structural features of the real image;

[0283] Based on the texture features and structural features of the target inpainted image and the texture features and structural features of the real image, the distance between the target inpainted image and the real image in the feature space is determined.

[0284] Through the training device of the above-mentioned image processing model, the texture features of the defective area in the sample defective image can be filled based on the texture features of the effective area in the sample defective image and the coarse structural features of the sample defective image, thereby obtaining the complete texture features of the sample defective image; further, based on the complete texture features and the coarse structural features, the refined structural features of the sample defective image can be obtained; after obtaining the predicted repaired image of the sample defective image based on the complete texture features and the refined structural features, the loss value is determined to achieve model training. Because the training device of the above-mentioned image processing model realizes the interactive guidance between texture features and structural features, the texture and structure of the defective area and the effective area in the repaired image can be consistent in terms of fineness, and the image repair effect is good.

[0285] It should be noted that the image processing model training device provided in the above embodiment only uses the division of the above functional modules as an example when executing the corresponding steps. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing model training device provided in the above embodiment and the image processing model training method embodiment are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0286] Figure 11 This is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application, see Figure 11 The device includes: an image input module 1101, a first feature extraction module 1102, a second feature extraction module 1103 and a repaired image acquisition module 1104.

[0287] An image input module 1101 is used to input an image to be repaired into an image processing model, where the image includes a valid area and a defective area, and the image processing model includes a texture feature layer and a structural feature layer;

[0288] A first feature extraction module 1102 is configured to extract a first texture feature of the image using a texture feature layer of the image processing model, and to extract a first structural feature of the image using a structural feature layer of the image processing model, wherein the image includes a valid area and a defective area, the first texture feature indicates a texture feature of the valid area of ​​the image, and the first structural feature indicates a rough structural feature of the image;

[0289] A second feature extraction module 1103 is configured to process the first texture feature and the first structural feature through the texture feature layer of the image processing model to obtain a second texture feature of the image, and process the second texture feature and the first structural feature through the structural feature layer of the image processing model to obtain a second structural feature of the image, wherein the second texture feature indicates texture features of a valid area and a defective area of ​​the image, and the second structural feature indicates a refined structural feature of the image;

[0290] The repaired image acquisition module 1104 is configured to acquire a repaired image of the image based on the second texture feature and the second structural feature.

[0291] In one possible implementation, the first feature extraction module 1102 is configured to:

[0292] Downsampling the image through the coding units in each texture feature layer to obtain first texture features corresponding to each texture feature layer;

[0293] Downsampling the first texture feature corresponding to the first texture feature layer through the encoding unit in each structural feature layer to obtain the first intermediate structural feature corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0294] The first intermediate structural features corresponding to the first structural feature layer are up-sampled by the decoding units in each structural feature layer to obtain the first structural features corresponding to each structural feature layer, where the first structural feature layer is the bottommost structural feature layer.

[0295] In one possible implementation, the second feature extraction module 1103 includes:

[0296] A first feature extraction unit is configured to upsample the first structural features corresponding to the second structural feature layer through the decoding units in each texture feature layer to obtain second intermediate structural features corresponding to each texture feature layer, where the second structural feature layer is the topmost structural feature layer;

[0297] The second feature extraction unit is used to process the second intermediate structural features and the first texture features corresponding to each texture feature layer through the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer.

[0298] In one possible implementation, the second feature extraction unit includes:

[0299] Based on the structural features of each pixel in the second intermediate structural feature corresponding to the target texture feature layer, obtaining an attention matrix corresponding to the target texture feature layer, the attention matrix indicating the affinity between any two pixels in the second intermediate structural feature corresponding to the target texture feature layer, the target texture feature layer being any texture feature layer;

[0300] Based on the attention matrix, the mask image corresponding to the target texture feature layer, and the first texture feature corresponding to the target texture feature layer, the second texture feature corresponding to the target texture feature layer is obtained.

[0301] In one possible implementation, the second feature extraction module 1103 includes:

[0302] a third feature extraction unit, configured to downsample the second texture features corresponding to the first texture feature layer through the encoding units in each structural feature layer to obtain third intermediate structural features corresponding to each structural feature layer, where the first texture feature layer is the bottommost texture feature layer;

[0303] The fourth feature extraction unit is used to process the third intermediate structural features and the first structural features corresponding to each structural feature layer through the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer.

[0304] In one possible implementation, the fourth feature extraction unit is configured to:

[0305] Obtaining a residual between a third intermediate structural feature and the first structural feature corresponding to a target structural feature layer, where the target structural feature layer is any structural feature layer;

[0306] Compress each feature channel in the residual to obtain the feature value of each feature channel;

[0307] Based on the eigenvalues ​​of each feature channel, the weight of each feature channel is obtained, and the weight of the feature channel indicates the importance of the feature channel to the structural feature;

[0308] Based on the weights of the respective feature channels, each feature channel in the third intermediate structural feature corresponding to the target structural feature layer is transformed to obtain the second structural feature corresponding to the target structural feature layer.

[0309] The image processing device described above can fill in the texture features of the defective area of ​​the image to be repaired based on the texture features and coarse structural features of the effective area in the image to be repaired, thereby obtaining the complete texture features of the image; further, based on the complete texture features and coarse structural features, the refined structural features of the image can be obtained; and further, based on the complete texture features and refined structural features, a repaired image of the image can be obtained. Because the image processing device described above implements interactive guidance between texture features and structural features, the texture and structure of the defective area and the effective area in the repaired image can be kept consistent in terms of refinement, resulting in a good image repair effect.

[0310] It should be noted that the image processing apparatus provided in the above embodiments, when executing the corresponding steps, is illustrated only by the division of the above-described functional modules. In actual applications, the above-described functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device can be divided into different functional modules to complete all or part of the functions described above. In addition, the image processing apparatus provided in the above embodiments and the image processing method embodiments are based on the same concept. The specific implementation process is detailed in the method embodiments and will not be repeated here.

[0311] An embodiment of the present application provides a computer device, which includes a processor and a memory, wherein the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the training method of the image processing model provided in the above embodiment, or the image processing method provided in the above embodiment.

[0312] Taking computer equipment as the terminal as an example, Figure 12 This is a schematic diagram of the structure of a terminal provided in an embodiment of the present application. The terminal may be a personal computer (PC), a mobile phone, a smart phone, a personal digital assistant (PDA), a wearable device, a pocket PC (PPC), a tablet computer, a smart car computer, a smart TV, a smart speaker, an intelligent voice interaction device, a smart home appliance, an in-vehicle terminal, etc. The terminal may also be referred to as a user device, a user terminal, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0313] Typically, the terminal includes: a processor 1201 and a memory 1202 .

[0314] The processor 1201 may include one or more processing cores, such as a 4-core processor, an 8-core processor, and the like. The processor 1201 may be implemented in at least one hardware form of digital signal processing (DSP), field-programmable gate array (FPGA), and programmable logic array (PLA). The processor 1201 may also include a main processor and a coprocessor. The main processor is a processor for processing data in an awake state, also known as a central processing unit (CPU); the coprocessor is a low-power processor for processing data in a standby state. In some embodiments, the processor 1201 may be integrated with a graphics processing unit (GPU), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1201 may also include an AI processor for processing computing operations related to machine learning.

[0315] The memory 1202 may include one or more computer-readable storage media, which may be non-transitory. The memory 1202 may also include a high-speed random access memory and a non-volatile memory, such as one or more disk storage devices and flash memory storage devices. In some embodiments, the non-transitory computer-readable storage medium in the memory 1202 is used to store at least one instruction, which is used to be executed by the processor 1201 so that the terminal implements the training method or image processing method of the image processing model provided in the method embodiment of the present application.

[0316] In some embodiments, the terminal may optionally include a peripheral device interface 1203 and at least one peripheral device. The processor 1201, memory 1202, and peripheral device interface 1203 may be connected via a bus or signal lines. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, signal lines, or circuit boards. Specifically, the peripheral device may include at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207, and a power supply 1208.

[0317] The peripheral device interface 1203 can be used to connect at least one input / output (I / O)-related peripheral device to the processor 1201 and the memory 1202. In some embodiments, the processor 1201, the memory 1202, and the peripheral device interface 1203 are integrated on the same chip or circuit board; in some other embodiments, any one or two of the processor 1201, the memory 1202, and the peripheral device interface 1203 can be implemented on separate chips or circuit boards, which is not limited in this embodiment.

[0318] The radio frequency circuit 1204 is used to receive and transmit radio frequency (RF) signals, also known as electromagnetic signals. The radio frequency circuit 1204 communicates with communication networks and other communication devices via electromagnetic signals. The radio frequency circuit 1204 converts electrical signals into electromagnetic signals for transmission, or converts received electromagnetic signals into electrical signals. Optionally, the radio frequency circuit 1204 includes: an antenna system, an RF transceiver, one or more amplifiers, a tuner, an oscillator, a digital signal processor, a codec chipset, a user identity module card, etc. The radio frequency circuit 1204 can communicate with other terminals via at least one wireless communication protocol. Such wireless communication protocols include, but are not limited to, metropolitan area networks, various generations of mobile communication networks (2G, 3G, 4G, and 5G), wireless local area networks, and / or wireless fidelity (WiFi) networks. In some embodiments, the radio frequency circuit 1204 may also include circuits related to near field communication (NFC), which is not limited in this application.

[0319] Display screen 1205 is used to display a user interface (UI). The UI may include graphics, text, icons, videos, or any combination thereof. When display screen 1205 is a touch screen display, it is also capable of collecting touch signals on or above the surface of display screen 1205. The touch signals can be input as control signals to processor 1201 for processing. In this case, display screen 1205 can also be used to provide virtual buttons and / or a virtual keyboard, also known as soft buttons and / or a soft keyboard. In some embodiments, display screen 1205 can be a single display screen, disposed on the front panel of the terminal; in other embodiments, display screen 1205 can be at least two, disposed on different surfaces of the terminal or in a foldable design; in still other embodiments, display screen 1205 can be a flexible display screen, disposed on a curved or foldable surface of the terminal. Display screen 1205 can also be configured as a non-rectangular irregular shape, i.e., a special-shaped screen. Display screen 1205 can be made of materials such as liquid crystal display (LCD) and organic light-emitting diode (OLED).

[0320] The camera assembly 1206 is used to capture images or videos. Optionally, the camera assembly 1206 includes a front camera and a rear camera. Typically, the front camera is arranged on the front panel of the terminal, and the rear camera is arranged on the back of the terminal. In some embodiments, there are at least two rear cameras, which are any one of a main camera, a depth of field camera, a wide-angle camera, and a telephoto camera, so as to realize the fusion of the main camera and the depth of field camera to realize the background blur function, the fusion of the main camera and the wide-angle camera to realize panoramic shooting and virtual reality (VR) shooting function or other fusion shooting functions. In some embodiments, the camera assembly 1206 may also include a flash. The flash can be a monochrome temperature flash or a dual-color temperature flash. A dual-color temperature flash refers to a combination of a warm light flash and a cold light flash, which can be used for light compensation at different color temperatures.

[0321] The audio circuit 1207 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals to be input into the processor 1201 for processing, or input into the radio frequency circuit 1204 to achieve voice communication. For the purpose of stereo acquisition or noise reduction, there may be multiple microphones, each disposed at different parts of the terminal. The microphone may also be an array microphone or an omnidirectional acquisition microphone. The speaker is used to convert electrical signals from the processor 1201 or the radio frequency circuit 1204 into sound waves. The speaker may be a traditional thin film speaker or a piezoelectric ceramic speaker. When the speaker is a piezoelectric ceramic speaker, it can not only convert electrical signals into sound waves audible to humans, but also convert electrical signals into sound waves inaudible to humans for purposes such as ranging. In some embodiments, the audio circuit 1207 may also include a headphone jack.

[0322] Power supply 1208 is used to power various components in the terminal. Power supply 1208 can be AC ​​power, DC power, disposable batteries, or rechargeable batteries. When power supply 1208 includes a rechargeable battery, the rechargeable battery can support wired charging or wireless charging. The rechargeable battery can also be used to support fast charging technology.

[0323] In some embodiments, the terminal further includes one or more sensors 1209 , including but not limited to: an acceleration sensor 1210 , a gyroscope sensor 1211 , a pressure sensor 1212 , an optical sensor 1213 , and a proximity sensor 1214 .

[0324] The accelerometer 1210 can detect the magnitude of acceleration along the three coordinate axes of the coordinate system established by the terminal. For example, the accelerometer 1210 can be used to detect the components of gravity acceleration along the three coordinate axes. The processor 1201 can control the display screen 1205 to display the user interface in a landscape or portrait view based on the gravity acceleration signal collected by the accelerometer 1210.

[0325] The gyroscope sensor 1211 can detect the terminal's body orientation and rotation angle. It can also work with the accelerometer 1210 to collect the user's 3D movements of the terminal. Based on the data collected by the gyroscope sensor 1211, the processor 1201 can implement the following functions: motion sensing (such as changing the UI based on the user's tilt operation), image stabilization during shooting, game control, and inertial navigation.

[0326] The pressure sensor 1212 can be set in the side frame of the terminal and / or the lower layer of the display screen 1205. When the pressure sensor 1212 is set in the side frame of the terminal, it can detect the user's grip signal of the terminal, and the processor 1201 performs left and right hand recognition or shortcut operations based on the grip signal collected by the pressure sensor 1212. When the pressure sensor 1212 is set in the lower layer of the display screen 1205, the processor 1201 controls the operable controls on the UI interface based on the user's pressure operation on the display screen 1205. Operable controls include at least one of button controls, scroll bar controls, icon controls, and menu controls.

[0327] Optical sensor 1213 is used to detect ambient light intensity. In one embodiment, processor 1201 can control the display brightness of display screen 1205 based on the ambient light intensity detected by optical sensor 1213. Specifically, when the ambient light intensity is high, the display brightness of display screen 1205 is increased; when the ambient light intensity is low, the display brightness of display screen 1205 is decreased. In another embodiment, processor 1201 can also dynamically adjust the shooting parameters of camera assembly 1206 based on the ambient light intensity detected by optical sensor 1213.

[0328] Proximity sensor 1214, also known as a distance sensor, is typically located on the front panel of the terminal. Proximity sensor 1214 is used to detect the distance between the user and the front of the terminal. In one embodiment, when proximity sensor 1214 detects that the distance between the user and the front of the terminal is gradually decreasing, processor 1201 controls display screen 1205 to switch from the screen-on state to the screen-off state. When proximity sensor 1214 detects that the distance between the user and the front of the terminal is gradually increasing, processor 1201 controls display screen 1205 to switch from the screen-off state to the screen-on state.

[0329] Those skilled in the art will understand that Figure 12 The structure shown in the figure does not constitute a limitation on the terminal, and may include more or fewer components than shown in the figure, or combine certain components, or adopt a different component arrangement.

[0330] Taking the computer device as a server as an example, Figure 13This is a structural diagram of a server provided in an embodiment of the present application. The server 1300 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1301 and one or more memories 1302, wherein the one or more memories 1302 store at least one computer program, and the at least one computer program is loaded and executed by the one or more processors 1301 to implement the above-mentioned image processing model training method or image processing method. Of course, the server 1300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The server 1300 may also include other components for implementing device functions, which will not be described here.

[0331] In an exemplary embodiment, a computer-readable storage medium is further provided, wherein the computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the image processing model training method provided in the above embodiment, or the image processing method provided in the above embodiment. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0332] In an exemplary embodiment, a computer program product or computer program is also provided, which includes at least one computer program, and the at least one computer program is loaded and executed by a computer device to implement the training method of the image processing model provided in the above embodiment, or the image processing method provided in the above embodiment.

[0333] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0334] The above are only optional embodiments of the present application and are not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application should be included in the scope of protection of the present application.

Claims

1. A training method for an image processing model, characterized in that: The method comprises: Inputting a sample defect image and a sample mask image into an image processing model, wherein the sample mask image is used to indicate a defect area and a valid area of ​​the sample defect image, and the image processing model includes a texture feature layer and a structural feature layer; Extracting a first texture feature of the sample defect image through the texture feature layer, and extracting a first structural feature of the sample defect image through the structural feature layer, wherein the first texture feature indicates a texture feature of the valid area, and the first structural feature indicates a rough structural feature of the sample defect image; Processing the first texture feature and the first structural feature through the texture feature layer to obtain a second texture feature of the sample defect image; processing the second texture feature and the first structural feature through the structural feature layer to obtain a second structural feature of the sample defect image, wherein the second texture feature indicates texture features of the valid area and the defect area, and the second structural feature indicates a refined structural feature of the sample defect image; Based on the second texture feature and the second structural feature, a predicted restoration image of the sample defective image is obtained, and based on the real image of the sample defective image, the predicted restoration image and the sample mask image, the image processing model is trained.

2. The method according to claim 1, characterized in that The extracting a first texture feature of the sample defect image through the texture feature layer and extracting a first structural feature of the sample defect image through the structural feature layer includes: Downsampling the sample defective image through the encoding unit in each of the texture feature layers to obtain first texture features corresponding to each of the texture feature layers; downsampling the first texture feature corresponding to the first texture feature layer by the encoding unit in each of the structural feature layers to obtain the first intermediate structural feature corresponding to each of the structural feature layers, where the first texture feature layer is the bottommost texture feature layer; The first intermediate structural features corresponding to the first structural feature layer are up-sampled by the decoding units in each of the structural feature layers to obtain the first structural features corresponding to each of the structural feature layers, where the first structural feature layer is the bottommost structural feature layer.

3. The method according to claim 1, characterized in that The step of processing the first texture feature and the first structural feature through the texture feature layer to obtain a second texture feature of the sample defect image includes: Upsampling the first structural features corresponding to the second structural feature layer by the decoding unit in each of the texture feature layers to obtain second intermediate structural features corresponding to each of the texture feature layers, where the second structural feature layer is the topmost structural feature layer; The second intermediate structural features and the first texture features corresponding to each texture feature layer are processed by the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer.

4. The method according to claim 3, characterized in that The processing of the second intermediate structural features and the first texture features corresponding to each of the texture feature layers by the texture enhancement unit in each of the texture feature layers to obtain the second texture features corresponding to each of the texture feature layers includes: obtaining, based on structural features of each pixel in a second intermediate structural feature corresponding to a target texture feature layer, an attention matrix corresponding to the target texture feature layer, the attention matrix indicating affinity between any two pixels in the second intermediate structural feature corresponding to the target texture feature layer, the target texture feature layer being any texture feature layer; Based on the attention matrix, the mask image corresponding to the target texture feature layer, and the first texture feature corresponding to the target texture feature layer, the second texture feature corresponding to the target texture feature layer is obtained.

5. The method according to claim 1, wherein The step of processing the second texture feature and the first structural feature through the structural feature layer to obtain the second structural feature of the sample defect image includes: downsampling the second texture features corresponding to the first texture feature layer by the encoding units in each of the structural feature layers to obtain third intermediate structural features corresponding to each of the structural feature layers, where the first texture feature layer is the bottommost texture feature layer; The third intermediate structural features and the first structural features corresponding to each structural feature layer are processed by the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer.

6. The method according to claim 5, characterized in that The step of processing the third intermediate structural features and the first structural features corresponding to each structural feature layer by the structural enhancement unit in each structural feature layer to obtain the second structural features corresponding to each structural feature layer includes: Obtaining a residual between a third intermediate structural feature and the first structural feature corresponding to a target structural feature layer, wherein the target structural feature layer is any structural feature layer; Compressing each characteristic channel in the residual to obtain a characteristic value of each characteristic channel; Based on the eigenvalues ​​of the respective feature channels, obtaining weights of the respective feature channels, wherein the weights of the feature channels indicate the importance of the feature channels to the structural features; Based on the weight of each feature channel, each feature channel in the third intermediate structural feature corresponding to the target structural feature layer is transformed to obtain the second structural feature corresponding to the target structural feature layer.

7. The method according to claim 1, characterized in that The training of the image processing model based on the real image of the sample defective image, the predicted repaired image and the sample mask image comprises: Inputting the predicted inpainted image into an image discriminator to obtain a discrimination result of the predicted inpainted image, and determining a first loss value of the image processing model based on the discrimination result of the predicted inpainted image, the predicted inpainted image, and the real image, wherein the discrimination result of the predicted inpainted image indicates a probability of inpainting failure for each pixel in the predicted inpainted image; Determining a second loss value of the image processing model based on a distance between a target restoration image and the real image in a feature space, wherein the target restoration image is obtained by merging the predicted restoration image and the sample defective image; Based on the first loss value and the second loss value, a target loss value is determined, and based on the target loss value, the image processing model is trained until a training end condition is met, thereby obtaining a trained image processing model.

8. The method according to claim 7, characterized in that The method further comprises: Determining a loss value of the image discriminator based on a discrimination result of the predicted restoration image and the sample mask image; The image discriminator is trained based on the loss value of the image discriminator.

9. The method according to claim 7, characterized in that The method further comprises: Merging the predicted restoration image and the sample defective image to obtain the target restoration image; Inputting the target restoration image into an image enhancement model to obtain texture features and structural features of the target restoration image; Inputting the real image into the image enhancement model to obtain texture features and structural features of the real image; Based on the texture features and structural features of the target inpainted image and the texture features and structural features of the real image, a distance between the target inpainted image and the real image in a feature space is determined.

10. An image processing method, characterized in that: The method comprises: Inputting an image to be repaired into an image processing model, wherein the image includes a valid area and a defective area, and the image processing model includes a texture feature layer and a structural feature layer; Extracting a first texture feature of the image through the texture feature layer, and extracting a first structural feature of the image through the structural feature layer, wherein the first texture feature indicates a texture feature of the valid area, and the first structural feature indicates a rough structural feature of the image; Processing the first texture feature and the first structural feature through the texture feature layer to obtain a second texture feature of the image; processing the second texture feature and the first structural feature through the structural feature layer to obtain a second structural feature of the image, wherein the second texture feature indicates texture features of the valid area and the defect area, and the second structural feature indicates a refined structural feature of the image; A restored image of the image is acquired based on the second texture feature and the second structural feature.

11. A training device for an image processing model, characterized in that: The device comprises: A sample input module, configured to input a sample defect image and a sample mask image into an image processing model, wherein the sample mask image is used to indicate a defect area and a valid area of ​​the sample defect image, and the image processing model includes a texture feature layer and a structural feature layer; a first feature extraction module, configured to extract a first texture feature of the sample defect image through the texture feature layer, and extract a first structural feature of the sample defect image through the structural feature layer, wherein the first texture feature indicates a texture feature of the valid area, and the first structural feature indicates a rough structural feature of the sample defect image; a second feature extraction module, configured to process the first texture feature and the first structural feature through the texture feature layer to obtain a second texture feature of the sample defect image, and process the second texture feature and the first structural feature through the structural feature layer to obtain a second structural feature of the sample defect image, wherein the second texture feature indicates texture features of the valid area and the defect area, and the second structural feature indicates a refined structural feature of the sample defect image; A model training module is used to obtain a predicted repair image of the sample defective image based on the second texture feature and the second structural feature, and to train the image processing model based on the real image of the sample defective image, the predicted repair image and the sample mask image.

12. The device according to claim 11, characterized in that The first feature extraction module is used to: Downsampling the sample defective image through the encoding unit in each of the texture feature layers to obtain first texture features corresponding to each of the texture feature layers; downsampling the first texture feature corresponding to the first texture feature layer by the encoding unit in each of the structural feature layers to obtain the first intermediate structural feature corresponding to each of the structural feature layers, where the first texture feature layer is the bottommost texture feature layer; The first intermediate structural features corresponding to the first structural feature layer are up-sampled by the decoding units in each of the structural feature layers to obtain the first structural features corresponding to each of the structural feature layers, where the first structural feature layer is the bottommost structural feature layer.

13. The device according to claim 11, characterized in that The second feature extraction module includes: a first feature extraction unit, configured to upsample the first structural features corresponding to the second structural feature layer through the decoding units in each of the texture feature layers to obtain second intermediate structural features corresponding to each of the texture feature layers, where the second structural feature layer is the topmost structural feature layer; The second feature extraction unit is used to process the second intermediate structural features and the first texture features corresponding to each texture feature layer through the texture enhancement unit in each texture feature layer to obtain the second texture features corresponding to each texture feature layer.

14. The device according to claim 13, characterized in that The second feature extraction unit is used for: obtaining, based on structural features of each pixel in a second intermediate structural feature corresponding to a target texture feature layer, an attention matrix corresponding to the target texture feature layer, the attention matrix indicating affinity between any two pixels in the second intermediate structural feature corresponding to the target texture feature layer, the target texture feature layer being any texture feature layer; Based on the attention matrix, the mask image corresponding to the target texture feature layer, and the first texture feature corresponding to the target texture feature layer, the second texture feature corresponding to the target texture feature layer is obtained.

15. The device according to claim 11, characterized in that The second feature extraction module includes: a third feature extraction unit, configured to downsample the second texture features corresponding to the first texture feature layer through the encoding units in each of the structural feature layers to obtain third intermediate structural features corresponding to each of the structural feature layers, where the first texture feature layer is the bottommost texture feature layer; The fourth feature extraction unit is used to process the third intermediate structural features and the first structural features corresponding to each of the structural feature layers through the structure enhancement unit in each of the structural feature layers to obtain the second structural features corresponding to each of the structural feature layers.

16. The device according to claim 15, characterized in that The fourth feature extraction unit is used for: Obtaining a residual between a third intermediate structural feature and the first structural feature corresponding to a target structural feature layer, wherein the target structural feature layer is any structural feature layer; Compressing each characteristic channel in the residual to obtain a characteristic value of each characteristic channel; Based on the feature values ​​of the feature channels, obtaining the weights of the feature channels, wherein the weights of the feature channels indicate the importance of the feature channels to the structural features; Based on the weight of each feature channel, each feature channel in the third intermediate structural feature corresponding to the target structural feature layer is transformed to obtain the second structural feature corresponding to the target structural feature layer.

17. The device according to claim 11, characterized in that The model training module includes: a first loss value determining unit, configured to input the predicted inpainted image into an image discriminator, obtain a discrimination result of the predicted inpainted image, and determine a first loss value of the image processing model based on the discrimination result of the predicted inpainted image, the predicted inpainted image, and the real image, wherein the discrimination result of the predicted inpainted image indicates a probability of inpainting failure for each pixel in the predicted inpainted image; a second loss value determining unit, configured to determine a second loss value of the image processing model based on a distance between a target restoration image and the real image in a feature space, wherein the target restoration image is obtained by merging the predicted restoration image and the sample defective image; A model training unit is used to determine a target loss value based on the first loss value and the second loss value, and train the image processing model based on the target loss value until a training end condition is met, thereby obtaining a trained image processing model.

18. The device according to claim 17, characterized in that The model training module is also used to: Determining a loss value of the image discriminator based on a discrimination result of the predicted restoration image and the sample mask image; The image discriminator is trained based on the loss value of the image discriminator.

19. The device according to claim 17, characterized in that The model training module is also used to: Merging the predicted restoration image and the sample defective image to obtain the target restoration image; Inputting the target restoration image into an image enhancement model to obtain texture features and structural features of the target restoration image; Inputting the real image into the image enhancement model to obtain texture features and structural features of the real image; Based on the texture features and structural features of the target inpainted image and the texture features and structural features of the real image, a distance between the target inpainted image and the real image in a feature space is determined.

20. An image processing device, characterized in that: The device comprises: An image input module is used to input an image to be repaired into an image processing model, wherein the image includes a valid area and a defective area, and the image processing model includes a texture feature layer and a structural feature layer; a first feature extraction module, configured to extract a first texture feature of the image through the texture feature layer, and extract a first structural feature of the image through the structural feature layer, wherein the first texture feature indicates a texture feature of the valid area, and the first structural feature indicates a rough structural feature of the image; a second feature extraction module, configured to process the first texture feature and the first structural feature through the texture feature layer to obtain a second texture feature of the image, and process the second texture feature and the first structural feature through the structural feature layer to obtain a second structural feature of the image, wherein the second texture feature indicates texture features of the valid area and the defect area, and the second structural feature indicates a refined structural feature of the image; The restoration image acquisition module is used to acquire a restoration image of the image based on the second texture feature and the second structural feature.

21. A computer device, characterized in that: The computer device includes a processor and a memory, the memory is used to store at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the training method of the image processing model as described in any one of claims 1 to 9, or the image processing method as described in claim 10.

22. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the training method of the image processing model according to any one of claims 1 to 9, or the image processing method according to claim 10.

23. A computer program product, characterized in that The computer program product includes at least one computer program, which is loaded and executed by a computer device to implement the training method of the image processing model according to any one of claims 1 to 9, or the image processing method according to claim 10.

Images