Image reconstruction model training method, device, equipment, medium and program product

Abstract

The present application discloses a training method, device, equipment, medium and program product for an image reconstruction model, which belongs to the field of image processing technology. It includes: obtaining a first sample image and at least two second sample images; adding the damage features corresponding to the at least two second sample images to the first sample image respectively to generate at least two single degraded images; fusing the at least two single degraded images to obtain multiple degraded images corresponding to the first sample image; performing image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images; calculating the loss function value based on the second sample image, the single degraded image, the first sample image and the predicted reconstructed image; and updating the model parameters of the model based on the loss function value. The model trained by the above method can simultaneously reconstruct multiple types of damage of the image, avoid the cumulative error caused by sequentially reconstructing images, and improve the image reconstruction accuracy of the model.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to a training method, apparatus, device, medium, and program product for an image reconstruction model. Background Art

[0002] High-quality 3D images can clearly display detailed information. For example, in medicine, high-quality 3D medical images facilitate diagnosis and analysis. However, image quality deteriorates during the formation, recording, processing, and transmission of images due to imperfections in imaging systems, recording equipment, transmission media, and processing methods.

[0003] In related technologies, deep convolutional neural networks are usually used to directly learn the mapping relationship between pairs of low-quality images and high-quality images, thereby generating high-quality three-dimensional images based on low-quality three-dimensional images.

[0004] However, in the case of a low-quality three-dimensional image having multiple damaged parts, such as noise and image missing, the above method usually reconstructs the damaged parts of the low-quality three-dimensional image in sequence, resulting in that the reconstruction error of the first stage is propagated to the subsequent stages, resulting in a larger overall image reconstruction error. Summary of the Invention

[0005] This application provides a training method, apparatus, device, medium, and program product for an image reconstruction model, which can obtain relatively accurate image reconstruction results. The technical solution is as follows:

[0006] According to one aspect of the present application, a method for training an image reconstruction model is provided, the method comprising:

[0007] Acquire a first sample image and at least two second sample images, where the second sample images are images with a single damage type, and the image quality of the first sample images is higher than that of the second sample images;

[0008] adding damage features corresponding to at least two of the second sample images to the first sample image to generate at least two single degraded images; fusing the at least two single degraded images to obtain a multiple degraded image corresponding to the first sample image, where the multiple degraded image refers to an image with at least two types of damage;

[0009] Performing image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images;

[0010] Calculating a loss function value based on the second sample image, the single degraded image, the first sample image, and the predicted reconstructed image;

[0011] The model parameters of the image reconstruction model are updated based on the loss function value.

[0012] In a possible implementation, the damage type includes at least one of a blur damage type, a noise damage type, and a bias damage type.

[0013] In a possible implementation, the method further includes:

[0014] Acquire a target image, wherein the target image refers to an image with multiple damage types;

[0015] Based on the trained reconstruction network layer, the target image is reconstructed to obtain a target reconstructed image, wherein the target reconstructed image refers to an image obtained by removing multiple types of damage in the target image;

[0016] The target reconstructed image is output.

[0017] According to one aspect of the present application, a training device for an image reconstruction model is provided, the device comprising:

[0018] an acquisition module, configured to acquire a first sample image and at least two second sample images, wherein the second sample images are images with a single damage type, and the image quality of the first sample image is higher than that of the second sample images;

[0019] a degradation module, configured to add damage features corresponding to at least two second sample images to the first sample image respectively, to generate at least two single degraded images;

[0020] a fusion module, configured to fuse at least two of the single degraded images to obtain a multiple degraded image corresponding to the first sample image, wherein the multiple degraded image refers to an image with at least two types of damage;

[0021] a reconstruction module, configured to perform image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images;

[0022] a calculation module, configured to calculate a loss function value based on the second sample image, the single degraded image, the first sample image, and the predicted reconstructed image;

[0023] An updating module is used to update the model parameters of the image reconstruction model based on the loss function value.

[0024] According to another aspect of the present application, a computer device is provided, which includes: a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the training method of the image reconstruction model described above.

[0025] According to another aspect of the present application, a computer storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the training method of the image reconstruction model as described above.

[0026] According to another aspect of the present application, a computer program product is provided, which includes a computer program stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes the training method of the image reconstruction model described above.

[0027] The beneficial effects of the technical solution provided by this application include at least:

[0028] By obtaining a first sample image and at least two second sample images; adding the damage features corresponding to the at least two second sample images to the first sample image in the degradation network layer to generate at least two single degraded images; fusing the at least two single degraded images to obtain multiple degraded images corresponding to the first sample image; then reconstructing the multiple degraded images in the reconstruction network layer to generate predicted reconstructed images corresponding to the multiple degraded images; the computer device calculates the loss function value based on the second sample image, the single degraded image, the first sample image and the predicted reconstructed image; and updating the model parameters of the image reconstruction model based on the loss function value. The training method of the image reconstruction model provided in the present application obtains multiple degraded images corresponding to the first sample image by simultaneously performing image damage of multiple damage types on the first sample image, and reconstructs the multiple degraded images with multiple damage types through the reconstruction network layer. The model trained by the above method can simultaneously reconstruct multiple damage types of low-quality images, avoiding the cumulative error caused by sequentially reconstructing the damage types in the low-quality image, thereby improving the image reconstruction accuracy of the trained image reconstruction model. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0030] Figure 1 is a schematic diagram of a training method for an image reconstruction model provided by an exemplary embodiment of the present application;

[0031] Figure 2 is a schematic diagram of the architecture of a computer system provided by an exemplary embodiment of the present application;

[0032] Figure 3 is a flowchart of a method for training an image reconstruction model provided by an exemplary embodiment of the present application;

[0033] Figure 4 is a flowchart of a method for training an image reconstruction model provided by an exemplary embodiment of the present application;

[0034] Figure 5 is a schematic diagram of a training method for an image reconstruction model provided by an exemplary embodiment of the present application;

[0035] Figure 6 is a schematic diagram of a reconstruction effect of an image reconstruction model provided by an exemplary embodiment of the present application;

[0036] Figure 7 is a schematic diagram of a reconstruction effect of an image reconstruction model provided by an exemplary embodiment of the present application;

[0037] Figure 8 is a schematic diagram of a training method for an image reconstruction model provided by an exemplary embodiment of the present application;

[0038] Figure 9 This is a framework diagram of image reconstruction model generation and image reconstruction provided by an exemplary embodiment of the present application;

[0039] Figure 10 is a flowchart of an image reconstruction method provided by an exemplary embodiment of the present application;

[0040] Figure 11 is a schematic diagram of an image reconstruction method provided by an exemplary embodiment of the present application;

[0041] Figure 12 is a block diagram of a training device for an image reconstruction model provided by an exemplary embodiment of the present application;

[0042] Figure 13 It is a structural diagram of a computer device provided by an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0043] In order to make the objectives, technical solutions and advantages of this application clearer, the implementation methods of this application will be further described in detail below with reference to the accompanying drawings.

[0044] In this document, "plurality" refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0045] For ease of understanding, the nouns involved in the embodiments of this application are explained below.

[0046] Artificial Intelligence (AI) refers to the theories, methods, techniques, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, to perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that seeks to understand the essence of intelligence and produce new intelligent machines that can respond in a manner similar to human intelligence. AI also studies the design principles and implementation methods of various intelligent machines, enabling them to possess the capabilities of perception, reasoning, and decision-making.

[0047] Artificial intelligence (AI) technology is a comprehensive discipline encompassing a wide range of fields, encompassing both hardware and software technologies. Foundational AI technologies generally include sensors, specialized AI chips, cloud computing, distributed storage, big data processing, operating / interaction systems, and mechatronics. AI software technologies primarily encompass computer vision, speech processing, natural language processing, and machine learning / deep learning.

[0048] Computer vision (CV) is the science of making machines "see." Specifically, it refers to using cameras and computers to replace the human eye in identifying and measuring objects, and then further processing the images to create images more suitable for human observation or transmission to instruments. As a scientific discipline, computer vision studies related theories and technologies, aiming to build artificial intelligence systems that can extract information from images or multidimensional data. Computer vision technologies generally include image segmentation, image recognition, image semantic understanding, image retrieval, optical character recognition (OCR), video processing, video semantic understanding, video content / action recognition, three-dimensional object reconstruction, three-dimensional (3D) technology, virtual reality, augmented reality, and simultaneous localization and mapping.

[0049] Machine learning (ML) is a multidisciplinary field that encompasses probability theory, statistics, approximation theory, convex analysis, and algorithmic complexity theory. It specifically studies how computers can simulate or implement human learning behaviors to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is at the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications span all areas of AI. Machine learning and deep learning typically include techniques such as artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and learning through demonstration.

[0050] Optionally, various pre-stored data involved in this application can be saved on the blockchain, such as image data of the first sample image, image data of the second sample image, parameters of the image reconstruction model, etc.

[0051] With the research and advancement of artificial intelligence technology, artificial intelligence technology has been studied and applied in many fields, such as common smart homes, smart wearable devices, virtual assistants, smart speakers, smart marketing, unmanned driving, autonomous driving, drones, robots, smart medical care, smart customer service, etc. It is believed that with the development of technology, artificial intelligence technology will be applied in more fields and play an increasingly important role.

[0052] The solution provided in the embodiments of the present application involves the fields of computer vision technology and machine learning technology. An image reconstruction model is obtained by training using computer vision technology and machine learning technology, and then the image reconstruction model is used to reconstruct images with various types of damage, which is specifically illustrated by the following embodiments.

[0053] The embodiment of the present application provides a technical solution for a training method of an image reconstruction model, such as Figure 1 As shown, the method can be executed by a computer device, which can be a terminal or a server.

[0054] Exemplarily, a computer device acquires a first sample image 101 and at least two second sample images. The at least two second sample images include at least two of a blurred sample image 102, a noisy sample image 104, and a biased sample image 103. The second sample images are images with a single type of damage, and the image quality of the first sample image is higher than that of the second sample images.

[0055] Optionally, the damage type includes at least one of a fuzzy damage type, a noise damage type, and a bias damage type, but is not limited thereto and is not specifically limited in the embodiments of the present application.

[0056] For example, the first sample image 101 refers to an image with high resolution, or an image that does not affect the expression of image content, or has little effect on the expression of image content.

[0057] The blurred sample image 102 refers to an image with blurred content.

[0058] The noisy sample image 104 refers to an image containing unnecessary image content or image content that has a negative impact on the analysis and understanding of the image content, that is, image noise. The image is the noisy sample image 104 .

[0059] The offset sample image 103 refers to an image having brightness differences due to offset.

[0060] It can be understood that the blur damage type, noise damage type, and bias damage type in the second sample image are all randomly set.

[0061] Exemplarily, the computer device extracts the first feature corresponding to the first sample image 101 through the first degradation encoder 105 , and respectively extracts the second features corresponding to at least two second sample images.

[0062] Based on the first and second features, the computer device extracts the damage feature from the second feature through a corresponding damage kernel extractor; the computer device adds the damage feature to the first feature of the first sample image 101 to obtain an intermediate first feature, and inputs the intermediate first feature into the first degradation decoder 109 for decoding processing to obtain a single degraded image corresponding to the first sample image.

[0063] For example, the computer device extracts features of the first sample image 101 through the first degradation encoder 105 to obtain a first feature; the computer device extracts features of the blurred sample image 102, the noisy sample image 104 and the biased sample image 103 through the first degradation encoder 105 to obtain blurred sample features, noisy sample features and biased sample features respectively.

[0064] The computer device inputs the first feature and the fuzzy sample feature into the fuzzy kernel extractor 106 for feature extraction to obtain a fuzzy damage feature in the fuzzy sample feature; the computer device inputs the first feature and the bias sample feature into the bias kernel extractor 107 for feature extraction to obtain a bias damage feature in the bias sample feature; the computer device inputs the first feature and the noise sample feature into the noise kernel extractor 108 for feature extraction to obtain a noise damage feature in the noise sample feature.

[0065] The computer device fuses the first feature and the blurred damage feature corresponding to the first sample image 101 to generate an intermediate first blurred feature; the computer device decodes the intermediate first blurred feature through the first degradation decoder 109 to generate a blurred degraded image 110 corresponding to the first sample image 101.

[0066] The computer device fuses the first feature and the biased damage feature corresponding to the first sample image 101 to generate an intermediate first biased feature; the computer device decodes the intermediate first biased feature through the first degradation decoder 109 to generate a biased degraded image 111 corresponding to the first sample image.

[0067] The computer device fuses the first feature and the noise corruption feature corresponding to the first sample image 101 to generate an intermediate first noise feature; the computer device decodes the intermediate first noise feature through the first degradation decoder 109 to generate a noise-degraded image 112 corresponding to the first sample image.

[0068] Exemplarily, the computer device obtains third features corresponding to at least two single degraded images through the second degradation encoder 113, and fuses the third features to obtain a degraded fused feature; the computer device decodes the degraded fused feature through the second degradation decoder 114 to generate multiple degraded images 115 corresponding to the first sample image 101.

[0069] For example, the computer device uses the second degradation encoder 113 to extract features from the blurred degraded image 110, the biased degraded image 111, and the noise-degraded image 112, respectively, and fuses the features corresponding to the blurred degraded image 110, the features corresponding to the biased degraded image 111, and the features corresponding to the noise-degraded image 112 to obtain degradation fusion features; the computer device uses the second degradation decoder 114 to decode the degradation fusion features to generate multiple degraded images 115 corresponding to the first sample image 101.

[0070] The computer device performs image reconstruction processing on the multiple degraded images based on the reconstruction encoder 116 and the reconstruction decoder 117 in the reconstruction network layer in the image reconstruction model, and generates predicted reconstructed images 118 corresponding to the multiple degraded images 115 .

[0071] Exemplarily, the computer device calculates a first loss function value based on a second feature corresponding to the second sample image and a third feature corresponding to the single degraded image, where the first loss function value includes a first blur loss function value, a first bias loss function value, and a first noise loss function value.

[0072] For example, the computer device calculates the first blur loss function value based on the second feature corresponding to the blurred sample image 102 and the third feature corresponding to the blurred degraded image 110; the computer device calculates the first bias loss function value based on the second feature corresponding to the biased sample image 103 and the third feature corresponding to the biased degraded image 111; the computer device calculates the first noise loss function value based on the second feature corresponding to the noisy sample image 104 and the third feature corresponding to the noise degraded image 112.

[0073] The first loss function value is used to measure the similarity between the second sample image and the single degraded image corresponding to the second sample image.

[0074] Exemplarily, the computer device calculates the second loss function value based on the first feature corresponding to the first sample image 101 and the fourth feature corresponding to the predicted reconstructed image 118 .

[0075] The second loss function value is used to measure the authenticity of the predicted reconstructed image.

[0076] Exemplarily, the computer device calculates the third loss function value based on the structural features corresponding to the multiple degraded images 115 and the structural features corresponding to the first sample image 101 .

[0077] The structural features corresponding to the multiple degraded images 115 refer to the structural features of the non-content portion of the multiple degraded images 115 .

[0078] The third loss function value is used to measure the similarity of the non-content parts between the multiple degraded images 115 and the first sample image 101 .

[0079] Exemplarily, the computer device calculates the fourth loss function value based on the content features and texture features corresponding to the first sample image 101 and the content features and texture features corresponding to the predicted reconstructed image 118 .

[0080] The fourth loss function value is used to measure the similarity between the first sample image and the predicted reconstructed image.

[0081] Exemplarily, the computer device updates the model parameters of the image reconstruction model based on the sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

[0082] In summary, the method provided in this embodiment obtains a first sample image and three second sample images; in the degradation network layer, the damage features corresponding to each of the three second sample images are added to the first sample image to generate three single degraded images; the three single degraded images are fused to obtain multiple degraded images corresponding to the first sample image; then, the multiple degraded images are reconstructed in the reconstruction network layer to generate predicted reconstructed images corresponding to the multiple degraded images; the computer device calculates the loss function value based on the three second sample images, the three single degraded images, the first sample image and the predicted reconstructed image; and updates the model parameters of the image reconstruction model based on the loss function value. The training method of the image reconstruction model provided in this application simultaneously performs image damage of multiple damage types on the first sample image to obtain multiple degraded images corresponding to the first sample image, and reconstructs the multiple degraded images with multiple damage types through the reconstruction network layer. The model trained by the above method can simultaneously reconstruct multiple damage types of low-quality images, avoiding the cumulative error caused by sequentially reconstructing low-quality images, thereby improving the image reconstruction accuracy of the trained image reconstruction model.

[0083] Figure 2 The schematic diagram of the computer system architecture provided by one embodiment of the present application is shown. The computer system may include: a terminal 100 and a server 200.

[0084] The terminal 100 can be an electronic device such as a mobile phone, a tablet computer, a vehicle-mounted terminal (vehicle computer), a wearable device, a personal computer (PC), an intelligent voice interaction device, a smart home appliance, a vehicle-mounted terminal, an aircraft, an unmanned vending terminal, etc. The terminal 100 can be installed with a client that runs a target application. The target application can be an image reconstruction application or other application that provides image reconstruction functions, and this application does not limit this. In addition, this application does not limit the form of the target application, including but not limited to an application (Application, App) installed in the terminal 100, a mini-program, etc., and can also be in the form of a web page.

[0085] Server 200 can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud computing services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, content delivery networks (CDNs), and big data and artificial intelligence platforms. Server 200 can be the backend server of the target application described above, used to provide backend services to the client of the target application.

[0086] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology, all based on the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require significant computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark and will need to be transmitted to backend systems for logical processing. Data of varying levels will be processed separately, and data from all industries will require a strong system backend, which can only be achieved through cloud computing.

[0087] In some embodiments, the aforementioned servers can also be implemented as nodes in a blockchain system. Blockchain is a novel application model for computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, a blockchain is a decentralized database, a series of data blocks generated using cryptographic methods. Each block contains information about a batch of network transactions, used to verify the validity of the information (to prevent counterfeiting) and generate the next block. A blockchain can include an underlying blockchain platform, a platform product and service layer, and an application service layer.

[0088] The terminal 100 and the server 200 may communicate with each other via a network, such as a wired or wireless network.

[0089] In the training method of the image reconstruction model provided in the embodiment of the present application, the execution subject of each step can be a computer device, which refers to an electronic device with data calculation, processing and storage capabilities. Figure 2 Taking the implementation environment of the scheme shown as an example, the training method of the image reconstruction model or the image reconstruction method can be executed by the terminal 100 (such as the client of the target application installed and running in the terminal 100 executes the training method of the image reconstruction model or the image reconstruction method), or the server 200 can execute the training method of the image reconstruction model or the image reconstruction method, or the terminal 100 and the server 200 can interact and cooperate to execute it, and this application does not limit this.

[0090] Figure 3 This is a flowchart of a method for training an image reconstruction model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment of the present invention. The method includes:

[0091] Step 302: Acquire a first sample image and at least two second sample images.

[0092] The second sample image refers to an image with a single damage type, and the image quality of the first sample image is higher than that of the second sample image.

[0093] For example, the first sample image is a high-resolution image, and the second sample image is a low-resolution image.

[0094] It is understandable that the images involved in the embodiments of the present application may be images of internal tissues of biological or non-biological organisms that cannot be directly seen by the human eye and are obtained through non-invasive methods.

[0095] For example, in the field of biomedicine, the images in the embodiments of the present application may be biological images (such as medical images). Biological images refer to images of the internal tissues of an organism or a part of an organism (such as a human body or a part of a human body) obtained in a non-invasive manner for the purpose of medical treatment or medical research. In one example, for the medical field, the images in the embodiments of the present application may be images of the heart, lungs, liver, stomach, large and small intestines, human brain, bones, blood vessels, etc.; they may also be images of non-human organs such as tumors. In addition, the images involved in the embodiments of the present application may be images generated based on imaging technologies such as X-ray technology, computerized tomography (CT), positron emission tomography (PET), nuclear magnetic resonance imaging (NMRI), and medical ultrasound. In addition, the images in the embodiments of the present application may also be images generated by visual imaging technology, such as what you see is what you get, such as images captured by a camera (such as a camera of a camera, a camera of a terminal, etc.).

[0096] Step 304: Add the damage features corresponding to the at least two second sample images to the first sample image respectively to generate at least two single degraded images.

[0097] A feature is a corresponding (essential) characteristic or property that distinguishes a certain type of object from other types of objects, or a collection of these characteristics or properties. In one possible implementation, a computer device can use a machine learning model to extract features from an image.

[0098] The damage feature refers to a feature corresponding to the damaged portion in the second sample image, for example, a feature corresponding to a blurred portion in the second sample image, or a feature corresponding to a noisy portion in the second sample image.

[0099] Exemplarily, the computer device extracts damage features corresponding to at least two second sample images, and adds the damage features corresponding to each second sample image to the first sample image to generate a single degraded image, so that the generated single degraded image contains damage features that are the same or similar to those of the second sample images.

[0100] A single degraded image refers to an image obtained by adding a single type of damage to the first sample image.

[0101] For example, the computer device extracts the blur damage features corresponding to the second sample image, and adds the blur damage features to the first sample image to obtain a blur degraded image corresponding to the first sample image.

[0102] Step 306: Fusing at least two single degraded images to obtain a multiple degraded image corresponding to the first sample image.

[0103] A multiply degraded image is an image with at least two types of damage.

[0104] Exemplarily, the computer device fuses at least two single degraded images to obtain an image with multiple damage types.

[0105] For example, a single degraded image is a blurred degraded image, a biased degraded image, and a noise-degraded image. The computer device fuses the blurred degraded image, the biased degraded image, and the noise-degraded image to generate multiple degraded images, so that the generated multiple degraded images have the same or similar blurred damage characteristics, noise damage characteristics, and bias damage characteristics.

[0106] Step 308: Perform image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images.

[0107] Image reconstruction processing refers to processing the damaged features in the multi-degraded images, for example, reducing or removing blurry damaged features, noise damaged features and bias damaged features in the multi-degraded images.

[0108] The predicted reconstructed image is an image obtained by reducing or removing the damaged features in multiple degraded images.

[0109] Exemplarily, a computer device performs reconstruction processing on the multiple degraded images to reduce or remove blur damage features, noise damage features, and bias damage features in the multiple degraded images, thereby generating predicted reconstructed images corresponding to the multiple degraded images.

[0110] Step 310: Calculate a loss function value based on the second sample image, the single degraded image, the first sample image, and the predicted reconstructed image.

[0111] The loss function value calculated based on the second sample image, the single degraded image, the first sample image and the predicted reconstructed image can be used to measure the training effect of the image reconstruction model.

[0112] Optionally, the loss function value is at least one of cross entropy, mean square error, and absolute difference, but is not limited thereto and is not limited to this embodiment of the present application.

[0113] Step 312: Update the model parameters of the image reconstruction model based on the loss function value.

[0114] Model parameter updating refers to updating the network parameters in the image reconstruction model, or updating the network parameters of each network module in the model, or updating the network parameters of each network layer in the model, but is not limited to this, and the embodiments of the present application do not limit this.

[0115] Optionally, based on the loss function value of the image reconstruction model, model parameters of the image reconstruction model are adjusted until the image reconstruction model or the image reconstruction model training system reaches a training stop condition, thereby obtaining a trained image reconstruction model. In some embodiments, before the image reconstruction model or the image reconstruction model training system reaches the training stop condition, model parameters of other learning models in the image reconstruction model training system are also continuously adjusted based on the training loss.

[0116] In summary, the method provided in this embodiment obtains a first sample image and at least two second sample images; the computer device adds the damage features corresponding to the at least two second sample images to the first sample image respectively to generate at least two single degraded images; the computer device fuses the at least two single degraded images to obtain multiple degraded images corresponding to the first sample image; and performs image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images; the computer device calculates the loss function value based on the second sample image, the single degraded image, the first sample image and the predicted reconstructed image; the computer device updates the model parameters of the image reconstruction model based on the loss function value. The training method of the image reconstruction model provided in this application obtains multiple degraded images corresponding to the first sample image by simultaneously performing image damage of multiple damage types on the first sample image, and reconstructs the multiple degraded images with multiple damage types through the reconstruction network layer. The model trained by the above method can simultaneously reconstruct multiple damage types of low-quality images, avoiding the cumulative error caused by sequentially reconstructing low-quality images, thereby improving the image reconstruction accuracy of the trained image reconstruction model.

[0117] An embodiment of the present application provides an image reconstruction model, which includes: a degraded network layer and a reconstruction network layer.

[0118] The computer device obtains a first sample image and at least two second sample images, and adds damage features corresponding to the at least two second sample images to the first sample image through a degradation network layer to generate at least two single degraded images; and fuses the at least two single degraded images to obtain a multiple degraded image corresponding to the first sample image.

[0119] The computer device performs image reconstruction processing on the multiple degraded images through a reconstruction network layer to generate predicted reconstructed images corresponding to the multiple degraded images.

[0120] Based on the image reconstruction model, the following image reconstruction model training method is provided.

[0121] Figure 4 This is a flowchart of a method for training an image reconstruction model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment of the present invention. The method includes:

[0122] Step 402: Acquire a first sample image and at least two second sample images.

[0123] The first sample image refers to an image with high resolution, or an image with a type of damage that does not affect the expression of image content, or an image with a type of damage that has little impact on the expression of image content.

[0124] The second sample image refers to an image with a single damage type, and the image quality of the first sample image is higher than that of the second sample image.

[0125] Optionally, the damage type includes at least one of a fuzzy damage type, a noise damage type, and a bias damage type, but is not limited thereto and is not limited in this embodiment of the present application.

[0126] The second sample image may be any one of a blurred sample image, a noisy sample image, and a biased sample image.

[0127] Blurry sample images refer to images with blurred content.

[0128] A noisy sample image refers to an image that contains unnecessary image content or has a negative impact on the analysis and understanding of the image content, that is, image noise. This image is a noisy sample image.

[0129] The biased sample image refers to an image with image brightness differences caused by bias.

[0130] For example, medical images often contain artifacts (i.e., image noise) due to interference from metal objects, which may affect doctors' judgments.

[0131] Step 404: Add the damage features corresponding to the at least two second sample images to the first sample image respectively to generate at least two single degraded images.

[0132] The damage feature refers to a feature corresponding to the damaged portion in the second sample image, for example, a feature corresponding to a blurred portion in the second sample image, or a feature corresponding to a noisy portion in the second sample image.

[0133] A single degraded image refers to an image obtained by adding a single type of damage to the first sample image.

[0134] Exemplarily, the computer device obtains a first feature corresponding to the first sample image, and obtains second features corresponding to at least two second sample images respectively; the computer device obtains a damage feature corresponding to the second sample image based on the first feature and the second feature; the computer device adds the damage feature to the first feature of the first sample image to obtain a single degraded image corresponding to the first sample image.

[0135] The first feature is used to characterize the image feature of the first sample image, and the second feature is used to characterize the image feature of the second sample image.

[0136] Exemplarily, the image reconstruction model includes a degradation network layer, which includes a first degradation encoder, a damage kernel extractor and a first degradation decoder.

[0137] The computer device extracts a first feature corresponding to the first sample image through a first degradation encoder, and respectively extracts second features corresponding to at least two second sample images.

[0138] The computer device determines the damage feature by comparing the first feature and the second feature, and decouples the damage feature corresponding to the second sample image from the second feature through a damage kernel extractor; the computer device adds the damage feature to the first feature of the first sample image to obtain an intermediate first feature, and inputs the intermediate first feature into a first degradation decoder for decoding processing to obtain a single degraded image corresponding to the first sample image.

[0139] For example, taking the blurred sample image, the noisy sample image and the biased sample image as examples of the second sample image, the computer device extracts the features of the first sample image through the first degradation encoder to obtain the first feature; the computer device extracts the features of the blurred sample image, the noisy sample image and the biased sample image respectively through the first degradation encoder to obtain the blurred sample feature, the noise sample feature and the biased sample feature respectively.

[0140] The computer device inputs the first feature and the fuzzy sample feature into a fuzzy kernel extractor for feature extraction, and obtains a fuzzy damage feature in the fuzzy sample feature; the computer device inputs the first feature and the bias sample feature into a bias kernel extractor for feature extraction, and obtains a bias damage feature in the bias sample feature; the computer device inputs the first feature and the noise sample feature into a noise kernel extractor for feature extraction, and obtains a noise damage feature in the noise sample feature.

[0141] The computer device fuses the first feature and the blurred damage feature corresponding to the first sample image to generate an intermediate first blurred feature; the computer device decodes the intermediate first blurred feature through a first degradation decoder to generate a blurred degraded image corresponding to the first sample image.

[0142] The computer device fuses the first feature and the biased damage feature corresponding to the first sample image to generate an intermediate first biased feature; the computer device decodes the intermediate first biased feature through a first degradation decoder to generate a biased degraded image corresponding to the first sample image.

[0143] The computer device fuses the first feature and the noise damage feature corresponding to the first sample image to generate an intermediate first noise feature; the computer device decodes the intermediate first noise feature through a first degradation decoder to generate a noise-degraded image corresponding to the first sample image.

[0144] Step 406: Obtain third features corresponding to at least two single degraded images, and fuse the third features to obtain multiple degraded images corresponding to the first sample image.

[0145] A multiply degraded image is an image with at least two types of damage.

[0146] The third feature is used to characterize the image feature of the single degraded image.

[0147] Exemplarily, the degradation network layer in the image reconstruction model also includes a second degradation encoder and a second degradation decoder; the computer device obtains third features corresponding to at least two single degraded images through the second degradation encoder, and fuses the third features to obtain degradation fusion features; the computer device decodes the degradation fusion features through the second degradation decoder to generate multiple degraded images corresponding to the first sample image.

[0148] For example, taking a blurred degraded image, a biased degraded image, and a noise-degraded image as examples of a single degraded image, the computer device uses a second degradation encoder to extract features from the blurred degraded image, the biased degraded image, and the noise-degraded image, respectively, and performs feature fusion on the features corresponding to the blurred degraded image, the features corresponding to the biased degraded image, and the features corresponding to the noise-degraded image to obtain degradation fusion features; the computer device uses a second degradation decoder to decode the degradation fusion features to generate multiple degraded images corresponding to the first sample image.

[0149] Step 408: Perform image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images.

[0150] Image reconstruction processing refers to processing the damaged features in the multi-degraded images to reduce or remove the blurry damaged features, noise damaged features and bias damaged features in the multi-degraded images.

[0151] The predicted reconstructed image is an image obtained by reducing or removing the damaged features in multiple degraded images.

[0152] Exemplarily, the image reconstruction model includes a reconstruction network layer, which includes a reconstruction encoder and a reconstruction decoder; the computer device inputs multiple degraded images into the reconstruction encoder for feature extraction to obtain image reconstruction features; the computer device decodes the image reconstruction features through the reconstruction decoder to generate predicted reconstructed images corresponding to the multiple degraded images.

[0153] Step 410: Calculate a first loss function value based on the second feature corresponding to the second sample image and the third feature corresponding to the single degraded image; calculate a second loss function value based on the first feature corresponding to the first sample image and the fourth feature corresponding to the predicted reconstructed image.

[0154] The fourth feature is used to characterize the image features of the predicted reconstructed image.

[0155] The first loss function value and the second loss function value calculated based on the second sample image, the single degraded image, the first sample image and the predicted reconstructed image can be used to measure the training effect of the image reconstruction model.

[0156] Optionally, the loss function value is at least one of cross entropy, mean square error, and absolute difference, but is not limited thereto and is not limited to this embodiment of the present application.

[0157] The first loss function value is used to measure the similarity between the second sample image and the single degraded image corresponding to the second sample image.

[0158] Exemplarily, the computer device calculates a first loss function value based on a second feature corresponding to the second sample image and a third feature corresponding to the single degraded image, where the first loss function value includes a first blur loss function value, a first bias loss function value, and a first noise loss function value.

[0159] Optionally, the computer device calculates the first loss function value based on the second feature corresponding to the i-th second sample image in the at least two second sample images and the third feature corresponding to the i-th single degraded image in the at least two single degraded images, where i is a positive integer.

[0160] For example, Figure 5 As shown, the computer device extracts features of the first sample image 501 through the first degradation encoder 505 to obtain a first feature. The computer device also extracts features of the blurred sample image 502, the noisy sample image 504, and the offset sample image 503 through the first degradation encoder 505 to obtain blurred sample features, noise sample features, and offset sample features, respectively. The computer device inputs the first feature and the blurred sample features into a blur kernel extractor 506 for feature extraction, obtaining a blur damage feature within the blurred sample features. The computer device inputs the first feature and the offset sample features into a bias kernel extractor 507 for feature extraction, obtaining a bias damage feature within the offset sample features. The computer device inputs the first feature and the noise sample features into a noise kernel extractor 508 for feature extraction, obtaining a noise damage feature within the noise sample features.

[0161] The computer device fuses the first feature corresponding to the first sample image 501 with the blur damage feature to generate an intermediate first blur feature; the computer device decodes the intermediate first blur feature through the first degradation decoder 509 to generate a blur degraded image 510 corresponding to the first sample image 501. The computer device fuses the first feature corresponding to the first sample image 501 with the bias damage feature to generate an intermediate first bias feature; the computer device decodes the intermediate first bias feature through the first degradation decoder 509 to generate a bias degraded image 511 corresponding to the first sample image 501. The computer device fuses the first feature corresponding to the first sample image 501 with the noise damage feature to generate an intermediate first noise feature; the computer device decodes the intermediate first noise feature through the first degradation decoder 509 to generate a noise degraded image 512 corresponding to the first sample image 501.

[0162] The computer device calculates a first blur loss function value 513 based on the second feature corresponding to the blurred sample image 502 and the third feature corresponding to the blurred degraded image 510;

[0163] The computer device calculates a first bias loss function value 514 based on the second feature corresponding to the biased sample image 503 and the third feature corresponding to the biased degraded image 511;

[0164] The computer device calculates a first noise loss function value 515 based on the second feature corresponding to the noisy sample image 504 and the third feature corresponding to the noise-degraded image 512 .

[0165] Exemplarily, the calculation formula of the first loss function value can be expressed as:

[0166]

[0167] Where, is the first loss function value, N is the number of data groups, R is the number of damage types, r is the damage type label, x is the first sample image, y is the second sample image, K is the damage kernel extractor, Ψ is the first degradation decoder, and E is the Charbonnier loss function.

[0168] The second loss function value is used to measure the authenticity of the predicted reconstructed image.

[0169] Exemplarily, the computer device calculates the second loss function value based on the first feature corresponding to the first sample image and the fourth feature corresponding to the predicted reconstructed image.

[0170] To make the generated image closer to the real image, the embodiment of the present application utilizes the concept of generative adversarial training. A first sample image and a predicted reconstructed image are input into a discriminator for discrimination, obtaining a discrimination result. The discriminator is used to discriminate between the first sample image and the predicted reconstructed image, and based on the discrimination result, an adversarial loss is determined, i.e., the value of the second loss function is determined. Ultimately, if the discriminator cannot distinguish whether a given image is the first sample image or the predicted reconstructed image, i.e., the predicted reconstructed image is close to the first sample image, the training is completed.

[0171] Exemplarily, the calculation formula of the second loss function value can be expressed as:

[0172]

[0173] Where, is the second loss function value, E is the Charbonnier loss function, D up is the discriminator, x is the first sample image, is a multi-degraded image, G up To rebuild the network layer.

[0174] In a possible implementation, the loss function value further includes a third loss function value and a fourth loss function value.

[0175] The third loss function value is used to measure the similarity of the non-content part between the multiple degraded images and the first sample image.

[0176] Exemplarily, the third loss function value is calculated based on the structural features corresponding to the multiple degraded images and the structural features corresponding to the first sample image.

[0177] Exemplarily, the calculation formula of the third loss function value can be expressed as:

[0178]

[0179] Where, is the third loss function value, x i is the i-th first sample image, is the i-th multi-degraded image, is the feature representation of the lth layer of the image.

[0180] The fourth loss function value is used to measure the similarity between the first sample image and the predicted reconstructed image.

[0181] Exemplarily, the fourth loss function value is calculated based on the content features and texture features corresponding to the first sample image and the content features and texture features corresponding to the predicted reconstructed image.

[0182] Exemplarily, the calculation formula of the fourth loss function value can be expressed as:

[0183]

[0184] Where, is the fourth loss function value, x i is the i-th first sample image, For the i-th predicted reconstructed image, G up To rebuild the network layer, is the content loss function value, is the texture loss function value, λ is the weight value, for example, λ is 0.9.

[0185] Step 412: Update the model parameters of the image reconstruction model based on the sum of the first loss function value and the second loss function value.

[0186] Model parameter updating refers to updating the network parameters in the image reconstruction model, or updating the network parameters of each network module in the model, or updating the network parameters of each network layer in the model, but is not limited to this, and the embodiments of the present application do not limit this.

[0187] In one possible implementation, the computer device updates the model parameters of the image reconstruction model based on the sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

[0188] For example, a linear combination constructed based on the sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value can be expressed as:

[0189]

[0190] Where, is the loss function value, is the first loss function value, is the third loss function value, is the second loss function value, is the fourth loss function value, α and β are weight factors, for example, α=400, β=100.

[0191] The model parameters of the image reconstruction model include at least one of network parameters of the first degradation encoder, network parameters of the damage kernel extractor, network parameters of the first degradation decoder, network parameters of the second degradation encoder, network parameters of the second degradation decoder, network parameters of the reconstruction encoder, and network parameters of the reconstruction decoder.

[0192] When obtaining the loss function value, the computer device updates the network parameters of the first degradation encoder, the network parameters of the damaged kernel extractor, the network parameters of the first degradation decoder, the network parameters of the second degradation encoder, the network parameters of the second degradation decoder, the network parameters of the reconstruction encoder and the network parameters of the reconstruction decoder in the image reconstruction model based on the loss function value to obtain the updated first degradation encoder, damaged kernel extractor, first degradation encoder, second degradation decoder, second degradation decoder, reconstruction encoder and reconstruction decoder, thereby obtaining a trained image reconstruction model.

[0193] In some embodiments, updating the model parameters of the image reconstruction model includes updating the network parameters of all network modules in the image reconstruction model, or fixing the network parameters of some network modules in the image reconstruction model and updating only the network parameters of the remaining network modules. For example, when updating the model parameters of the image reconstruction model, the network parameters of the first degradation encoder, the second degradation encoder, and the reconstruction encoder in the image reconstruction model are fixed, and only the network parameters of the damage kernel extractor, the first degradation decoder, the second degradation decoder, and the reconstruction decoder are updated.

[0194] In summary, the method provided in this embodiment obtains a first sample image and at least two second sample images; the computer device adds the damage features corresponding to the at least two second sample images to the first sample image respectively to generate at least two single degraded images; the computer device fuses the at least two single degraded images to obtain multiple degraded images corresponding to the first sample image; and performs image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images; the computer device calculates the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value based on the second sample image, the single degraded image, the first sample image and the predicted reconstructed image; the computer device updates the model parameters of the image reconstruction model based on the sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value. The training method of the image reconstruction model provided in the present application simultaneously performs image damage of multiple damage types on a first sample image to obtain multiple degraded images corresponding to the first sample image, and reconstructs the multiple degraded images with multiple damage types through a reconstruction network layer. The model trained by the above method can simultaneously reconstruct multiple damage types of low-quality images, avoiding the cumulative error caused by sequentially reconstructing low-quality images, thereby improving the image reconstruction accuracy of the trained image reconstruction model.

[0195] Figure 6 It is a schematic diagram of the reconstruction effect of the image reconstruction model provided by an exemplary embodiment of the present application.

[0196] In the medical field, medical images have become an important auxiliary tool for medical auxiliary diagnosis. Reconstructing medical images and displaying the detailed information in medical images more clearly based on the results of image reconstruction can better assist medical staff in medical diagnosis, such as Figure 6 As shown in the figure, a control experiment was designed to compare the comparative effects between the scheme provided by this application and the reference scheme. The experiment used the MNIST dataset, and the reference schemes included Reference Scheme 1, Reference Scheme 2, and Reference Scheme 3. The reference schemes used denoising, deblurring, and debiasing on multiple degraded images, but the order of the processing was different.

[0197] For example, in reference scheme one, by selecting different processing orders to perform denoising, deblurring and debiasing on multiple degraded images, six sets of image reconstruction results are finally obtained. By comparing the six sets of image reconstruction results in reference scheme one with the first sample image, and comparing the predicted reconstructed image obtained based on the scheme provided in this application with the first sample image, it can be obtained that: the scheme provided in this application can produce more realistic image reconstruction results with higher visual effects for medical images.

[0198] In the field of image defect repair, high-quality images can reflect more realistic views and provide higher visual effects. Figure 7 As shown in the figure, a control experiment was designed to compare the comparative effects between the scheme provided by this application and the reference scheme. The experiment used the MNIST dataset, and the reference schemes included Reference Scheme 1, Reference Scheme 2, and Reference Scheme 3. The reference schemes used denoising, deblurring, and debiasing on multiple degraded images, but the order of the processing was different.

[0199] For example, for different reference schemes, different processing orders are selected to perform denoising, deblurring and debiasing on multiple degraded images. Finally, each reference scheme obtains six sets of image reconstruction results. The six sets of image reconstruction results in different reference schemes are compared with the first sample image, and the predicted reconstructed image obtained based on the scheme provided in this application is compared with the first sample image. It can be obtained that: the scheme provided in this application can achieve better defect repair effect for defective images.

[0200] Figure 8 Schematic diagram of a training method for an image reconstruction model provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment of the present invention. The method includes:

[0201] In the medical field, medical images have become an important auxiliary tool for medical auxiliary diagnosis. Reconstructing medical images and displaying the detailed information in medical images more clearly based on the results of image reconstruction can better assist medical staff in medical diagnosis, such as Figure 8 As shown, a first sample image 801 and at least two second sample images 802 are input into a degradation network layer 803 to perform image degradation processing on the first sample image 801 to obtain an image degradation result: a multiple degraded image 804.

[0202] Image degradation refers to the decline in image quality during the process of image formation, recording, processing and transmission due to imperfections in the imaging system, recording equipment, transmission media and processing methods. This phenomenon is called image degradation.

[0203] In the embodiment of the present application, the image degradation processing performed on the first sample image 801 is to simultaneously add multiple damage types to the first sample image 801. For example, random blur processing, random noise processing, and random offset processing are added to the first sample image 801, that is, multiple types of defects are added to the first sample image 801.

[0204] When reconstructing the multiple degraded images 804, the reconstruction network layer 805 simultaneously reconstructs multiple types of defects in the multiple degraded images 804, that is, removes blur, removes noise, and removes bias. The reconstruction network layer 805 reconstructs the multiple degraded images 804 into high-quality images, that is, predicts the reconstructed images 806.

[0205] During the image reconstruction process, the accuracy of image reconstruction affects the accuracy of information display in medical images, and thus affects the accuracy of medical diagnosis made by medical staff based on the displayed information. Therefore, in the medical image-assisted diagnosis scenario, the image reconstruction model obtained by using the training method of the image reconstruction model provided in this application can improve the accuracy of image reconstruction of medical images, thereby more clearly displaying the detailed information in the medical images and improving the accuracy of medical-assisted diagnosis.

[0206] The training method of the image reconstruction model involved in this application can be implemented based on the image reconstruction model. The solution includes an image reconstruction model generation stage and an image reconstruction stage. Figure 9 This is a framework diagram of an image reconstruction model generation and image reconstruction shown in an exemplary embodiment of the present application. Figure 9 As shown, during the image reconstruction model generation phase, the image reconstruction model generation device 910 obtains an image reconstruction model using a pre-set training sample dataset (including a first sample image and at least two second sample images), and then generates an image reconstruction result based on the image reconstruction model. During the image reconstruction phase, the image reconstruction device 920 processes the input target image based on the image reconstruction model to obtain an image reconstruction result for the target image.

[0207] Among them, the above-mentioned image reconstruction model generation device 910 and image reconstruction device 920 can be computer devices. For example, the computer device can be a fixed computer device such as a personal computer or a server, or the computer device can also be a mobile computer device such as a tablet computer or an e-book reader.

[0208] Optionally, the above-mentioned image reconstruction model generation device 910 and image reconstruction device 920 can be the same device, or the image reconstruction model generation device 910 and the image reconstruction device 920 can also be different devices. Moreover, when the image reconstruction model generation device 910 and the image reconstruction device 920 are different devices, the image reconstruction model generation device 910 and the image reconstruction device 920 can be the same type of device, such as the image reconstruction model generation device 910 and the image reconstruction device 920 can both be servers; or the image reconstruction model generation device 910 and the image reconstruction device 920 can also be different types of devices, such as the image reconstruction device 920 can be a personal computer or a terminal, and the image reconstruction model generation device 910 can be a server, etc. The embodiment of the present application does not limit the specific types of the image reconstruction model generation device 910 and the image reconstruction device 920.

[0209] The above embodiment describes the training method of the image reconstruction model, and the image reconstruction method will be described next.

[0210] Figure 10 This is a flowchart of an image reconstruction method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment of the present invention. The method includes:

[0211] Step 1002: Acquire a target image.

[0212] The target image refers to an image with various types of damage.

[0213] The method of acquiring the target image includes at least one of the following situations:

[0214] 1. The computer device receives the target image. For example, the terminal is the terminal that initiates the image scanning. The terminal scans the image and sends the target image to the server after the scanning is completed.

[0215] 2. The computer device obtains a target image from a stored database, such as at least one target image in the MNIST segmentation dataset or a public brain MRI dataset.

[0216] It is worth noting that the above-mentioned method of acquiring the target image is only an illustrative example and is not limited to this embodiment of the present application.

[0217] Step 1004: Based on the trained reconstruction network layer, perform image reconstruction processing on the target image to obtain a target reconstructed image.

[0218] The computer device performs image reconstruction processing on the target image based on the trained reconstruction network layer. That is, the reconstruction network layer simultaneously reconstructs multiple types of defects in the target image, that is, removes blur, removes noise, and removes bias. The reconstruction network layer reconstructs the target image into a high-quality image, that is, the target reconstructed image.

[0219] Step 1006: Output the target reconstructed image.

[0220] The computer device outputs the target reconstructed image.

[0221] To verify the effectiveness of the image reconstruction model trained using the solution provided in the embodiment of this application, a control experiment was designed to compare the effectiveness of the solution provided in this application with the reference solution. The experiment used the MNIST dataset, which consists of 60,000 training and 10,000 test examples, all of which are 28×28 images. All images were pre-registered, and all images were divided into 90% for training and 10% for testing, with 20% of the training images retained as the validation set for both datasets.

[0222] The reference solutions include Reference Solution 1, Reference Solution 2, and Reference Solution 3. The reference solutions perform denoising (DN), deblurring (DB), and N4 offset correction on low-quality images, but in a different order. The evaluation metrics used are Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) to assess image reconstruction performance.

[0223] Table 1 Comparison of average performance of repair methods with different schemes and orders

[0224]

[0225]

[0226] The experimental results are shown in Table 1. It can be seen from Table 1 that the reconstruction results of the scheme provided in this application achieve a PSNR of 35.52dB and an SSIM of 0.9482 on the MNIST dataset. The performance on this dataset is better than the reference scheme, and it has high applicability and stability.

[0227] In summary, the method provided in this embodiment obtains a target image and, based on the trained reconstruction network layer, performs image reconstruction processing on the target image to obtain a high-quality target reconstructed image. This application can obtain a relatively accurate image reconstruction result based on the trained reconstruction network layer.

[0228] Figure 11 is a schematic diagram of an image reconstruction method provided by an exemplary embodiment of the present application. The method can be executed by a computer device, which can be Figure 2 The terminal 100 or the server 200 in the embodiment of the present invention. The method includes:

[0229] When the first front end 1101 receives a target image that needs to be reconstructed, the target image refers to an image with multiple loss types. The first front end 1101 uploads the target image to the computer device 1102 for image reconstruction processing. The image reconstruction processing process of the target image by the computer device 1102 can be referred to the description in the aforementioned embodiment and will not be repeated here.

[0230] After the computer device 1102 performs image reconstruction processing on the target image, the computer device 1102 outputs the image reconstruction result to the second front end 1103 .

[0231] Optionally, the first front end 1101 and the second front end 1103 may be the same front end or different front ends, which is not limited in this embodiment of the present application.

[0232] Figure 12 A schematic diagram of a training device for an image reconstruction model provided by an exemplary embodiment of the present application is shown. The device can be implemented as all or part of a computer device through software, hardware, or a combination of both. The device includes:

[0233] The acquisition module 1201 is configured to acquire a first sample image and at least two second sample images, where the second sample images are images with a single damage type, and the image quality of the first sample image is higher than that of the second sample image.

[0234] The degradation module 1202 is configured to add damage features corresponding to at least two second sample images to the first sample image respectively, to generate at least two single degraded images.

[0235] The fusion module 1203 is configured to fuse at least two of the single degraded images to obtain a multiple degraded image corresponding to the first sample image, where the multiple degraded image refers to an image with at least two types of damage.

[0236] The reconstruction module 1204 is configured to perform image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images.

[0237] The calculation module 1205 is configured to calculate a loss function value based on the second sample image, the single degraded image, the first sample image, and the predicted reconstructed image.

[0238] The updating module 1206 is configured to update the model parameters of the image reconstruction model based on the loss function value.

[0239] In one possible implementation, the degradation module 1202 is further configured to obtain a first feature corresponding to the first sample image and respectively obtain second features corresponding to at least two second sample images; obtain the damage feature corresponding to the second sample image based on the first feature and the second feature; and add the damage feature to the first feature of the first sample image to obtain the single degraded image corresponding to the first sample image.

[0240] The image reconstruction model includes a degradation network layer, which includes a first degradation encoder, a damage kernel extractor and a first degradation decoder.

[0241] In one possible implementation, the degradation module 1202 is further configured to extract a first feature corresponding to the first sample image through the first degradation encoder, and to extract at least two second features corresponding to the second sample images through the first degradation encoder; determine the damage feature by comparing the first feature and the second feature, and decouple the damage feature corresponding to the second sample image from the second feature through the damage kernel extractor; add the damage feature to the first feature of the first sample image to obtain an intermediate first feature, and input the intermediate first feature into the first degradation decoder for decoding processing to obtain the single degraded image corresponding to the first sample image.

[0242] In a possible implementation, the fusion module 1203 is further configured to obtain third features corresponding to at least two of the single degraded images, and fuse the third features to obtain the multiple degraded images corresponding to the first sample image.

[0243] The degradation network layer in the image reconstruction model further includes a second degradation encoder and a second degradation decoder.

[0244] In one possible implementation, the fusion module 1203 is further configured to obtain, through the second degradation encoder, third features corresponding to at least two of the single degraded images, and fuse the third features to obtain a degraded fused feature; and decode the degraded fused feature through the second degradation decoder to generate the multiple degraded images corresponding to the first sample image.

[0245] The image reconstruction model includes a reconstruction network layer, and the reconstruction network layer includes a reconstruction encoder and a reconstruction decoder.

[0246] In one possible implementation, the reconstruction module 1204 is further configured to input the multiple degraded images into the reconstruction encoder for feature extraction to obtain the image reconstruction features; and decode the image reconstruction features through the reconstruction decoder to generate the predicted reconstructed images corresponding to the multiple degraded images.

[0247] The loss function value includes a first loss function value and a second loss function value, the first loss function value is used to measure the similarity between the second sample image and the single degraded image corresponding to the second sample image, and the second loss function value is used to measure the authenticity of the predicted reconstructed image.

[0248] In a possible implementation, the calculation module 1205 is further configured to calculate the first loss function value based on the second feature corresponding to the second sample image and the third feature corresponding to the single degraded image.

[0249] In a possible implementation, the calculation module 1205 is further configured to calculate the second loss function value based on the first feature corresponding to the first sample image and the fourth feature corresponding to the predicted reconstructed image.

[0250] In a possible implementation, the updating module 1206 is further configured to update the model parameters of the image reconstruction model based on the sum of the first loss function value and the second loss function value.

[0251] In one possible implementation, the calculation module 1205 is further used to calculate the first loss function value based on the second feature corresponding to the i-th second sample image in at least two of the second sample images and the third feature corresponding to the i-th single degraded image in at least two of the single degraded images, where i is a positive integer.

[0252] The loss function value also includes a third loss function value and a fourth loss function value. The third loss function value is used to measure the similarity of the non-content part between the multiple degraded images and the first sample image, and the fourth loss function value is used to measure the similarity between the first sample image and the predicted reconstructed image.

[0253] In a possible implementation, the calculation module 1205 is further configured to calculate a third loss function value based on the structural features corresponding to the multiple degraded images and the structural features corresponding to the first sample image.

[0254] In a possible implementation, the calculation module 1205 is further configured to calculate the fourth loss function value based on content features and texture features corresponding to the first sample image, and content features and texture features corresponding to the predicted reconstructed image.

[0255] In a possible implementation, the updating module 1206 is further configured to update the model parameters of the image reconstruction model based on the sum of the first loss function value, the second loss function value, the third loss function value, and the fourth loss function value.

[0256] The damage type includes at least one of a blur damage type, a noise damage type, and a bias damage type.

[0257] In a possible implementation, the acquisition module 1201 is further configured to acquire a target image, where the target image refers to an image with multiple types of damage.

[0258] In one possible implementation, the reconstruction module 1204 is further used to perform image reconstruction processing on the target image based on the trained reconstruction network layer to obtain a target reconstructed image, where the target reconstructed image refers to an image obtained by removing multiple types of damage in the target image; and output the target reconstructed image.

[0259] Figure 13 The following is a block diagram of a computer device 1300 according to an exemplary embodiment of the present application. The computer device can be implemented as the server in the aforementioned solution of the present application. The image computer device 1300 includes a central processing unit (CPU) 1301, a system memory 1304 including a random access memory (RAM) 1302 and a read-only memory (ROM) 1303, and a system bus 1305 connecting the system memory 1304 and the CPU 1301. The image computer device 1300 also includes a mass storage device 1306 for storing an operating system 1309, application programs 1310, and other program modules 1311.

[0260] The mass storage device 1306 is connected to the central processing unit 1301 via a mass storage controller (not shown) connected to the system bus 1305. The mass storage device 1306 and its associated computer-readable media provide non-volatile storage for the image computing device 1300. In other words, the mass storage device 1306 may include a computer-readable medium (not shown) such as a hard disk or a compact disc read-only memory (CD-ROM) drive.

[0261] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically Erasable Programmable Read-Only Memory (EEPROM), flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, tape cassettes, magnetic tape, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage media are not limited to the above-mentioned ones. The above-mentioned system memory 1304 and mass storage device 1306 can be collectively referred to as memory.

[0262] According to various embodiments of the present disclosure, the image computer device 1300 may also be connected to a remote computer on a network such as the Internet for operation. That is, the image computer device 1300 may be connected to a network 1308 via a network interface unit 1307 connected to the system bus 1305. Alternatively, the network interface unit 1307 may be used to connect to other types of networks or remote computer systems (not shown).

[0263] The memory also includes at least one computer program, which is stored in the memory. The central processing unit 1301 implements all or part of the steps of the image reconstruction model training method shown in the above embodiments by executing the at least one program.

[0264] An embodiment of the present application also provides a computer device, which includes a processor and a memory, wherein the memory stores at least one program, and the at least one program is loaded and executed by the processor to implement the training method of the image reconstruction model provided by the above-mentioned method embodiments.

[0265] An embodiment of the present application also provides a computer-readable storage medium, which stores at least one computer program. The at least one computer program is loaded and executed by a processor to implement the training method of the image reconstruction model provided by the above-mentioned method embodiments.

[0266] An embodiment of the present application also provides a computer program product, which includes a computer program stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes to implement the training method of the image reconstruction model provided in the above-mentioned method embodiments.

[0267] It is understandable that in the specific implementation methods of this application, the data involved, historical data, and portraits and other data related to user data processing related to user identity or characteristics, when the above embodiments of this application are applied to specific products or technologies, need to obtain user permission or consent, and the collection, use and processing of relevant data need to comply with relevant laws, regulations and standards of relevant countries and regions.

[0268] It should be understood that the term "plurality" used herein refers to two or more. "And / or" describes a relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, or B exists alone. The character " / " generally indicates an "or" relationship between the associated objects.

[0269] Those skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware, or may be accomplished by a program to instruct the relevant hardware, and the program may be stored in a computer-readable storage medium, which may be a read-only memory, a disk, or an optical disk, etc.

[0270] The above description is merely an optional embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent switches, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. A training method for an image reconstruction model, characterized in that: The method comprises: Acquire a first sample image and at least two second sample images, wherein the second sample images are images with damaged portions of a single damage type, and the image quality of the first sample images is higher than that of the second sample images; Acquire a first feature corresponding to the first sample image, and respectively acquire at least two second features corresponding to the second sample images, wherein the first feature is used to characterize an image feature of the first sample image, and the second feature is used to characterize an image feature of the second sample image; Based on the first feature and the second feature, obtaining the damage feature corresponding to the second sample image, where the damage feature refers to a feature corresponding to the damaged portion in the second sample image; adding the damage feature to the first feature of the first sample image to obtain the single degraded image corresponding to the first sample image; fusing at least two of the single degraded images to obtain a multiple degraded image corresponding to the first sample image, wherein the multiple degraded image refers to an image with at least two types of damage; Performing image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images; Calculating a loss function value based on the second sample image, the single degraded image, the first sample image, and the predicted reconstructed image; The model parameters of the image reconstruction model are updated based on the loss function value.

2. The method according to claim 1, characterized in that The image reconstruction model includes a degradation network layer, and the degradation network layer includes a first degradation encoder; The obtaining of a first feature corresponding to the first sample image, and respectively obtaining of second features corresponding to at least two second sample images, includes: A first feature corresponding to the first sample image is extracted by the first degradation encoder, and at least two second features corresponding to the second sample image are respectively extracted by the first degradation encoder.

3. The method according to claim 1, characterized in that The image reconstruction model includes a degradation network layer, and the degradation network layer includes a damage kernel extractor; The obtaining, based on the first feature and the second feature, the damage feature corresponding to the second sample image includes: The damage feature is determined by comparing the first feature and the second feature, and the damage feature corresponding to the second sample image is obtained by decoupling the second feature through the damage kernel extractor.

4. The method according to claim 1, wherein The image reconstruction model includes a degradation network layer, and the degradation network layer includes a first degradation decoder; Adding the damage feature to the first feature of the first sample image to obtain the single degraded image corresponding to the first sample image includes: The damage feature is added to the first feature of the first sample image to obtain an intermediate first feature, and the intermediate first feature is input into the first degradation decoder for decoding to obtain the single degraded image corresponding to the first sample image.

5. The method according to claim 1, wherein The fusing of at least two of the single degraded images to obtain multiple degraded images corresponding to the first sample image includes: Acquire third features corresponding to at least two of the single degraded images, and fuse the third features to obtain the multiple degraded images corresponding to the first sample image, wherein the third features are used to characterize image features of the single degraded image.

6. The method according to claim 5, characterized in that The degradation network layer in the image reconstruction model further includes a second degradation encoder and a second degradation decoder; The acquiring of the third features corresponding to at least two of the single degraded images and fusing the third features to obtain the multiple degraded images corresponding to the first sample image includes: acquiring, by the second degradation encoder, third features corresponding to at least two of the single degraded images, and fusing the third features to obtain a degradation fusion feature; The degradation fusion feature is decoded by the second degradation decoder to generate the multiple degraded images corresponding to the first sample image.

7. The method according to claim 1, characterized in that The image reconstruction model includes a reconstruction network layer, and the reconstruction network layer includes a reconstruction encoder and a reconstruction decoder; The performing image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images includes: Inputting the multiple degraded images into the reconstruction encoder for feature extraction to obtain the image reconstruction features; The image reconstruction feature is decoded by the reconstruction decoder to generate the predicted reconstructed image corresponding to the multiple degraded images.

8. The method according to claim 1, characterized in that The loss function value includes a first loss function value and a second loss function value, the first loss function value is used to measure the similarity between the second sample image and the single degraded image corresponding to the second sample image, and the second loss function value is used to measure the authenticity of the predicted reconstructed image; The calculating of the loss function value based on the second sample image, the single degraded image, the first sample image, and the predicted reconstructed image includes: Calculating the first loss function value based on the second feature corresponding to the second sample image and the third feature corresponding to the single degraded image; calculating the second loss function value based on a first feature corresponding to the first sample image and a fourth feature corresponding to the predicted reconstructed image, wherein the fourth feature is used to characterize an image feature of the predicted reconstructed image; The updating of the model parameters of the image reconstruction model based on the loss function value includes: The model parameters of the image reconstruction model are updated based on the sum of the first loss function value and the second loss function value.

9. The method according to claim 8, characterized in that The calculating the first loss function value based on the second feature corresponding to the second sample image and the third feature corresponding to the single degraded image includes: The first loss function value is calculated based on the second feature corresponding to the i-th second sample image in at least two of the second sample images and the third feature corresponding to the i-th single degraded image in at least two of the single degraded images, where i is a positive integer.

10. The method according to claim 8, characterized in that The loss function value further includes a third loss function value and a fourth loss function value, the third loss function value is used to measure the similarity of the non-content part between the multiple degraded images and the first sample image, and the fourth loss function value is used to measure the similarity between the first sample image and the predicted reconstructed image; the method further includes: Calculating a third loss function value based on the structural features corresponding to the multiple degraded images and the structural features corresponding to the first sample image; Calculating the fourth loss function value based on the content features and texture features corresponding to the first sample image and the content features and texture features corresponding to the predicted reconstructed image; The updating of the model parameters of the image reconstruction model based on the sum of the first loss function value and the second loss function value includes: The model parameters of the image reconstruction model are updated based on the sum of the first loss function value, the second loss function value, the third loss function value and the fourth loss function value.

11. A training device for an image reconstruction model, characterized in that: The device comprises: an acquisition module, configured to acquire a first sample image and at least two second sample images, wherein the second sample images are images with damaged portions of a single damage type, and the image quality of the first sample images is higher than that of the second sample images; a degradation module, configured to obtain a first feature corresponding to the first sample image and respectively obtain at least two second features corresponding to the second sample image, wherein the first feature is used to characterize an image feature of the first sample image, and the second feature is used to characterize an image feature of the second sample image; obtain a damage feature corresponding to the second sample image based on the first feature and the second feature, wherein the damage feature refers to a feature corresponding to the damaged portion of the second sample image; and add the damage feature to the first feature of the first sample image to obtain the single degraded image corresponding to the first sample image; a fusion module, configured to fuse at least two of the single degraded images to obtain a multiple degraded image corresponding to the first sample image, wherein the multiple degraded image refers to an image with at least two types of damage; a reconstruction module, configured to perform image reconstruction processing on the multiple degraded images to generate predicted reconstructed images corresponding to the multiple degraded images; a calculation module, configured to calculate a loss function value based on the second sample image, the single degraded image, the first sample image, and the predicted reconstructed image; An updating module is used to update the model parameters of the image reconstruction model based on the loss function value.

12. A computer device, characterized in that: The computer device includes: a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the training method of the image reconstruction model according to any one of claims 1 to 10.

13. A computer storage medium, characterized in that At least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by the processor to implement the training method of the image reconstruction model according to any one of claims 1 to 10.

14. A computer program product, characterized in that The computer program product includes a computer program, which is stored in a computer-readable storage medium; the computer program is read and executed from the computer-readable storage medium by a processor of a computer device, so that the computer device executes the training method of the image reconstruction model as described in any one of claims 1 to 10.

Images