Image processing method, device, computer equipment and storage medium

Abstract

The embodiment of the present application discloses an image processing method, device, computer equipment and storage medium, which belongs to the field of computer technology. The method comprises: determining a spatial transformation feature based on a first scale of a first image space and a second scale of a second image space, upsampling a first image feature corresponding to a first image belonging to the first image space to obtain a second image feature; transforming the second image feature based on the spatial transformation feature to obtain a third image feature; and reconstructing the third image feature to obtain a second image. The embodiment of the present application improves the clarity of the image.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of computer technology, and in particular to an image processing method, apparatus, computer equipment, and storage medium. Background Art

[0002] With the development of computer technology, images are used more and more widely. However, image information is lost during the process of image shooting or transmission, resulting in low image clarity and poor display effect. Therefore, it is urgent to provide an image processing method to improve the clarity of the image. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, apparatus, computer equipment and storage medium, which can improve the clarity of an image. The technical solution is as follows:

[0004] In one aspect, an image processing method is provided, the method comprising:

[0005] Determining a spatial transformation feature based on a first scale of the first image space and a second scale of the second image space, wherein the spatial transformation feature represents a spatial transformation condition when any image belonging to the first image space is transformed into the second image space, the second scale is larger than the first scale, and a resolution of the second image space is higher than a resolution of the first image space;

[0006] Upsampling a first image feature corresponding to a first image belonging to the first image space to obtain a second image feature belonging to the second scale;

[0007] Based on the spatial transformation feature, transform the second image feature to obtain a third image feature;

[0008] The third image feature is reconstructed to obtain a second image belonging to the second image space.

[0009] In a possible implementation, the generation sub-model includes m first decoding layers and m second decoding layers;

[0010] The calling of the generative sub-model in the adversarial model to reconstruct the sample coding features corresponding to the plurality of sample image blocks and the m first sample decoding features to obtain a second predicted image belonging to the second image space; and the calling of the discriminative sub-model in the adversarial model to discriminate the second predicted image and the sample image to obtain a discriminative result, including:

[0011] Calling the first first decoding layer to decode the sample coding features corresponding to the multiple sample image blocks to obtain the first second sample decoding feature corresponding to each sample image block; fusing the second sample decoding feature with the first first sample decoding feature to obtain the first fused feature;

[0012] Calling the jth first decoding layer to decode the j-1th fusion feature to obtain the jth second sample decoding feature corresponding to each sample image block; until calling the mth first decoding layer to obtain the mth second sample decoding feature corresponding to each sample image block, j is an integer greater than 1 and not greater than m;

[0013] calling the second decoding layer, splicing the mth second sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a second sample splicing feature, and convolving the second sample splicing feature to obtain the second predicted image;

[0014] The discriminant sub-model is called to discriminate the second predicted image and the sample image to obtain the discrimination result.

[0015] In another possible implementation, the first image is an image extracted from a region of an original image containing biological features; after the second image block features corresponding to the multiple image blocks are spliced ​​according to positions of the multiple image blocks in the second image to obtain spliced ​​features, and the spliced ​​features are convolved to obtain a fourth image belonging to the second image space, the method further includes:

[0016] scaling the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of a region of the background image containing the biological feature is the second scale;

[0017] The fourth image is combined with the background image according to the position of the first image in the original image to obtain a fifth image.

[0018] In another aspect, an image processing device is provided, the device comprising:

[0019] a determining module, configured to determine a spatial transformation feature based on a first scale of the first image space and a second scale of the second image space, wherein the spatial transformation feature represents a spatial transformation condition when any image belonging to the first image space is transformed into the second image space, the second scale being larger than the first scale, and a resolution of the second image space being higher than a resolution of the first image space;

[0020] an upsampling module, configured to upsample a first image feature corresponding to a first image belonging to the first image space to obtain a second image feature belonging to the second scale;

[0021] A feature transformation module, used for transforming the second image feature based on the spatial transformation feature to obtain a third image feature;

[0022] An image reconstruction module is used to reconstruct the third image feature to obtain a second image belonging to the second image space.

[0023] In one possible implementation, the spatial transformation feature includes a position offset feature, and the position offset feature represents the position offset of a pixel point when any image belonging to the first image space is transformed to the second image space; the feature transformation module is used to transform the second image feature based on the position offset feature to obtain the third image feature.

[0024] In another possible implementation, the determining module includes:

[0025] a factor determination unit, configured to determine a ratio between the second scale and the first scale as an upsampling factor, wherein the upsampling factor represents an upsampling factor used to transform any pixel point from the first image space to the second image space;

[0026] a pixel point transformation unit, configured to transform the plurality of first pixel points into the first image space based on the coordinates of the plurality of first pixel points in the second image space and the upsampling factor, so as to obtain coordinates of a plurality of second pixel points;

[0027] a relative distance determining unit, configured to determine a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points, the relative distances representing distances between positions of first pixel points and positions of corresponding second pixel points in the second image space;

[0028] The offset feature determining unit is used to determine the position offset feature based on the coordinates of the plurality of second pixel points and the plurality of relative distances.

[0029] In another possible implementation manner, the offset feature determining unit is used to:

[0030] The coordinates of the plurality of second pixel points and the plurality of relative distances constitute a feature matrix, wherein the feature matrix includes a plurality of first sub-features, each of which includes a coordinate of the second pixel point and a corresponding relative distance;

[0031] The feature matrix is ​​transformed to obtain the position shift feature, where the position shift feature includes a plurality of second sub-features obtained by respectively transforming the plurality of first sub-features.

[0032] In another possible implementation, the spatial transformation feature further includes a scale transformation weight, where the scale transformation weight represents a scale transformation condition when any image belonging to the first image space is transformed into the second image space;

[0033] The feature transformation module is used to:

[0034] Based on the position offset feature, adjusting the position of the feature in the second image feature to obtain a fourth image feature;

[0035] Based on the scale transformation weight, compressive convolution and dilated convolution are performed on the fourth image feature to obtain the third image feature.

[0036] In another possible implementation, the feature transformation module is used to:

[0037] Based on the scale transformation weight, respectively adjusting the compression convolution parameters and the expansion convolution parameters;

[0038] Based on the adjusted compression convolution parameter and the adjusted expansion convolution parameter, performing compression convolution and expansion convolution on the fourth image feature to obtain a fifth image feature;

[0039] The fifth image feature is fused with the second image feature to obtain the third image feature.

[0040] In another possible implementation, the determining module includes:

[0041] a factor determination unit, configured to determine a ratio between the second scale and the first scale as an upsampling factor, wherein the upsampling factor represents an upsampling factor used to transform any pixel point from the first image space to the second image space;

[0042] a pixel point transformation unit, configured to transform the plurality of first pixel points into the first image space based on the coordinates of the plurality of first pixel points in the second image space and the upsampling factor, so as to obtain coordinates of a plurality of second pixel points;

[0043] a relative distance determining unit, configured to determine a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points, the relative distances representing distances between positions of first pixel points and positions of corresponding second pixel points in the second image space;

[0044] a feature vector determination unit, configured to form a feature matrix from the coordinates of the plurality of second pixel points and the plurality of relative distances, wherein the feature matrix includes a plurality of first sub-features, each of which includes a coordinate of a second pixel point and a corresponding relative distance; and transform the feature matrix to obtain a feature vector;

[0045] A first transformation unit, used for performing a first transformation on the feature vector to obtain the position offset feature;

[0046] The second transformation unit is used to perform a second transformation on the feature vector, normalize the transformed feature vector, and obtain the scale transformation weight, where the scale transformation weight includes weight features of the multiple second pixel points.

[0047] In another possible implementation, the method is implemented based on a reconstruction model, wherein the reconstruction model includes a spatial transformation sub-model and a reconstruction sub-model;

[0048] The determining module is used to call the spatial transformation sub-model to determine the spatial transformation feature based on the first scale and the second scale;

[0049] The upsampling module is used to call the reconstruction sub-model to upsample the first image feature to obtain the second image feature;

[0050] The feature transformation module is used to transform the second image feature based on the spatial transformation feature to obtain the third image feature;

[0051] The image reconstruction module is used to reconstruct the third image feature to obtain the second image.

[0052] In another possible implementation, the first image is an image extracted from a region of the original image that contains biological features; and the device further includes:

[0053] A first scaling module, configured to scale the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of a region of the background image containing the biological feature is the second scale;

[0054] The first combining module is used to combine the second image with the background image according to the position of the first image in the original image to obtain a third image.

[0055] In another possible implementation, the device further includes:

[0056] A feature acquisition module, used to divide the second image into a plurality of image blocks, and acquire a first image block feature corresponding to each image block;

[0057] An image block processing module, configured to encode and decode the first image block features corresponding to each image block m times based on a self-attention mechanism, to obtain the second image block features corresponding to each image block, where m is an integer greater than 1;

[0058] A splicing convolution module is used to splice the second image block features corresponding to the multiple image blocks according to the positions of the multiple image blocks in the second image to obtain splicing features, and convolve the splicing features to obtain a fourth image belonging to the second image space.

[0059] In another possible implementation, the feature acquisition module includes:

[0060] A feature extraction unit, used to extract features from each of the image blocks to obtain a third image block feature corresponding to each of the image blocks;

[0061] A position fusion unit is used to fuse the third image block feature corresponding to each image block with the position feature to obtain the first image block feature corresponding to each image block, wherein the position feature indicates the position of the image block in the second image.

[0062] In another possible implementation manner, the process of acquiring the fourth image based on the second image is implemented based on a degradation removal model, wherein the degradation removal model includes a first encoding sub-model, m second encoding sub-models based on a self-attention mechanism, m first decoding sub-models based on a self-attention mechanism, and a second decoding sub-model, where m is an integer greater than 1;

[0063] The feature acquisition module is used to call the first encoding sub-model, divide the second image into the plurality of image blocks, and acquire the first image block feature corresponding to each image block;

[0064] Furthermore, the m second encoding sub-models are connected in sequence, the m first decoding sub-models are connected in sequence, and the mth second encoding sub-model is connected to the first first decoding sub-model, and the image block processing module is used to:

[0065] Calling m second encoding sub-models to encode first image block features corresponding to the multiple image blocks m times to obtain encoding features corresponding to the multiple image blocks;

[0066] Calling m first decoding sub-models to decode the encoding features corresponding to the multiple image blocks m times to obtain decoding features corresponding to the multiple image blocks;

[0067] The splicing convolution module is used to call the second decoding sub-model, splice the decoding features corresponding to the multiple image blocks according to the positions of the multiple image blocks in the second image, obtain the splicing features, and convolve the splicing features to obtain the fourth image.

[0068] In another possible implementation, the image block processing module is used to:

[0069] For any image block, calling the first second encoding sub-model, based on the similarity between the first image block feature corresponding to the image block and the first image block features corresponding to the multiple image blocks, performing weighted fusion on the first image block features corresponding to the multiple image blocks to obtain a first fused image block feature corresponding to the image block;

[0070] The first fused image block feature is fused with the first image block feature corresponding to the image block to obtain a first encoding feature corresponding to the image block.

[0071] In another possible implementation, the image block processing module is used to:

[0072] For any image block, calling the i-th second coding sub-model, based on the similarity between the i-1th coding feature corresponding to the image block and the i-1th coding features corresponding to the multiple image blocks, performing weighted fusion on the i-1th coding features corresponding to the multiple image blocks to obtain the i-th fused image block feature corresponding to the image block;

[0073] The i-th fused image block feature is fused with i-1 coding features corresponding to the image block to obtain the i-th coding feature corresponding to the image block, where i is an integer greater than 1 and not greater than m.

[0074] In another possible implementation, the device further includes:

[0075] A sample acquisition module, used for acquiring a sample image belonging to the second image space;

[0076] A prediction module, used for calling the degradation removal model to reconstruct the sample image to obtain a first predicted image belonging to the second image space;

[0077] A training module is used to train the degradation removal model based on the sample image and the first predicted image, so as to increase the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image.

[0078] In another possible implementation, the prediction module is used to:

[0079] Calling the first encoding sub-model, dividing the sample image into the plurality of sample image blocks, and obtaining a first image block feature corresponding to each sample image block;

[0080] Calling m second encoding sub-models to encode the first image block features corresponding to the multiple sample image blocks m times to obtain sample encoding features corresponding to the multiple sample image blocks;

[0081] Calling m first decoding sub-models to decode the sample coding features corresponding to the multiple sample image blocks m times, to obtain m first sample decoding features corresponding to each sample image block;

[0082] calling the second decoding sub-model, splicing the mth first sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a first sample splicing feature, and convolving the first sample splicing feature to obtain the first predicted image;

[0083] The training module is used to:

[0084] calling a generation sub-model in the adversarial model to reconstruct the sample encoding features corresponding to the plurality of sample image blocks and the m first sample decoding features to obtain a second predicted image belonging to the second image space; calling a discrimination sub-model in the adversarial model to discriminate the second predicted image and the sample image to obtain a discrimination result, wherein the discrimination result indicates whether the second predicted image is similar to the sample image;

[0085] Based on the sample image, the first predicted image and the discrimination result, the degradation removal model and the adversarial model are trained so that the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image increases, and the similarity between the second predicted image obtained by calling the trained generation sub-model and the sample image increases.

[0086] In another possible implementation, the generation sub-model includes m first decoding layers and m second decoding layers;

[0087] The training module is used to:

[0088] Calling the first first decoding layer to decode the sample coding features corresponding to the multiple sample image blocks to obtain the first second sample decoding feature corresponding to each sample image block; fusing the second sample decoding feature with the first first sample decoding feature to obtain the first fused feature;

[0089] Calling the jth first decoding layer to decode the j-1th fusion feature to obtain the jth second sample decoding feature corresponding to each sample image block; until calling the mth first decoding layer to obtain the mth second sample decoding feature corresponding to each sample image block, j is an integer greater than 1 and not greater than m;

[0090] calling the second decoding layer, splicing the mth second sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a second sample splicing feature, and convolving the second sample splicing feature to obtain the second predicted image;

[0091] The discriminant sub-model is called to discriminate the second predicted image and the sample image to obtain the discrimination result.

[0092] In another possible implementation, the first image is an image extracted from a region of the original image that contains biological features; and the device further includes:

[0093] A second scaling module, configured to scale the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of a region of the background image containing the biological feature is the second scale;

[0094] The second combining module is used to combine the fourth image with the background image according to the position of the first image in the original image to obtain a fifth image.

[0095] On the other hand, a computer device is provided, comprising a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the image processing method described in the above aspects.

[0096] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the image processing method described in the above aspects.

[0097] On the other hand, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the operations performed by the image processing method as described in the above aspects are implemented.

[0098] The solution provided in the embodiment of the present application represents the spatial transformation of the image when it is transformed from the first image space to the second image space based on the spatial transformation characteristics between the first image space and the second image space, so that the first image feature corresponding to the first image is scaled and then transformed based on the spatial transformation characteristics, taking into account the image changes in the spatial dimension, thereby obtaining image features suitable for the second image space, and then reconstructing the image belonging to the second image space. Since the scale of the second image space is larger and the resolution is higher, the scale and resolution of the image are improved through the above-mentioned image transformation, thereby improving the clarity of the image. Moreover, the first image space and the second image space in this method can be image spaces of any scale, without having to limit the second scale to an integer multiple of the first scale, so it is more applicable and can meet the image processing needs in a variety of scenarios. BRIEF DESCRIPTION OF THE DRAWINGS

[0099] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0100] Figure 1 It is a structural diagram of an implementation environment provided by an embodiment of the present application;

[0101] Figure 2 is a flow chart of an image processing method provided by an embodiment of the present application;

[0102] Figure 3 is a flowchart of another image processing method provided by an embodiment of the present application;

[0103] Figure 4 is a flowchart of another image processing method provided by an embodiment of the present application;

[0104] Figure 5 is a structural schematic diagram of a reconstruction model provided in an embodiment of the present application;

[0105] Figure 6 It is a structural schematic diagram of a spatial transformation sub-model and a reconstruction sub-model in a reconstruction model provided in an embodiment of the present application;

[0106] Figure 7 is a flowchart of another image processing method provided by an embodiment of the present application;

[0107] Figure 8 is a flowchart of another image processing method provided by an embodiment of the present application;

[0108] Fig. 9 is a schematic diagram of a model in another image processing method provided in an embodiment of the present application;

[0109] Fig.10 It is a structural schematic diagram of a degradation removal model provided in an embodiment of the present application;

[0110] Fig.11 is a structural diagram of a second encoding sub-model provided in an embodiment of the present application;

[0111] Fig.12 is a flow chart of a method for training a degradation removal model provided in an embodiment of the present application;

[0112] Fig.13 is a schematic diagram of a joint training degradation removal model and an adversarial model provided in an embodiment of the present application;

[0113] Fig.14 This is an operational flow chart of a joint training provided in an embodiment of the present application;

[0114] Fig.15 is an operation flow chart of repairing an image provided by an embodiment of the present application;

[0115] Fig.16 is a schematic diagram of a face image provided in an embodiment of the present application;

[0116] Fig.17 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0117] Fig.18 is a schematic diagram of the structure of a terminal provided in an embodiment of the present application;

[0118] Fig.19 It is a structural diagram of a server provided in an embodiment of the present application. DETAILED DESCRIPTION

[0119] In order to make the objectives, technical solutions and advantages of the embodiments of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.

[0120] The terms "first", "second", "third", "fourth", "fifth", etc. used in this application may be used herein to describe various concepts, but unless otherwise specified, these concepts are not limited by these terms. These terms are only used to distinguish one concept from another. For example, a first image may be referred to as a second image, and similarly, a second image may be referred to as a first image without departing from the scope of this application.

[0121] The terms "at least one", "a plurality of", "each", and "any" used in this application include one, two or more than two, a plurality of includes two or more than two, each refers to each of the corresponding plurality, and any refers to any one of the plurality. For example, a plurality of pixels includes three pixels, and each refers to each of the three pixels, and any refers to any one of the three pixels, which can be the first pixel, the second pixel, or the third pixel.

[0122] The data involved in this application (including but not limited to images, image features, etc.) are authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.

[0123] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0124] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics and other technologies. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0125] Computer Vision (CV) is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision such as using cameras and computers to replace human eyes to identify and measure targets, and further perform graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology usually includes image processing, image recognition, image semantic understanding, image retrieval, OCR (Optical Character Recognition), video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D (3-Dimension) technology, virtual reality, augmented reality, simultaneous positioning and mapping, autonomous driving, smart transportation and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition.

[0126] Machine Learning (ML) is a multi-disciplinary subject that involves probability theory, statistics, approximation theory, convex analysis, algorithm complexity theory and other disciplines. It specializes in studying how computers simulate or implement human learning behavior to acquire new knowledge or skills and reorganize existing knowledge structures to continuously improve their performance. Machine learning is the core of artificial intelligence and the fundamental way to make computers intelligent. Its applications are spread across all areas of artificial intelligence. Machine learning and deep learning usually include artificial neural networks, belief networks, reinforcement learning, transfer learning, inductive learning, and teaching learning.

[0127] The solution provided in the embodiment of the present application is based on artificial intelligence machine learning technology, which can train reconstruction models and degradation removal models, and implement an image processing method using the trained reconstruction models and degradation removal models.

[0128] Before describing the embodiments of the present application in detail, the keywords involved in the embodiments of the present application are described as follows:

[0129] (1) CNN (Convolutional Neural Network): It is a feed-forward neural network whose artificial neurons can respond to surrounding units within a certain coverage area. Convolutional neural networks include convolutional layers and pooling layers.

[0130] (2) GAN (Generative Adversarial Networks): It is a deep learning model and one of the most promising methods for unsupervised learning on complex distributions in recent years. Generative adversarial networks consist of two parts: a generative model and a discriminative model. The two models compete with each other to learn and produce good outputs.

[0131] (3) Self-attention mechanism: The attention mechanism imitates the internal process of biological observation behavior, that is, a mechanism that aligns internal experience and external sensations to increase the observation precision of some areas. The attention mechanism can quickly extract important features of sparse data, and is therefore widely used in natural language processing tasks, computer vision tasks, etc. The self-attention mechanism is an improvement on the attention mechanism, which reduces dependence on external information and is better at capturing the internal correlation of data or features.

[0132] (4) Transformer model: It is a model based on the self-attention mechanism to handle semantic sequence related problems.

[0133] (5) Resolution: Also known as resolution, it determines the fineness of image details. The higher the resolution of an image, the more pixels it contains and the clearer the image.

[0134] (6) Super-resolution: It is a method of improving the resolution of the original image through hardware or software methods. The process of obtaining a high-resolution image through a series of low-resolution images is called super-resolution reconstruction.

[0135] (7) Image noise: refers to unnecessary or redundant interference information present in image data.

[0136] (8) Image upsampling: refers to the technology of adjusting the size of an image to enlarge it. The resolution of the upsampled image will become higher.

[0137] (9) Kaiming initialization: A method for initializing neural network parameters proposed by Kaiming He. The initialization follows a uniform distribution U~(-bound,bound), where

[0138] (10) Fréchet Inception distance: also known as FID, is a method for evaluating the quality of images generated by GAN. FID is calculated by calculating the Fréchet distance between two Gaussian functions fitted to the feature representation of the Inception network.

[0139] (11) NIQE (Natural Image Quality Evaluator): It is an image quality evaluation method that uses a 96×96 resolution image block with no overlap to normalize the image contrast and calculate the mean local contrast of each image block.

[0140] The image processing method provided in the embodiment of the present application is performed by a computer device, and optionally, the computer device is a terminal or a server. Optionally, the server is an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Server. Optionally, the terminal is a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, an intelligent voice interaction device, a smart home appliance, and a car terminal, etc., but is not limited to this.

[0141] In some embodiments, the computer program involved in the embodiments of the present application may be deployed and executed on a computer device, or on multiple computer devices located at one location, or on multiple computer devices distributed at multiple locations and interconnected by a communication network. Multiple computer devices distributed at multiple locations and interconnected by a communication network may constitute a blockchain system.

[0142] Figure 1 is a schematic diagram of an implementation environment provided by an embodiment of the present application. Figure 1 The implementation environment includes a terminal 101 and a server 102. The terminal 101 and the server 102 are directly or indirectly connected via wired or wireless communication, and this application does not limit this.

[0143] The server 102 is used to provide an image restoration function, and the terminal 101 obtains the image to be restored and sends the image to the server 102. The server 102 uses the image processing method provided in the embodiment of the present application to restore the image, thereby obtaining an image with higher resolution and larger scale, thereby improving the clarity of the image.

[0144] In one possible implementation, a target application provided by the server 102 is installed on the terminal 101, and the terminal 101 can implement functions such as image processing and message interaction through the target application. Optionally, the target application is a target application in the operating system of the terminal 101, or a target application provided by a third party. For example, the target application is an image processing application, which has an image processing function. Of course, the image processing application can also have other functions, such as a message interaction function, a navigation function, a game function, etc.

[0145] The terminal 101 sends the image to be repaired to the server 102 through the target application. The server 102 receives the image sent by the terminal 101, repairs the image to obtain an image with higher resolution and larger scale, and sends it to the terminal 101, which displays it through the target application.

[0146] In other embodiments, after the terminal acquires the image, the image processing method provided in the embodiment of the present application is used to repair the image to obtain an image with higher resolution and larger scale, thereby improving the clarity of the image.

[0147] Figure 2 is a flowchart of an image processing method provided in an embodiment of the present application, the method being executed by a computer device, such as Figure 2 As shown, the method includes:

[0148] 201. A computer device determines a spatial transformation feature based on a first scale of a first image space and a second scale of a second image space.

[0149] In an embodiment of the present application, in order to facilitate image transformation, a first image space and a second image space are first determined, wherein the first image space has a first scale, the second image space has a second scale, the second scale is larger than the first scale, and the resolution of the second image space is higher than the resolution of the first image space. Therefore, the size of the image belonging to the second image space is larger than the size of the image belonging to the first image space, and the resolution of the image belonging to the second image space is higher than the resolution of the image belonging to the first image space. Then, when any image is transformed from the first image space to the second image space, the scale of the image can be increased and the resolution of the image can be improved, thereby improving the clarity of the image.

[0150] The first scale and the second scale are arbitrary scales, for example, the first scale is 512×512, and the second scale is 1024×1024. In addition, the resolution of the first image space and the second image space is arbitrary resolution, which is not limited in the embodiment of the present application.

[0151] For example, when a computer device obtains a first image to be processed, it can determine the image space to which the first image belongs as the first image space, and also determine the second image space based on the scale and resolution to which the image needs to be transformed, so as to subsequently transform the first image from the first image space to the second image space. This transformation process does not limit the specific scale and resolution of the first image space and the second image space, so the method provided in the embodiment of the present application can be implemented for any image space.

[0152] The spatial transformation feature represents the spatial transformation of any image belonging to the first image space when it is transformed to the second image space, and the spatial transformation feature can reflect the changes in spatial dimensions that occur when an image is transformed in different image spaces. Therefore, in the embodiment of the present application, in order to perform image transformation, it is first necessary to determine the spatial transformation feature required for transforming from the first image space to the second image space.

[0153] 202. The computer device performs scale upsampling on the first image feature corresponding to the first image belonging to the first image space to obtain the second image feature belonging to the second scale.

[0154] The embodiment of the present application takes a first image belonging to a first image space as an example to illustrate the process of transforming the first image to a second image space. First, since the second scale is larger than the first scale, if the first image is to be transformed to the second image space, the scale of the first image needs to be increased first.

[0155] Therefore, a first image feature corresponding to the first image is obtained, and the first image feature is used to represent the first image. The first image feature belongs to a first scale, and then the first image feature is upsampled so that the obtained second image feature belongs to a second scale. Subsequently, the second image feature with a larger scale can be used to generate an image with a larger scale.

[0156] 203. The computer device transforms the second image feature based on the spatial transformation feature to obtain a third image feature.

[0157] Since the spatial transformation feature represents the spatial transformation of any image belonging to the first image space when it is transformed to the second image space, the second image feature obtained after upscaling is transformed based on the spatial transformation feature, and the obtained third image feature can represent the image obtained by transforming the first image from the first image space to the second image space.

[0158] 204. The computer device reconstructs the third image feature to obtain a second image belonging to the second image space.

[0159] The third image feature can represent the second image. Therefore, the third image feature is reconstructed, and the obtained image including the third image feature is the second image obtained by repairing the first image.

[0160] In one possible implementation, a method for obtaining a first image feature from a first image matches a method for reconstructing a third image feature to obtain a second image. For example, if a feature extraction algorithm is used to obtain the first image feature from the first image, an inverse algorithm of the feature extraction algorithm is used to reconstruct the third image feature to obtain the second image.

[0161] The method provided in the embodiment of the present application represents the spatial transformation of an image when it is transformed from the first image space to the second image space based on the spatial transformation characteristics between the first image space and the second image space, so that the first image characteristics corresponding to the first image are up-sampled and then transformed based on the spatial transformation characteristics, taking into account the image changes in the spatial dimension, thereby obtaining image characteristics suitable for the second image space, and then reconstructing the image belonging to the second image space. Since the scale of the second image space is larger and the resolution is higher, the scale and resolution of the image are improved through the above-mentioned image transformation, thereby improving the clarity of the image. Moreover, the first image space and the second image space in this method can be image spaces of any scale, without having to limit the second scale to an integer multiple of the first scale, so it is more applicable and can meet image processing requirements in a variety of scenarios.

[0162] exist Figure 2 Based on the embodiment shown, the following Figure 3 The illustrated embodiment will take the spatial transformation feature including the position shift feature as an example to explain the image processing process in detail.

[0163] Figure 3 is a flowchart of an image processing method provided in an embodiment of the present application, the method being executed by a computer device, such as Figure 3 As shown, the method includes:

[0164] 301. The computer device determines a ratio between the second scale and the first scale as an upsampling factor.

[0165] The embodiments of the present application involve a first image space and a second image space, wherein the first image space is the image space to which the first image to be restored belongs, and the second image space is the target image space, that is, the image space to which the first image needs to be transformed.

[0166] Among them, the second scale is the scale of the second image space, the first scale is the scale of the first image space, the second scale is greater than the first scale, and the ratio between the second scale and the first scale can reflect the scale change of the pixel points between the first image space and the second image space. Therefore, the ratio is used as the upsampling factor. The upsampling factor represents the upsampling factor used to transform any pixel point from the first image space to the second image space, and can be used when upsampling the image.

[0167] In a possible implementation, the first scale and the second scale both include height and width, for example, the height is the X-axis scale, and the width is the Y-axis scale. The upsampling factor includes a height upsampling factor and a width upsampling factor, and the ratio of the height of the second image space to the height of the first image space is determined as the height upsampling factor, and the ratio of the width of the second image space to the width of the first image space is determined as the width upsampling factor.

[0168] Optionally, the first scale, the second scale, the height upsampling factor, and the width upsampling factor satisfy the following relationship:

[0169]

[0170] Among them, S x represents the width upsampling factor, W′ represents the width of the second image space, W represents the width of the first image space, S y represents the height upsampling factor, H′ represents the height of the second image space, and H represents the height of the first image space. In this optional solution, the first scale is W×H, and the second scale is W′×H′.

[0171] 302. The computer device transforms the plurality of first pixel points into the first image space based on the coordinates of the plurality of first pixel points in the second image space and an upsampling factor to obtain coordinates of the plurality of second pixel points.

[0172] In the embodiment of the present application, the image space is represented by the coordinates of multiple pixel points, wherein, when the coordinates of each pixel point in the image space have been determined, the size and number of the pixel points have also been determined, wherein the size and number of the pixel points determine the scale of the image space, and the number of pixels within the unit scale determines the resolution of the image space. If an image is generated according to the coordinates of multiple pixel points in the image space so that the coordinates of each pixel point in the image are consistent with the image space, then the image is an image belonging to the image space. The difference between an image and an image space is that the pixel value of each pixel point in the image space has not yet been determined, and there are multiple possibilities, while the pixel value of each pixel point in the image has been determined, and the pixel value represents the color in the image.

[0173] In the embodiment of the present application, the first image space includes the coordinates of multiple second pixel points, and the second image space includes the coordinates of multiple first pixel points. The upsampling scale factor is the scaling ratio between the second image space and the first image space, and the coordinates of the first pixel point in the second image space are transformed by the upsampling factor to obtain the coordinates of the second pixel point corresponding to the first pixel point in the first image space.

[0174] In a possible implementation, since the upsampling scale factor is a ratio of the second scale to the first scale, the coordinates of the first pixel point in the second image space are divided by the upsampling scale factor to obtain the coordinates of the second pixel point corresponding to the first pixel point in the first image space.

[0175] Conversely, the coordinates of the second pixel point in the first image space are multiplied by the up-sampling scale factor to obtain the coordinates of the first pixel point corresponding to the second pixel point in the second image space.

[0176] In another possible implementation, the coordinates of the center of a pixel are taken as the coordinates of the pixel, and the height and width of the pixel are 1, for example, 0.5 represents half the scale of the pixel, and the coordinate change caused by the movement of the center of the pixel also needs to be considered when performing the transformation. Therefore, the coordinates of the first pixel, the upsampling factor, and the coordinates of the second pixel satisfy the following relationship:

[0177]

[0178]

[0179] The coordinates of the first pixel are (x, y), the coordinates of the second pixel are (x′, y′), S x represents the width upsampling factor, S y Indicates the height upsampling factor.

[0180] 303. The computer device determines a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points.

[0181] The relative distance represents the distance between the position of a first pixel point and the position of a corresponding second pixel point in the second image space.

[0182] In a possible implementation, the relative distance includes a first relative distance and a second relative distance, the first relative distance is the relative distance between the first pixel point and the second pixel point on the X-axis, and the second relative distance is the relative distance between the first pixel point and the second pixel point on the Y-axis.

[0183] Optionally, the coordinates of the first pixel point, the upsampling factor, the coordinates of the second pixel point corresponding to the first pixel point, and the relative distance satisfy the following relationship:

[0184]

[0185]

[0186] The coordinates of the first pixel are (x, y), the coordinates of the second pixel are (x′, y′), S x represents the width upsampling factor, S y represents the height upsampling factor. R(x) is the first relative distance between the first pixel and the second pixel, R(y) is the second relative distance between the first pixel and the second pixel, Used to indicate rounding down.

[0187] 304. The computer device determines a position offset feature based on the coordinates of the plurality of second pixel points and the plurality of relative distances.

[0188] In the embodiment of the present application, the coordinates of the plurality of second pixel points are obtained by transforming the coordinates of the plurality of first pixel points in the second image space to the first image space, and the relative distance is the relative distance between the first pixel points and the second pixel points corresponding to each other in the first image space. The position offset feature determined based on the coordinates of the plurality of second pixel points and the plurality of relative distances can reflect the required offset of the position of the pixel point when the pixel point in any image belonging to the first image space is transformed to the second image space. Therefore, the position offset feature represents the position offset of the pixel point when any image belonging to the first image space is transformed to the second image space.

[0189] In one possible implementation, the position offset feature includes a first offset and a second offset of the multiple second pixel points, the first offset indicating the offset required for the second pixel point on the X-axis, and the second offset indicating the offset required for the second pixel point on the Y-axis.

[0190] The embodiment of the present application transforms the first pixel point in the second image space to the first image space based on the upsampling factor, and determines the relative distance between the coordinates of the pixel points before and after the transformation in the first image space, thereby determining the position offset feature based on the coordinates of the second pixel point after the transformation and the relative distance. The position offset feature can accurately represent the position offset of the pixel point when any image belonging to the first image space is transformed to the second image space, thereby improving the accuracy of the space transformation feature and further improving the accuracy of image processing.

[0191] It should be noted that the embodiment of the present application is described by taking the spatial transformation feature including the position offset feature as an example, while in another embodiment, it is not necessary to perform the above steps 301-304, but other methods are adopted to determine the spatial transformation feature.

[0192] 305. The computer device upsamples the first image feature corresponding to the first image belonging to the first image space to obtain a second image feature belonging to a second scale.

[0193] 306. The computer device transforms the second image feature based on the position offset feature to obtain a third image feature.

[0194] In one possible implementation, step 306 includes: adding the position offset feature to the second image feature to obtain a third image feature, or transforming in other ways, which is not limited in the present embodiment. For example, the second image feature includes the coordinates (x′, y′) of each pixel point in the first image and a feature value, the feature value is used to indicate the pixel value of the pixel point, which can be a value such as 0 or 1. The second image feature includes the first offset δ x and the second offset δ y , then the position offset feature is added to the second image feature, that is, the coordinates (x′, y′) of each pixel point are superimposed with the first offset δ x and the second offset δ y , thus obtaining the offset coordinates (x′+δ x , y′+δ y ).

[0195] In addition, the embodiment of the present application is described in that the spatial transformation feature includes a position offset feature. In this case, the spatial transformation feature may also include other features. Therefore, in addition to determining the position offset feature using the above steps 301-304, other spatial transformation features may also be obtained, thereby transforming the second image feature based on the obtained spatial transformation feature to obtain a third image feature.

[0196] 307. The computer device reconstructs the third image feature to obtain a second image belonging to the second image space.

[0197] Here, step 307 is similar to the above step 204 and will not be described in detail here.

[0198] The method provided in the embodiment of the present application represents the spatial transformation of an image when it is transformed from the first image space to the second image space based on the spatial transformation characteristics between the first image space and the second image space, so that the first image characteristics corresponding to the first image are upsampled and then transformed based on the spatial transformation characteristics, taking into account the image changes in the spatial dimension, thereby obtaining image characteristics suitable for the second image space, and then reconstructing an image belonging to the second image space. Since the scale of the second image space is larger and the resolution is higher, the scale and resolution of the image are improved through the above-mentioned image transformation, thereby improving the clarity of the image.

[0199] Furthermore, based on the upsampling factor, the first pixel point in the second image space is transformed into the first image space, and the relative distance between the coordinates of the pixel points before and after the transformation is determined in the first image space, so as to determine the position offset feature based on the coordinates of the second pixel point after the transformation and the relative distance. The position offset feature can accurately represent the position offset of the pixel point when any image belonging to the first image space is transformed into the second image space, thereby improving the accuracy of the space transformation feature and further improving the accuracy of image processing.

[0200] In the above Figure 3 Based on the embodiment shown in the figure, in addition to the position offset feature, the spatial transformation feature can also include a scale transformation weight, as follows Figure 4 The illustrated embodiment will take the spatial transformation features including the position offset features and the scale transformation weights as an example to explain the image processing process in detail.

[0201] Figure 4 is a flowchart of an image processing method provided in an embodiment of the present application, the method being executed by a computer device, such as Figure 4 As shown, the method includes:

[0202] 401. The computer device determines a ratio between the second scale and the first scale as an upsampling factor.

[0203] 402. The computer device transforms the plurality of first pixel points into the first image space based on the coordinates of the plurality of first pixel points in the second image space and an upsampling factor to obtain coordinates of the plurality of second pixel points.

[0204] 403. The computer device determines a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points.

[0205] Steps 401-403 are similar to the above steps 301-303 and will not be described again.

[0206] After the coordinates of the plurality of second pixel points and the plurality of relative distances are determined, the position offset features and the scale transformation weights can be determined based on the coordinates of the plurality of second pixel points and the plurality of relative distances. The process of determining the position offset features is detailed in steps 404 and 405 below, and the process of determining the scale transformation weight is detailed in steps 404 and 406 below.

[0207] 404. The computer device forms a feature matrix with the coordinates of the plurality of second pixel points and the plurality of relative distances, wherein the feature matrix includes a plurality of first sub-features, each of which includes the coordinates of a second pixel point and the corresponding relative distance, and transforms the feature matrix to obtain a feature vector.

[0208] Each row in the feature matrix is ​​a first sub-feature, or each column is a first sub-feature. In addition, each first sub-feature includes the coordinates of a second pixel point and a relative distance, and the relative distance is the distance between the position of the first pixel point in the second image space and the position of the corresponding second pixel point. By transforming the feature matrix into a feature vector, the feature matrix can be reduced in dimension, so that the number of dimensions of the features in the feature matrix is ​​reduced, the size of the feature matrix is ​​reduced, and thus the amount of calculation is reduced.

[0209] 405. The computer device performs a first transformation on the feature vector to obtain a position offset feature.

[0210] For example, the position offset feature includes the first offset and the second offset of the plurality of second pixel points, the first offset represents the offset required for the second pixel point on the X axis, and the second offset represents the offset required for the second pixel point on the Y axis. That is, it is necessary to obtain features of two dimensions, namely, the first offset and the second offset, at the position corresponding to each second pixel point, so it is necessary to transform the feature vector so that the feature dimension at the position corresponding to each second pixel point is 2.

[0211] 406. The computer device performs a second transformation on the feature vector, and normalizes the transformed feature vector to obtain a scale transformation weight, where the scale transformation weight includes weight features of multiple second pixel points.

[0212] The scale transformation weight represents the scale transformation condition when any image belonging to the first image space is transformed to the second image space.

[0213] The first transformation in step 406 and the second transformation in step 405 may be transformations of the same manner or of different manners. The second transformation in step 406 is used to transform the feature matrix into a weight feature, which may be regarded as an adjustment coefficient of the scale of the second pixel.

[0214] 407. The computer device performs upscaling on the first image feature corresponding to the first image belonging to the first image space to obtain a second image feature belonging to a second scale.

[0215] This step 407 is similar to the above step 202 and will not be described again.

[0216] 408. The computer device adjusts the position of the feature in the second image feature based on the position offset feature to obtain a fourth image feature.

[0217] The second image feature includes multiple positions, and the feature at each position is used to represent the feature of the pixel point corresponding to the position. The second image feature is processed based on the position offset feature, and the feature in the second image feature can be offset, so that the fourth image feature obtained by processing represents the image after the position of the pixel point is offset.

[0218] In a possible implementation, the position offset feature includes the first offset and the second offset of each second pixel point, and the second image feature includes the horizontal coordinate and the vertical coordinate of each second pixel point, and the feature corresponding to each second pixel point. Then, in the order of the second pixels, the horizontal coordinate of each second pixel point is superimposed with the first offset, and the vertical coordinate is superimposed with the second offset, to obtain the offset coordinate of the second pixel point, and the feature corresponding to the second pixel point is adjusted to the position where the offset coordinate of the second pixel point is located. After the processing of multiple second pixels is completed, the adjusted fourth image feature can be obtained.

[0219] 409. The computer device performs compression convolution and expansion convolution on the fourth image feature based on the scale transformation weight to obtain the third image feature.

[0220] In one possible implementation, step 409 includes:

[0221] Based on the scale transformation weight, the compression convolution parameters and the expansion convolution parameters are adjusted respectively, and based on the adjusted compression convolution parameters and the adjusted expansion convolution parameters, the fourth image feature is compressed and expanded to obtain a fifth image feature, and the fifth image feature is fused with the second image feature to obtain a third image feature.

[0222] Since the compressed convolution parameters and the dilated convolution parameters are adjusted based on the scale transformation weight, the adjusted compressed convolution parameters and the adjusted dilated convolution parameters contain scale transformation information. In this way, the fifth image feature obtained after performing compressed convolution and dilated convolution on the fourth image feature also contains scale transformation information, and can represent the image after scale transformation.

[0223] Furthermore, in order to avoid information loss due to scale transformation, the fifth image feature is fused with the second image feature, so that the fused third image feature contains both scale transformation information and original image information.

[0224] In one possible implementation, based on the scale transformation weight, the compressed convolution parameters and the dilated convolution parameters are adjusted respectively, including: performing a dot multiplication of the scale transformation weight and the compressed convolution parameter to obtain the adjusted compressed convolution parameter, and performing a dot multiplication of the scale transformation weight and the dilated convolution parameter to obtain the adjusted dilated convolution parameter.

[0225] 410. The computer device reconstructs the third image feature to obtain a second image belonging to the second image space.

[0226] It should be noted that the embodiment of the present application is illustrated by taking the spatial transformation features including position offset features and scale transformation weights as an example, while in another embodiment, other methods can also be adopted to transform the second image features to obtain the third image features based on the spatial transformation features.

[0227] The method provided in the embodiment of the present application represents the spatial transformation of an image when it is transformed from the first image space to the second image space based on the spatial transformation characteristics between the first image space and the second image space, so that the first image characteristics corresponding to the first image are upsampled and then transformed based on the spatial transformation characteristics, taking into account the image changes in the spatial dimension, thereby obtaining image characteristics suitable for the second image space, and then reconstructing an image belonging to the second image space. Since the scale of the second image space is larger and the resolution is higher, the scale and resolution of the image are improved through the above-mentioned image transformation, thereby improving the clarity of the image.

[0228] In addition, the spatial transformation features include position offset features and scale transformation weights. The position offset features can accurately represent the position offset of pixels when any image belonging to the first image space is transformed to the second image space. The scale transformation weights can accurately represent the scale transformation of any image belonging to the first image space when it is transformed to the second image space. The spatial transformation features are more accurate. After adjusting the position of the features based on the position offset features, compressed convolution and expanded convolution are performed based on the scale transformation weights, so that accurate image features can be obtained, and then an accurate target image can be obtained, thereby improving the accuracy of image processing.

[0229] In the above Figures 2 to 4On the basis of the embodiment shown, the embodiment of the present application also provides a method for improving the clarity of a biometric image. In order to improve the clarity of the biometric in the process of processing the image, and to ensure the accuracy of the biometric, and to avoid affecting the subsequent biometric recognition accuracy, a first image is first extracted from the original image, and the first image is an image extracted from the area of ​​the original image containing the biometric. If the biometric is a face feature, the first image is a face image corresponding to the face area in the original image, or if the biometric is a fingerprint feature, the first image is a fingerprint image corresponding to the fingerprint area in the original image, etc.

[0230] After the first image is extracted, the scale of the first image is the first scale. Figures 2 to 7 The method provided in the illustrated embodiment improves the clarity of the first image to obtain the second image, and the scale of the obtained second image is the second scale. Then, in order to restore the second image to the original image, the original image is first scaled according to the ratio between the second scale and the first scale to obtain the background image. The scale of the area containing the biometric features in the background image is the second scale, which is equivalent to reserving space for the second image. Then, the second image and the background image can be combined according to the position of the first image in the original image to obtain the third image. The background of the original image is retained in the third image, and the clarity of the area containing the biometric features is improved.

[0231] In the above Figures 2 to 4 Based on the embodiment shown, in a possible implementation, the image processing method is implemented based on a reconstruction model, and the reconstruction model includes a spatial transformation sub-model and a reconstruction sub-model. Figure 2 Each step in the illustrated embodiment may be performed by the spatial transformation sub-model and the reconstruction sub-model, wherein the spatial transformation sub-model is used to determine the spatial transformation feature, and the reconstruction sub-model is used to perform image reconstruction based on the spatial transformation feature. Accordingly, the method includes:

[0232] The spatial transformation submodel is called to determine the spatial transformation feature based on the first scale and the second scale. The reconstruction submodel is called to upsample the first image feature to obtain the second image feature, transform the second image feature based on the spatial transformation feature to obtain the third image feature, and reconstruct the third image feature to obtain the second image.

[0233] It should be noted that this implementation can be combined with the above Figures 2 to 4 The illustrated embodiments are combined in any manner, for example, calling the spatial transformation sub-model to execute the above steps 301-304 or the above steps 401-406, calling the reconstruction sub-model to execute the above steps 305-307 or execute the above steps 407-410.

[0234] In another possible implementation, Figure 5 As shown, the reconstruction model includes a feature extraction sub-model, a space transformation sub-model and a reconstruction sub-model; wherein the feature extraction sub-model is used to extract image features, the space transformation sub-model is used to determine the space transformation features, and the reconstruction sub-model is used to reconstruct the image based on the space transformation features. Accordingly, the method includes:

[0235] The spatial transformation sub-model is called to determine the spatial transformation features based on the first scale and the second scale, the feature extraction sub-model is called to extract the first image features corresponding to the first image, the reconstruction sub-model is called to upsample the first image features to obtain the second image features, the second image features are transformed based on the spatial transformation features to obtain the third image features, and the third image features are reconstructed to obtain the second image.

[0236] In some embodiments, Figure 5 As shown, the feature extraction sub-model includes multiple convolutional layers ( Figure 5 Taking only four convolutional layers as an example, the multiple convolutional layers are cascaded, and the method includes: obtaining a first image, calling multiple convolutional layers included in the feature extraction sub-model, performing multiple convolution processes on the first image, and obtaining first image features corresponding to the first image. In addition, the structure of the feature extraction sub-model can also be other structures, which is not limited in the embodiments of the present application.

[0237] Among them, the above-mentioned multiple convolution layers can be two-dimensional convolution layers, the number of convolution kernels of the four two-dimensional convolution layers are 64, 64, 64, and 64 respectively, the size of the convolution kernel is 3, and the dimension of the obtained first image feature is H×W×64, where H represents the height of the first image and W represents the width of the first image.

[0238] In some embodiments, Figure 6 As shown, the spatial transformation submodel includes at least two first fully connected layers ( Figure 6 Taking only two first fully connected layers as an example), the second fully connected layer, the third fully connected layer and the normalization layer, at least two first fully connected layers are cascaded, and the last first fully connected layer is connected to the second fully connected layer and the third fully connected layer respectively, the third fully connected layer is connected to the normalization layer, and the normalization layer includes a softmax function (normalized exponential function). The reconstruction submodel includes an upsampling layer, a first fusion layer, a compressed convolution layer, an expanded convolution layer, a second fusion layer and a convolution layer, the compressed convolution layer corresponds to a compressed convolution parameter, and the expanded convolution layer corresponds to an expanded convolution parameter.

[0239] In the above Figure 6 Based on the model structure shown, Figure 7 As shown, the method includes:

[0240] 701. A computer device calls a feature extraction sub-model to extract first image features corresponding to a first image.

[0241] 702. The computer device calls the spatial transformation sub-model and determines the ratio between the second scale and the first scale as an upsampling factor.

[0242] The upsampling factor represents the upsampling factor used to transform any pixel point from the first image space to the second image space.

[0243] 703. The computer device calls the spatial transformation sub-model, transforms the multiple first pixel points into the first image space based on the coordinates of the multiple first pixel points in the second image space and the upsampling factor, obtains the coordinates of the multiple second pixel points, and determines the multiple relative distances based on the coordinates of the multiple first pixel points, the upsampling factor and the coordinates of the multiple second pixel points.

[0244] The relative distance represents the distance between the position of a first pixel point and the position of a corresponding second pixel point in the second image space.

[0245] 704. The computer device calls the space transformation sub-model to form a feature matrix with the coordinates of the plurality of second pixel points and the plurality of relative distances. The feature matrix includes a plurality of first sub-features, and each first sub-feature includes the coordinates of a second pixel point and the corresponding relative distance.

[0246] 705. The computer device calls at least two first fully connected layers to transform the feature matrix to obtain a feature vector, and calls the second fully connected layer to perform a first transformation on the scale vector to obtain a position offset feature.

[0247] The position shift feature indicates the position shift of a pixel point when any image belonging to the first image space is transformed into the second image space.

[0248] 706. The computer device calls the third fully connected layer, performs a second transformation on the feature vector, and then uses a softmax function to perform normalization processing to obtain a scale transformation weight.

[0249] The scale transformation weight represents the scale transformation of any image belonging to the first image space when it is transformed to the second image space. The scale transformation weight includes weight features of multiple second pixel points.

[0250] 707. The computer device calls the upsampling layer to upsample the first image feature to obtain a second image feature of a second scale.

[0251] 708. The computer device calls the first fusion layer, and adjusts the position of the feature in the second image feature based on the position offset feature to obtain a fourth image feature.

[0252] 709. The computer device adjusts the compression convolution parameters and the dilation convolution parameters based on the scale transformation weights, calls the compression convolution layer and the dilation convolution layer, and performs compression convolution and dilation convolution on the fourth image feature based on the adjusted compression convolution parameters and the adjusted dilation convolution parameters to obtain the fifth image feature.

[0253] The compressed convolution parameter and the expanded convolution parameter can be initialized using Kaiming initialization and then adjusted based on the scale transformation weight. Optionally, the compressed convolution parameter is the size of the convolution kernel used in the compressed convolution process, and the expanded convolution parameter is the size of the convolution kernel used in the expanded convolution process.

[0254] It should be noted that the first image includes the coordinates and pixel information of the pixel points, and the pixel information includes information on multiple channels, such as the brightness values ​​on the three channels of RGB (Red, Green, Blue). After the image feature of the first image (i.e., the fourth image feature mentioned above) is compressed and expanded, the number of channels of the pixel points changes, for example, from 3 channels to 64 channels. Therefore, the fifth image feature is the image feature obtained after the transformation in the channel dimension.

[0255] 710. The computer device calls the second fusion layer to fuse the fifth image feature with the second image feature to obtain a third image feature.

[0256] Among them, the second image feature is the image feature obtained after transformation in the spatial dimension, and the fifth image feature is the image feature obtained after transformation in the channel dimension. Fusion of the two can obtain image features after transformation in both the spatial dimension and the channel dimension.

[0257] like Figure 6 As shown, the second fusion layer can be used to perform an addition operation, that is, to perform a pixel-by-pixel addition operation on the fifth image feature and the second image feature, that is, to add the features corresponding to each pixel to obtain the third image feature.

[0258] 711. The computer device calls the convolution layer to convolve the third image feature to obtain a second image belonging to the second image space.

[0259] For example, the convolution layer is a two-dimensional convolution layer, the number of convolution kernels is 64, and the convolution kernel size is 3. By convolving the third image feature, a second image belonging to a high-resolution space can be reconstructed, thereby improving the resolution of the image.

[0260] Moreover, since the third image feature is an image feature after both the spatial dimension and the channel dimension are transformed, the second image obtained based on the third image feature is an image obtained after image super-resolution processing in both the spatial dimension and the channel dimension, which can better meet the user's image restoration needs.

[0261] In addition, before using the reconstruction model, the reconstruction model needs to be trained. During the training process, sample data can be first obtained, and the sample data includes an original sample image and a reconstructed sample image after the original sample image has improved definition. For example, the original sample image belongs to the first image space, and the reconstructed sample image belongs to the second image space. Then, the reconstruction model is trained based on the sample data. After one or more trainings, the reconstruction model can learn the ability to improve image definition, thereby obtaining an accurate reconstruction model.

[0262] The operation process of the embodiment of the present application is divided into two stages: the Coarse stage and the Refine stage, and the reconstructed model can be called the Coarse stage model. During the iterative training process, the current number of iterative training times is obtained to determine whether the number of iterative training times is greater than the iteration number threshold of the Coarse stage. If the number of iterative training times is not greater than the iteration number threshold of the Coarse stage, the loss function of the Coarse stage is calculated, and the gradient value is returned through the back propagation algorithm, so as to update the model parameters of the reconstructed model of the Coarse stage based on the gradient value. If the number of iterative training times is greater than the iteration number threshold of the Coarse stage, the reconstruction model training is completed.

[0263] It should be noted that in the above Figures 2 to 7 Based on the embodiment shown, after the computer device processes the first image to obtain the second image, since the second image may contain noise and other problems, the image quality is still not good enough, so the second image can be further processed to improve the image quality.

[0264] Accordingly, if Figure 8 As shown, the method also includes:

[0265] 801. A computer device divides a second image into multiple image blocks, and obtains a first image block feature corresponding to each image block.

[0266] The scale of the second image is the second scale, and the scale of the image block can be any scale smaller than the second scale. The resolution of the second image is the resolution of the second image space, and the scale of the image block can be any resolution smaller than the resolution of the second image. For example, the resolution of the image block is 16*16.

[0267] In one possible implementation, the process of obtaining the first image block feature corresponding to each image block includes: extracting features from each image block to obtain the third image block feature corresponding to each image block; and fusing the third image block feature corresponding to each image block with the position feature to obtain the first image block feature corresponding to each image block.

[0268] Among them, the position feature indicates the position of the image block in the second image. By fusing the original third image block feature of the image block with the position feature, not only the content of the image block itself but also the position of the image block in the second image is considered, so the obtained third image block feature is more accurate.

[0269] 802. Based on the self-attention mechanism, encode and decode the first image block features corresponding to each image block m times respectively to obtain the second image block features corresponding to each image block, where m is an integer greater than 1.

[0270] The above encoding is used to extract content features of an image block and remove other features in the image block that are irrelevant to the content features, thereby removing noise.

[0271] By encoding and decoding each image block multiple times based on the self-attention mechanism, the information of other image blocks in the second image can be considered for each image block, thereby improving the accuracy of the image block features of the image block, so that the image block features can accurately represent the content of the image block and avoid being mixed with noise features.

[0272] 803. According to the positions of the multiple image blocks in the second image, second image block features corresponding to the multiple image blocks are spliced ​​to obtain spliced ​​features, and the spliced ​​features are convolved to obtain a fourth image belonging to the second image space.

[0273] After processing the image block features of each image block, the second image block features corresponding to the multiple image blocks are spliced ​​according to the positions of the multiple image blocks in the second image, and the obtained spliced ​​features are the updated image features. Then, the spliced ​​features are convolved to achieve image reconstruction and obtain an updated fourth image. Compared with the second image, the fourth image has reduced noise.

[0274] In a possible implementation, the process of acquiring the fourth image based on the second image is implemented based on a degradation removal model, for example, Fig. 9 As shown, the computer device calls Figure 5 After the reconstruction model is shown to generate the second image, the degradation removal model is called to generate the fourth image.

[0275] In another possible implementation, Fig.10As shown, the degradation removal model includes a first encoding sub-model, m second encoding sub-models based on a self-attention mechanism, m first decoding sub-models based on a self-attention mechanism, and a second decoding sub-model, and the m second encoding sub-models are connected in sequence, the m first decoding sub-models are connected in sequence, and the mth second encoding sub-model is connected to the first first decoding sub-model, where m is an integer greater than 1. The process of obtaining a fourth image based on the second image includes:

[0276] The first encoding sub-model is called to divide the second image into multiple image blocks, and the first image block features corresponding to each image block are obtained. The m second encoding sub-models are called to encode the first image block features corresponding to the multiple image blocks m times to obtain the encoding features corresponding to the multiple image blocks. The m first decoding sub-models are called to decode the encoding features corresponding to the multiple image blocks m times to obtain the decoding features corresponding to the multiple image blocks. The second decoding sub-model is called to splice the decoding features corresponding to the multiple image blocks according to the positions of the multiple image blocks in the second image to obtain the spliced ​​features, and the spliced ​​features are convolved to obtain the fourth image.

[0277] Since the m second encoding sub-models and the m first decoding sub-models are models based on the self-attention mechanism, the information of other image blocks will be considered in the process of processing each image block, so the accuracy of the extracted features can be guaranteed, thereby ensuring the accuracy of the subsequent image. Among them, the m second encoding sub-models and the m first decoding sub-models can be Transformer models or other models based on the self-attention mechanism.

[0278] Taking facial images as an example, since the left and right parts of an actual face may not be completely symmetrical, the facial area in the collected facial image is also asymmetrical. After being processed by the above-mentioned m second encoding sub-models and m first decoding sub-models, since each image block refers to each other's information, the facial area in the restored facial image is symmetrical, thereby improving the quality of the facial image.

[0279] In a possible implementation, the first encoding process in the m encodings includes:

[0280] For any image block, the first second encoding sub-model is called, and based on the similarity between the first image block feature corresponding to the image block and the first image block features corresponding to multiple image blocks, the first image block features corresponding to the multiple image blocks are weightedly fused to obtain the first fused image block feature corresponding to the image block; the first fused image block feature is fused with the first image block feature corresponding to the image block to obtain the first encoding feature corresponding to the image block.

[0281] In a possible implementation, the i-th encoding process in the m-encoding process includes:

[0282] For any image block, call the i-th second coding sub-model, and based on the similarity between the i-1th coding feature corresponding to the image block and the i-1th coding features corresponding to multiple image blocks, perform weighted fusion on the i-1th coding features corresponding to the multiple image blocks to obtain the i-th fused image block feature corresponding to the image block; fuse the i-th fused image block feature with the i-1th coding feature corresponding to the image block to obtain the i-th coding feature corresponding to the image block, where i is an integer greater than 1 and not greater than m.

[0283] The above encoding method uses a self-attention mechanism to determine the image features of each image block, which can comprehensively consider the similarities between multiple image blocks, so that the obtained fused image block features incorporate the influence of other image blocks and are more accurate. Moreover, the obtained fused image block features are fused with the original first image block features, which not only incorporates the influence of other image blocks, but also avoids losing the global information of the original image block. Therefore, the obtained encoding features will contain more comprehensive information and have higher accuracy.

[0284] In another possible implementation, the m second encoding sub-models and the m first decoding sub-models use the same structure. Taking the second encoding sub-model as an example, Fig.11 As shown, the second encoding sub-model includes a first vector normalization layer, a windowed multi-head self-attention layer, a first fusion layer, a second vector normalization layer, a multi-layer perceptron and a second fusion layer.

[0285] P represents the first image block feature of multiple image blocks output by the first encoding sub-model, and P is sequentially input into the m second encoding sub-models connected in series for feature extraction. In the second encoding sub-model, P is firstly input into the first vector normalization layer for normalization to obtain the normalized feature P1, and then the normalized feature P1 is input into the windowed multi-head self-attention layer, and the feature of P1 of each image block is extracted by the self-attention mechanism to obtain the attention feature P2. P and P2 are fused to obtain the global feature P3 representing the global information of the image block, and the global feature P3 includes both the information of the image block itself and the influence of other image blocks on the image block.

[0286] The global feature P3 is then input into the second vector normalization layer to obtain the normalized feature P4, the normalized feature P4 is input into the multi-layer perceptron to obtain the multi-layer perceptual feature P5, and the multi-layer perceptual feature P5 is then fused with the global feature P3 to obtain the encoding feature P6 of the image block.

[0287] In this implementation, after normalization processing is performed, the first image block feature corresponding to the image block is adjusted to a specified feature range, and then processed using a windowed multi-head self-attention mechanism to improve accuracy.

[0288] It should be noted that the embodiment of the present application also provides a method for training a degradation removal model, such as Fig.12 As shown, the training method is executed by a computer device, which is a terminal or a server, and the computer device that executes the training method and the computer device that executes the above-mentioned image processing method can be the same device or different devices, which is not limited in this embodiment of the present application.

[0289] The training method includes:

[0290] 1201. Obtain a sample image belonging to a second image space.

[0291] 1202. Call the degradation removal model to reconstruct the sample image to obtain a first predicted image belonging to the second image space.

[0292] 1203. Based on the sample image and the first predicted image, a degradation removal model is trained to increase the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image.

[0293] The degradation removal model is used to reconstruct another image based on the input image, and the content of the reconstructed image is similar to that of the input image. Therefore, when training the degradation removal model, the training goal is to update the model parameters of the degradation removal model so that if the trained degradation removal model re-obtains the first predicted image based on the sample image, the similarity between the new first predicted image and the sample image increases, that is, the performance of the degradation removal model is improved through training.

[0294] Based on the above-mentioned embodiment, the degradation removal model can also be trained in combination with the adversarial model. Fig.13 , the method comprises the following steps 1-3:

[0295] 1. Call the degradation removal model to obtain the first predicted image, sample encoding features corresponding to multiple sample image blocks, and m first sample decoding features.

[0296] That is, the first encoding sub-model is called to divide the sample image into multiple sample image blocks, and the first image block features corresponding to each sample image block are obtained; m second encoding sub-models are called to encode the first image block features corresponding to the multiple sample image blocks m times to obtain the sample encoding features corresponding to the multiple sample image blocks; m first decoding sub-models are called to decode the sample encoding features corresponding to the multiple sample image blocks m times to obtain m first sample decoding features corresponding to each sample image block; the second decoding sub-model is called to splice the mth first sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain the first sample splicing features; the first sample splicing features are convolved to obtain the first predicted image.

[0297] The embodiment of the present application takes a sample image as an example. Through the above step 1, a first predicted image can be determined based on the sample image. The first predicted image is the first predicted image obtained after improving the image quality of the sample image. The first predicted image should meet the following requirements: the first predicted image is similar to the sample image, and the image quality of the first predicted image is higher than the image quality of the sample image.

[0298] However, considering that the degradation removal model is not accurate enough during the training process, the first predicted image currently obtained is also not accurate enough.

[0299] 2. Call the generative sub-model in the adversarial model to reconstruct the sample encoding features corresponding to the multiple sample image blocks and the m first sample decoding features to obtain a second predicted image belonging to the second image space; call the discriminative sub-model in the adversarial model to discriminate the second predicted image and the sample image to obtain a discriminative result, which indicates whether the second predicted image is similar to the sample image.

[0300] 3. Based on the sample image, the first predicted image and the discrimination result, the degradation removal model and the adversarial model are trained so that the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image increases, and the similarity between the second predicted image obtained by calling the trained generation sub-model and the sample image increases.

[0301] Among them, the adversarial model includes a generative sub-model and a discriminative sub-model. The generative sub-model is used to reconstruct the image based on the input features, while the discriminative sub-model is used to determine whether the reconstructed image is similar to the original image, so as to determine whether the degradation removal model has learned the ability to reconstruct the image.

[0302] Among them, the sample coding features corresponding to the sample image block can represent the content of the sample image block, and the sample decoding features corresponding to the sample image block can also represent the content of the sample image block to a certain extent. Therefore, in the embodiment of the present application, the sample coding features corresponding to the sample image block and the m sample decoding features are comprehensively considered and used together for training. After the sample coding features and the m first sample decoding features corresponding to each sample image block are reconstructed to obtain the second predicted image belonging to the second image space, if the degradation removal model has learned the ability to reconstruct images and the reconstructed image is accurate enough, then the first predicted image should be similar to the sample image, and the discrimination result obtained by the discriminant submodel should indicate that the second predicted image is similar to the sample image. If the degradation removal model is not accurate enough, the first predicted image may not be similar to the sample image, and the discrimination result obtained by the discriminant submodel may indicate that the second predicted image is not similar to the sample image.

[0303] Therefore, based on the sample image and the first predicted image, as well as the discrimination result, the degradation removal model and the adversarial model are trained so that the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image increases, and the similarity between the second predicted image obtained by calling the trained generative sub-model and the sample image increases. In this way, the capabilities of the degradation removal model and the adversarial model can be improved, and the accuracy can be improved.

[0304] In a possible implementation, the generation sub-model includes m first decoding layers and m second decoding layers, and the process of calling the generation sub-model to obtain the second predicted image includes:

[0305] Calling the first first decoding layer to decode the sample coding features corresponding to the multiple sample image blocks to obtain the first second sample decoding feature corresponding to each sample image block; fusing the second sample decoding feature with the first first sample decoding feature to obtain the first fused feature;

[0306] Call the jth first decoding layer to decode the j-1th fusion feature to obtain the jth second sample decoding feature corresponding to each sample image block; until the mth first decoding layer is called to obtain the mth second sample decoding feature corresponding to each sample image block, where j is an integer greater than 1 and not greater than m;

[0307] The second decoding layer is called to splice the mth second sample decoding features corresponding to the multiple sample image blocks according to their positions in the sample image to obtain a second sample splicing feature, and the second sample splicing feature is convolved to obtain a second predicted image.

[0308] It should be noted that the above steps are only an example of jointly training the degradation removal model and the adversarial model. The reconstruction model can be trained first, and after the reconstruction model is trained, the degradation removal model and the adversarial model are jointly trained, and then the reconstruction model and the degradation removal model are called to implement the method provided in the embodiment of the present application. Alternatively, the reconstruction model, the degradation removal model and the adversarial model can also be jointly trained.

[0309] The operation flow of the embodiment of the present application is divided into two stages: the Coarse stage and the Refine stage. The reconstruction model can be called the Coarse stage model, and the degradation removal model and the adversarial model can be called the Refine stage model. Then, the reconstruction model is trained first, and then the degradation removal model and the adversarial model are trained.

[0310] See also Fig.14 , based on low-quality face images for training, first for the reconstruction model of the Coarse stage, based on multiple low-quality face images for iterative training, each iterative training process, obtain the current number of iterative training times, and determine whether the number of iterative training times is greater than the number of iterations threshold of the Coarse stage. If the number of iterative training times is not greater than the number of iterations threshold of the Coarse stage, then calculate the loss function of the Coarse stage, and back-propagate the gradient value through the back-propagation algorithm, so as to update the model parameters of the reconstruction model of the Coarse stage based on the gradient value. If the number of iterative training times is greater than the number of iterations threshold of the Coarse stage, the iterative training of the reconstruction model is completed, and it is considered that the reconstruction model has the preliminary ability to reconstruct images.

[0311] After that, iterative training is continued based on multiple low-quality face images. During each iterative training process, the reconstruction model of the Coarse stage is called to obtain the features corresponding to the face image. At this time, it is determined by judging that the current number of iterative training is greater than the iteration number threshold of the Coarse stage, and the reconstruction model will not be trained again. Instead, the features obtained by the reconstruction model of the Coarse stage will be input into the degradation removal model of the Refine stage, and the degradation removal model and the adversarial model will be used for processing, so as to calculate the loss function of the Refine stage, and the gradient value will be returned through the back propagation algorithm, so as to update the model parameters of the model of the Refine stage based on the gradient value. And obtain the current number of iterative training, and judge whether the number of iterative training is greater than the iteration number threshold of the Refine stage. If the number of iterative training is not greater than the iteration number threshold of the Refine stage, the next iterative training is performed. If the number of iterative training is greater than the iteration number threshold of the Refine stage, the training ends.

[0312] In a possible implementation, the embodiment of the present application also provides a method for improving the clarity of a biometric image. For an image containing a biometric, in order to improve the clarity of the biometric in the process of processing the image, and to ensure the accuracy of the biometric, and to avoid affecting the subsequent biometric recognition accuracy, a first image is first extracted from the original image, and the first image is an image extracted from the area containing the biometric in the original image. If the biometric is a face feature, the first image is a face image corresponding to the face area in the original image, or if the biometric is a fingerprint feature, the first image is a fingerprint image corresponding to the fingerprint area in the original image, etc.

[0313] After the first image is extracted, the scale of the first image is the first scale. Figures 2 to 7 The method provided in the embodiment shown in the figure improves the clarity of the first image to obtain a second image, and the scale of the obtained second image is the second scale. Figure 8 In the illustrated embodiment, the second image is processed to obtain a fourth image, thereby improving the image quality.

[0314] Then, in order to restore the fourth image to the original image, the original image is first scaled according to the ratio between the second scale and the first scale to obtain a background image, and the scale of the area containing the biological features in the background image is the second scale, which is equivalent to reserving space for the fourth image. Then, according to the position of the first image in the original image, the fourth image and the background image can be combined to obtain the fifth image. The background of the original image is retained in the fifth image, and the clarity of the area containing the biological features is improved.

[0315] Therefore, in one example, taking a face image as an example, the operation flow of the embodiment of the present application can be as follows: Fig.15 As shown, firstly, the input original image is obtained, then face detection is performed on the original image to obtain a face image, and the original image is scaled in proportion to obtain a background image. For the face image, the reconstruction model and the degradation removal model in the embodiment of the present application are called to process to obtain a repaired face image, and then the face image is combined with the background image to obtain a complete image after the final repair.

[0316] On multiple data sets, the input image, the image restored by the method provided in the embodiment of the present application, and the image restored by the method provided by other related technologies are compared, where the related technologies include: DeblurGANv2 (a method for blind motion blur removal based on the GAN method), PSFRGAN (a generative adversarial network), GFP-GAN (another generative adversarial network), etc.

[0317] The input image and the image processed by the above methods are as follows Fig.16 As shown, the upper left corner of each image shows an enlarged image of the white box area. Fig.16 It can be seen that for the same input image, the image restored by the method provided in the embodiment of the present application has higher clarity.

[0318] In addition, the comparison results of the above-mentioned methods are shown in Table 1 below. Table 1 is a comparison of the FID and NIQE metrics of the images restored by the embodiments of the present application and the related art. As can be seen from Table 1, compared with the related art, the embodiments of the present application have improved the FID index by 0.6 and 39.51 on the LFW-Test and CelebChild datasets respectively; and the NIQE index by 0.056 and 0.011 on the CelebChild and WebPhoto datasets respectively.

[0319] Table 1

[0320]

[0321] Fig.17 is a schematic diagram of the structure of an image processing device provided in an embodiment of the present application, such as Fig.17 As shown, the device comprises:

[0322] A determination module 1701 is used to determine a spatial transformation feature based on a first scale of the first image space and a second scale of the second image space, where the spatial transformation feature represents a spatial transformation condition when any image belonging to the first image space is transformed into the second image space, the second scale is larger than the first scale, and the resolution of the second image space is higher than the resolution of the first image space;

[0323] An upsampling module 1702 is used to upsample the first image feature corresponding to the first image belonging to the first image space to obtain a second image feature belonging to a second scale;

[0324] A feature transformation module 1703 is used to transform the second image feature based on the spatial transformation feature to obtain a third image feature;

[0325] The image reconstruction module 1704 is used to reconstruct the third image feature to obtain a second image belonging to the second image space.

[0326] In one possible implementation, the spatial transformation feature includes a position offset feature, which represents the position offset of a pixel point when any image belonging to the first image space is transformed to the second image space; the feature transformation module 1703 is used to transform the second image feature based on the position offset feature to obtain a third image feature.

[0327] In another possible implementation, the determining module 1701 includes:

[0328] A factor determination unit, used to determine the ratio between the second scale and the first scale as an upsampling factor, where the upsampling factor represents an upsampling factor used to transform any pixel point from the first image space to the second image space;

[0329] A pixel point transformation unit, configured to transform the plurality of first pixel points into the first image space based on the coordinates of the plurality of first pixel points in the second image space and an upsampling factor, so as to obtain the coordinates of the plurality of second pixel points;

[0330] a relative distance determination unit, configured to determine a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points, the relative distances representing the distances between the positions of the first pixel points and the positions of the corresponding second pixel points in the second image space;

[0331] The offset feature determining unit is used to determine the position offset feature based on the coordinates of the plurality of second pixel points and the plurality of relative distances.

[0332] In another possible implementation manner, the offset feature determination unit is configured to:

[0333] The coordinates of the plurality of second pixel points and the plurality of relative distances constitute a feature matrix, wherein the feature matrix includes a plurality of first sub-features, each of which includes the coordinates of a second pixel point and the corresponding relative distance;

[0334] The feature matrix is ​​transformed to obtain a position shift feature, where the position shift feature includes a plurality of second sub-features obtained by respectively transforming a plurality of first sub-features, and a feature dimension of each second sub-feature is smaller than a feature dimension of a corresponding first sub-feature.

[0335] In another possible implementation, the spatial transformation feature further includes a scale transformation weight, where the scale transformation weight represents a scale transformation condition when any image belonging to the first image space is transformed into the second image space;

[0336] The feature transformation module 1703 is used to:

[0337] Based on the position offset feature, the position of the feature in the second image feature is adjusted to obtain a fourth image feature;

[0338] Based on the scale transformation weight, compression convolution and expansion convolution are performed on the fourth image feature to obtain the third image feature.

[0339] In another possible implementation, the feature transformation module 1703 is used to:

[0340] Based on the scale transformation weights, the compressed convolution parameters and the expanded convolution parameters are adjusted respectively;

[0341] Based on the adjusted compressed convolution parameter and the adjusted expanded convolution parameter, performing compressed convolution and expanded convolution on the fourth image feature to obtain a fifth image feature;

[0342] The fifth image feature is fused with the second image feature to obtain a third image feature.

[0343] In another possible implementation, the determining module 1701 includes:

[0344] A factor determination unit, used to determine the ratio between the second scale and the first scale as an upsampling factor, where the upsampling factor represents an upsampling factor used to transform any pixel point from the first image space to the second image space;

[0345] A pixel point transformation unit, configured to transform the plurality of first pixel points into the first image space based on the coordinates of the plurality of first pixel points in the second image space and an upsampling factor, so as to obtain the coordinates of the plurality of second pixel points;

[0346] a relative distance determination unit, configured to determine a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points, the relative distances representing the distances between the positions of the first pixel points and the positions of the corresponding second pixel points in the second image space;

[0347] A feature vector determination unit, configured to form a feature matrix from the coordinates of the plurality of second pixel points and the plurality of relative distances, wherein the feature matrix includes a plurality of first sub-features, each of which includes the coordinates of a second pixel point and the corresponding relative distance; transform the feature matrix to obtain a feature vector;

[0348] A first transformation unit, used for performing a first transformation on the feature vector to obtain a position shift feature;

[0349] The second transformation unit is used to perform a second transformation on the feature vector, normalize the transformed feature vector, and obtain a scale transformation weight, where the scale transformation weight includes weight features of multiple second pixel points.

[0350] In another possible implementation, the method is implemented based on a reconstruction model, and the reconstruction model includes a spatial transformation sub-model and a reconstruction sub-model;

[0351] A determination module 1701 is used to call the spatial transformation sub-model to determine the spatial transformation feature based on the first scale and the second scale;

[0352] An upsampling module 1702 is used to call a reconstruction sub-model to upsample the first image feature to obtain a second image feature;

[0353] A feature transformation module 1703 is used to transform the second image feature based on the spatial transformation feature to obtain a third image feature;

[0354] The image reconstruction module 1704 is used to reconstruct the third image feature to obtain a second image.

[0355] In another possible implementation, the first image is an image extracted from a region of the original image containing biological features; the device further includes:

[0356] A first scaling module is used to scale the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of the region containing the biological feature in the background image is the second scale;

[0357] The first combining module is used to combine the second image with the background image according to the position of the first image in the original image to obtain a third image.

[0358] In another possible implementation, the device further includes:

[0359] A feature acquisition module, used to divide the second image into a plurality of image blocks, and acquire a first image block feature corresponding to each image block;

[0360] An image block processing module, used for encoding and decoding the first image block features corresponding to each image block m times based on a self-attention mechanism, to obtain the second image block features corresponding to each image block, where m is an integer greater than 1;

[0361] The splicing convolution module is used to splice the second image block features corresponding to the multiple image blocks according to their positions in the second image to obtain splicing features, and convolve the splicing features to obtain a fourth image belonging to the second image space.

[0362] In another possible implementation, the feature acquisition module includes:

[0363] A feature extraction unit, used to extract features from each image block respectively to obtain a third image block feature corresponding to each image block;

[0364] The position fusion unit is used to fuse the third image block feature corresponding to each image block with the position feature to obtain the first image block feature corresponding to each image block, and the position feature indicates the position of the image block in the second image.

[0365] In another possible implementation, the process of acquiring the fourth image based on the second image is implemented based on a degradation removal model, where the degradation removal model includes a first encoding sub-model, m second encoding sub-models based on a self-attention mechanism, m first decoding sub-models based on a self-attention mechanism, and a second decoding sub-model, where m is an integer greater than 1;

[0366] A feature acquisition module, used for calling the first encoding sub-model, dividing the second image into a plurality of image blocks, and acquiring a first image block feature corresponding to each image block;

[0367] Furthermore, the m second encoding sub-models are connected in sequence, the m first decoding sub-models are connected in sequence, and the mth second encoding sub-model is connected to the first first decoding sub-model, and the image block processing module is used to:

[0368] Calling m second encoding sub-models to encode first image block features corresponding to the multiple image blocks m times, to obtain encoding features corresponding to the multiple image blocks;

[0369] Calling m first decoding sub-models to decode the encoding features corresponding to the multiple image blocks m times to obtain decoding features corresponding to the multiple image blocks;

[0370] The splicing convolution module is used to call the second decoding sub-model, splice the decoding features corresponding to the multiple image blocks according to their positions in the second image, obtain splicing features, and convolve the splicing features to obtain a fourth image.

[0371] In another possible implementation, the image block processing module is used to:

[0372] For any image block, calling the first second encoding sub-model, based on the similarity between the first image block feature corresponding to the image block and the first image block features corresponding to the multiple image blocks, performing weighted fusion on the first image block features corresponding to the multiple image blocks to obtain the first fused image block feature corresponding to the image block;

[0373] The first fused image block feature is fused with the first image block feature corresponding to the image block to obtain the first encoding feature corresponding to the image block.

[0374] In another possible implementation, the image block processing module is used to:

[0375] For any image block, call the i-th second coding sub-model, and based on the similarity between the i-1th coding feature corresponding to the image block and the i-1th coding features corresponding to multiple image blocks, perform weighted fusion on the i-1th coding features corresponding to the multiple image blocks to obtain the i-th fused image block feature corresponding to the image block;

[0376] The i-th fused image block feature is fused with the i-1 coding features corresponding to the image block to obtain the i-th coding feature corresponding to the image block, where i is an integer greater than 1 and not greater than m.

[0377] In another possible implementation, the device further includes:

[0378] A sample acquisition module, used for acquiring a sample image belonging to the second image space;

[0379] A prediction module, used for calling the degradation removal model to reconstruct the sample image to obtain a first predicted image belonging to the second image space;

[0380] The training module is used to train the degradation removal model based on the sample image and the first predicted image, so as to increase the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image.

[0381] In another possible implementation, the prediction module is used to:

[0382] Calling the first encoding sub-model, dividing the sample image into a plurality of sample image blocks, and obtaining a first image block feature corresponding to each sample image block;

[0383] Calling m second encoding sub-models to encode first image block features corresponding to the plurality of sample image blocks m times, to obtain sample encoding features corresponding to the plurality of sample image blocks;

[0384] Calling m first decoding sub-models, decoding the sample coding features corresponding to the multiple sample image blocks m times, and obtaining m first sample decoding features corresponding to each sample image block;

[0385] Calling the second decoding sub-model, splicing the mth first sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a first sample splicing feature, and convolving the first sample splicing feature to obtain a first predicted image;

[0386] Training modules for:

[0387] The generation sub-model in the adversarial model is called to reconstruct the sample encoding features corresponding to the plurality of sample image blocks and the m first sample decoding features to obtain a second predicted image belonging to the second image space; the discrimination sub-model in the adversarial model is called to discriminate the second predicted image and the sample image to obtain a discrimination result, and the discrimination result indicates whether the second predicted image is similar to the sample image;

[0388] Based on the sample image, the first predicted image and the discrimination result, the degradation removal model and the adversarial model are trained so that the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image increases, and the similarity between the second predicted image obtained by calling the trained generation sub-model and the sample image increases.

[0389] In another possible implementation, the generation sub-model includes m first decoding layers and m second decoding layers;

[0390] Training modules for:

[0391] Calling the first first decoding layer to decode the sample coding features corresponding to the multiple sample image blocks to obtain the first second sample decoding feature corresponding to each sample image block; fusing the second sample decoding feature with the first first sample decoding feature to obtain the first fused feature;

[0392] Call the jth first decoding layer to decode the j-1th fusion feature to obtain the jth second sample decoding feature corresponding to each sample image block; until the mth first decoding layer is called to obtain the mth second sample decoding feature corresponding to each sample image block, where j is an integer greater than 1 and not greater than m;

[0393] Calling the second decoding layer, splicing the mth second sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a second sample splicing feature, and convolving the second sample splicing feature to obtain a second predicted image;

[0394] The discriminant sub-model is called to discriminate the second predicted image and the sample image to obtain a discriminant result.

[0395] In another possible implementation, the first image is an image extracted from a region of the original image containing biological features; the device further includes:

[0396] A second scaling module is used to scale the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of the area containing the biological feature in the background image is the second scale;

[0397] The second combining module is used to combine the fourth image with the background image according to the position of the first image in the original image to obtain the fifth image.

[0398] It should be noted that: when the image processing device provided in the above embodiment processes an image, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the computer device is divided into different functional modules to complete all or part of the functions described above. In addition, the image processing device provided in the above embodiment and the image processing method embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0399] An embodiment of the present application further provides a computer device, which includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the image processing method of the above embodiment.

[0400] Optionally, the computer device is provided as a terminal. Fig.18 A structural block diagram of a terminal 1800 provided by an exemplary embodiment of the present application is shown.

[0401] The terminal 1800 includes: a processor 1801 and a memory 1802 .

[0402] The processor 1801 may include one or more processing cores, such as a 4-core processor, an 8-core processor, etc. In some embodiments, the processor 1801 may be integrated with a GPU (Graphics Processing Unit), which is responsible for rendering and drawing the content to be displayed on the display screen. In some embodiments, the processor 1801 may also include an AI (Artificial Intelligence) processor, which is used to process computing operations related to machine learning.

[0403] The memory 1802 may include one or more computer-readable storage media, which may be non-transitory. In some embodiments, the non-transitory computer-readable storage medium in the memory 1802 is used to store at least one computer program, which is used to be executed by the processor 1801 to implement the image processing method provided in the method embodiment of the present application.

[0404] In some embodiments, the terminal 1800 may also optionally include: a peripheral device interface 1803 and at least one peripheral device. The processor 1801, the memory 1802 and the peripheral device interface 1803 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1803 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1804, a display screen 1805, a camera assembly 1806, an audio circuit 1807 and a power supply 1808.

[0405] The peripheral device interface 1803 may be used to connect at least one peripheral device related to I / O (Input / Output) to the processor 1801 and the memory 1802 .

[0406] The radio frequency circuit 1804 is used to receive and transmit RF (Radio Frequency) signals, also known as electromagnetic signals. The radio frequency circuit 1804 communicates with a communication network and other communication devices through electromagnetic signals.

[0407] Display screen 1805 is used to display UI (User Interface). The UI may include graphics, text, icons, videos, and any combination thereof. When display screen 1805 is a touch display screen, display screen 1805 also has the ability to collect touch signals on the surface or above the surface of display screen 1805. The touch signal can be input to processor 1801 as a control signal for processing. At this time, display screen 1805 can also be used to provide virtual buttons and / or virtual keyboards, also known as soft buttons and / or soft keyboards.

[0408] The camera assembly 1806 is used to collect images or videos. Optionally, the camera assembly 1806 includes a front camera and a rear camera. The front camera is set on the front panel of the terminal, and the rear camera is set on the back of the terminal.

[0409] The audio circuit 1807 may include a microphone and a speaker. The microphone is used to collect sound waves from the user and the environment, and convert the sound waves into electrical signals and input them to the processor 1801 for processing, or input them to the RF circuit 1804 to achieve voice communication.

[0410] The power supply 1808 is used to supply power to various components in the terminal 1800 .

[0411] Those skilled in the art will understand that Fig.18 The structure shown in the figure does not constitute a limitation on the terminal 1800, and the terminal 1800 may include more or less components than those shown in the figure, or combine some components, or adopt a different component arrangement.

[0412] Optionally, the computer device is provided as a server. Fig.19 This is a schematic diagram of the structure of a server provided in an embodiment of the present application. The server 1900 may have relatively large differences due to different configurations or performances, and may include one or more processors (Central Processing Units, CPU) 1901 and one or more memories 1902, wherein at least one computer program is stored in the memory 1902, and at least one computer program is loaded and executed by the processor 1901 to implement the methods provided in the above-mentioned various method embodiments. Of course, the server may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output, and the server may also include other components for implementing device functions, which will not be described in detail here.

[0413] The server 1900 may be used to execute the steps executed by the computer device in the above-mentioned image processing method.

[0414] An embodiment of the present application further provides a computer-readable storage medium, in which at least one computer program is stored. The at least one computer program is loaded and executed by a processor to implement the operations performed by the image processing method of the above embodiment.

[0415] An embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the operations performed by the image processing method described in the above aspects.

[0416] A person skilled in the art will understand that all or part of the steps to implement the above embodiments may be accomplished by hardware or by instructing related hardware through a program, and the program may be stored in a computer-readable storage medium, and the above-mentioned storage medium may be a read-only memory, a disk or an optical disk, etc.

[0417] The above description is only an optional embodiment of the embodiments of the present application and is not intended to limit the embodiments of the present application. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the embodiments of the present application should be included in the protection scope of the present application.

Claims

1. An image processing method, characterized in that: The method comprises: Determine, based on a first scale of the first image space and a second scale of the second image space, a spatial transformation feature, wherein the spatial transformation feature represents a spatial transformation situation when any image belonging to the first image space is transformed into the second image space, the second scale is greater than the first scale, and a resolution of the second image space is higher than a resolution of the first image space, the spatial transformation feature includes a position offset feature and a scale transformation weight, the position offset feature represents a position offset situation of a pixel point when any image belonging to the first image space is transformed into the second image space, and the scale transformation weight represents a scale transformation situation when any image belonging to the first image space is transformed into the second image space; Upsampling a first image feature corresponding to a first image belonging to the first image space to obtain a second image feature belonging to the second scale; Based on the position offset feature, adjusting the position of the feature in the second image feature to obtain a fourth image feature; Based on the scale transformation weight, respectively adjusting the compression convolution parameters and the expansion convolution parameters; Based on the adjusted compression convolution parameter and the adjusted expansion convolution parameter, performing compression convolution and expansion convolution on the fourth image feature to obtain a fifth image feature; fusing the fifth image feature with the second image feature to obtain a third image feature; The third image feature is reconstructed to obtain a second image belonging to the second image space.

2. The method according to claim 1, characterized in that: The determining of the spatial transformation feature based on the first scale of the first image space and the second scale of the second image space includes: Determine a ratio between the second scale and the first scale as an upsampling factor, where the upsampling factor represents an upsampling factor used to transform any pixel point from the first image space to the second image space; Based on the coordinates of the plurality of first pixel points in the second image space and the upsampling factor, transform the plurality of first pixel points into the first image space to obtain the coordinates of the plurality of second pixel points; Determine a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points, the relative distances representing distances between positions of the first pixel points and positions of the corresponding second pixel points in the second image space; The position offset feature is determined based on the coordinates of the plurality of second pixel points and the plurality of relative distances.

3. The method according to claim 2, characterized in that The determining the position offset feature based on the coordinates of the plurality of second pixel points and the plurality of relative distances comprises: The coordinates of the plurality of second pixel points and the plurality of relative distances constitute a feature matrix, wherein the feature matrix includes a plurality of first sub-features, each of which includes a coordinate of the second pixel point and a corresponding relative distance; The feature matrix is ​​transformed to obtain the position shift feature, where the position shift feature includes a plurality of second sub-features obtained by respectively transforming the plurality of first sub-features.

4. The method according to claim 1, characterized in that: The determining of the spatial transformation feature based on the first scale of the first image space and the second scale of the second image space includes: Determine a ratio between the second scale and the first scale as an upsampling factor, where the upsampling factor represents an upsampling factor used to transform any pixel point from the first image space to the second image space; Based on the coordinates of the plurality of first pixel points in the second image space and the upsampling factor, transform the plurality of first pixel points into the first image space to obtain the coordinates of the plurality of second pixel points; Determine a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points, the relative distances representing distances between positions of the first pixel points and positions of the corresponding second pixel points in the second image space; The coordinates of the plurality of second pixel points and the plurality of relative distances constitute a feature matrix, wherein the feature matrix includes a plurality of first sub-features, each of which includes a coordinate of the second pixel point and a corresponding relative distance; Transforming the characteristic matrix to obtain a characteristic vector; Performing a first transformation on the feature vector to obtain the position offset feature; A second transformation is performed on the feature vector, and the transformed feature vector is normalized to obtain the scale transformation weight, where the scale transformation weight includes weight features of the plurality of second pixel points.

5. The method according to claim 1, characterized in that The method is implemented based on a reconstruction model, wherein the reconstruction model includes a space transformation sub-model and a reconstruction sub-model; The determining of the spatial transformation feature based on the first scale of the first image space and the second scale of the second image space includes: Calling the spatial transformation sub-model to determine the spatial transformation feature based on the first scale and the second scale; The method further comprises: The reconstruction submodel is called to upsample the first image feature to obtain the second image feature; based on the spatial transformation feature, the second image feature is transformed to obtain the third image feature; and the third image feature is reconstructed to obtain the second image.

6. The method according to any one of claims 1 to 5, characterized in that: The first image is an image extracted from a region of the original image containing biological features; after reconstructing the third image features to obtain a second image belonging to the second image space, the method further includes: scaling the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of a region of the background image containing the biological feature is the second scale; According to the position of the first image in the original image, the second image is combined with the background image to obtain a third image.

7. The method according to any one of claims 1 to 5, characterized in that: After reconstructing the third image feature to obtain the second image belonging to the second image space, the method further includes: Divide the second image into a plurality of image blocks, and obtain a first image block feature corresponding to each image block; Based on the self-attention mechanism, respectively encode and decode the first image block features corresponding to each image block m times to obtain the second image block features corresponding to each image block, where m is an integer greater than 1; According to the positions of the multiple image blocks in the second image, second image block features corresponding to the multiple image blocks are spliced ​​to obtain spliced ​​features, and the spliced ​​features are convolved to obtain a fourth image belonging to the second image space.

8. The method according to claim 7, characterized in that The obtaining of the first image block feature corresponding to each image block includes: Extracting features from each of the image blocks respectively to obtain third image block features corresponding to each of the image blocks; The third image block feature corresponding to each image block is fused with the position feature respectively to obtain the first image block feature corresponding to each image block, wherein the position feature indicates the position of the image block in the second image.

9. The method according to claim 7, characterized in that: The process of acquiring the fourth image based on the second image is implemented based on a degradation removal model, wherein the degradation removal model includes a first encoding sub-model, m second encoding sub-models based on a self-attention mechanism, m first decoding sub-models based on a self-attention mechanism, and a second decoding sub-model, where m is an integer greater than 1; The step of dividing the second image into a plurality of image blocks and obtaining a first image block feature corresponding to each image block includes: Calling the first encoding sub-model, dividing the second image into the plurality of image blocks, and obtaining a first image block feature corresponding to each image block; Furthermore, m second encoding sub-models are connected in sequence, m first decoding sub-models are connected in sequence, and the mth second encoding sub-model is connected to the first first decoding sub-model, and the first image block features corresponding to each image block are encoded and decoded m times respectively to obtain the second image block features corresponding to each image block, including: Calling m second encoding sub-models to encode first image block features corresponding to the multiple image blocks m times to obtain encoding features corresponding to the multiple image blocks; Calling m first decoding sub-models to decode the encoding features corresponding to the multiple image blocks m times to obtain decoding features corresponding to the multiple image blocks; The method of splicing second image block features corresponding to the multiple image blocks according to positions of the multiple image blocks in the second image to obtain splicing features, and convolving the splicing features to obtain a fourth image belonging to the second image space includes: The second decoding sub-model is called to splice the decoding features corresponding to the multiple image blocks according to the positions of the multiple image blocks in the second image to obtain the spliced ​​features, and convolute the spliced ​​features to obtain the fourth image.

10. The method according to claim 9, characterized in that The first encoding process in the m encodings includes: For any image block, calling the first second encoding sub-model, based on the similarity between the first image block feature corresponding to the image block and the first image block features corresponding to the multiple image blocks, performing weighted fusion on the first image block features corresponding to the multiple image blocks to obtain a first fused image block feature corresponding to the image block; The first fused image block feature is fused with the first image block feature corresponding to the image block to obtain a first encoding feature corresponding to the image block.

11. The method according to claim 10, characterized in that The i-th encoding process in the m encodings includes: For any image block, calling the i-th second coding sub-model, based on the similarity between the i-1th coding feature corresponding to the image block and the i-1th coding features corresponding to the multiple image blocks, performing weighted fusion on the i-1th coding features corresponding to the multiple image blocks to obtain the i-th fused image block feature corresponding to the image block; The i-th fused image block feature is fused with i-1 coding features corresponding to the image block to obtain the i-th coding feature corresponding to the image block, where i is an integer greater than 1 and not greater than m.

12. The method according to claim 9, characterized in that The process of training the degradation removal model includes: Acquire a sample image belonging to the second image space; Calling the degradation removal model to reconstruct the sample image to obtain a first predicted image belonging to the second image space; The degradation removal model is trained based on the sample image and the first predicted image, so that the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image is increased.

13. The method according to claim 12, characterized in that The calling the degradation removal model to reconstruct the sample image to obtain a first predicted image belonging to the second image space includes: Calling the first encoding sub-model, dividing the sample image into a plurality of sample image blocks, and obtaining a first image block feature corresponding to each sample image block; Calling m second encoding sub-models to encode the first image block features corresponding to the multiple sample image blocks m times to obtain sample encoding features corresponding to the multiple sample image blocks; Calling m first decoding sub-models to decode the sample coding features corresponding to the multiple sample image blocks m times, to obtain m first sample decoding features corresponding to each sample image block; calling the second decoding sub-model, splicing the mth first sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a first sample splicing feature, and convolving the first sample splicing feature to obtain the first predicted image; The step of training the degradation removal model based on the sample image and the first predicted image so as to increase the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image includes: calling a generation sub-model in the adversarial model to reconstruct the sample encoding features corresponding to the plurality of sample image blocks and the m first sample decoding features to obtain a second predicted image belonging to the second image space; calling a discrimination sub-model in the adversarial model to discriminate the second predicted image and the sample image to obtain a discrimination result, wherein the discrimination result indicates whether the second predicted image is similar to the sample image; Based on the sample image, the first predicted image and the discrimination result, the degradation removal model and the adversarial model are trained so that the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image increases, and the similarity between the second predicted image obtained by calling the trained generation sub-model and the sample image increases.

14. The method according to claim 13, characterized in that The generation sub-model includes m first decoding layers and m second decoding layers; The generating sub-model in the adversarial model is called to reconstruct the sample encoding features corresponding to the multiple sample image blocks and the m first sample decoding features to obtain a second predicted image belonging to the second image space; Calling the discriminant sub-model in the adversarial model to discriminate the second predicted image and the sample image to obtain a discrimination result, including: Calling the first first decoding layer to decode the sample coding features corresponding to the multiple sample image blocks to obtain the first second sample decoding feature corresponding to each sample image block; fusing the second sample decoding feature with the first first sample decoding feature to obtain the first fused feature; Calling the jth first decoding layer to decode the j-1th fusion feature to obtain the jth second sample decoding feature corresponding to each sample image block; until calling the mth first decoding layer to obtain the mth second sample decoding feature corresponding to each sample image block, j is an integer greater than 1 and not greater than m; calling the second decoding layer, splicing the mth second sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a second sample splicing feature, and convolving the second sample splicing feature to obtain the second predicted image; The discriminant sub-model is called to discriminate the second predicted image and the sample image to obtain the discrimination result.

15. The method according to claim 7, characterized in that The first image is an image extracted from a region of the original image containing biological features; after the second image block features corresponding to the multiple image blocks are spliced ​​according to the positions of the multiple image blocks in the second image to obtain spliced ​​features, and the spliced ​​features are convolved to obtain a fourth image belonging to the second image space, the method further includes: scaling the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of a region of the background image containing the biological feature is the second scale; The fourth image is combined with the background image according to the position of the first image in the original image to obtain a fifth image.

16. An image processing device, characterized in that: The device comprises: a determination module, configured to determine a spatial transformation feature based on a first scale of a first image space and a second scale of a second image space, wherein the spatial transformation feature indicates a spatial transformation condition when any image belonging to the first image space is transformed to the second image space, the second scale is larger than the first scale, and a resolution of the second image space is higher than a resolution of the first image space, the spatial transformation feature includes a position offset feature and a scale transformation weight, the position offset feature indicates a position offset condition of a pixel point when any image belonging to the first image space is transformed to the second image space, and the scale transformation weight indicates a scale transformation condition when any image belonging to the first image space is transformed to the second image space; an upsampling module, configured to upsample a first image feature corresponding to a first image belonging to the first image space to obtain a second image feature belonging to the second scale; a feature transformation module, configured to adjust the position of the feature in the second image feature based on the position offset feature to obtain a fourth image feature; adjust the compression convolution parameter and the expansion convolution parameter based on the scale transformation weight; perform compression convolution and expansion convolution on the fourth image feature based on the adjusted compression convolution parameter and the adjusted expansion convolution parameter to obtain a fifth image feature; and fuse the fifth image feature with the second image feature to obtain a third image feature; An image reconstruction module is used to reconstruct the third image feature to obtain a second image belonging to the second image space.

17. The device according to claim 16, characterized in that The determining module comprises: a factor determination unit, configured to determine a ratio between the second scale and the first scale as an upsampling factor, wherein the upsampling factor represents an upsampling factor used to transform any pixel point from the first image space to the second image space; a pixel point transformation unit, configured to transform the plurality of first pixel points into the first image space based on the coordinates of the plurality of first pixel points in the second image space and the upsampling factor, so as to obtain coordinates of a plurality of second pixel points; a relative distance determining unit, configured to determine a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points, the relative distances representing distances between positions of first pixel points and positions of corresponding second pixel points in the second image space; The offset feature determining unit is used to determine the position offset feature based on the coordinates of the plurality of second pixel points and the plurality of relative distances.

18. The device according to claim 17, characterized in that The offset feature determination unit is used to: The coordinates of the plurality of second pixel points and the plurality of relative distances constitute a feature matrix, wherein the feature matrix includes a plurality of first sub-features, each of which includes a coordinate of the second pixel point and a corresponding relative distance; The feature matrix is ​​transformed to obtain the position shift feature, where the position shift feature includes a plurality of second sub-features obtained by respectively transforming the plurality of first sub-features.

19. The device according to claim 16, characterized in that The determining module comprises: a factor determination unit, configured to determine a ratio between the second scale and the first scale as an upsampling factor, wherein the upsampling factor represents an upsampling factor used to transform any pixel point from the first image space to the second image space; a pixel point transformation unit, configured to transform the plurality of first pixel points into the first image space based on the coordinates of the plurality of first pixel points in the second image space and the upsampling factor, so as to obtain coordinates of a plurality of second pixel points; a relative distance determining unit, configured to determine a plurality of relative distances based on the coordinates of the plurality of first pixel points, the upsampling factor, and the coordinates of the plurality of second pixel points, the relative distances representing distances between positions of first pixel points and positions of corresponding second pixel points in the second image space; a feature vector determination unit, configured to form a feature matrix from the coordinates of the plurality of second pixel points and the plurality of relative distances, wherein the feature matrix includes a plurality of first sub-features, each of which includes a coordinate of a second pixel point and a corresponding relative distance; and transform the feature matrix to obtain a feature vector; A first transformation unit, used for performing a first transformation on the feature vector to obtain the position offset feature; The second transformation unit is used to perform a second transformation on the feature vector, normalize the transformed feature vector, and obtain the scale transformation weight, where the scale transformation weight includes weight features of the multiple second pixel points.

20. The device according to claim 16, characterized in that The device is implemented based on a reconstruction model, and the reconstruction model includes a space transformation sub-model and a reconstruction sub-model; The determining module is used to call the spatial transformation sub-model to determine the spatial transformation feature based on the first scale and the second scale; The upsampling module is used to call the reconstruction sub-model to upsample the first image feature to obtain the second image feature; The feature transformation module is used to transform the second image feature based on the spatial transformation feature to obtain the third image feature; The image reconstruction module is used to reconstruct the third image feature to obtain the second image.

21. The device according to any one of claims 16 to 20, characterized in that: The first image is an image extracted from a region of the original image containing biological features; the device further comprises: A first scaling module, configured to scale the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of a region of the background image containing the biological feature is the second scale; The first combining module is used to combine the second image with the background image according to the position of the first image in the original image to obtain a third image.

22. The device according to any one of claims 16 to 20, characterized in that The device also includes: A feature acquisition module, used to divide the second image into a plurality of image blocks, and acquire a first image block feature corresponding to each image block; An image block processing module, configured to encode and decode the first image block features corresponding to each image block m times based on a self-attention mechanism, to obtain the second image block features corresponding to each image block, where m is an integer greater than 1; A splicing convolution module is used to splice the second image block features corresponding to the multiple image blocks according to the positions of the multiple image blocks in the second image to obtain splicing features, and convolve the splicing features to obtain a fourth image belonging to the second image space.

23. The device according to claim 22, characterized in that The feature acquisition module comprises: A feature extraction unit, used to extract features from each of the image blocks respectively to obtain a third image block feature corresponding to each of the image blocks; A position fusion unit is used to fuse the third image block feature corresponding to each image block with the position feature to obtain the first image block feature corresponding to each image block, wherein the position feature indicates the position of the image block in the second image.

24. The device according to claim 22, characterized in that The process of acquiring the fourth image based on the second image is implemented based on a degradation removal model, wherein the degradation removal model includes a first encoding sub-model, m second encoding sub-models based on a self-attention mechanism, m first decoding sub-models based on a self-attention mechanism, and a second decoding sub-model, where m is an integer greater than 1; The feature acquisition module is used to call the first encoding sub-model, divide the second image into the plurality of image blocks, and acquire the first image block feature corresponding to each image block; Furthermore, the m second encoding sub-models are connected in sequence, the m first decoding sub-models are connected in sequence, and the mth second encoding sub-model is connected to the first first decoding sub-model, and the image block processing module is used to: Calling m second encoding sub-models to encode first image block features corresponding to the multiple image blocks m times to obtain encoding features corresponding to the multiple image blocks; Calling m first decoding sub-models to decode the encoding features corresponding to the multiple image blocks m times to obtain decoding features corresponding to the multiple image blocks; The splicing convolution module is used to call the second decoding sub-model, splice the decoding features corresponding to the multiple image blocks according to the positions of the multiple image blocks in the second image, obtain the splicing features, and convolve the splicing features to obtain the fourth image.

25. The device according to claim 24, characterized in that The image block processing module is used to: For any image block, calling the first second encoding sub-model, based on the similarity between the first image block feature corresponding to the image block and the first image block features corresponding to the multiple image blocks, performing weighted fusion on the first image block features corresponding to the multiple image blocks to obtain a first fused image block feature corresponding to the image block; The first fused image block feature is fused with the first image block feature corresponding to the image block to obtain a first encoding feature corresponding to the image block.

26. The device according to claim 25, characterized in that The image block processing module is used to: For any image block, calling the i-th second coding sub-model, based on the similarity between the i-1th coding feature corresponding to the image block and the i-1th coding features corresponding to the multiple image blocks, performing weighted fusion on the i-1th coding features corresponding to the multiple image blocks to obtain the i-th fused image block feature corresponding to the image block; The i-th fused image block feature is fused with i-1 coding features corresponding to the image block to obtain the i-th coding feature corresponding to the image block, where i is an integer greater than 1 and not greater than m.

27. The device according to claim 24, characterized in that The device also includes: A sample acquisition module, used for acquiring a sample image belonging to the second image space; A prediction module, used for calling the degradation removal model to reconstruct the sample image to obtain a first predicted image belonging to the second image space; A training module is used to train the degradation removal model based on the sample image and the first predicted image, so as to increase the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image.

28. The device according to claim 27, characterized in that The prediction module is used to: Calling the first encoding sub-model, dividing the sample image into a plurality of sample image blocks, and obtaining a first image block feature corresponding to each sample image block; Calling m second encoding sub-models to encode the first image block features corresponding to the multiple sample image blocks m times to obtain sample encoding features corresponding to the multiple sample image blocks; Calling m first decoding sub-models to decode the sample coding features corresponding to the multiple sample image blocks m times, to obtain m first sample decoding features corresponding to each sample image block; calling the second decoding sub-model, splicing the mth first sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a first sample splicing feature, and convolving the first sample splicing feature to obtain the first predicted image; The training module is used to: Calling a generation sub-model in the adversarial model to reconstruct the sample encoding features corresponding to the multiple sample image blocks and the m first sample decoding features to obtain a second predicted image belonging to the second image space; calling a discriminant sub-model in the adversarial model to discriminate the second predicted image and the sample image to obtain a discriminant result, wherein the discriminant result indicates whether the second predicted image is similar to the sample image; Based on the sample image, the first predicted image and the discrimination result, the degradation removal model and the adversarial model are trained so that the similarity between the first predicted image obtained by calling the trained degradation removal model and the sample image increases, and the similarity between the second predicted image obtained by calling the trained generation sub-model and the sample image increases.

29. The device according to claim 28, characterized in that The generation sub-model includes m first decoding layers and m second decoding layers; The training module is used to: Calling the first first decoding layer to decode the sample coding features corresponding to the multiple sample image blocks to obtain the first second sample decoding feature corresponding to each sample image block; Fusing the second sample decoding feature with the first first sample decoding feature to obtain a first fused feature; Calling the j-th first decoding layer to decode the j-1-th fusion feature to obtain the j-th second sample decoding feature corresponding to each sample image block; Until the mth first decoding layer is called to obtain the mth second sample decoding feature corresponding to each sample image block, j is an integer greater than 1 and not greater than m; calling the second decoding layer, splicing the mth second sample decoding features corresponding to the multiple sample image blocks according to the positions of the multiple sample image blocks in the sample image to obtain a second sample splicing feature, and convolving the second sample splicing feature to obtain the second predicted image; The discriminant sub-model is called to discriminate the second predicted image and the sample image to obtain the discrimination result.

30. The device according to claim 22, characterized in that The first image is an image extracted from a region of the original image containing biological features; the device further comprises: A second scaling module, configured to scale the original image according to a ratio between the second scale and the first scale to obtain a background image, wherein the scale of a region of the background image containing the biological feature is the second scale; The second combining module is used to combine the fourth image with the background image according to the position of the first image in the original image to obtain a fifth image.

31. A computer device, characterized in that: The computer device includes a processor and a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor to implement the operations performed by the image processing method according to any one of claims 1 to 15.

32. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor to implement the operations performed by the image processing method according to any one of claims 1 to 15.

33. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the operations performed by the image processing method according to any one of claims 1 to 15 are implemented.

Images