Image reconstruction model training method, image reconstruction method, device and equipment

Abstract

The present application relates to an image reconstruction model training method, an image reconstruction method, a device and equipment, and relates to the field of image processing technology. The method comprises: obtaining a training image set, which includes a sample image to be reconstructed, a reconstructed sample set and a standard image corresponding to the sample image; performing multi-level feature extraction on each image in the reconstructed sample set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed sample set; generating a feature matrix of each image based on the feature information of multiple levels corresponding to each image through the image reconstruction model; performing image reconstruction on the sample image to be reconstructed based on the feature matrix of each image through the image reconstruction model to obtain a reconstruction result of the sample image to be reconstructed; and training the image reconstruction model based on the reconstruction result and the standard image. Through the above method, the image reconstruction effect of the image reconstruction model obtained through training can be improved.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of image processing technology, and in particular to an image reconstruction model training method, an image reconstruction method, a device and equipment. Background Art

[0002] Image super-resolution reconstruction, as a research issue with important application value, has attracted widespread attention in the image field of academia and industry. In particular, based on the rapid development of deep learning technology in recent years, many model algorithms using convolutional neural networks have emerged, striving to quickly and effectively retain image information and reconstruct details through image post-processing.

[0003] However, the accuracy of images reconstructed using existing convolutional neural network model algorithms still needs to be improved, the degree of retention of local information is poor, and there is a problem of obvious distortion of image details. Summary of the invention

[0004] The embodiment of the present application provides an image reconstruction model training method, an image reconstruction method, an apparatus and a device, which can obtain feature information of different receptive fields from each image when extracting features from each image included in a reconstruction sample set in a training image set, thereby enhancing the effect of feature representation of each image, and thus improving the image reconstruction effect of the image reconstruction model obtained by training. The technical solution is as follows:

[0005] In one aspect, a method for training an image reconstruction model is provided, the method comprising:

[0006] Acquire a training image set, wherein the training image set includes a sample image to be reconstructed, a reconstructed sample set, and a standard image corresponding to the sample image; the reconstructed sample set includes a first sample image obtained by upsampling the sample image to be reconstructed, a reference image corresponding to the sample image to be reconstructed, and a second sample image obtained by first downsampling and then upsampling the reference image; the resolution of the standard image is higher than that of the sample image;

[0007] Performing multi-level feature extraction on each image in the reconstructed sample set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed sample set; the size of the minimum feature extraction window for feature extraction in each level is different;

[0008] Generate a feature matrix for each image in the reconstructed sample set based on the feature information of multiple levels corresponding to each image in the reconstructed sample set by the image reconstruction model; wherein the feature matrix of the first sampled image is a query matrix, the feature matrix of the second sampled image is a key matrix, and the feature matrix of the reference image is a value matrix;

[0009] Reconstructing the sample image to be reconstructed based on the characteristic matrix of each image in the reconstructed sample set by using the image reconstruction model to obtain a reconstruction result of the sample image to be reconstructed;

[0010] Based on the reconstruction result and the standard image, parameters of the image reconstruction model are adjusted to train the image reconstruction model.

[0011] In another aspect, an image reconstruction method is provided, the method comprising:

[0012] Acquire an image to be reconstructed and a reconstructed image set, wherein the reconstructed image set includes a third sampled image obtained by upsampling the image to be reconstructed, a reference image corresponding to the image to be reconstructed, and a fourth sampled image obtained by downsampling and then upsampling the reference image;

[0013] Performing multi-level feature extraction on each image in the reconstructed image set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed image set; the size of the minimum feature extraction window for feature extraction in each level is different;

[0014] Generate a feature matrix for each image in the reconstructed image set based on the feature information of multiple levels corresponding to each image in the reconstructed image set through the image reconstruction model; wherein the feature matrix of the third sampled image is a query matrix, the feature matrix of the fourth sampled image is a key matrix, and the feature matrix of the reference image is a value matrix;

[0015] The image to be reconstructed is reconstructed by using the image reconstruction model based on the characteristic matrix of each image in the image set to be reconstructed, so as to obtain a reconstructed image of the image to be reconstructed.

[0016] In another aspect, an image reconstruction model training device is provided, the device comprising:

[0017] A training set acquisition module is used to acquire a training image set, wherein the training image set includes a sample image to be reconstructed, a reconstructed sample set, and a standard image corresponding to the sample image; the reconstructed sample set includes a first sample image obtained by upsampling the sample image to be reconstructed, a reference image corresponding to the sample image to be reconstructed, and a second sample image obtained by first downsampling and then upsampling the reference image; the resolution of the standard image is higher than that of the sample image;

[0018] A feature extraction module is used to perform multi-level feature extraction on each image in the reconstructed sample set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed sample set; the size of the minimum feature extraction window for feature extraction in each level is different;

[0019] A feature matrix generation module, used to generate a feature matrix of each image in the reconstructed sample set based on the feature information of multiple levels corresponding to each image in the reconstructed sample set through the image reconstruction model; wherein the feature matrix of the first sampled image is a query matrix, the feature matrix of the second sampled image is a key matrix, and the feature matrix of the reference image is a value matrix;

[0020] An image reconstruction module, used to reconstruct the sample image to be reconstructed based on the characteristic matrix of each image in the reconstruction sample set through the image reconstruction model, so as to obtain a reconstruction result of the sample image to be reconstructed;

[0021] A parameter adjustment module is used to adjust the parameters of the image reconstruction model based on the reconstruction result and the standard image, so as to train the image reconstruction model.

[0022] In a possible implementation, the feature matrix generation module includes:

[0023] The feature extraction submodule is used to perform multi-level inter-window feature extraction on each image to obtain inter-window feature information corresponding to each image at multiple levels; the inter-window feature information is used to characterize feature information between adjacent minimum feature extraction windows during multi-level feature extraction;

[0024] The feature matrix generation submodule is used to generate a feature matrix for each image in the reconstructed sample set based on the feature information of multiple levels corresponding to each image and the inter-window feature information corresponding to each image at multiple levels through the image reconstruction model.

[0025] In a possible implementation, the feature extraction submodule includes:

[0026] A window moving unit, used to move the position of each minimum feature extraction window in a target level to obtain a moved feature extraction window; the target level is one of the multiple levels containing at least two minimum feature extraction windows;

[0027] A transfer and splicing unit, used for performing a transfer and splicing operation on the moved feature extraction window, so that the window arrangement after the transfer and splicing is the same as the window arrangement before the position is moved;

[0028] The feature extraction unit is used to perform inter-window feature extraction on each image of the target level based on the transferred and spliced ​​feature extraction windows to obtain the inter-window feature information corresponding to each image at the target level.

[0029] In a possible implementation, the feature extraction unit is used to:

[0030] Perform feature extraction on each image based on the feature extraction window after transfer and splicing, and obtain feature information of each image under the feature extraction window after transfer and splicing;

[0031] Construct the mask matrix;

[0032] Based on the mask matrix, feature information of each image under the feature extraction window after transfer and splicing is screened to obtain the inter-window feature information corresponding to each image at the target level.

[0033] In a possible implementation, the feature matrix generation module is used to weightedly integrate feature information of multiple levels corresponding to a target image and inter-window feature information corresponding to the target image at multiple levels based on weights corresponding to different levels to obtain a feature matrix of the target image, where the target image is one of the images in the reconstructed sample set.

[0034] In a possible implementation, the image reconstruction module includes:

[0035] A first calculation submodule, configured to perform similarity calculation on the query matrix and the key matrix to obtain a similarity matrix between the first sampled image and the second sampled image;

[0036] A second calculation submodule is used to perform attention mechanism calculation on the similarity matrix and the value matrix to obtain a variable relationship matrix between the first sampled image and the reference image;

[0037] The image reconstruction submodule performs image reconstruction based on the variable relationship matrix and the sample image to be reconstructed to obtain a reconstruction result output by the image reconstruction model.

[0038] In another aspect, an image reconstruction device is provided, the device comprising:

[0039] An image acquisition module, used to acquire an image to be reconstructed and a reconstructed image set, wherein the reconstructed image set includes a third sampled image obtained by upsampling the image to be reconstructed, a reference image corresponding to the image to be reconstructed, and a fourth sampled image obtained by downsampling and then upsampling the reference image;

[0040] An extraction module, used for performing multi-level feature extraction on each image in the reconstructed image set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed image set; the size of the minimum feature extraction window for feature extraction in each level is different;

[0041] A generating module, configured to generate a feature matrix for each image in the reconstructed image set based on the feature information of multiple levels corresponding to each image in the reconstructed image set through the image reconstruction model; wherein the feature matrix of the third sampling image is a query matrix, the feature matrix of the fourth sampling image is a key matrix, and the feature matrix of the reference image is a value matrix;

[0042] The reconstruction module is used to reconstruct the image to be reconstructed based on the characteristic matrix of each image in the image set to be reconstructed by using the image reconstruction model to obtain a reconstructed image of the image to be reconstructed.

[0043] On the other hand, a computer device is provided, which includes a processor and a memory, wherein the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the above-mentioned image reconstruction model training method and / or image reconstruction method.

[0044] On the other hand, a computer-readable storage medium is provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement the above-mentioned image reconstruction model training method and / or image reconstruction method.

[0045] On the other hand, a computer program product is provided, which includes at least one computer program, and the computer program is loaded and executed by a processor to implement the image reconstruction model training method and / or the image reconstruction method provided in the above-mentioned various optional implementations.

[0046] The technical solution provided by this application may have the following beneficial effects:

[0047] The image reconstruction model training method provided in the embodiment of the present application performs multi-level feature extraction on each image in the reconstruction sample set through the image reconstruction model, and generates a feature matrix of each image based on the feature information of multiple levels corresponding to each image, so as to reconstruct the sample image to be reconstructed based on the feature matrix of each image in the reconstruction sample set, and finally adjusts the parameters of the image reconstruction model based on the sample image and the standard image corresponding to the sample image, so as to train the image reconstruction model; in the above process, when performing feature extraction on each image included in the reconstructed sample set in the training image set, feature extraction is performed by multi-level feature extraction, and the size of the minimum feature extraction window for feature extraction in different levels is different, so that feature information of different receptive fields can be obtained from each image, thereby enhancing the feature representation effect of each image, and then reconstructing the sample image based on the feature matrix generated by the feature information of multiple levels of each image, which can improve the image reconstruction effect of the image reconstruction model obtained by training.

[0048] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0050] Figure 1 A flowchart of an image reconstruction model training method provided by an exemplary embodiment of the present application is shown;

[0051] Figure 2 A schematic diagram of the minimum feature extraction window at each level provided by an exemplary embodiment of the present application is shown;

[0052] Figure 3 A flowchart of an image reconstruction model training method provided by an exemplary embodiment of the present application is shown;

[0053] Figure 4 A schematic diagram showing the re-division of a feature extraction window provided by an exemplary embodiment of the present application is shown;

[0054] Figure 5 A schematic diagram of transfer splicing provided by an exemplary embodiment of the present application is shown;

[0055] Figure 6 A schematic diagram showing a masking operation provided by an exemplary embodiment of the present application is shown;

[0056] Figure 7 A schematic diagram showing an image reconstruction process of an image reconstruction model provided by an exemplary embodiment of the present application;

[0057] Figure 8 A flowchart of an image reconstruction method provided by an exemplary embodiment of the present application is shown;

[0058] Fig. 9 A block diagram of an image reconstruction model training device provided by an exemplary embodiment of the present application is shown;

[0059] Fig.10 A block diagram of an image reconstruction device provided by an exemplary embodiment of the present application is shown;

[0060] Fig.11 A structural block diagram of a computer device shown in an exemplary embodiment of the present application is shown;

[0061] Fig.12 A structural block diagram of a computer device shown in an exemplary embodiment of the present application is shown. DETAILED DESCRIPTION

[0062] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present application. Instead, they are merely examples of devices and methods consistent with some aspects of the present application as detailed in the appended claims.

[0063] Image super resolution reconstruction is a technology that restores a high-resolution image from a low-resolution image or image sequence. In order to improve the accuracy of the reconstructed high-resolution image, improve the degree of retention of local information, and restore image details as much as possible, the embodiment of the present application provides a new image reconstruction model and a training method for the image reconstruction model; Figure 1 A flowchart of an image reconstruction model training method provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device, and the computer device can be implemented as a server or a terminal, such as Figure 1 As shown, the training method of the image reconstruction model may include the following steps:

[0064] Step 110, obtaining a training image set, which includes a sample image to be reconstructed, a reconstructed sample set, and a standard image corresponding to the sample image; the reconstructed sample set includes a first sampling image obtained by upsampling the sample image to be reconstructed, a reference image corresponding to the sample image to be reconstructed, and a second sampling image obtained by first downsampling and then upsampling the reference image; the resolution of the standard image is higher than that of the sample image.

[0065] In an embodiment of the present application, the image reconstruction model has four input images, namely, a sample image to be reconstructed, a first sampling image, a reference image and a second sampling image; wherein the reference image is an image having a certain similarity with the sample image to be reconstructed; the second sampling image corresponding to the reference image is an image obtained by first downsampling and then upsampling the reference image, so as to establish an association relationship between the reference image and the image to be reconstructed, and to evaluate the positions at which the reference image and the image to be reconstructed are similar, so as to draw texture reference at similar positions; schematically, the first sampling image LR↑ is an image obtained by upsampling the sample image to be reconstructed by bicubic x4, and the second sampling image Ref↑↓ is the reference image that is first downsampled by bicubic x4 and then upsampled by bicubic x4, so as to maintain domain-consistent domain consistency with LR↑.

[0066] The standard image is an image with a higher resolution corresponding to the sample image, and is used for comparison with the reconstruction result of the image reconstruction model to train the image reconstruction model.

[0067] Step 120, performing multi-level feature extraction on each image in the reconstructed sample set through the image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed sample set; the size of the minimum feature extraction window for feature extraction in each level is different.

[0068] The process of extracting features from an image is the process of performing a convolution operation on an image block within a feature extraction window in units of the feature extraction window; wherein the number of levels for feature extraction can be set based on actual needs, and the present application does not impose any restrictions on this; the size of the minimum feature extraction window for feature extraction in each level is different, and further, the size of the minimum feature extraction window for feature extraction in each level can be gradually reduced according to a preset segmentation multiple, schematically, Figure 2 FIG. 4 shows a schematic diagram of the minimum feature extraction window at each level provided by an exemplary embodiment of the present application. Figure 2As shown, the minimum feature extraction window in the first level is a global window, that is, a window covering the entire image is used on the input image, and a convolution operation is performed on the image blocks therein to obtain the feature representation of the image in the global feature dimension, and the scope of influence is all image blocks; the minimum feature extraction window in the second level is a 1 / 4 window of the global window, that is, the global window is divided into 2*2, and then the image blocks under each window are convolved, and the feature representation of the corresponding image blocks is affected, so that richer information is extracted at the 1 / 4 local level to enhance the feature representation; the minimum feature extraction window in the third level is 1 / 16 of the global window, that is, each 1 / 4 window is divided into 2*2, that is, the global window is divided into 4*4, and then the image blocks under each window are convolved, and so on. It should be noted that the above division of the minimum feature extraction windows of each level is only schematic, and the size of the minimum extraction window of each level can be set based on actual needs, and this application does not limit this.

[0069] Step 130, generating a feature matrix for each image in the reconstructed sample set based on multiple levels of feature information corresponding to each image in the reconstructed sample set through an image reconstruction model; wherein the feature matrix of the first sampled image is a query matrix, the feature matrix of the second sampled image is a key matrix, and the feature matrix of the reference image is a value matrix.

[0070] Schematically, taking one of the images as an example, its corresponding feature matrix is ​​the integration of the feature information of the image at multiple levels, that is, linking the cross-level feature information of the same image together, which has the effect of strengthening the feature representation, thereby improving the quality of image reconstruction.

[0071] Step 140 , reconstructing the sample image to be reconstructed based on the feature matrix of each image in the reconstruction sample set through the image reconstruction model to obtain a reconstruction result of the sample image to be reconstructed.

[0072] Step 150 , based on the reconstruction result and the standard image, adjusting parameters of the image reconstruction model to train the image reconstruction model.

[0073] In an embodiment of the present application, the computer device can calculate the loss function based on the reconstruction result and the standard image, and adjust the parameters of the image reconstruction model according to the function value of the loss function.

[0074] Optionally, during the model training process, steps 110 to 150 can be repeatedly executed until the training result condition is met, the training is terminated and a trained image reconstruction model is obtained; wherein the training result condition may refer to the number of training times reaching a training times threshold, or the image reconstruction model converges, etc.

[0075] In summary, the image reconstruction model training method provided in the embodiment of the present application performs multi-level feature extraction on each image in the reconstruction sample set through the image reconstruction model, and generates a feature matrix of each image based on the feature information of multiple levels corresponding to each image, so as to reconstruct the sample image to be reconstructed based on the feature matrix of each image in the reconstruction sample set, and finally adjusts the parameters of the image reconstruction model based on the sample image and the standard image corresponding to the sample image, so as to train the image reconstruction model; in the above process, when performing feature extraction on each image included in the reconstructed sample set in the training image set, feature extraction is performed by multi-level feature extraction, and the size of the minimum feature extraction window for feature extraction at different levels is different, so that feature information of different receptive fields can be obtained from each image, thereby enhancing the feature representation effect of each image, and then the sample image is reconstructed based on the feature matrix generated based on the feature information of multiple levels of each image, which can improve the image reconstruction effect of the image reconstruction model obtained by training.

[0076] In one possible implementation, after performing multi-level feature extraction on each image in the reconstructed sample set and obtaining the feature information of multiple levels corresponding to each image in the reconstructed sample set, a feature matrix of each image in the reconstructed sample set can be generated based on the feature information of multiple levels. In another possible implementation, since when performing feature extraction on an image based on the minimum feature extraction window of each level, the convolution operation is performed in units of the minimum feature extraction window, when there are multiple minimum feature extraction windows, information cannot be transferred between windows. Therefore, in an embodiment of the present application, in order to ensure the comprehensiveness of feature extraction, when performing multi-level feature extraction, multi-level inter-window feature extraction can also be performed to obtain the feature matrix of the image by integrating the multi-level feature information and the multi-level inter-window feature information; based on this, Figure 3 A flowchart of an image reconstruction model training method provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device, and the computer device can be implemented as a server or a terminal, such as Figure 3 As shown, the training method of the image reconstruction model may include the following steps:

[0077] Step 310, obtaining a training image set, which includes a sample image to be reconstructed, a reconstructed sample set, and a standard image corresponding to the sample image; the reconstructed sample set includes a first sampling image obtained by upsampling the sample image to be reconstructed, a reference image corresponding to the sample image to be reconstructed, and a second sampling image obtained by first downsampling and then upsampling the reference image; the resolution of the standard image is higher than that of the sample image.

[0078] In an embodiment of the present application, training parameters may be pre-set in a computer device so as to pre-process images in an image set based on the training parameters to obtain a training image set and to train an image reconstruction model; illustratively, the training parameters may include: downsampling and upsampling sizes of the image, super-resolution reconstruction multiples, batch size in training, feature extraction window size and segmentation multiples, number of iterations and learning rate, etc.

[0079] The input end of the image reconstruction model includes three channels, namely channel Q (ordinary reconstructed image channel), channel K (reference image reconstruction channel) and channel V (reference image channel). Therefore, after preprocessing the images in the image set, the corresponding images in the training image set are used as the input of the corresponding channels, that is, the first sampled image is used as the input of channel Q, the second sampled image is used as the input of channel K, and the reference image is used as the input of channel V.

[0080] Furthermore, after obtaining the training image set, the images in the training image set can be standardized to eliminate pixel deviations, and data enhancement operations such as image denoising can be appropriately used to enhance effective information in the image to facilitate subsequent feature extraction.

[0081] It should be noted that the computer device can pre-process the images in the image set based on the preset batch size in training to obtain multiple sets of training image sets, and load them into the CPU video memory to train the image reconstruction model.

[0082] Step 320, performing multi-level feature extraction on each image in the reconstructed sample set through the image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed sample set; the size of the minimum feature extraction window for feature extraction in each level is different.

[0083] Schematically, taking the case of performing two-level feature extraction on images in a reconstructed image set with a segmentation multiple of 4 as an example, the computer device first performs a feature extraction operation on the input three-channel image in units of a global window through an image reconstruction model, and obtains feature information of this level for each image block under the corresponding window; then, the image reconstruction model performs a feature extraction operation on the input three-channel image in units of a 1 / 4 window, and obtains feature information of this level for each image block under the corresponding window; thereby obtaining feature information of two levels corresponding to each image.

[0084] Step 330, performing multi-level inter-window feature extraction on each image to obtain inter-window feature information corresponding to each image at multiple levels; the inter-window feature information is used to represent feature information between adjacent minimum feature extraction windows during multi-level feature extraction.

[0085] In the embodiment of the present application, multi-level inter-window feature extraction is performed on each image to obtain the inter-window feature information corresponding to each image at multiple levels, including:

[0086] The position of each minimum feature extraction window in the target level is moved to obtain the moved feature extraction window; the target level is one of the multiple levels including at least two minimum feature extraction windows;

[0087] Performing a transfer and splicing operation on the moved feature extraction window so that the window arrangement after the transfer and splicing is the same as the window arrangement before the position movement;

[0088] Based on the transferred and spliced ​​feature extraction windows, inter-window feature extraction is performed on each image at the target level to obtain the inter-window feature information corresponding to each image at the target level.

[0089] When extracting feature information between windows of an image, it is necessary to re-divide the feature extraction window so that the image blocks at the edge of the minimum feature extraction window at the corresponding level are in the same feature extraction window to compensate for the problem that information cannot be exchanged between windows; in the embodiment of the present application, the feature extraction window can be re-divided by moving the position of the minimum feature extraction window; Figure 4 FIG. 4 shows a schematic diagram of re-dividing a feature extraction window provided by an exemplary embodiment of the present application. Figure 4 As shown in , the global window is an 8*8 window, which can cover 64 image blocks; assuming that the minimum feature extraction window of the Lth layer is 1 / 4 of the global window, as shown in Figure 4 As shown in part A of FIG, in the Lth layer, four 1 / 4 windows are used to extract features from the image, and information cannot be exchanged between the windows. When inter-window feature extraction is performed on the image in the Lth layer, the minimum feature extraction window can be moved, as shown in FIG. Figure 4 As shown in , it can be understood that the minimum feature extraction window is shifted from the upper left corner to the right and downward by 2 image blocks, respectively, to obtain the following Figure 4 The window arrangement shown in part B of the figure forms a feature extraction window after movement, that is, 9 feature extraction windows of different sizes. In the feature extraction windows formed after movement, feature extraction can be performed between windows. For example, for the 2*4 window in the first row and second column, it can enable the two windows in the first row of the Lth layer to exchange information. For the 4*4 window in the second row and second column, it can enable the four windows in the Lth layer to exchange information, and so on. This solves the problem that different windows cannot exchange information, and can extract feature information of more receptive fields of the image. It should be noted that Figure 4 The movement method in the figure is only for illustration. The movement methods corresponding to different levels may be different, and the application does not impose any restrictions on this.

[0090] Furthermore, in order not to introduce additional computational complexity, after obtaining the feature extraction window formed after the movement, the feature extraction window formed after the movement can be transferred and spliced, so that the window arrangement of the feature extraction window after the transfer and splicing is the same as the window arrangement before the position movement, thereby not performing additional computational complexity and not introducing additional computational complexity; Figure 4 Take the window arrangement of the feature extraction window obtained as an example, Figure 5 A schematic diagram of transfer splicing provided by an exemplary embodiment of the present application is shown. Figure 5 As shown, in Figure 5 Based on the window arrangement shown in part A of FIG, the surrounding windows are transferred and spliced ​​to the corresponding positions, so that several surrounding windows can be spliced ​​into a 1 / 4 window, for example, in Figure 5 In part B of the figure, the two 2*4 windows in the lower left corner can be spliced ​​into a 1 / 4 window, so that while the feature extraction between windows is realized, the amount of calculation is consistent with that under the corresponding ordinary window arrangement.

[0091] For the feature extraction window after transfer and splicing, it is necessary to extract the feature information inside the window, without extracting the feature information between windows. For example, Figure 5 In part B of the figure, the feature information between the two 4*2 windows that make up the 1 / 4 window in the upper right corner is obtained. Therefore, when performing feature extraction, a masking operation is required to erase the calculated deviation information. Based on this, inter-window feature extraction is performed on each image at the target level based on the feature extraction window after transfer and splicing, and the inter-window feature information corresponding to each image at the target level is obtained, including:

[0092] Perform feature extraction on each image based on the feature extraction window after transfer and splicing, and obtain feature information of each image under the feature extraction window after transfer and splicing;

[0093] Construct the mask matrix;

[0094] Based on the mask matrix, the feature information of each image under the feature extraction window after transfer and splicing is screened to obtain the inter-window feature information corresponding to each image at the target level.

[0095] In the embodiment of the present application, different mask matrices may be constructed corresponding to different types of feature extraction windows obtained after transfer and splicing; Figure 6 FIG. 1 shows a schematic diagram of a masking operation provided by an exemplary embodiment of the present application, such as Figure 6 As shown, Figure 5Taking the feature extraction window after transfer and splicing in as an example, for the splicing window 1 in the upper right corner (the window obtained by splicing two 4*2 windows), the left side is the original image block, and the right side is the spliced ​​image block. There is no adjacent relationship between the two in the original image. Therefore, when the convolution operation is performed in the splicing window 1, invalid feature information will be generated, such as feature 610. At this time, the feature information can be filtered based on the constructed mask matrix 620, and the valid feature information is retained and the invalid feature information is deleted; similarly, for the splicing window 2 in the lower left corner (the window obtained by splicing two 2*4 windows), the upper side is the original image block, and the lower side is the spliced ​​image block. There is no adjacent relationship between the two in the original image. Therefore, when the convolution operation is performed in the splicing window 1, invalid feature information will be generated, such as feature 630. At this time, the feature information can be filtered based on the constructed mask matrix 640, and the valid feature information is retained and the invalid feature information is deleted. It should be noted that, corresponding to different splicing windows, the mask matrix can be constructed in different forms, and this application does not limit this.

[0096] Step 340 , generating a feature matrix for each image in the reconstructed sample set based on the feature information of multiple levels corresponding to each image and the feature information between windows corresponding to each image at multiple levels through the image reconstruction model.

[0097] Taking the generation process of the feature matrix of any target image in the image sample set as an example, the process can be implemented as follows: the feature information of multiple levels corresponding to the target image and the inter-window feature information corresponding to the target image at multiple levels are weightedly integrated based on the weights corresponding to different levels to obtain the feature matrix of the target image, where the target image is one of the images in the reconstructed sample set.

[0098] Among them, the weights corresponding to different levels can be set differently based on actual needs; for example, the weight of the level whose minimum feature extraction window is the global window is 0.6, and the weight of the level whose minimum feature extraction window is the 1 / 4 window is 0.4; it should be noted that the above setting of the level weights is only for illustration, and this application does not impose any restrictions on this.

[0099] In order to improve the computational efficiency of the model, the image reconstruction model in the present application is constructed with a lighter network structure, which is reflected in the fact that the image reconstruction model includes a similarity calculation module and an attention mechanism calculation module, and reduces the amount of calculation through masking operations, thereby improving the effect of reconstructing the image by the image reconstruction model while reducing the amount of model calculation.

[0100] Step 350 , reconstructing the sample image to be reconstructed based on the characteristic matrix of each image in the reconstruction sample set through the image reconstruction model to obtain a reconstruction result of the sample image to be reconstructed.

[0101] The process of image reconstruction for the sample image to be reconstructed can be implemented as follows:

[0102] The similarity calculation is performed on the query matrix and the key matrix to obtain a similarity matrix between the first sampled image and the second sampled image.

[0103] The attention mechanism is calculated on the similarity matrix and the value matrix to obtain the variable relationship matrix between the first sampled image and the reference image.

[0104] Image reconstruction is performed based on the variable relationship matrix and the sample image to be reconstructed to obtain a reconstruction result output by the image reconstruction model.

[0105] The feature matrix of the first sampled image is a query matrix, the feature matrix of the second sampled image is a key matrix, and the feature matrix of the reference image is a value matrix.

[0106] The computer device performs similarity calculation on the query matrix corresponding to the first sampling image and the key matrix corresponding to the second sampling image through the similarity calculation module in the image reconstruction model, and obtains a similarity matrix used to characterize the similarity between each image block in the first sampling image and the image block in the second sampling image; the computer device performs attention mechanism calculation on the value matrix corresponding to the reference image and the similarity matrix through the attention mechanism calculation module in the image reconstruction model, and obtains a variable relationship matrix used to characterize the correlation between the image blocks in the first sampling image and the image blocks in the reference image; finally, based on the variable relationship matrix, with reference to the reference image, the details of each image block in the image to be reconstructed are reconstructed, and then after residual fusion, the reconstruction result of the image reconstruction model corresponding to the sample image to be reconstructed is obtained and output.

[0107] Figure 7 A schematic diagram showing an image reconstruction process of an image reconstruction model provided by an exemplary embodiment of the present application is shown in FIG. Figure 7As shown, the image reconstruction model has four input images, namely, the image to be reconstructed LR, the first sampled image SR obtained by upsampling the image to be reconstructed, the reference image Ref, and the second sampled image Ref-SR obtained by first downsampling and then upsampling the reference image; in the feature extraction stage, multi-level feature extraction and multi-level window feature extraction are performed on the first sampled image, the reference image and the second sampled image, respectively, and feature weighted fusion is performed to obtain the query matrix Q corresponding to the first sampled image, the value matrix V of the reference image and the key matrix K of the second sampled image; in the image reconstruction stage, the similarity calculation module 710 of the image reconstruction model performs similarity calculation on the query matrix Q and the key matrix K to obtain the similarity matrix, and the attention mechanism calculation module 720 of the image reconstruction model performs attention mechanism calculation on the similarity matrix and the value matrix to obtain the variable relationship matrix; in the residual fusion stage, the variable relationship matrix and the sample image to be reconstructed are used to perform detail reconstruction, and the residual fusion module 720 performs residual fusion to obtain the reconstruction result.

[0108] Step 360: Based on the reconstruction result and the standard image, adjust the parameters of the image reconstruction model to train the image reconstruction model.

[0109] In an embodiment of the present application, the computer device can calculate the loss function based on the reconstruction result and the standard image, perform back propagation according to the calculated loss function value, and adjust the parameters of the image reconstruction model. Schematically, the loss function can be a mean square error loss function (MSE), or can also be implemented as any other loss function; optionally, the computer device can use an optimizer to adjust the parameters of the image reconstruction model.

[0110] The above training process is repeated until the training completion condition is met, and the image reconstruction model training is determined to be completed, and a trained image reconstruction model is obtained; in a possible implementation method, when the training round reaches the number of iterations set in the training parameters, the image reconstruction model training is determined to be completed.

[0111] Optionally, the computer device can also obtain a test set and test the image reconstruction model through the test set, using the peak signal-to-noise ratio PSNR and structural similarity SSIM in the super-resolution reconstruction task as evaluation indicators, and preliminarily evaluating the quality of the image reconstruction model by evaluating the calculated indicator values; if the evaluation result does not meet the requirements, it is necessary to adjust the hyperparameters in the image reconstruction model and retrain the image reconstruction model; if the evaluation result meets the requirements, the model parameters can be saved to complete the image reconstruction model training process.

[0112] In summary, the image reconstruction model training method provided in the embodiment of the present application performs multi-level feature extraction on each image in the reconstruction sample set through the image reconstruction model, and generates a feature matrix of each image based on the feature information of multiple levels corresponding to each image, so as to reconstruct the sample image to be reconstructed based on the feature matrix of each image in the reconstruction sample set, and finally adjusts the parameters of the image reconstruction model based on the sample image and the standard image corresponding to the sample image, so as to train the image reconstruction model; in the above process, when performing feature extraction on each image included in the reconstructed sample set in the training image set, feature extraction is performed by multi-level feature extraction, and the size of the minimum feature extraction window for feature extraction at different levels is different, so that feature information of different receptive fields can be obtained from each image, thereby enhancing the feature representation effect of each image, and then the sample image is reconstructed based on the feature matrix generated based on the feature information of multiple levels of each image, which can improve the image reconstruction effect of the image reconstruction model obtained by training.

[0113] After obtaining the trained image reconstruction model, image reconstruction can be performed based on the image reconstruction model. Figure 8 A flowchart of an image reconstruction method provided by an exemplary embodiment of the present application is shown. The method can be executed by a computer device, and the computer device can implement a server or a terminal. The computer device can be configured with a trained image reconstruction model; Figure 8 As shown, the image reconstruction method may include the following steps:

[0114] Step 810, obtaining an image to be reconstructed and a reconstructed image set, wherein the reconstructed image set includes a third sampling image obtained by upsampling the image to be reconstructed, a reference image corresponding to the image to be reconstructed, and a fourth sampling image obtained by downsampling and then upsampling the reference image.

[0115] Step 820, performing multi-level feature extraction on each image in the reconstructed image set through the image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed image set; the size of the minimum feature extraction window for feature extraction in each level is different.

[0116] Step 830, generating a feature matrix for each image in the reconstructed image set based on multiple levels of feature information corresponding to each image in the reconstructed image set through an image reconstruction model; wherein the feature matrix of the third sampling image is a query matrix, the feature matrix of the fourth sampling image is a key matrix, and the feature matrix of the reference image is a value matrix.

[0117] Step 840 , reconstructing the image to be reconstructed based on the characteristic matrix of each image in the image set to be reconstructed by using the image reconstruction model to obtain a reconstructed image of the image to be reconstructed.

[0118] The process of performing image reconstruction through the image reconstruction model can refer to the relevant content of the above-mentioned image reconstruction model training process, which will not be repeated here.

[0119] To summarize, the image reconstruction method provided in the embodiment of the present application performs multi-level feature extraction on each image in the reconstructed image set through an image reconstruction model, and generates a feature matrix of each image based on the obtained feature information of multiple levels corresponding to each image, so as to reconstruct the image to be reconstructed based on the feature matrix of each image in the reconstructed sample set; in the above process, when performing feature extraction on each image included in the reconstructed image set, feature extraction is performed by multi-level feature extraction, and the size of the minimum feature extraction window for feature extraction at different levels is different, so that feature information of different receptive fields can be obtained from each image, thereby enhancing the feature representation effect of each image, and then performing image reconstruction based on the feature matrix generated by the feature information of multiple levels of each image, which can improve the image reconstruction effect.

[0120] Fig. 9 FIG. 1 shows a block diagram of an image reconstruction model training device provided by an exemplary embodiment of the present application. The image reconstruction model training device can perform the following steps: Figure 1 or Figure 3 All or part of the steps of the embodiment shown, such as Fig. 9 As shown, the image reconstruction model training device includes:

[0121] The training set acquisition module 910 is used to acquire a training image set, wherein the training image set includes a sample image to be reconstructed, a reconstructed sample set, and a standard image corresponding to the sample image; the reconstructed sample set includes a first sample image obtained by upsampling the sample image to be reconstructed, a reference image corresponding to the sample image to be reconstructed, and a second sample image obtained by first downsampling and then upsampling the reference image; the resolution of the standard image is higher than that of the sample image;

[0122] The feature extraction module 920 is used to perform multi-level feature extraction on each image in the reconstructed sample set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed sample set; the size of the minimum feature extraction window for feature extraction in each level is different;

[0123] The feature matrix generation module 930 is used to generate a feature matrix for each image in the reconstructed sample set based on the feature information of multiple levels corresponding to each image in the reconstructed sample set through the image reconstruction model; wherein the feature matrix of the first sampled image is a query matrix, the feature matrix of the second sampled image is a key matrix, and the feature matrix of the reference image is a value matrix;

[0124] An image reconstruction module 940 is used to perform image reconstruction on the sample image to be reconstructed based on the characteristic matrix of each image in the reconstruction sample set through the image reconstruction model to obtain a reconstruction result of the sample image to be reconstructed;

[0125] A parameter adjustment module is used to adjust the parameters of the image reconstruction model based on the reconstruction result and the standard image, so as to train the image reconstruction model.

[0126] In a possible implementation, the feature matrix generating module 930 includes:

[0127] The feature extraction submodule is used to perform multi-level inter-window feature extraction on each image to obtain inter-window feature information corresponding to each image at multiple levels; the inter-window feature information is used to characterize feature information between adjacent minimum feature extraction windows during multi-level feature extraction;

[0128] The feature matrix generation submodule is used to generate a feature matrix for each image in the reconstructed sample set based on the feature information of multiple levels corresponding to each image and the inter-window feature information corresponding to each image at multiple levels through the image reconstruction model.

[0129] In a possible implementation, the feature extraction submodule includes:

[0130] A window moving unit, used to move the position of each minimum feature extraction window in a target level to obtain a moved feature extraction window; the target level is one of the multiple levels containing at least two minimum feature extraction windows;

[0131] A transfer and splicing unit, used for performing a transfer and splicing operation on the moved feature extraction window, so that the window arrangement after the transfer and splicing is the same as the window arrangement before the position is moved;

[0132] The feature extraction unit is used to perform inter-window feature extraction on each image of the target level based on the transferred and spliced ​​feature extraction windows to obtain the inter-window feature information corresponding to each image at the target level.

[0133] In a possible implementation, the feature extraction unit is used to:

[0134] Perform feature extraction on each image based on the feature extraction window after transfer and splicing, and obtain feature information of each image under the feature extraction window after transfer and splicing;

[0135] Construct the mask matrix;

[0136] Based on the mask matrix, feature information of each image under the feature extraction window after transfer and splicing is screened to obtain the inter-window feature information corresponding to each image at the target level.

[0137] In one possible implementation, the feature matrix generation module 930 is used to weightedly integrate feature information of multiple levels corresponding to a target image and inter-window feature information corresponding to the target image at multiple levels based on weights corresponding to different levels to obtain a feature matrix of the target image, where the target image is one of the images in the reconstructed sample set.

[0138] In a possible implementation, the image reconstruction module 940 includes:

[0139] A first calculation submodule, configured to perform similarity calculation on the query matrix and the key matrix to obtain a similarity matrix between the first sampled image and the second sampled image;

[0140] A second calculation submodule is used to perform attention mechanism calculation on the similarity matrix and the value matrix to obtain a variable relationship matrix between the first sampled image and the reference image;

[0141] The image reconstruction submodule performs image reconstruction based on the variable relationship matrix and the sample image to be reconstructed to obtain a reconstruction result output by the image reconstruction model.

[0142] In summary, the image reconstruction model training device provided in the embodiment of the present application performs multi-level feature extraction on each image in the reconstruction sample set through the image reconstruction model, and generates a feature matrix of each image based on the feature information of multiple levels corresponding to each image, so as to reconstruct the sample image to be reconstructed based on the feature matrix of each image in the reconstruction sample set, and finally adjusts the parameters of the image reconstruction model based on the sample image and the standard image corresponding to the sample image, so as to train the image reconstruction model; in the above process, the device can perform feature extraction by multi-level feature extraction when extracting features from each image included in the reconstructed sample set in the training image set, and the size of the minimum feature extraction window for feature extraction at different levels is different, so that feature information of different receptive fields can be obtained from each image, thereby enhancing the feature representation effect of each image, and then reconstructing the sample image based on the feature matrix generated by the feature information of multiple levels of each image, which can improve the image reconstruction effect of the image reconstruction model.

[0143] Fig.10 FIG. 1 shows a block diagram of an image reconstruction device provided by an exemplary embodiment of the present application. The image reconstruction device can perform the following steps: Figure 8All or part of the steps of the embodiment shown, such as Fig.10 As shown, the image reconstruction device comprises:

[0144] An image acquisition module 1010 is used to acquire an image to be reconstructed and a reconstructed image set, wherein the reconstructed image set includes a third sampled image obtained by upsampling the image to be reconstructed, a reference image corresponding to the image to be reconstructed, and a fourth sampled image obtained by downsampling and then upsampling the reference image;

[0145] An extraction module 1020 is used to perform multi-level feature extraction on each image in the reconstructed image set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed image set; the size of the minimum feature extraction window for feature extraction in each level is different;

[0146] A generating module 1030 is used to generate a feature matrix for each image in the reconstructed image set based on the feature information of multiple levels corresponding to each image in the reconstructed image set through the image reconstruction model; wherein the feature matrix of the third sampled image is a query matrix, the feature matrix of the fourth sampled image is a key matrix, and the feature matrix of the reference image is a value matrix;

[0147] The reconstruction module 1040 is used to reconstruct the image to be reconstructed based on the characteristic matrix of each image in the image set to be reconstructed by using the image reconstruction model to obtain a reconstructed image of the image to be reconstructed.

[0148] To summarize, the image reconstruction device provided in the embodiment of the present application performs multi-level feature extraction on each image in the reconstructed image set through an image reconstruction model, and generates a feature matrix for each image based on the obtained feature information of multiple levels corresponding to each image, so as to reconstruct the image to be reconstructed based on the feature matrix of each image in the reconstructed sample set; in the above process, when the device extracts features from each image included in the reconstructed image set, it performs feature extraction by multi-level feature extraction, and the size of the minimum feature extraction window for feature extraction at different levels is different, so that feature information of different receptive fields can be obtained from each image, thereby enhancing the feature representation effect of each image, and then performing image reconstruction based on the feature matrix generated by the feature information of multiple levels of each image, which can improve the image reconstruction effect.

[0149] Fig.11The structural block diagram of a computer device 1100 shown in an exemplary embodiment of the present application is shown. The computer device can be implemented as a server in the above-mentioned solution of the present application. The computer device 1100 includes a central processing unit (CPU) 1101, a system memory 1104 including a random access memory (RAM) 1102 and a read-only memory (ROM) 1103, and a system bus 1105 connecting the system memory 1104 and the central processing unit 1101. The computer device 1100 also includes a mass storage device 1106 for storing an operating system 1109, an application program 1110 and other program modules 1111.

[0150] The computer device may be configured with a predetermined training environment, for example, a development environment of Python 3.6 (and above) and a deep framework of PyTorch 1.4 (and above) installed. Since the algorithm used in this application is a model algorithm based on deep learning, the model training process may be performed in a GPU (Graphics Processing Unit) environment. At this time, the GPU version of Pytorch 1.4 (and above) and the corresponding version of the CUDA (Compute Unified Device Architecture) parallel computing architecture may be installed in the computer device.

[0151] Without loss of generality, the computer-readable medium may include computer storage media and communication media. Computer storage media include volatile and non-volatile, removable and non-removable media implemented by any method or technology for storing information such as computer-readable instructions, data structures, program modules or other data. Computer storage media include RAM, ROM, Erasable Programmable Read Only Memory (EPROM), Electronically-Erasable Programmable Read-Only Memory (EEPROM) flash memory or other solid-state storage technology, CD-ROM, Digital Versatile Disc (DVD) or other optical storage, cassettes, tapes, disk storage or other magnetic storage devices. Of course, those skilled in the art will appreciate that the computer storage medium is not limited to the above. The above-mentioned system memory 1104 and mass storage device 1106 can be collectively referred to as memory.

[0152] According to various embodiments of the present application, the computer device 1100 can also be connected to a remote computer on the network through a network such as the Internet. That is, the computer device 1100 can be connected to the network 1108 through the network interface unit 1107 connected to the system bus 1105, or the network interface unit 1107 can be used to connect to other types of networks or remote computer systems (not shown).

[0153] The memory also includes at least one instruction, at least one program, code set or instruction set, and the at least one instruction, at least one program, code set or instruction set is stored in the memory. The central processing unit 1101 implements all or part of the steps in the image reconstruction model training method or image reconstruction method shown in the above-mentioned embodiments by executing the at least one instruction, at least one program, code set or instruction set.

[0154] Fig.12 The following is a block diagram of a computer device 1200 according to an exemplary embodiment of the present application. The computer device 1200 may be implemented as the above-mentioned terminal, such as a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart watch, a TV, etc. The computer device 1200 may also be referred to as a user device, a portable terminal, a laptop terminal, a desktop terminal, or other names.

[0155] Typically, the computer device 1200 includes a processor 1201 and a memory 1202 .

[0156] In some embodiments, the computer device 1200 may further optionally include: a peripheral device interface 1203 and at least one peripheral device. The processor 1201, the memory 1202 and the peripheral device interface 1203 may be connected via a bus or a signal line. Each peripheral device may be connected to the peripheral device interface 1203 via a bus, a signal line or a circuit board. Specifically, the peripheral device includes: at least one of a radio frequency circuit 1204, a display screen 1205, a camera assembly 1206, an audio circuit 1207 and a power supply 1208.

[0157] In some embodiments, the computer device 1200 further includes one or more sensors 1209 , including but not limited to: an acceleration sensor 1210 , a gyroscope sensor 1211 , a pressure sensor 1212 , an optical sensor 1213 , and a proximity sensor 1214 .

[0158] Those skilled in the art will understand that Fig.12 The structure shown in the figure does not constitute a limitation on the computer device 1200, and the computer device 1200 may include more or less components than those shown in the figure, or combine some components, or adopt a different arrangement of components.

[0159] In an exemplary embodiment, a computer-readable storage medium is also provided, in which at least one computer program is stored, and the computer program is loaded and executed by a processor to implement all or part of the steps in the above-mentioned image reconstruction model training method or image reconstruction method. For example, the computer-readable storage medium can be a read-only memory (ROM), a random access memory (RAM), a compact disc read-only memory (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, etc.

[0160] In an exemplary embodiment, a computer program product is also provided, the computer program product comprising at least one computer program, the computer program being loaded and executed by a processor Figure 1 or Figure 3 The image reconstruction model training method shown in any embodiment or Figure 8 All or part of the steps of the image reconstruction method shown in the embodiment.

[0161] Those skilled in the art will readily appreciate other embodiments of the present application after considering the specification and practicing the invention disclosed herein. The present application is intended to cover any modification, use or adaptation of the present application, which follows the general principles of the present application and includes common knowledge or customary techniques in the art that are not disclosed in the present application. The specification and examples are intended to be exemplary only, and the true scope and spirit of the present application are indicated by the following claims.

[0162] It should be understood that the present application is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present application is limited only by the appended claims.

Claims

1. A method for training an image reconstruction model, characterized in that: The method comprises: Acquire a training image set, wherein the training image set includes a sample image to be reconstructed, a reconstructed sample set, and a standard image corresponding to the sample image to be reconstructed; the reconstructed sample set includes a first sampling image obtained by upsampling the sample image to be reconstructed, a reference image corresponding to the sample image to be reconstructed, and a second sampling image obtained by first downsampling and then upsampling the reference image; the resolution of the standard image is higher than that of the sample image to be reconstructed; Performing multi-level feature extraction on each image in the reconstructed sample set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed sample set; the size of the minimum feature extraction window for feature extraction in each level is different and gradually becomes smaller according to a preset segmentation multiple; the feature extraction refers to performing a convolution operation on the image block in the feature extraction window with the feature extraction window as a unit; Generate a feature matrix for each image in the reconstructed sample set based on the feature information of multiple levels corresponding to each image in the reconstructed sample set and the inter-window information corresponding to each image at multiple levels by using the image reconstruction model; Among them, multi-level inter-window feature extraction is performed on each image to obtain the inter-window feature information corresponding to each image at multiple levels; the inter-window feature information is used to characterize the feature information between adjacent minimum feature extraction windows during multi-level feature extraction; the inter-window feature information is the effective feature information retained after filtering the feature information extracted after window transfer and splicing through a mask matrix; the mask matrix is ​​constructed based on the correspondence of different types of feature extraction windows after transfer and splicing; The feature information of multiple levels corresponding to the target image and the inter-window feature information corresponding to the target image at the multiple levels are weighted and integrated based on the weights corresponding to different levels to obtain a feature matrix of the target image, wherein the target image is one of the images in the reconstructed sample set; and the weights corresponding to different levels are set differently; The feature matrix of the first sampled image is a query matrix, the feature matrix of the second sampled image is a key matrix, and the feature matrix of the reference image is a value matrix; Reconstructing the sample image to be reconstructed based on the characteristic matrix of each image in the reconstructed sample set by using the image reconstruction model to obtain a reconstruction result of the sample image to be reconstructed; Based on the reconstruction result and the standard image, parameters of the image reconstruction model are adjusted to train the image reconstruction model.

2. The method according to claim 1, characterized in that The multi-level inter-window feature extraction is performed on each image to obtain the inter-window feature information corresponding to each image at multiple levels, including: The position of each minimum feature extraction window in the target level is moved to obtain a moved feature extraction window; the target level is one of the multiple levels including at least two minimum feature extraction windows; Performing a transfer and splicing operation on the moved feature extraction window so that the window arrangement after the transfer and splicing is the same as the window arrangement before the position movement; Based on the transferred and spliced ​​feature extraction windows, inter-window feature extraction is performed on each image at the target level to obtain inter-window feature information corresponding to each image at the target level.

3. The method according to claim 2, characterized in that The inter-window feature extraction is performed on each image of the target level based on the feature extraction window after transfer and splicing to obtain the inter-window feature information corresponding to each image at the target level, including: Perform feature extraction on each image based on the feature extraction window after transfer and splicing, and obtain feature information of each image under the feature extraction window after transfer and splicing; Construct the mask matrix; Based on the mask matrix, feature information of each image under the feature extraction window after transfer and splicing is screened to obtain the inter-window feature information corresponding to each image at the target level.

4. The method according to claim 1, characterized in that The image reconstruction is performed based on the characteristic matrix of each image in the reconstruction sample set by the image reconstruction model to obtain the reconstruction result of the sample image to be reconstructed, including: Performing similarity calculation on the query matrix and the key matrix to obtain a similarity matrix between the first sampled image and the second sampled image; Performing attention mechanism calculation on the similarity matrix and the value matrix to obtain a variable relationship matrix between the first sampled image and the reference image; Image reconstruction is performed based on the variable relationship matrix and the sample image to be reconstructed to obtain a reconstruction result output by the image reconstruction model.

5. An image reconstruction method, characterized in that: The method comprises: Acquire an image to be reconstructed and a reconstructed image set, wherein the reconstructed image set includes a third sampled image obtained by upsampling the image to be reconstructed, a reference image corresponding to the image to be reconstructed, and a fourth sampled image obtained by downsampling and then upsampling the reference image; Performing multi-level feature extraction on each image in the reconstructed image set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed image set; the size of the minimum feature extraction window for feature extraction in each level is different and gradually becomes smaller according to a preset segmentation multiple; the feature extraction refers to performing a convolution operation on the image blocks in the feature extraction window with the feature extraction window as a unit; Generate, by the image reconstruction model, a feature matrix for each image in the reconstructed image set based on feature information of multiple levels corresponding to each image in the reconstructed image set and feature information between windows corresponding to each image at multiple levels; Among them, multi-level inter-window feature extraction is performed on each image to obtain the inter-window feature information corresponding to each image at multiple levels; the inter-window feature information is used to characterize the feature information between adjacent minimum feature extraction windows during multi-level feature extraction; the inter-window feature information is the effective feature information retained after filtering the extracted feature information through a mask matrix; the mask matrix is ​​constructed based on the correspondence of different types of feature extraction windows after transfer splicing; The feature information of multiple levels corresponding to one of the images in the reconstructed image set and the feature information between windows corresponding to the images at the multiple levels are weighted and integrated based on the weights corresponding to the different levels to obtain the feature matrix of the image; the weights corresponding to the different levels are set differently; The feature matrix of the third sampled image is a query matrix, the feature matrix of the fourth sampled image is a key matrix, and the feature matrix of the reference image is a value matrix; The image to be reconstructed is reconstructed by using the image reconstruction model based on the characteristic matrices of each image in the reconstructed image set to obtain a reconstructed image of the image to be reconstructed.

6. An image reconstruction model training device, characterized in that: The device comprises: A training set acquisition module is used to acquire a training image set, wherein the training image set includes a sample image to be reconstructed, a reconstructed sample set, and a standard image corresponding to the sample image to be reconstructed; the reconstructed sample set includes a first sampled image obtained by upsampling the sample image to be reconstructed, a reference image corresponding to the sample image to be reconstructed, and a second sampled image obtained by first downsampling and then upsampling the reference image; the resolution of the standard image is higher than that of the sample image to be reconstructed; A feature extraction module is used to perform multi-level feature extraction on each image in the reconstructed sample set through an image reconstruction model to obtain feature information of multiple levels corresponding to each image in the reconstructed sample set; the size of the minimum feature extraction window for feature extraction in each level is different and gradually decreases according to a preset segmentation multiple; the feature extraction refers to performing a convolution operation on the image blocks in the feature extraction window with the feature extraction window as a unit; A feature matrix generation module, used to generate a feature matrix for each image in the reconstructed sample set based on the feature information of multiple levels corresponding to each image in the reconstructed sample set and the window information corresponding to each image at multiple levels through the image reconstruction model; Among them, multi-level inter-window feature extraction is performed on each image to obtain the inter-window feature information corresponding to each image at multiple levels; the inter-window feature information is used to characterize the feature information between adjacent minimum feature extraction windows during multi-level feature extraction; the inter-window feature information is the effective feature information retained after filtering the feature information extracted after window transfer and splicing through a mask matrix; the mask matrix is ​​constructed based on the correspondence of different types of feature extraction windows after transfer and splicing; The feature information of multiple levels corresponding to the target image and the inter-window feature information corresponding to the target image at the multiple levels are weighted and integrated based on the weights corresponding to different levels to obtain a feature matrix of the target image, wherein the target image is one of the images in the reconstructed sample set; and the weights corresponding to different levels are set differently; The feature matrix of the first sampled image is a query matrix, the feature matrix of the second sampled image is a key matrix, and the feature matrix of the reference image is a value matrix; An image reconstruction module, used to reconstruct the sample image to be reconstructed based on the characteristic matrix of each image in the reconstruction sample set through the image reconstruction model, so as to obtain a reconstruction result of the sample image to be reconstructed; A parameter adjustment module is used to adjust the parameters of the image reconstruction model based on the reconstruction result and the standard image, so as to train the image reconstruction model.

7. A computer device, characterized in that: The computer device includes a processor and a memory, the memory stores at least one computer program, and the at least one computer program is loaded and executed by the processor to implement the image reconstruction model training method as described in any one of claims 1 to 4, and / or the image reconstruction method as described in claim 5.

8. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, which is loaded and executed by a processor to implement the image reconstruction model training method as described in any one of claims 1 to 4, and / or the image reconstruction method as described in claim 5.

Images