Image processing method and device, storage medium, and electronic device

Abstract

The embodiment of the present invention provides an image processing method and device, a storage medium, and an electronic device, the method comprising: dividing the acquired N frames of image into M image blocks, wherein M and N are both natural numbers greater than 1; determining K key image blocks from the M image blocks, wherein K is a natural number less than M; using the K key image blocks and the preset images in the N frames of image, determining the similarity between each image block in the K key image blocks and each image block in the preset image; reconstructing the N frames of image using the similarity to determine the target image. Through the present invention, the problem of low efficiency of super-resolution reconstruction of video in the related art is solved, and the effect of improving the efficiency of super-resolution reconstruction of video is achieved.

Description

Technical Field

[0001] The present invention relates to the field of computers, and in particular to an image processing method and device, a storage medium, and an electronic device. Background Art

[0002] In the prior art, in the process of video super-resolution reconstruction, the entire image is generally divided into multiple image blocks, and all the multiple image blocks are sent to the network model for reconstruction. For example, in a video super-resolution network model based on the Transformer network model, all the multiple image blocks are input to learn the relationship between image position and information in order to reconstruct the image.

[0003] It can be seen that the video super-resolution reconstruction method in the related art not only increases the workload of calculation, but also causes information redundancy, thereby reducing the efficiency of video super-resolution reconstruction. Summary of the invention

[0004] The embodiments of the present invention provide an image processing method and device, a storage medium, and an electronic device to at least solve the problem of low efficiency of super-resolution reconstruction of videos in the related art.

[0005] According to one embodiment of the present invention, there is provided an image processing method, comprising: dividing N acquired image frames into M image blocks, wherein both M and N are natural numbers greater than 1; determining K key image blocks from the M image blocks, wherein K is a natural number less than M; using the K key image blocks and preset images in the N image frames, determining the similarity between each image block in the K key image blocks and each image block in the preset image; and reconstructing the N image frames using the similarities to determine a target image.

[0006] According to another embodiment of the present invention, there is provided an image processing device, comprising: a first division module, used to divide the acquired N frames of images into M image blocks, wherein M and N are both natural numbers greater than 1; a first determination module, used to determine K key image blocks from the M image blocks, wherein K is a natural number less than M; a second determination module, used to determine the similarity between each image block in the K key image blocks and each image block in the preset image by using the K key image blocks and the preset image in the N frames of images; and a first reconstruction module, used to reconstruct the N frames of images by using the similarity to determine the target image.

[0007] In an exemplary embodiment, the first division module includes: a first division unit, configured to divide N frames of image according to a preset size to determine M image blocks, wherein the M image blocks are arranged in a time series.

[0008] In an exemplary embodiment, the above-mentioned first determination module includes: a first calculation unit, used to calculate the similarity between each image block in a preset image and other image blocks; a first determination unit, used to select image blocks ranked in the top K from the calculated similarities to obtain K key image blocks.

[0009] In an exemplary embodiment, the above-mentioned second determination module includes: a first input unit, used to associate each key image block among the K key image blocks and each image block in the preset image and input them into the first target network model to obtain the similarity between each image block among the K key image blocks output by the first target network model and each image block in the preset image, wherein the first target network model includes a convolution layer and a minimum absolute deviation loss function layer.

[0010] In an exemplary embodiment, the above-mentioned device also includes: a third determination module, which is used to associate each key image block in the K key image blocks with each image block in the preset image and input them into the first target network model, and before obtaining the similarity between each image block in the K key image blocks output by the first target network model and each image block in the preset image, the association between each sample key image block in the K sample key image blocks and each sample image block in the sample image is determined as an input parameter; a fourth determination module, which is used to determine the overlap rate between each sample key image block in the K sample key image blocks and each sample image block in the sample image as an output parameter; and a first training module, which is used to train the first network model using the input parameters and the output parameters to obtain a first target network model.

[0011] In an exemplary embodiment, the above-mentioned device also includes: a first input module, which is used to use K key image blocks and preset images in N frames of images to determine the similarity between each image block in the K key image blocks and each image block in the preset image, and then input the K key image blocks and the preset image into the convolution layer in the second target network model to obtain the image features of each key image block output by the second target network model.

[0012] In an exemplary embodiment, the above-mentioned first reconstruction module includes: a second input module, used to input the similarity between each image block of K key image blocks and each image block in a preset image, as well as the image features of each key image block, into a third target network model to obtain a target image output by the third target network model, wherein the target image includes K key image blocks.

[0013] In an exemplary embodiment, the second target network model and the third target network model both include an adversarial loss function layer, a VGG perceptual constraint loss function layer, and a content loss function layer.

[0014] According to yet another embodiment of the present invention, a computer-readable storage medium is provided, in which a computer program is stored, wherein the computer program is configured to execute the steps of any one of the above method embodiments when run.

[0015] According to yet another embodiment of the present invention, there is provided an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0016] Through the present invention, K key image blocks are selected from M image blocks of N frames of images; the similarity between each image block in the K key image blocks and each image block in the preset image is calculated, and the N frames of images are reconstructed using the similarity. Thus, the purpose of selecting the most similar image blocks and performing video super-resolution reconstruction is achieved. The amount of calculation is reduced and the efficiency of video super-resolution reconstruction is improved. Therefore, the problem of low efficiency of video super-resolution reconstruction in the related art can be solved, and the effect of improving the efficiency of video super-resolution reconstruction can be achieved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] Figure 1 is a schematic diagram of the structure of a VSR-Transformer network model according to an embodiment of the present invention;

[0018] Figure 2 is a hardware structure block diagram of a mobile terminal of an image processing method according to an embodiment of the present invention;

[0019] Figure 3 is a flowchart of an image processing method according to an embodiment of the present invention;

[0020] Figure 4 is a schematic diagram of determining a training set according to an embodiment of the present invention;

[0021] Figure 5 2 is a schematic diagram of a network structure for implementing video super-resolution based on a Transformer network model according to an embodiment of the present invention;

[0022] Figure 6 is a structural block diagram of an image processing apparatus according to an embodiment of the present invention. DETAILED DESCRIPTION

[0023] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings and in combination with the embodiments.

[0024] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0025] First, the related technologies involved in the present invention are described:

[0026] The Transformer network model is a network model with a self-attention mechanism, which mainly includes multiple encoder layers and multiple decoder layers; each encoder layer includes a self-attention mechanism layer and a feedforward neural network layer. The self-attention mechanism layer uses the input information vector to generate three vectors, namely, Q vector (query), K vector (primary key), and V vector (value), and calculates the self-attention weight of the information according to the three vectors. The matrix determined based on the self-attention weight is input into the feedforward neural network layer, and the feedforward neural network layer outputs a result with the same dimension as the matrix (sequence length × image block vector). The decoder layer also includes a self-attention mechanism layer and a feedforward neural network layer, which includes calculating the self-attention score of the encoder layer output result. The difference from the encoder layer is that after the self-attention mechanism layer, the self-attention score is calculated again with the output result of the encoder layer after the attention mechanism score is calculated, and then enters the feedforward neural network layer.

[0027] The VSR-Transformer network model is a video super-resolution network model based on Transformer. Figure 1 As shown in the figure, it includes feature extraction, Transformer encoder and reconstruction network. Specifically, given a video sequence, a stacked residual module is used to extract image features from the video sequence; then VSR-Transformer encodes the image features into a continuous expression sequence; finally, the reconstruction network performs high-resolution video reconstruction from the expression sequence.

[0028] The method embodiments provided in the embodiments of the present application can be executed in a mobile terminal, a computer terminal or a similar computing device. Taking running on a mobile terminal as an example, Figure 2 FIG. 1 is a hardware structure block diagram of a mobile terminal of an image processing method according to an embodiment of the present invention. Figure 2 As shown, the mobile terminal may include one or more ( Figure 2 Only one is shown in the figure) a processor 202 (the processor 202 may include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 204 for storing data, wherein the mobile terminal may also include a transmission device 206 and an input / output device 208 for communication functions. It can be understood by those skilled in the art that Figure 2 The structure shown is only for illustration and does not limit the structure of the mobile terminal. Figure 2 More or fewer components as shown, or with Figure 2 Different configurations shown.

[0029] The memory 204 can be used to store computer programs, for example, software programs and modules of application software, such as the computer program corresponding to the image processing method in the embodiment of the present invention. The processor 202 executes various functional applications and data processing by running the computer program stored in the memory 204, that is, to implement the above method. The memory 204 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 204 may further include a memory remotely arranged relative to the processor 202, and these remote memories may be connected to the mobile terminal via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0030] The transmission device 206 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the mobile terminal. In one example, the transmission device 206 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 206 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0031] This embodiment provides an image processing method. Figure 3 is a flowchart of an image processing method according to an embodiment of the present invention. Figure 3 As shown, the process includes the following steps:

[0032] Step S302, dividing the acquired N frames of image into M image blocks, where M and N are both natural numbers greater than 1;

[0033] Step S304, determining K key image blocks from the M image blocks, where K is a natural number less than M;

[0034] Step S306, using the K key image blocks and the preset image in the N frames of image, determining the similarity between each image block in the K key image blocks and each image block in the preset image;

[0035] Step S308, reconstructing N frames of images using similarity to determine the target image.

[0036] In this embodiment, the values ​​of N, M and K can be flexibly set based on the actual reference scene or image requirements. For example, 3 frames of images can be divided into 45 image blocks, and 9 key image blocks are selected from the 45 image blocks. The similarity between the 9 key image blocks and the image blocks in the preset image is calculated.

[0037] The execution subject of the above steps may be a terminal, a server, a specific processor provided in the terminal or the server, or a processor or processing device provided relatively independently from the terminal or the server, but is not limited thereto.

[0038] Through the above steps, K key image blocks are selected from M image blocks of N frames of images; the similarity between each image block in the K key image blocks and each image block in the preset image is calculated, and the N frames of images are reconstructed using the similarity. Thus, the purpose of selecting the most similar image blocks and performing video super-resolution reconstruction is achieved. The amount of calculation is reduced and the efficiency of video super-resolution reconstruction is improved. Therefore, the problem of low efficiency of video super-resolution reconstruction in the related art can be solved, and the effect of improving the efficiency of video super-resolution reconstruction can be achieved.

[0039] In an exemplary embodiment, the acquired N frames of images are divided into M image blocks, including:

[0040] S10, dividing N frames of image according to a preset size to determine M image blocks, wherein the M image blocks are arranged in a time series.

[0041] In this embodiment, the values ​​of N and M are not limited and can be selected based on actual conditions. Here, N is 3 and M is 45 as an example for explanation. The three frames of images are represented as a sequence t0, t1 and t2 according to the shooting time, and the three frames of images are divided into square or rectangular image blocks with a side length of w (w includes but is not limited to any natural number greater than 3, such as 64). Use p(ti, j) to represent the image block, where ti is used to represent the shooting time of the image block, the above i includes 0, 1, 2, and the above j is used to represent the position of the image block from top to bottom and from left to right. For example, the first image block in the first row from the left of t1 is represented as p(t1, 0). Through the above division method, the three frames of images can be divided into 45 image blocks. It should be noted that the above division method is only an example. In actual applications, other division methods can also be used, for example, 3 frames of images are divided into 27 image blocks. For example, 5 frames of images are divided.

[0042] In an exemplary embodiment, determining K key image blocks from M image blocks includes:

[0043] S11, calculating the similarity between each image block in the preset image and other image blocks, where the other image blocks include image blocks other than the image blocks in the preset image among the M image blocks;

[0044] S12, selecting the top K image blocks from the calculated similarities to obtain K key image blocks.

[0045] In this embodiment, the preset image includes but is not limited to an image in N frames of images. For example, taking N as 3 as an example, the second frame of the three frames of images is determined as the preset image (it should be noted that this determination method is only an example. In actual applications, the first frame of the image can also be determined as the preset image, or the third frame of the image can be determined as the preset image). The second frame of the image is represented as t1 according to the time sequence of shooting, and the image block in the second frame of the image is represented as p(t1, j), where j=0, 1, 2, 3, 4... Take p(t1, j) as the image block ref of the preset image, and calculate the similarity between ref and other image blocks respectively. Select the image blocks ranked in the top K from the calculated similarities to determine K key image blocks.

[0046] In an exemplary embodiment, using K key image blocks and preset images in N frames of images, determining the similarity between each image block in the K key image blocks and each image block in the preset image includes:

[0047] S21, associating each key image block in the K key image blocks and each image block in the preset image and inputting them into the first target network model to obtain the similarity between each image block in the K key image blocks output by the first target network model and each image block in the preset image, wherein the first target network model includes a convolution layer and a minimum absolute deviation loss function layer.

[0048] In this embodiment, the first target network model includes but is not limited to a convolutional neural network and a recursive neural network, wherein the convolution layer in the first target network model is used to output image features in each image block, and the minimum absolute value deviation loss function layer is used to detect the accuracy of the output of the first target network model.

[0049] In this embodiment, for example, each key image block is connected in series with each image block in the preset image as the input of the first network model, and the output of the first network model includes the similarity probability between each key image block and each image block in the preset image. The value range of the similarity probability is between 0 and 1.

[0050] Furthermore, each key image block is connected in series with each image block in the preset image to include [K+1, w, w], and the output includes [1, K+1], where [w, w] is used to represent the length and width of each key image block. This achieves the purpose of further learning a similarity probability value for the filtered image blocks.

[0051] In this embodiment, the first target network model may be a model in the encoder layer of the Transformer network model, thereby realizing the function of learning the features of Q*K. Figure 5As shown, each key image block in the extracted K key image blocks and each image block in the preset image are associated and input into the first target network model in the encoder layer of the Transformer network model to learn the features of Q*K. Compared with the VSR-Transformer network model in the related art, all image blocks are input to learn the features of Q*K. The amount of input parameters is reduced, thereby reducing the amount of calculation.

[0052] In an exemplary embodiment, before associating each key image block in the K key image blocks and each image block in the preset image and inputting them into the first target network model, and obtaining the similarity between each image block in the K key image blocks output by the first target network model and each image block in the preset image, the method further includes:

[0053] S22, determining the association between each sample key image block in the K sample key image blocks and each sample image block in the sample image as an input parameter;

[0054] S23, determining the overlap ratio between each sample key image block in the K sample key image blocks and each sample image block in the sample image as an output parameter;

[0055] S24, training the first network model using the input parameters and the output parameters to obtain a first target network model.

[0056] In this embodiment, for example, the sample image block may be determined in the following manner: randomly cutting out a w*w image block as the sample image block.

[0057] The sample key image block can be determined in the following manner: within the range of w*w, K seed points are randomly selected, and K w*w sample key image blocks are cut out using the K seed points as center points.

[0058] The overlap rate can be determined by dividing the overlap area of ​​each sample image block and the sample key image block by w*w to determine the overlap rate. Figure 4 As shown, the black frame is a sample image block, and the gray frame is a sample key image block among K sample key image blocks; the area of ​​the intersection region divided by the total area of ​​the black frame is the overlap rate.

[0059] In this embodiment, the loss function for training the first network model is a minimum absolute deviation loss function.

[0060] In an exemplary embodiment, after determining the similarity between each image block in the K key image blocks and each image block in the preset image using the K key image blocks and the preset image in the N frames of image, the method further includes:

[0061] S31, inputting K key image blocks and a preset image into a second target network model to obtain image features of each key image block output by the second target network model.

[0062] In this embodiment, the second target network model includes but is not limited to a convolutional neural network and a recursive neural network. The second target network model and the first target network model may be the same network model or different network models. The second target network model includes at least a convolutional layer, an adversarial loss function layer, a VGG perceptual constraint loss function layer, and a content loss function layer, wherein the convolutional layer is used to output the image features of each key image block; the adversarial loss function layer, the VGG perceptual constraint loss function layer, and the content loss function layer can all be used to detect the accuracy of the output of the second target network model.

[0063] In this embodiment, for example, the output of the second target network model includes [K+1, w, w], where [w, w] is used to represent the length and width of each key image block.

[0064] In this embodiment, the second target network model may be a model in the encoder layer of the Transformer network model, thereby realizing the function of learning V features. Figure 5 As shown, K key image blocks and preset images are input into the second target network model in the encoder layer of the Transformer network model to learn V features, wherein the second target network model can be a V feature extraction network.

[0065] In an exemplary embodiment, reconstructing N frames of images using similarity to determine a target image includes:

[0066] S41, inputting the similarity between each image block in the K key image blocks and each image block in the preset image, as well as the image features of each key image block, into the third target network model to obtain a target image output by the third target network model, wherein the target image includes K key image blocks.

[0067] In this embodiment, the third target network model includes but is not limited to a convolutional neural network and a recursive neural network. The third target network model and the second target network model may be the same network model or different network models. For example, the following example takes the first target network model as CNN_1, the second target network model as CNN_2, and the third target network model as CNN_3, and all three target network models are included in the Transformer network model. Figure 5As shown, it is a schematic diagram of the network structure for realizing video super-resolution based on the Transformer network model. The network structure includes three functions: feature extraction, Transformer encoder and reconstruction network. Among them, CNN_1 and CNN_2 are included in the encoder layer of the Transformer network model, which are used to realize the function of learning Q, K, and V in the Transformer encoder. CNN_3 is included in the reconstruction network in the Transformer network model, which is used to realize the function of the reconstruction network.

[0068] In addition, the training process of CNN_1, CNN_2 and CNN_3 in this embodiment includes but is not limited to: training CNN_1; fixing CNN_1 to train CNN_2 and CNN_3; training CNN_1, CNN_2 and CNN_3.

[0069] In this embodiment, the third target network model includes at least n Resblocks and a deconvolution upsampling convolution layer, an adversarial loss function layer, a VGG (Visual Geometry Group Network) perceptual constraint loss function layer and a content loss function layer, wherein the convolution layer is used to output image features of each key image block; the adversarial loss function layer, the VGG perceptual constraint loss function layer and the content loss function layer can all be used to detect the accuracy of the output of the second target network model.

[0070] In this embodiment, for example, the output of the first target network model is multiplied by the output of the second target network model and input into the third target network model; the dimensions include [K+1, w, w].

[0071] In summary, this embodiment can perform video super-resolution reconstruction more efficiently and obtain better video super-resolution effect.

[0072] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, a magnetic disk, or an optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the methods described in each embodiment of the present invention.

[0073] In this embodiment, an image processing device is also provided, which is used to implement the above-mentioned embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can implement a combination of software and / or hardware of a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0074] Figure 6 is a structural block diagram of an image processing device according to an embodiment of the present invention. Figure 6 As shown, the device comprises:

[0075] A first division module 62, configured to divide the acquired N frames of image into M image blocks, wherein M and N are both natural numbers greater than 1;

[0076] A first determination module 64 is used to determine K key image blocks from M image blocks, where K is a natural number less than M;

[0077] A second determination module 66 is used to determine the similarity between each image block in the K key image blocks and each image block in the preset image by using the K key image blocks and the preset image in the N frames of images;

[0078] The first reconstruction module 68 is used to reconstruct N frames of images using similarity to determine a target image.

[0079] In an exemplary embodiment, the first division module 62 includes:

[0080] The first division unit is used to divide N frames of image according to a preset size to determine M image blocks, wherein the M image blocks are arranged in a time series.

[0081] In an exemplary embodiment, the first determining module 64 includes:

[0082] A first calculation unit, used to calculate the similarity between each image block in a preset image and other image blocks;

[0083] The first determining unit is used to select the top K image blocks from the calculated similarities to obtain K key image blocks.

[0084] In an exemplary embodiment, the second determining module 66 includes:

[0085] The first input unit is used to associate each key image block in the K key image blocks with each image block in the preset image and input them into the first target network model to obtain the similarity between each image block in the K key image blocks output by the first target network model and each image block in the preset image, wherein the first target network model includes a convolution layer and a minimum absolute deviation loss function layer.

[0086] In an exemplary embodiment, the above device further comprises:

[0087] A third determination module is used to determine the association between each sample key image block in the K sample key image blocks and each sample image block in the sample image as an input parameter before associating each key image block in the K key image blocks and each image block in the preset image and inputting each key image block in the K key image blocks and each image block in the preset image into the first target network model to obtain the similarity between each image block in the K key image blocks output by the first target network model and each image block in the preset image;

[0088] a fourth determination module, configured to determine the coincidence rate between each sample key image block in the K sample key image blocks and each sample image block in the sample image as an output parameter;

[0089] The first training module is used to train the first network model using the input parameters and the output parameters to obtain a first target network model.

[0090] In an exemplary embodiment, the above device further comprises:

[0091] The first input module is used to input the K key image blocks and the preset images into the convolution layer in the second target network model after determining the similarity between each image block in the K key image blocks and each image block in the preset images using the K key image blocks and the preset images in the N frames of images, so as to obtain the image features of each key image block output by the second target network model.

[0092] In an exemplary embodiment, the first reconstruction module 68 comprises:

[0093] The second input module is used to input the similarity between each image block in the K key image blocks and each image block in the preset image, as well as the image features of each key image block, into the third target network model to obtain a target image output by the third target network model, wherein the target image includes K key image blocks.

[0094] In an exemplary embodiment, the second target network model and the third target network model both include an adversarial loss function layer, a VGG perceptual constraint loss function layer, and a content loss function layer.

[0095] It should be noted that the above modules can be implemented by software or hardware. For the latter, it can be implemented in the following ways, but not limited to: the above modules are all located in the same processor; or the above modules are located in different processors in any combination.

[0096] An embodiment of the present invention further provides a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the steps of any of the above method embodiments when running.

[0097] In this embodiment, the computer-readable storage medium may be configured to store a computer program for executing the above steps.

[0098] In an exemplary embodiment, the computer-readable storage medium may include, but is not limited to, various media that can store computer programs, such as a USB flash drive, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk.

[0099] An embodiment of the present invention further provides an electronic device, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0100] In an exemplary embodiment, the electronic device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0101] In an exemplary embodiment, the processor may be configured to execute the above steps through a computer program.

[0102] For specific examples in this embodiment, reference may be made to the examples described in the above embodiments and exemplary implementation modes, and this embodiment will not be described in detail herein.

[0103] Obviously, those skilled in the art should understand that the above modules or steps of the present invention can be implemented by a general computing device, they can be concentrated on a single computing device, or distributed on a network composed of multiple computing devices, they can be implemented by a program code executable by a computing device, so that they can be stored in a storage device and executed by the computing device, and in some cases, the steps shown or described can be executed in a different order than here, or they can be made into individual integrated circuit modules, or multiple modules or steps therein can be made into a single integrated circuit module for implementation. Thus, the present invention is not limited to any specific combination of hardware and software.

[0104] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, the present invention may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. An image processing method, characterized in that: include: Dividing the acquired N frames of image into M image blocks, wherein both M and N are natural numbers greater than 1; Determine K key image blocks from the M image blocks, where K is a natural number smaller than M; Determine the similarity between each image block in the K key image blocks and each image block in the preset image by using the K key image blocks and the preset image in the N frames of image; Reconstructing the N frames of images using the similarity to determine a target image; Among them, the N frames of images are reconstructed using the similarity, and determining the target image includes: inputting the similarity between each image block in the K key image blocks and each image block in the preset image, as well as the image features of each key image block, into the third target network model to obtain the target image output by the third target network model, wherein the target image includes the K key image blocks, and the third target network model includes at least n Resblocks and a deconvolution upsampling convolution layer, an adversarial loss function layer, a VGG perceptual constraint loss function layer, and a content loss function layer, the convolution layer is used to output the image features of each of the key image blocks, and the adversarial loss function layer, the VGG perceptual constraint loss function layer, and the content loss function layer are all used to detect the accuracy of the output of the second target network model.

2. The method according to claim 1, characterized in that The acquired N frames of images are divided into M image blocks, including: The N frames of image are divided according to a preset size to determine the M image blocks, wherein the M image blocks are arranged in a time series.

3. The method according to claim 1, characterized in that Determining K key image blocks from the M image blocks includes: Calculating the similarity between each image block and other image blocks in the preset image; The top K image blocks are selected from the calculated similarities to obtain the K key image blocks.

4. The method according to claim 1, characterized in that: Determining the similarity between each image block in the K key image blocks and each image block in the preset image by using the K key image blocks and the preset image in the N frames of image includes: Associating each of the K key image blocks and each image block in the preset image into a first target network model, and obtaining the similarity between each of the K key image blocks output by the first target network model and each image block in the preset image, wherein the first target network model includes a convolutional layer and a minimum absolute deviation loss function layer.

5. The method according to claim 4, characterized in that Before associating each key image block in the K key image blocks and each image block in the preset image and inputting them into the first target network model to obtain the similarity between each image block in the K key image blocks and each image block in the preset image output by the first target network model, the method further includes: Determine the association between each sample key image block in the K sample key image blocks and each sample image block in the sample image as an input parameter; Determine the overlap rate between each sample key image block in the K sample key image blocks and each sample image block in the sample image as an output parameter; The first network model is trained using the input parameters and the output parameters to obtain the first target network model.

6. The method according to claim 1 or 4, characterized in that: After determining the similarity between each image block in the K key image blocks and each image block in the preset image by using the K key image blocks and the preset image in the N frames of image, the method further includes: The K key image blocks and the preset image are input into a second target network model to obtain image features of each key image block output by the second target network model.

7. The method according to claim 1, characterized in that Both the second target network model and the third target network model include an adversarial loss function layer, a VGG perceptual constraint loss function layer and a content loss function layer.

8. An image processing device, characterized in that: include: A first division module, used for dividing the acquired N frames of image into M image blocks, wherein both M and N are natural numbers greater than 1; A first determination module, configured to determine K key image blocks from the M image blocks, wherein K is a natural number smaller than M; A second determination module is used to determine the similarity between each image block in the K key image blocks and each image block in the preset image by using the K key image blocks and the preset image in the N frames of images; A first reconstruction module, used to reconstruct the N frames of images using the similarity to determine a target image; Among them, the first reconstruction module is also used to input the similarity between each image block in the K key image blocks and each image block in the preset image, as well as the image features of each key image block, into the third target network model to obtain the target image output by the third target network model, wherein the target image includes the K key image blocks, and the third target network model includes at least n Resblocks and a deconvolution upsampling convolution layer, an adversarial loss function layer, a VGG perceptual constraint loss function layer and a content loss function layer, the convolution layer is used to output the image features of each of the key image blocks, and the adversarial loss function layer, the VGG perceptual constraint loss function layer and the content loss function layer are all used to detect the accuracy of the output of the second target network model.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program implements the method described in any one of claims 1 to 7 when executed by a processor.

10. An electronic device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to perform the method according to any one of claims 1 to 7.

Images