Data processing method and apparatus, terminal device, and computer-readable storage medium

Abstract

The application is suitable for the technical field of data processing, and provides a data processing method and device, terminal equipment and computer readable storage medium. The method comprises the following steps: obtaining to-be-processed data, wherein the to-be-processed data comprises multiple images; performing deblurring processing, down-sampling processing and feature extraction processing on the to-be-processed data to obtain first feature data; performing feature alignment processing and feature fusion processing on the first feature data to obtain second feature data; performing feature reconstruction processing and up-sampling processing on the second feature data to obtain a reconstruction up-sampling result; and performing superposition processing on the reconstruction up-sampling result and a reference image in the multiple images to obtain a de-compression blurred image. The application can reduce the calculation amount in the de-compression blurred processing process and the noise of the de-compression blurred image, make the texture structure of the de-compression blurred image more accurate, and improve the de-compression blurred processing efficiency of the image data.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of data processing, and particularly relates to a data processing method and device, a terminal device and a computer readable storage medium. BACKGROUND

[0002] With the development of science and technology, image and video data have gradually become the most important information transmission carrier in people's life.

[0003] In the process of image and video transmission, the storage space occupied by the image and video is large due to the large amount of information. Therefore, people usually compress the image or video data during transmission to reduce the data transmission cost. However, the compression process will bring certain compression noise to the image and video data, and the clarity of the compressed image and video will decrease.

[0004] The related image and video optimization processing method has the problems of large amount of calculation, low processing efficiency, large noise of processed video and image data, and inaccurate texture structure. SUMMARY

[0005] The embodiments of the present application provide a data processing method and device, a terminal device and a computer readable storage medium, which can solve the problems of large amount of calculation, low processing efficiency, large noise of processed video and image data, and inaccurate texture structure of the related image and video optimization processing method.

[0006] In a first aspect, the embodiments of the present application provide a data processing method, comprising:

[0007] obtaining to-be-processed data, the to-be-processed data comprising a plurality of images compressed from a plurality of original images with the same image size by using different image size compression ratios;

[0008] performing deblurring processing, downsampling processing and feature extraction processing on the to-be-processed data to obtain first feature data;

[0009] performing feature alignment processing and feature fusion processing on the first feature data to obtain second feature data;

[0010] performing feature reconstruction processing and upsampling processing on the second feature data to obtain a reconstruction and upsampling result;

[0011] superimposing the reconstruction and upsampling result and a reference image in the plurality of images to obtain a de-compressed blurred image.

[0012] In a second aspect, the embodiments of the present application provide a data processing device, comprising:

[0013] The first acquisition module is used to acquire data to be processed, which includes multiple frames of images. The multiple frames of images are obtained by compressing multiple original images of the same size using different image size compression ratios.

[0014] The extraction module is used to perform deblurring, downsampling, and feature extraction on the data to be processed to obtain the first feature data;

[0015] The fusion processing module is used to perform feature alignment and feature fusion processing on the first feature data to obtain the second feature data.

[0016] The reconstruction module is used to perform feature reconstruction and upsampling on the second feature data to obtain the reconstruction and upsampling results.

[0017] The overlay processing module is used to overlay the reconstructed upsampling results with reference images from multiple frames to obtain a decompressed and blurred image.

[0018] Thirdly, embodiments of this application provide a terminal device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the data processing method as described in any of the first aspects above.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the data processing method as described in any of the first aspects above.

[0020] Fifthly, embodiments of this application provide a computer program product that, when run on a terminal device, causes the terminal device to execute any of the data processing methods described in the first aspect above.

[0021] This application embodiment obtains the second feature data corresponding to the data to be processed through multiple feature extractions, and performs feature reconstruction and upsampling processing on the second feature data to obtain the reconstruction upsampling result. The reconstruction upsampling result is superimposed on the reference image in the data to be processed to obtain the decompressed blurred image. This reduces the amount of computation in the decompressed blur processing process and the noise in the decompressed blurred image, making the texture structure of the decompressed blurred image more accurate and improving the efficiency of image data decompressed blur processing.

[0022] It is understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0023] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0024] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application;

[0025] Figure 2 This is a schematic diagram of a decompressed and blurred image using the data processing method provided in this application embodiment;

[0026] Figure 3 This is a flowchart illustrating step S102 of the data processing method provided in this application embodiment;

[0027] Figure 4 This is a schematic diagram of the structure of a trained hierarchical attention network model provided in an embodiment of this application;

[0028] Figure 5 This is another schematic diagram of a decompressed and blurred image obtained based on a trained hierarchical attention network model or EDVR model, as provided in the embodiments of this application.

[0029] Figure 6 This is a flowchart illustrating step S105 of the data processing method provided in this application embodiment;

[0030] Figure 7 This is a schematic diagram of the structure of the data processing apparatus provided in the embodiments of this application;

[0031] Figure 8 This is a schematic diagram of the structure of the terminal device provided in the embodiments of this application. Detailed Implementation

[0032] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0033] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0034] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0035] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0036] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0037] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0038] The data processing method provided in this application can be applied to terminal devices such as mobile phones, tablets, and laptops. This application does not impose any restrictions on the specific type of terminal device.

[0039] Figure 1 A schematic flowchart of the data processing method provided in this application is shown. It is provided as an example and not as a limitation. The method can be applied to the aforementioned laptop computer.

[0040] S101. Obtain the data to be processed, which includes multiple frames of images. The multiple frames of images are obtained by compressing multiple original images of the same size using different image size compression ratios.

[0041] Among them, multiple frames can be a number of consecutive images in a video, and there is no limitation here.

[0042] Specifically, the data to be processed is acquired. This data is not limited to multiple frames of images.

[0043] For example, the data to be processed can be set as a combination of multiple sets of 5 consecutive images. Using multiple frames as input data can reduce image flickering problems that occur during the actual deblurring process of data containing multiple frames.

[0044] S102. Perform deblurring, downsampling, and feature extraction on the data to be processed to obtain the first feature data.

[0045] Specifically, the data to be processed is preprocessed to extract the first feature data. Preprocessing methods include, but are not limited to, deblurring, downsampling, and feature extraction.

[0046] S103. Perform feature alignment and feature fusion processing on the first feature data to obtain the second feature data.

[0047] Specifically, the first feature data of the extracted data to be processed is subjected to feature alignment processing, and the aligned first feature data is subjected to feature fusion processing to obtain the fused second feature data.

[0048] S104. Perform feature reconstruction and upsampling processing on the second feature data to obtain the reconstruction and upsampling results.

[0049] Specifically, the second feature data is reconstructed and upsampled using a trained network model (i.e., the fused feature data is further optimized) to obtain the corresponding reconstruction and upsampling results. The network model mentioned above includes, but is not limited to, a hierarchical attention network model.

[0050] By using feature reconstruction and upsampling, problems such as texture detail distortion and inaccuracy that occur after deblurring multi-frame image data can be reduced.

[0051] S105. The reconstructed upsampling result is superimposed with the reference image in the multi-frame image to obtain the decompressed and blurred image.

[0052] Specifically, the reconstructed upsampling result output by the aforementioned network model is superimposed with a reference image from multiple frames. This process fuses the reconstructed upsampling result with the reference image from the multiple frames, improving the clarity of the image data and resulting in a decompressed and blurred image. The reference image refers to a frame of image data selected from multiple frames that meets preset conditions. The preset conditions can be specifically set according to actual needs.

[0053] This embodiment obtains the second feature data of the data to be processed through multiple feature extractions, and performs feature reconstruction and upsampling processing on the second feature data to obtain the reconstruction and upsampling result. The reconstruction and upsampling result is superimposed on the reference image in the data to be processed to obtain the decompressed and blurred image. This reduces the amount of computation in the decompressed and blurred processing process and the noise of the decompressed and blurred image, making the texture structure of the decompressed and blurred image more accurate and improving the efficiency of image data decompressed and blurred processing.

[0054] like Figure 2 The diagram illustrates a data processing method for decompressing and blurring images.

[0055] like Figure 3 As shown, in one embodiment, step S102 includes:

[0056] S1021. Input the data to be processed into the deblurring network for deblurring processing, and output the first feature information;

[0057] The deblurring network stores the deblurring algorithm, which is used to process the data to be processed to obtain the first feature information. The deblurring algorithm can be DeblurGAN, SRN-DeblurNet, DeblurGAN-v2, EDVR, PSS-NSC, etc.

[0058] S1022. Perform N-fold downsampling on the first feature information to obtain downsampled feature information, where N is an integer greater than 1;

[0059] S1023. The downsampled feature information is processed by multiple stacked residual modules to obtain the first feature data.

[0060] Specifically, the data to be processed is input into a deblurring network, where it is deblurred, and first feature information is output. This first feature information is then downsampled by a factor of N to obtain downsampled feature information. Furthermore, multiple stacked residual modules are used to extract features from this downsampled feature information, resulting in the first feature data of the data to be processed. The resolution of the first feature information is consistent with the resolution of the data to be processed.

[0061] The network structure of the deblurring network model is set to be the same as that of the deblurring module in the deformable convolution-based video restoration, deblurring, and super-resolution network model (EDVR), with 128 feature channels. The downsampling factor N and the number of stacked residual modules can be set according to actual needs, where N is an integer greater than 1.

[0062] In this embodiment, to improve feature extraction efficiency and reduce the computational load of the model, the downsampling factor N can be set to 4. The number of stacked residual modules is 5.

[0063] By employing multiple feature extraction processes, the computational load for deblurring multiple frames of images is reduced, thereby improving processing efficiency.

[0064] In one embodiment, step S103 includes:

[0065] The first feature data is aligned using a pyramid cascade and deformable modules to obtain the second feature information.

[0066] The second feature information is processed by the feature fusion module to obtain the second feature data.

[0067] Specifically, the first feature data is aligned using a pyramid cascading and deformable convolution (PCD) module to obtain the second feature information. The second feature information is then fused using a feature fusion module to obtain fused second feature data, thereby improving the accuracy of texture structure in the decompressed and blurred image. The structure of the aforementioned deformable module is identical to that of the feature alignment module in the video restoration, deblurring, and super-resolution network model based on deformable convolution.

[0068] In one embodiment, feature reconstruction and upsampling are performed on the second feature data to obtain a reconstructed upsampling result, including:

[0069] The second feature data is input into the trained hierarchical attention network model for feature reconstruction, and the reconstruction result is output.

[0070] The reconstruction results are upsampled N times to obtain the reconstructed upsampled results.

[0071] Specifically, when performing feature reconstruction processing on the second feature data using the trained hierarchical attention network model, the second feature data is optimized to make it closer to the feature data of image frames with closer adjacent distances between multi-frame images and intermediate frames, thereby obtaining the output reconstruction result. This further reduces the flickering problem that occurs during the deblurring process. The reconstruction result is then upsampled N times again to obtain the reconstruction upsampled result, ensuring that the resolution of the upsampled reconstruction result is consistent with the resolution of the input data to be processed.

[0072] In one embodiment, the trained hierarchical attention network model includes a hierarchical convolutional neural network model and a channel attention mechanism model.

[0073] Specifically, the trained hierarchical attention network models include hierarchical convolutional neural network models and channel attention mechanism models. The L1 function is used as the loss function in the trained hierarchical attention network models, and the Adam optimizer is used to optimize the regression network.

[0074] Specifically, the loss function L1 in the trained hierarchical attention network model can be expressed as:

[0075]

[0076] Where p represents the output of the trained hierarchical attention network model; y represents the input data of the trained hierarchical attention network model. H represents the height of the output, W represents the width of the output, and C represents the number of channels of the trained hierarchical attention network model.

[0077] like Figure 4 As shown, a schematic diagram of a trained hierarchical attention network model is provided.

[0078] Figure 4 The provided trained hierarchical attention network model is a model consisting of one convolutional neural network model sequentially connected to one hierarchical convolutional neural network model, one channel attention mechanism model, and another convolutional neural network model. The channel attention mechanism model includes an average pooling layer, a linear sigmoid activation function, a rectified linear unit (ReLU) function, and a feature fusion function (concat).

[0079] In other words, the structure of the trained hierarchical attention network model is similar to that of the EDVR (Earthquake Dynamics Video Restoration, Deblurring, and Super-Resolution Network) model, which is based on deformable convolutions. Most of the module parameters are the same as those in the EDVR model. The difference lies in the fact that the reconstruction module in the trained hierarchical attention network model consists of residual blocks constructed from hierarchical convolutions and channel attention mechanisms stacked 24 times. This reduces the number of parameters in the model, thereby reducing its computational cost.

[0080] Table 1

[0081]

[0082]

[0083] Table 1 provides a comparison of the deblurring performance based on a trained hierarchical attention network model or an EDVR model. Specifically, using the Reds training dataset as the training dataset and the Reds test dataset as the test dataset, the table compares the performance of generating 1280x720px video frames based on the trained hierarchical attention network model and the EDVR model. As shown in Table 1, when the number of parameters in the trained hierarchical attention network model is smaller than that of the EDVR model, the trained hierarchical attention network model achieves better deblurring performance and faster inference speed.

[0084] like Figure 5 As shown, another schematic diagram is provided for obtaining a decompressed and blurred image based on a trained hierarchical attention network model or an EDVR model.

[0085] like Figure 6 As shown, in one embodiment, step S105 includes:

[0086] S1051. Select a reference image that meets the preset conditions from multiple frames of images;

[0087] S1052. The reconstructed upsampling result is superimposed on the reference image to obtain a decompressed and blurred image.

[0088] Specifically, a reference image that meets preset conditions is selected from multiple frames of images, and the reconstructed upsampling result is superimposed on the reference image to obtain a decompressed and blurred image.

[0089] Specifically, the preset conditions can be set according to the actual situation. For example, the middle frame image in a multi-frame image can be set as the reference image; or the first frame image in a multi-frame image can be set as the reference image.

[0090] In one embodiment, before acquiring the data to be processed, the above method further includes:

[0091] Acquire multiple blurred images and multiple sharp images, with each blurred image and the sharp image corresponding to a specific one;

[0092] Multiple blurred images are divided into multiple training data and test datasets. Multiple training data are compressed using different image size compression ratios to obtain training datasets. Each training dataset contains multiple frames of continuous blurred images.

[0093] The untrained hierarchical attention network model is trained using the training dataset, the test dataset, and multiple clear images to obtain the trained hierarchical attention network model.

[0094] Specifically, multiple blurred image data and corresponding multiple sharp image data are acquired. The blurred images are divided into multiple training and test datasets (each test dataset contains multiple test datasets, each containing multiple frames of consecutive blurred images, and each training dataset also contains multiple frames of consecutive blurred images). (The number of frames in the training dataset is determined based on the number of frames in the image data to be processed; for example, if the image to be processed consists of 5 consecutive frames, each training dataset contains 5 consecutive blurred images). Based on different compression ratios for the multiple image sizes, the training datasets are compressed using multimedia processing tools (such as ffmpeg) to obtain the corresponding training datasets. Using the multiple sharp image data as labels, the hierarchical attention network model is trained using the training and test datasets to obtain the trained hierarchical attention network model. Sharp images refer to high-resolution image data that has not undergone blurring or compression. Blurred images refer to low-resolution image data after fusion processing.

[0095] Specifically, during training, the intermediate blurred image in a series of consecutive blurred images is used as a reference blurred image. The reference blurred image is then cropped, and the cropped reference blurred image is used as input data to the hierarchical attention network model. The clear image corresponding to the reference blurred image is used as a label to train the hierarchical attention network model.

[0096] This embodiment trains a training dataset after compressing blurred images with multiple image size compression ratios. The resulting trained hierarchical attention network model improves the generalization performance of the trained hierarchical attention network model, enabling it to eliminate compression noise caused by different image size compression ratios.

[0097] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0098] Corresponding to the data processing method in the above embodiments, Figure 7 A structural block diagram of a data processing apparatus provided in an embodiment of this application is shown. For ease of explanation, only the parts related to the embodiments of this application are shown.

[0099] Reference Figure 7 The data processing device 100 includes:

[0100] The first acquisition module 101 is used to acquire data to be processed, which includes multiple frames of images. The multiple frames of images are obtained by compressing multiple original images of the same size using different image size compression ratios.

[0101] Extraction module 102 is used to perform deblurring, downsampling and feature extraction on the data to be processed to obtain the first feature data;

[0102] The fusion processing module 103 is used to perform feature alignment processing and feature fusion processing on the first feature data to obtain the second feature data;

[0103] Reconstruction module 104 is used to perform feature reconstruction and upsampling processing on the second feature data to obtain reconstruction and upsampling results;

[0104] The overlay processing module 105 is used to overlay the reconstructed upsampling result with the reference image in the multi-frame image to obtain a decompressed and blurred image.

[0105] In one embodiment, the extraction module 102 includes:

[0106] The deblurring unit is used to input the data to be processed into the deblurring network for deblurring processing and output the first feature information.

[0107] The first downsampling processing unit is used to perform N-fold downsampling processing on the first feature information to obtain downsampled feature information, where N is an integer greater than 1;

[0108] The feature extraction unit is used to perform feature extraction processing on the downsampled feature information through multiple stacked residual modules to obtain the first feature data.

[0109] In one embodiment, the fusion processing module 103 includes:

[0110] The feature alignment processing unit is used to perform feature alignment processing on the first feature data through pyramid cascading and deformable modules to obtain the second feature information;

[0111] The feature fusion processing unit is used to perform feature fusion processing on the second feature information through the feature fusion module to obtain the second feature data.

[0112] In one embodiment, the reconstruction module 104 includes:

[0113] The feature reconstruction processing unit is used to input the second feature data into the trained hierarchical attention network model for feature reconstruction processing and output the reconstruction result.

[0114] The second downsampling processing unit is used to perform N-fold upsampling on the reconstruction result to obtain the reconstruction upsampling result.

[0115] In one embodiment, the trained hierarchical attention network model includes a hierarchical convolutional neural network model and a channel attention mechanism model.

[0116] In one embodiment, the overlay processing module 105 includes:

[0117] The selection unit is used to select a reference image that meets preset conditions from multiple frames of images;

[0118] The overlay processing unit is used to overlay the reconstructed upsampling result with the reference image to obtain a decompressed and blurred image.

[0119] In one embodiment, the data processing apparatus further includes:

[0120] The acquisition module is used to acquire multiple blurred images and multiple clear images, with each blurred image and the clear image corresponding to a specific one.

[0121] The compression processing module is used to divide multiple blurred images into multiple sets of training data and test datasets. It compresses the multiple sets of training data using multiple image size compression ratios with different compression ratios to obtain the training dataset. Each set of training data contains multiple frames of continuous blurred images.

[0122] The training module is used to train an untrained hierarchical attention network model based on the training dataset, the test dataset, and multiple clear images, to obtain a trained hierarchical attention network model.

[0123] This embodiment obtains the second feature data corresponding to the data to be processed through multiple feature extractions, and performs feature reconstruction and upsampling processing on the second feature data to obtain the reconstruction upsampling result. The reconstruction upsampling result is superimposed on the reference image in the data to be processed to obtain the decompressed blurred image. This reduces the amount of computation in the decompressed blur processing process and the noise in the decompressed blurred image, making the texture structure of the decompressed blurred image more accurate and improving the efficiency of image data decompressed blur processing.

[0124] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0125] Figure 8 This is a schematic diagram of the structure of the terminal device provided in this embodiment. Figure 8 As shown, the terminal device 8 of this embodiment includes: at least one processor 80 ( Figure 8(Only one is shown in the diagram), memory 81, and computer program 82 stored in memory 81 and executable on at least one processor 80, wherein processor 80 executes computer program 82 to implement the steps in any of the above-described data processing method embodiments.

[0126] Terminal device 8 can be a computing device such as a desktop computer, laptop, handheld computer, or cloud server. This terminal device may include, but is not limited to, a processor 80 and a memory 81. Those skilled in the art will understand that... Figure 8 This is merely an example of terminal device 8 and does not constitute a limitation on terminal device 8. It may include more or fewer components than shown in the figure, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0127] The processor 80 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0128] In some embodiments, memory 81 may be an internal storage unit of the terminal device 8, such as a hard disk or memory of the terminal device 8. In other embodiments, memory 81 may be an external storage device of the terminal device 8, such as a plug-in hard disk, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the terminal device 8. Furthermore, memory 81 may include both internal storage units and external storage devices of the terminal device 8. Memory 81 is used to store operating systems, applications, bootloaders, data, and other programs, such as program code for computer programs. Memory 81 can also be used to temporarily store data that has been output or will be output.

[0129] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0130] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0131] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0132] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0133] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0134] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0135] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0136] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0137] The above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A data processing method, characterized in that, include: Acquire data to be processed, which includes multiple frames of images. The multiple frames of images are obtained by compressing multiple original images of the same size using different image size compression ratios. The multiple frames of images are multiple images with consecutive frames in a video. The data to be processed is subjected to deblurring, downsampling, and feature extraction to obtain the first feature data; The first feature data is subjected to feature alignment and feature fusion processing to obtain the second feature data; The second feature data is subjected to feature reconstruction and upsampling processing to obtain the reconstruction and upsampling result; The reconstructed upsampling result is superimposed with a reference image from the multi-frame images to obtain a decompressed and blurred image. The reference image refers to a frame of image data that meets preset conditions selected from the multi-frame images.

2. The method as described in claim 1, characterized in that, The process of deblurring, downsampling, and feature extraction of the data to be processed to obtain first feature data includes: The data to be processed is input into a deblurring network for deblurring, and the first feature information is output. The first feature information is downsampled by N times to obtain downsampled feature information, where N is an integer greater than 1; The downsampled feature information is processed by multiple stacked residual modules to obtain the first feature data.

3. The method as described in claim 2, characterized in that, The step of performing feature alignment and feature fusion processing on the first feature data to obtain the second feature data includes: The first feature data is aligned using a pyramid cascade and deformable module to obtain the second feature information. The second feature information is processed by the feature fusion module to obtain the second feature data.

4. The method as described in claim 3, characterized in that, The step of performing feature reconstruction and upsampling processing on the second feature data to obtain the reconstruction and upsampling result includes: The second feature data is input into a trained hierarchical attention network model for feature reconstruction, and the reconstruction result is output. The reconstruction result is upsampled N times to obtain the reconstruction upsampled result.

5. The method as described in claim 4, characterized in that, The trained hierarchical attention network model includes a hierarchical convolutional neural network model and a channel attention mechanism model.

6. The method as described in claim 5, characterized in that, The step of overlaying the reconstructed upsampling result with the reference image in the multi-frame image to obtain the decompressed and blurred image includes: Select a reference image that meets the preset conditions from the multiple frames of images; The reconstructed upsampling result is superimposed on the reference image to obtain a decompressed and blurred image.

7. The method according to any one of claims 1 to 6, characterized in that, Before acquiring the data to be processed, the method further includes: Acquire multiple blurred images and multiple clear images, wherein the multiple blurred images and the multiple clear images correspond one-to-one; The multiple blurred images are divided into multiple sets of training data and test datasets. The multiple sets of training data are compressed using multiple image size compression ratios with different values ​​to obtain the training dataset. Each set of training data contains multiple frames of continuous blurred images. The untrained hierarchical attention network model is trained using the training dataset, the test dataset, and the multiple clear images to obtain the trained hierarchical attention network model.

8. A data processing apparatus, characterized in that, include: The first acquisition module is used to acquire data to be processed, the data to be processed includes multiple frames of images, the multiple frames of images are obtained by compressing multiple original images of the same image size using different image size compression ratios, and the multiple frames of images are multiple images with consecutive frame numbers in a video. The extraction module is used to perform deblurring, downsampling, and feature extraction on the data to be processed to obtain the first feature data; The fusion processing module is used to perform feature alignment and feature fusion processing on the first feature data to obtain the second feature data. The reconstruction module is used to perform feature reconstruction processing and upsampling processing on the second feature data to obtain the reconstruction upsampling result; The overlay processing module is used to overlay the reconstructed upsampling result with a reference image in the multi-frame images to obtain a decompressed and blurred image. The reference image refers to a frame of image data selected from the multi-frame images that meets preset conditions.

9. A terminal device, characterized in that, The terminal device includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the method as described in any one of claims 1 to 7.

Images