Image processing method and device and related equipment

Through an image processing method combining multi-feature extraction module and attention mechanism, the quantum convolution network is used to remove interferences such as rain and snow in the light field images, solving the problem of poor image recovery effect under severe weather conditions, and achieving a more comprehensive Feature extraction and image recovery, suitable for computer vision systems.

CN119027679APending Publication Date: 2024-11-26CHINA MOBILE (SUZHOU) SOFTWARE TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202410994926.0
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-07-24
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

When the image is restored under severe weather conditions, the interference objects such as rain, snow, fog, etc. remain and the background texture details are missing, resulting in poor image recovery effect.

Method used

Multiple feature extraction modules are used to convolution process the light field image, and channel features and spatial features are extracted through convolution kernels of different sizes, combined with residual blocks, channel attention modules and spatial attention modules to perform feature stitching and image recovery. , using quantum convolution networks and long-term memory networks to improve image recovery effect.

Benefits of technology

It significantly improves the recovery effect of light field images, removes interferences such as rain and snow, restores background texture details, improves image quality, and reduces calculation time. It is suitable for applications such as monitoring and autonomous navigation systems.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027679A_ABST
    Figure CN119027679A_ABST
Patent Text Reader

Abstract

The invention provides an image processing method and device and related equipment, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the feature extraction of a to-be-processed light field image according to a plurality of feature extraction modules, obtaining a plurality of feature data, enabling the plurality of feature data to be in one-to-one correspondence with the plurality of feature extraction modules, and enabling the plurality of feature extraction modules to be in one-to-one correspondence with the plurality of feature extraction modules; the feature data is used for indicating channel features of the light field image and spatial features of the light field image, and in the plurality of feature extraction modules, convolution kernels included in different feature extraction modules are different in size; performing feature splicing on the multiple pieces of feature data to obtain target feature data for indicating a target object; and performing image restoration on the light field image according to the target feature data to obtain a target image. According to the method, more comprehensive image feature data can be obtained based on the use of the convolution kernels of different sizes and the acquisition of the channel features and the spatial features, so that the image restoration effect of the light field image is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present disclosure relates to the technical field of artificial intelligence, and in particular to an image processing method, apparatus and related equipment. Background Art

[0002] Harsh weather conditions such as rain, snow, and fog can degrade the quality of images / videos captured outdoors and affect the performance of computer vision systems.

[0003] In order to reduce the impact of severe weather conditions on the performance of computer vision systems, relevant technologies propose: converting the original image into a four-dimensional light field image, and using different sub-views of different perspectives corresponding to the light field image to achieve feature recognition of interfering objects such as rain, snow, and fog in images / videos captured outdoors and cleaning of corresponding image parts.

[0004] It was found in the application that the images restored by related technologies have the following problems: interference objects such as rain, snow, fog, etc. remain in the restored images, and the background texture details of the restored images are missing.

[0005] It can be seen that the image restoration measures provided by the related art have poor image restoration effects. Summary of the invention

[0006] The purpose of the present disclosure is to provide an image processing method, an apparatus and related equipment, so as to solve the technical problem that the image restoration measures provided by the related art have poor image restoration effects.

[0007] In a first aspect, an embodiment of the present disclosure provides an image processing method, the method comprising:

[0008] According to the plurality of feature extraction modules, feature extraction is performed on the light field image to be processed to obtain a plurality of feature data, wherein the plurality of feature data corresponds to the plurality of feature extraction modules one by one, the feature data is used to indicate the channel feature of the light field image and the spatial feature of the light field image, and among the plurality of feature extraction modules, different feature extraction modules include convolution kernels of different sizes;

[0009] Performing feature splicing on the plurality of feature data to obtain target feature data for indicating the target object;

[0010] The light field image is restored according to the target feature data to obtain a target image.

[0011] In a second aspect, an embodiment of the present disclosure provides an image processing device, the device comprising:

[0012] A feature extraction module, configured to perform feature extraction on the light field image to be processed according to a plurality of feature extraction modules to obtain a plurality of feature data, wherein the plurality of feature data corresponds to the plurality of feature extraction modules in a one-to-one manner, and the feature data is used to indicate a channel feature of the light field image and a spatial feature of the light field image, and among the plurality of feature extraction modules, different feature extraction modules include convolution kernels of different sizes;

[0013] A feature splicing module, used for performing feature splicing on the plurality of feature data to obtain target feature data for indicating a target object;

[0014] The image processing module is used to perform image restoration on the light field image according to the target feature data to obtain a target image.

[0015] In a third aspect, the present disclosure provides an electronic device, comprising a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program, when executed by the processor, implements the steps of the method described in the first aspect.

[0016] In a fourth aspect, the present disclosure provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the steps of the method described in the first aspect are implemented.

[0017] In a fifth aspect, the present disclosure provides a computer program product, comprising computer instructions, which, when executed by a processor, implement the steps of the method described in the first aspect.

[0018] In the present disclosure, based on multiple different convolution kernels included in multiple feature extraction modules, convolution processing is performed on the light field image to avoid interference caused by the perspective phenomenon, and image features of the light field image at different depths are fully extracted to obtain channel features corresponding to different depths for representing the image characteristics of the light field image in color, and spatial features for representing the image characteristics of the light field image in space, which can make the obtained image features more comprehensive and accurate. Therefore, image restoration processing of the light field image based on the obtained image features can improve the image restoration effect of the light field image. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flowchart of an image processing method provided by an embodiment of the present disclosure;

[0020] Figure 2 is a structural diagram of a residual feature attention module provided by an embodiment of the present disclosure;

[0021] Figure 3is a structural diagram of a channel attention module provided by an embodiment of the present disclosure;

[0022] Figure 4 is a schematic diagram of the structure of a spatial attention module provided by an embodiment of the present disclosure;

[0023] Figure 5 is a structural diagram of a hybrid network provided by an embodiment of the present disclosure;

[0024] Figure 6 is a schematic diagram of the structure of a dense quantum convolution module provided by an embodiment of the present disclosure;

[0025] Figure 7 It is a schematic diagram of a process of removing rain from a light field image based on a quantum convolutional neural network provided by an embodiment of the present disclosure;

[0026] Figure 8 is a structural schematic diagram of an image processing device provided by an embodiment of the present disclosure;

[0027] Fig. 9 It is a schematic diagram of an electronic device provided by an embodiment of the present disclosure. DETAILED DESCRIPTION

[0028] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0029] The present disclosure provides an image processing method. Figure 1 As shown, the method includes:

[0030] Step 101: extract features from a light field image to be processed according to a plurality of feature extraction modules to obtain a plurality of feature data.

[0031] Among them, the multiple feature data and the multiple feature extraction modules correspond one to one, the feature data are used to indicate the channel features of the light field image and the spatial features of the light field image, and among the multiple feature extraction modules, different feature extraction modules include convolution kernels of different sizes.

[0032] Exemplarily, in order to normalize the image size of light field images of different specifications, the light field image can be decomposed into sub-aperture views and saved as a four-dimensional tensor I of (C, U*V, H, W), wherein C is 3, representing RGB channels; (U*V) represents the angular resolution, which is set to (5*5); (H, W) represents the spatial resolution, which is set to (64, 64), and the four-dimensional tensor I is subsequently input into the multiple feature extraction modules respectively to obtain the multiple feature data.

[0033] It should be understood that the smaller the size of the convolution kernel, the smaller the receptive field of the convolution kernel, and therefore the stronger the ability to express the detail features in the image, which can be used to perceive the corresponding nearby part of the image; correspondingly, the larger the size of the convolution kernel, the larger the receptive field of the convolution kernel, and therefore the stronger the ability to express the global features in the image, which can be used to perceive the corresponding distant part of the image; by setting convolution kernels of different sizes, the image detail features corresponding to the near and the image global features corresponding to the far in the light field image can be perceived respectively, thereby obtaining more comprehensive image features.

[0034] Step 102: perform feature splicing on the plurality of feature data to obtain target feature data for indicating the target object.

[0035] Step 103: Perform image restoration on the light field image according to the target feature data to obtain a target image.

[0036] The aforementioned target object refers to an interference object present in the light field image, such as rain, snow, frost, etc.; the process of performing image restoration on the light field image can be understood as the process of removing the aforementioned interference object from the light field image, that is, the target image can be understood as the light field image after removing the target object.

[0037] The present application performs convolution processing on light field images based on multiple different convolution kernels included in multiple feature extraction modules to avoid interference caused by perspective phenomena, and fully extracts image features of the light field images at different depths to obtain channel features corresponding to different depths for representing image characteristics of the light field images in color, and spatial features for representing image characteristics of the light field images in space, which can make the obtained image features more comprehensive and accurate. Therefore, image restoration processing of the light field images based on the obtained image features can improve the image restoration effect of the light field images.

[0038] In one embodiment, each of the feature extraction modules includes Q residual blocks connected in series, where Q is an integer greater than 1;

[0039] According to the pre-set multiple feature extraction modules, feature extraction is performed on the light field image to be processed to obtain multiple feature data, including:

[0040] Performing convolution processing on the light field image to obtain an initial feature map;

[0041] Processing the initial feature map based on Q residual blocks in a target feature extraction module to obtain feature data corresponding to the target feature extraction module, where the target feature extraction module is any one of the multiple feature extraction modules;

[0042] Among them, the input data of the 1st residual block of the target feature extraction module is the initial feature map, the input data of the qth residual block of the target feature extraction module is obtained by splicing the output data of the q-1th residual block of the target feature extraction module and the initial feature map, the output data of the Qth residual block of the target feature extraction module is the feature data corresponding to the target feature extraction module, and q is an integer greater than 1 and less than or equal to Q.

[0043] Furthermore, each of the residual blocks includes a channel submodule corresponding to the channel attention mechanism and a spatial submodule corresponding to the spatial attention mechanism;

[0044] Processing the initial feature map based on the Q residual blocks in the target feature extraction module to obtain feature data corresponding to the target feature extraction module includes:

[0045] Performing convolution processing on input data of the target residual block according to a convolution kernel included in the target residual block to obtain a first feature map, wherein the target residual block is any one of the Q residual blocks included in the target feature extraction module;

[0046] Performing channel feature extraction on the first feature map according to the channel submodule included in the target residual block to obtain a second feature map;

[0047] Performing spatial feature extraction on the second feature map according to the spatial submodule included in the target residual block to obtain a third feature map;

[0048] Convolving the third feature map according to the convolution kernel included in the target residual block to obtain output data of the target residual block.

[0049] In this application, if Figure 2As shown, the feature extraction module can also be called a residual feature attention module (Residual Feature Attention Module, RFAM), each RFAM includes Q residual blocks, each residual block is composed of a 3D convolution layer, a channel attention module (Channel Attention Module, CAM), a spatial attention module (Spatial Attention Module, SAM) and a 3D convolution layer connected in series. It should be understood that for different residual blocks in the same RFAM, the convolution kernel used in the 3D convolution layer is the same convolution kernel. For example, Q can be set to 6.

[0050] The channel submodule may be the aforementioned CAM, and the space submodule may be the aforementioned SAM.

[0051] For example, the channel attention module can be Figure 3 As shown, average pooling and maximum pooling operations are performed on the channel dimension of the input feature map (i.e., the first feature map mentioned above) to generate two different channel context features, wherein the average pooling operation retains the feature information of the "important" and "less important" elements in the feature map by averaging the feature points in the neighborhood of the feature map; the maximum pooling operation retains the feature information of the "important" elements in the feature map by taking the maximum of the feature points in the neighborhood of the feature map.

[0052] The two different context features generated by the average pooling and maximum pooling operations are input into a shared multilayer perceptron (MLP) to learn the correlation and weights between channels through the multilayer perceptron. Then, based on the element-wise summation operation, the output feature vectors are merged to obtain a channel attention map (i.e., the aforementioned second feature map) for representing the global features of the light field image.

[0053] The spatial attention module can be Figure 4 As shown, average pooling and maximum pooling operations are performed on the spatial dimension of the input feature map (i.e., the channel attention map) to generate two two-dimensional mappings, and then the two maps are concatenated in the spatial dimension and convolved through a standard convolutional layer to generate a two-dimensional spatial attention map (i.e., the aforementioned third feature map) for representing the local features of the light field image.

[0054] In view of the fact that there is a brightness difference between the area corresponding to the target object (such as rain streaks) in the light field image and the normal area in the light field image, CAM is introduced to adaptively select and weight the feature representations of different channels, so that the global color and brightness representation of the light field image can be obtained, which facilitates the subsequent distinction of the image part corresponding to the target object in the light field image; at the same time, in view of the uneven distribution of the target object in the light field image, SAM is introduced to adaptively select and weight the feature representations of different spatial positions, so that the local texture details of the light field image can be obtained, which facilitates the subsequent determination of the area including the target object in the light field image.

[0055] In an example, the four-dimensional tensor I corresponding to the light field image can be convolved, and the number of channels can be mapped to 32 to obtain the initial feature map F. If the convolution kernels used by multiple feature extraction modules are set to 3×3×3, 5×5×5, and 7×7×7, respectively, the acquisition and splicing process of multiple feature data can be expressed as:

[0056]

[0057] In the above formula, F 3×3×3 is the feature data obtained after feature extraction of RFAM corresponding to the 3×3×3 convolution kernel, F 5 ×5×5 is the feature data obtained after feature extraction of RFAM corresponding to the 5×5×5 convolution kernel, F 7×7×7 is the feature data obtained after feature extraction of RFAM corresponding to the 7×7×7 convolution kernel, I rain It is the target feature data obtained by concatenating multiple feature data.

[0058] In one embodiment, performing image restoration on the light field image according to the target feature data to obtain the target image includes:

[0059] In the channel dimension, the target feature data and the light field image are spliced ​​to obtain data to be restored;

[0060] Image restoration is performed on the data to be restored according to a hybrid network to obtain a target image, wherein the hybrid network includes a quantum convolutional network and a long short-term memory network.

[0061] Furthermore, the quantum convolution network includes a plurality of groups of recursive modules connected in sequence, and each of the recursive modules includes a memory module corresponding to the long short-term memory network and a quantum module corresponding to the quantum convolution network connected in series.

[0062] In this embodiment, feature extraction and calculation tasks corresponding to the image restoration operation are performed based on the quantum convolutional network. The advantages of the quantum convolutional network in feature extraction and processing large-scale, high-dimensional data are utilized to effectively improve the performance and computing efficiency of the network, shorten the calculation time, and improve the image restoration efficiency. Among them, since the image restoration operation involves iterative cycles, in order to reduce the adverse effects of gradient disappearance during the iterative process, a long short-term memory network is introduced to perform corresponding processing on the data output by the quantum convolutional network in each iteration, so as to cyclically transmit rain streaks and rain-free background information (assuming that the target object is rain) in multiple iterations, thereby improving the image quality of the target image finally output.

[0063] In one example, assuming that the target object is rain, the obtained rain streak feature I rain The four-dimensional tensor I of the light field image is concatenated in the channel dimension and input into the hybrid network.

[0064] The structural diagram of the hybrid network is as follows: Figure 5 As shown, Figure 5 The Dense Quantum Convolution Module (DQCM) layer can be understood as a quantum module corresponding to the quantum convolution network. Figure 5 The LSTM layer can be understood as a memory module corresponding to the long short-term memory network.

[0065] The recursive dense quantum convolution module (DQCM) is used to remove rain streaks and restore the background. Figure 6 As shown in the figure, each dense block in the DQCM layer consists of two convolutional layers with 3×3×3 convolutional kernels, two convolutional layers with 1×1×1 convolutional kernels, and one quantum convolutional layer. By utilizing the quantum superposition and entanglement characteristics of quantum convolution, the global information of the image can be better extracted and the network computing efficiency can be improved.

[0066] The quantum convolution layer used in this example encodes the input data into quantum states and evolves the quantum states using a parameterized quantum circuit consisting of interleaved single-qubit layers and double-qubit layers, where the single-qubit layer consists of y gates, each containing an adjustable parameter, and the double-qubit layer consists of controlled-NOT (CNOT) gates on the nearest neighbor qubit pairs, with a filter window size of m×n. The output after evolution is |ψ 0 >, take the observed value N=mn.

[0067] The final feature map is:

[0068]

[0069] In order to preserve long-term memory and alleviate the gradient vanishing problem, the LSTM layer is used to transfer rain streaks and rain-free background information in each cycle.

[0070] The calculation process of the LSTM layer can be expressed as:

[0071]

[0072] f t =σ(W f ·[h t-1 ,X t ]+b f ) (7)

[0073] i t =σ(W i ·[h t-1 ,X t ]+b i ) (8)

[0074]

[0075] o t =σ(W o ·[h t-1 ,X t ]+b o ) (11)

[0076] h t =o t *tanh(C t ) (12)

[0077] Among them, t represents the iteration order; f, i, o and h represent the forget gate, input gate, output gate and hidden state respectively; C is the cell state; σ is the Sigmoid activation function; W and b are the weights and biases of the corresponding operations.

[0078] Through the cyclic iteration of the above hybrid network, rain streaks are gradually removed and background texture details are restored to obtain a high-quality rain-free light field image I derain (i.e. the target image).

[0079] In one embodiment, after performing image restoration on the light field image according to the target feature data to obtain the target image, the method further includes:

[0080] In the case where the light field image is a training image, a mean square error between pixels of a true value image of the light field image and pixels of the target image is calculated to obtain a first loss value, and a structural similarity between the true value image of the light field image and the target image is calculated to obtain a second loss value;

[0081] Performing weighted calculation on the first loss value and the second loss value to obtain a target loss value;

[0082] Model parameters in a target model are optimized according to the target loss value, wherein the target model includes the feature extraction module and the hybrid network.

[0083] The true value image of the light field image is a light field image that does not include the target object.

[0084] Among them, the first loss value corresponding to the mean square error (MSE) can be used to evaluate the difference between the target image and the true image of the light field image at the image pixel level; the second loss value corresponding to the structural similarity (StructureSimilarityIndexMeasure, SSIM) can be used to evaluate the structural similarity between the target image and the true image of the light field image. By weighted calculation, the similarity between the target image and the true image of the light field image is comprehensively quantified from multiple aspects such as image pixels, image brightness, contrast and structure, which can simulate the perception effect of the human eye on the image and improve the training effect of the model parameters in the target model.

[0085] Exemplarily, the formula for calculating the target loss value can be expressed as:

[0086]

[0087] in:

[0088]

[0089] I gt is the true value image used to represent the light field image, M is I derain The total number of pixels, N is I gt The total number of pixels. μ, σ 2 are the mean and variance of the image; C1 and C2 are constants.

[0090] For ease of understanding, the following examples are provided:

[0091] This example provides a light field image deraining method based on quantum convolutional neural network. Figure 7 As shown in FIG, the light field image deraining method includes two stages: rain streak extraction and image restoration.

[0092] In the rain streak extraction stage, the light field image is input into the multi-scale rain streak extraction module. The multi-scale rain streak extraction module includes multiple residual feature attention modules to obtain the rain streak information in the light field image by connecting convolutional layers and channel attention modules and spatial attention modules of different scales in a residual manner.

[0093] In the image restoration stage, the acquired rain streak features and light field images are connected in the channel dimension and input into the image restoration module for feature extraction through the dense quantum convolution module. At the same time, LSTM is used to retain long-term memory, recursively remove rain streaks and restore the rain-free light field image.

[0094] Among them, in the multi-scale rain streak extraction module, convolutional layers with different receptive fields are used to extract rain streak features of different scales. At the same time, different feature representations are adaptively selected through the channel attention module and the spatial attention module, which can better extract rain streaks with complex shapes. In the image restoration stage, the advantages of quantum computing in feature extraction and processing large-scale, high-dimensional data are utilized, and dense quantum convolutional neural networks are used to gradually remove rain streaks and restore the rain-free background, which can effectively improve the performance and computational efficiency of the overall network.

[0095] Compared with traditional two-dimensional images, light field images capture the direction and position of each incident light ray through a plenoptic camera, and also contain the spatial and angular information of the target scene. They have been widely used in computer vision tasks such as new viewpoint generation, target detection, and three-dimensional reconstruction.

[0096] However, light field images collected in rainy weather lack background information due to the occlusion of rain streaks. At the same time, the high-dimensional characteristics of light field images increase the time cost of processing light field images.

[0097] The method proposed in this application can effectively solve the above problems, remove rain streaks and restore the image texture details of the light field image, while improving the network computing efficiency, promoting the application of light field images in outdoor vision systems such as monitoring and autonomous navigation systems, and broadening the application of quantum algorithms in the field of computer vision in image classification, target detection, face recognition and other scenarios.

[0098] The present disclosure also provides an image processing device, such as Figure 8 As shown, the image processing device 800 includes:

[0099] A feature extraction module 801 is used to perform feature extraction on the light field image to be processed according to multiple feature extraction modules to obtain multiple feature data, wherein the multiple feature data correspond to the multiple feature extraction modules one by one, and the feature data is used to indicate the channel feature of the light field image and the spatial feature of the light field image. Among the multiple feature extraction modules, different feature extraction modules include convolution kernels of different sizes;

[0100] A feature splicing module 802 is used to perform feature splicing on the plurality of feature data to obtain target feature data for indicating a target object;

[0101] The image processing module 803 is used to perform image restoration on the light field image according to the target feature data to obtain a target image.

[0102] In one embodiment, each of the feature extraction modules includes Q residual blocks connected in series, where Q is an integer greater than 1;

[0103] The feature extraction module 801 includes:

[0104] An initial convolution unit, used to perform convolution processing on the light field image to obtain an initial feature map;

[0105] A residual processing unit, used to process the initial feature map based on Q residual blocks in a target feature extraction module to obtain feature data corresponding to the target feature extraction module, wherein the target feature extraction module is any one of the multiple feature extraction modules;

[0106] Among them, the input data of the 1st residual block of the target feature extraction module is the initial feature map, the input data of the qth residual block of the target feature extraction module is obtained by splicing the output data of the q-1th residual block of the target feature extraction module and the initial feature map, the output data of the Qth residual block of the target feature extraction module is the feature data corresponding to the target feature extraction module, and q is an integer greater than 1 and less than or equal to Q.

[0107] In one embodiment, each of the residual blocks includes a channel submodule corresponding to the channel attention mechanism and a spatial submodule corresponding to the spatial attention mechanism;

[0108] The residual processing unit is specifically used for:

[0109] Performing convolution processing on input data of the target residual block according to a convolution kernel included in the target residual block to obtain a first feature map, wherein the target residual block is any one of the Q residual blocks included in the target feature extraction module;

[0110] Performing channel feature extraction on the first feature map according to the channel submodule included in the target residual block to obtain a second feature map;

[0111] Performing spatial feature extraction on the second feature map according to the spatial submodule included in the target residual block to obtain a third feature map;

[0112] Convolving the third feature map according to the convolution kernel included in the target residual block to obtain output data of the target residual block.

[0113] In one embodiment, the image processing module 803 is specifically used to:

[0114] In the channel dimension, the target feature data and the light field image are spliced ​​to obtain data to be restored;

[0115] Image restoration is performed on the data to be restored according to a hybrid network to obtain a target image, wherein the hybrid network includes a quantum convolutional network and a long short-term memory network.

[0116] In one embodiment, the quantum convolution network includes a plurality of groups of recursive modules connected in sequence, and each of the recursive modules includes a memory module corresponding to the long short-term memory network and a quantum module corresponding to the quantum convolution network connected in series.

[0117] In one embodiment, the apparatus 800 further includes:

[0118] a loss value calculation module, configured to calculate, when the light field image is a training image, a mean square error between pixels of a true value image of the light field image and pixels of the target image to obtain a first loss value, and to calculate a structural similarity between the true value image of the light field image and the target image to obtain a second loss value;

[0119] A loss value weighting module, used for performing weighted calculation on the first loss value and the second loss value to obtain a target loss value;

[0120] A parameter optimization module is used to optimize model parameters in a target model according to the target loss value, wherein the target model includes the feature extraction module and the hybrid network.

[0121] The image processing device 800 provided in the embodiment of the present disclosure can implement each process in the above-mentioned image processing method embodiment, and will not be described again here to avoid repetition.

[0122] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device and a readable storage medium.

[0123] Fig. 9 A schematic block diagram of an example electronic device 900 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0124] like Fig. 9As shown, the device 900 includes a computing unit 901, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 902 or a computer program loaded from a storage unit 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the device 900 can also be stored. The computing unit 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0125] A number of components in the device 900 are connected to the I / O interface 905, including: an input unit 906, such as a keyboard, a mouse, etc.; an output unit 907, such as various types of displays, speakers, etc.; a storage unit 908, such as a disk, an optical disk, etc.; and a communication unit 909, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 909 allows the device 900 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0126] The computing unit 901 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 901 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSP), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 901 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly included in a machine-readable medium, such as a storage unit 908. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 900 via the ROM 902 and / or the communication unit 909. When the computer program is loaded into the RAM 903 and executed by the computing unit 901, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 901 may be configured to execute the image processing method in any other appropriate manner (eg, by means of firmware).

[0127] Various embodiments of the systems and techniques described above herein may be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0128] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0129] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0130] As used herein, the term "machine-readable medium" refers to any computer program product, apparatus, and / or device (e.g., disk, optical disk, memory, programmable logic device (PLD)) for providing machine instructions and / or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal for providing machine instructions and / or data to a programmable processor.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0132] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0133] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0134] The present application also provides a computer program product, including computer instructions, which, when executed by a processor, implement the above Figure 1 The various processes of the method embodiment shown can achieve the same technical effect, and will not be described again here to avoid repetition.

[0135] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0136] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. An image processing method, characterized in that: The method comprises: According to the plurality of feature extraction modules, feature extraction is performed on the light field image to be processed to obtain a plurality of feature data, wherein the plurality of feature data corresponds to the plurality of feature extraction modules one by one, the feature data is used to indicate the channel feature of the light field image and the spatial feature of the light field image, and among the plurality of feature extraction modules, different feature extraction modules include convolution kernels of different sizes; Performing feature splicing on the plurality of feature data to obtain target feature data for indicating the target object; The light field image is restored according to the target feature data to obtain a target image.

2. The method according to claim 1, characterized in that Each of the feature extraction modules includes Q residual blocks connected in series, where Q is an integer greater than 1; According to the pre-set multiple feature extraction modules, feature extraction is performed on the light field image to be processed to obtain multiple feature data, including: Performing convolution processing on the light field image to obtain an initial feature map; Processing the initial feature map based on Q residual blocks in a target feature extraction module to obtain feature data corresponding to the target feature extraction module, where the target feature extraction module is any one of the multiple feature extraction modules; Among them, the input data of the 1st residual block of the target feature extraction module is the initial feature map, the input data of the qth residual block of the target feature extraction module is obtained by splicing the output data of the q-1th residual block of the target feature extraction module and the initial feature map, the output data of the Qth residual block of the target feature extraction module is the feature data corresponding to the target feature extraction module, and q is an integer greater than 1 and less than or equal to Q.

3. The method according to claim 2, characterized in that Each of the residual blocks includes a channel submodule corresponding to the channel attention mechanism and a spatial submodule corresponding to the spatial attention mechanism; Processing the initial feature map based on the Q residual blocks in the target feature extraction module to obtain feature data corresponding to the target feature extraction module includes: Performing convolution processing on input data of the target residual block according to a convolution kernel included in the target residual block to obtain a first feature map, wherein the target residual block is any one of the Q residual blocks included in the target feature extraction module; Performing channel feature extraction on the first feature map according to the channel submodule included in the target residual block to obtain a second feature map; Performing spatial feature extraction on the second feature map according to the spatial submodule included in the target residual block to obtain a third feature map; Convolving the third feature map according to the convolution kernel included in the target residual block to obtain output data of the target residual block.

4. The method according to claim 1, characterized in that: The step of performing image restoration on the light field image according to the target feature data to obtain a target image includes: In the channel dimension, the target feature data and the light field image are spliced ​​to obtain data to be restored; Image restoration is performed on the data to be restored according to a hybrid network to obtain a target image, wherein the hybrid network includes a quantum convolutional network and a long short-term memory network.

5. The method according to claim 4, characterized in that The quantum convolution network includes a plurality of groups of recursive modules connected in sequence, and each of the recursive modules includes a memory module corresponding to the long short-term memory network and a quantum module corresponding to the quantum convolution network connected in series.

6. The method according to claim 4, characterized in that After performing image restoration on the light field image according to the target feature data to obtain the target image, the method further includes: In the case where the light field image is a training image, a mean square error between pixels of a true value image of the light field image and pixels of the target image is calculated to obtain a first loss value, and a structural similarity between the true value image of the light field image and the target image is calculated to obtain a second loss value; Performing weighted calculation on the first loss value and the second loss value to obtain a target loss value; Model parameters in a target model are optimized according to the target loss value, wherein the target model includes the feature extraction module and the hybrid network.

7. An image processing device, characterized in that: The device comprises: A feature extraction module, configured to perform feature extraction on the light field image to be processed according to a plurality of feature extraction modules to obtain a plurality of feature data, wherein the plurality of feature data corresponds to the plurality of feature extraction modules in a one-to-one manner, and the feature data is used to indicate a channel feature of the light field image and a spatial feature of the light field image, and among the plurality of feature extraction modules, different feature extraction modules include convolution kernels of different sizes; A feature splicing module, used for performing feature splicing on the plurality of feature data to obtain target feature data for indicating a target object; The image processing module is used to perform image restoration on the light field image according to the target feature data to obtain a target image.

8. An electronic device, characterized in that: The method comprises a processor, a memory, and a computer program stored in the memory and executable on the processor, wherein the computer program implements the steps of the method according to any one of claims 1 to 6 when executed by the processor.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, characterized in that The method comprises computer instructions, which, when executed by a processor, implement the steps of the method according to any one of claims 1 to 6.