Image processing method and apparatus

Abstract

An image processing method and device. According to the method, after receiving image data of a target image, the image data can be processed based on network parameters to obtain enhanced image feature data of the target image, and the target image is processed based on the enhanced image feature data. Wherein, the target image is a low-quality image, and the network parameters are used to indicate the corresponding relationship between the feature data of the low-quality image and the feature data of a clear image. The method can improve the processing effect of the low-quality image.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer, and particularly relates to an image processing method and device. BACKGROUND

[0002] In recent years, with the rapid development of deep learning technology, breakthroughs have been made in the research of high-level vision represented by image classification, target recognition and semantic segmentation, and these breakthroughs are largely due to the emergence of large image databases such as ImageNet and PASCAL VOC. However, the images in these databases are usually clear and lossless high-quality images. However, in the actual imaging process, because of light (low light, overexposure, etc.), weather factors (rain, snow, fog, etc.), noise and motion interference factors will destroy the structure and statistical information of the image, resulting in low-quality images. Therefore, in actual visual applications, the computer vision system may need to process a low-quality image.

[0003] In the process of low-quality image processing, a commonly used method is to first use an image enhancement algorithm to enhance the degraded image, improve the image quality of the low-quality image, and then perform recognition and other processing on the image with improved quality. However, the main purpose of the existing image enhancement algorithm is to obtain an enhanced image with good visual perception effect, and the object is a person. However, this kind of algorithm cannot guarantee that the computer network can extract complete structure or statistical features from the enhanced image, so the recognition accuracy is low. SUMMARY

[0004] The image processing method and device provided by the present application can improve the processing effect of low-quality images.

[0005] In a first aspect, an image processing method is provided. According to the method, after receiving image data of a target image, the image data can be processed based on network parameters to obtain enhanced image feature data of the target image, and the target image is processed based on the enhanced image feature data. Wherein, the target image is a low-quality image, and the network parameters are used to indicate the corresponding relationship between the feature data of the low-quality image and the feature data of a clear image.

[0006] It can be learned from the description of the image processing method provided in the embodiments of the present application that, in the embodiments of the present application, the low-quality image itself is not processed in advance, but in the image processing process, the image data of the low-quality image is processed by using the set network parameters, the enhanced image feature data of the low-quality image is obtained, and the low-quality image is processed based on the enhanced image feature data. Since the network parameters reflect the corresponding relationship between the feature data of the low-quality image and the feature data of the clear image, that is, the feature of the low-quality target image is processed by using the connection between the low-quality image feature and the clear image feature in the processing process, the feature data of the low-quality image can be enhanced, the network recognizability of the feature data of the target image can be improved, and the processing effect of the low-quality image can be improved, for example, the recognition accuracy of the low-quality image can be improved.

[0007] In combination with the first aspect, in a first possible implementation, in the process of obtaining the enhanced image feature data of the target image, the feature data of the target image can be obtained according to the image data, and neural network calculation is performed on the feature data and the image data based on the network parameters to obtain residual data, and then the enhanced image feature data of the target image can be obtained according to the residual data and the feature data. The feature data is the feature data obtained by performing N-layer neural network calculation on the image data, N is greater than 0 and less than a preset threshold, and the residual data is used to indicate the deviation between the feature data of the target image and the feature data of the clear image.

[0008] According to this way, since the non-classical receptive field structure of the retina is referred to in the processing process, and the bipolar cell photosensitive principle of the retina is simulated, the high-frequency information in the target image is enhanced and the low-frequency information in the target image is maintained, so that the enhanced image feature data is easier to be recognized or extracted, the processing effect is better, for example, the recognition accuracy of the low-quality image can be improved. And the robustness (or stability) of the network can be stronger. Further, since the low-quality image is processed by using the feature drift attribute of the image in the embodiments of the present application, the processing process does not need the supervision of the semantic signal (used to indicate the image content), and the network parameters are less.

[0009] In combination with the above embodiments, in a possible implementation, the neural network calculation of the feature data and the image data based on the network parameters includes center-periphery convolution calculation of the feature data and the image data based on the network parameters. According to this way, the bipolar cell photosensitive principle of the retina can be simulated, and the processing effect of the image is better.

[0010] In any of the above embodiments, in a further possible implementation, the performing neural network calculation on the feature data and the image data based on the set network parameters comprises: performing at least a first level center-periphery convolution calculation, a second level center-periphery convolution calculation, and a third level center-periphery convolution calculation on the feature data and the image data based on the set network parameters. According to this manner, the structure of the non-classical receptive field of the retina and the photoreception principle of the bipolar cell of the retina can be simulated, and the image recognition accuracy and effect can be improved.

[0011] In any of the above embodiments, in a further possible implementation, input data of the first level center-periphery convolution calculation comprises the feature data and the image data, input data of the second level center-periphery convolution calculation comprises a calculation result of the first level center-periphery convolution calculation, and input data of the third level center-periphery convolution calculation comprises a calculation result of the second level center-periphery convolution calculation.

[0012] In any of the above embodiments, in a further possible implementation, the residual data is obtained based on a calculation result of the first level center-periphery convolution calculation, a calculation result of the second level center-periphery convolution calculation, and a calculation result of the third level center-periphery convolution calculation.

[0013] In any of the above embodiments, in a further possible implementation, the first level center-periphery convolution calculation is used to simulate the response of the central region of the human eye retina to the target image, the second level center-periphery convolution calculation is used to simulate the response of the peripheral region of the human eye retina to the target image, and the third level center-periphery convolution calculation is used to simulate the response of the edge region of the human eye retina to the target image.

[0014] In any of the above embodiments, in a further possible implementation, the first level center-periphery convolution calculation comprises: performing a first convolution operation on the feature data and the image data based on a first convolution kernel to obtain a first intermediate result, wherein the weight value of the central region of the first convolution kernel is 0; performing a second convolution operation on the feature data and the image data based on a second convolution kernel to obtain a second intermediate result, wherein the second convolution kernel only comprises the weight value of the central region, and the first convolution kernel and the second convolution kernel have the same size; and obtaining the calculation result of the first level center-periphery convolution based on the first intermediate result and the second intermediate result.

[0015] In combination with any of the foregoing embodiments, in a further possible implementation, the second-level center-surround convolution calculation includes: performing a third convolution operation on the calculation result of the first-level center-surround convolution based on a third convolution kernel, to obtain a third intermediate result, where the third convolution kernel has a center region with a value of 0; performing a fourth convolution operation on the calculation result of the first-level center-surround convolution based on a fourth convolution kernel, to obtain a fourth intermediate result, where the fourth convolution kernel only includes a weight value of the center region, and the third convolution kernel and the fourth convolution kernel have the same size; and obtaining the calculation result of the second-level center-surround convolution based on the third intermediate result and the fourth intermediate result.

[0016] In combination with any of the foregoing embodiments, in a further possible implementation, the third-level center-surround convolution calculation includes: performing a fifth convolution operation on the calculation result of the second-level center-surround convolution based on a fifth convolution kernel, to obtain a fifth intermediate result, where the fifth convolution kernel has a center region with a weight value of 0; performing a sixth convolution operation on the calculation result of the second-level center-surround convolution based on a sixth convolution kernel, to obtain a sixth intermediate result, where the sixth convolution kernel only includes a weight value of the center region, and the fifth convolution kernel and the sixth convolution kernel have the same size; and obtaining the calculation result of the third-level center-surround convolution based on the fifth intermediate result and the sixth intermediate result.

[0017] In combination with any of the foregoing embodiments, in a further possible implementation, the method can be performed by a neural network device, and the network parameters are obtained after training.

[0018] In a second aspect, the present application provides an image recognition device, which includes function modules for implementing the image processing method in the first aspect or any of the implementation manners of the first aspect.

[0019] In a third aspect, the present application provides an image recognition device, which includes a neural network for implementing the image processing method in the first aspect or any of the implementation manners of the first aspect.

[0020] In a fourth aspect, the present application further provides a computer program product, which includes program code including instructions executed by a computer to implement the image processing method in the first aspect or any of the implementation manners of the first aspect.

[0021] In a fifth aspect, the present application further provides a computer-readable storage medium for storing program code including instructions executed by a computer to implement the image processing method in the first aspect or any of the implementation manners of the first aspect. BRIEF DESCRIPTION OF DRAWINGS

[0022] In order to make the technical solution of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiment description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, not all embodiments.

[0023] Figure 1 A structural schematic diagram of an image processing device provided by an embodiment of the present application is shown in FIG. 1.

[0024] Figure 2 A schematic diagram of a neural network layer in a neural network system provided by an embodiment of the present application is shown in FIG. 2.

[0025] Figure 3A A flowchart of an image processing method provided by an embodiment of the present application is shown in FIG. 3.

[0026] Figure 3B A flowchart of another image processing method provided by an embodiment of the present application is shown in FIG. 4.

[0027] Figure 4 A signal schematic diagram of an image processing method provided by an embodiment of the present application is shown in FIG. 5.

[0028] Figure 5 A structural schematic diagram of a non-classical receptive field in a human eye retina provided by an embodiment of the present application is shown in FIG. 6.

[0029] Figure 6 A structural schematic diagram of a feature drift module provided by an embodiment of the present application is shown in FIG. 7.

[0030] Figure 7 A schematic diagram of a center-peripheral convolution mechanism provided by an embodiment of the present application is shown in FIG. 8.

[0031] Figure 8 A training schematic diagram of a neural network system provided by an embodiment of the present application is shown in FIG. 9.

[0032] Figure 9 A schematic diagram of another image processing device provided by an embodiment of the present application is shown in FIG. 10. DETAILED DESCRIPTION

[0033] In order to make the technical solution of the embodiments of the present application or the prior art clearer, the accompanying drawings needed in the embodiment description will be briefly introduced. Obviously, the accompanying drawings in the following description are only some embodiments of the present application, not all embodiments.

[0034] Figure 1 A structural schematic diagram of an image processing device provided by an embodiment of the present application is shown in FIG. 1. Figure 1As shown, the image processing apparatus 100 can include a control module 105 and a neural network circuit 110. The control module 105 can include a processor 1052 and a memory 1054. The processor 1052 is an operation core and a control core of the control module 105. The processor 1052 can include one or more processor cores. The processor 1052 can be a very large scale integrated circuit. An operating system and other software programs are installed in the processor 1052, so that the processor 1052 can realize access to the memory 1054, cache, disk and peripheral devices (such as the neural network circuit in the image processing apparatus 100). It can be understood that, in the embodiments of the present application, the core in the processor 1052 can be a central processing unit (CPU), and can also be an application specific integrated circuit (ASIC). Figure 1

[0035] The memory 1054 can be used as a cache of the processor 1052. The memory 1054 can be connected to the processor 1052 through a double data rate (DDR) bus. The memory 1054 is usually used to store various running software in the operating system, input and output data, and information exchanged with the external storage, etc. In order to improve the access speed of the processor 1052, the memory 1054 needs to have the advantage of fast access speed. In the traditional computer system architecture, dynamic random access memory (DRAM) is usually used as the memory 1054. The processor 1052 can access the memory 1054 at high speed through a memory controller (not shown in the figure), and perform read and write operations on any storage unit in the memory 1054. Figure 1

[0036] ​​The neural network circuit 110 is used to perform artificial neural network computations. Those skilled in the art will understand that an artificial neural network (ANN), often simply called a neural network (NN) or a neural network-like system, is a mathematical or computational model in machine learning and cognitive science that mimics the structure and function of biological neural networks (the central nervous system of animals, especially the brain) for estimating or approximating functions. Artificial neural networks can include convolutional neural networks (CNNs), deep neural networks (DNNs), multilayer perceptrons (MLPs), and other types of neural networks. Neural networks are commonly used for image recognition, image classification, and speech recognition.

[0037] In this embodiment of the invention, the neural network circuit 110 may include one or more neural network chips 115 (which may be simply referred to as chips 115) for performing artificial neural network calculations. The one or more chips 115 are used to perform neural network calculations. The neural network circuit 110 is connected to the control module 105. Figure 1 As shown, the neural network circuit 110 can be connected to the control module 105 via the connection bus 106. The connection bus 106 can be a peripheral component interconnect express (PCIe) bus, or other connection cables (such as network cables). Here, the connection method between the neural network circuit 110 and the control module 105 is not limited. Through the connection between the control module 105 and the neural network circuit 110, the processor 1052 can access the neural network circuit 110 via the connection bus 106. For example, when the processor 1052 accesses the neural network circuit 110 through an interface (… Figure 1 After receiving the image data to be processed (not shown), the control module 105 can send the image data to the chip 115 in the neural network circuit 110 via the connection bus 106, and receive the processing result of the neural network circuit 110 via the connection bus 106. Furthermore, the control module 105 can also monitor the operating status of the neural network circuit 110 via the connection bus 106.

[0038] A neural network system may include multiple neural network layers. In this embodiment of the invention, a neural network layer is a logical layer concept; one neural network layer refers to one neural network operation to be performed. Neural network layers may include convolutional layers, pooling layers, etc. Figure 2 As shown, a neural network system can include n neural network layers (also known as an n-layer neural network), where n is an integer greater than or equal to 2.Figure 2 Part of the neural network layers in the neural network system are shown as follows. Figure 2 As shown, the neural network system can include a first layer 202, a second layer 204, a third layer 206, a fourth layer 208, a fifth layer 210, and an n-th layer 212. The first layer 202 can perform a convolution operation, the second layer 204 can perform a pooling operation on the output data of the first layer 302, the third layer 206 can perform a convolution operation on the output data of the second layer 204, the fourth layer 208 can perform a convolution operation on the output result of the third layer 206, the fifth layer 210 can perform a summation operation on the output data of the second layer 204 and the output data of the fourth layer 208, and so on. It can be understood that, Figure 2 The above is only a simple example and illustration of the neural network layers in the neural network system, and does not limit the specific operations of each neural network layer. For example, the fourth layer 208 can also be a pooling operation, and the fifth layer 210 can also be a convolution operation or a pooling operation or other neural network operations.

[0039] In actual applications, when the neural network system processes an image, the processing result of the image can be obtained by performing the calculation of multiple neural network layers on the image data. In the embodiments of the present application, the number of neural network chips performing neural network calculation is not limited. It should be noted that, Figure 1 The above is only a schematic diagram of an image processing device. The image processing device in the embodiments of the present application can also be a server, a computer or other computing device capable of performing neural network calculation. These computing devices can include a central processing unit (CPU) or a graphic processing unit (GPU) or other computing nodes for performing neural network calculation. These computing devices can not include the neural network chip specially used for performing neural network calculation as shown above. In the embodiments of the present application, the specific structure of the image processing device is not limited, as long as it includes a neural network capable of implementing the image processing method provided by the embodiments of the present application. In the embodiments of the present application, the computer, the server and the image processing device containing the neural network as shown above can be referred to as a neural network device or a neural network system. In the embodiments of the present application, the image processing can include image classification, target recognition, semantic segmentation and other image processing methods. The image processing method provided by the embodiments of the present application can be applied to the scenes such as automatic driving, smartphone photographing and intelligent monitoring system, and can improve the processing accuracy of the image processing device on low-quality images. Figure 1 Figure 1

[0040] The following will be described in combination with Figure 3A , Figure 3B and Figure 4 ​​The detailed description of the embodiment of the present application provides how the image processing device processes the low-quality image. Figure 3A The flowchart of the image processing method provided by the embodiment of the present application is shown in Figure 3B The flowchart of another image processing method provided by the embodiment of the present application is shown in Figure 3A And Figure 3B The difference between Figure 3B Some steps in Figure 3A Some steps in the specific embodiment. Figure 4 The signal diagram of the image processing method provided by the embodiment of the present application is shown in Figure 3A And Figure 3B The image processing method shown in Figure 1 The neural network circuit 110 in can be executed. In the process of processing the image by the image processing device 100, the input image data is mainly processed by the neural network chip 115 in the neural network circuit 110. For example, the neural network circuit 110 can perform convolution, pooling and other neural network calculations on the image data to obtain the image processing result. In combination with Figure 3A , Figure 3B And Figure 4 The image processing method provided by the embodiment of the present application can include the following steps.

[0041] In step 302, the image data of the target image is received, and the target image is a low-quality image. In the embodiment of the present application, the low-quality image refers to a low-quality image generated due to the interference of light (such as low light, overexposure, etc.), weather (such as rain, snow, fog, etc.) and target relative motion, etc. during the image imaging process, which destroys the inherent color and structure information of the image. Simply put, the low-quality image is an image whose image quality is lower than a preset threshold. As Figure 4 As shown in

[0042] In step 303, the image data is processed based on network parameters to obtain enhanced image feature data of the target image. The network parameters are used to indicate the corresponding relationship between the feature data of the low-quality image and the feature data of the clear image. In the embodiment of the present application, the trained network parameters are set in the neural network system, which are obtained by training a plurality of low-quality images and clear images. The network parameters can indicate the corresponding relationship between the feature data of the low-quality image and the feature data of the clear image. Therefore, after determining that the input image data is the image data of the low-quality image, the input image data can be calculated based on the set network parameters, so as to enhance the feature data of the target image and obtain the enhanced image feature data. The following will be described in detail in combination with Figure 3B the description of how to obtain the enhanced image feature data of the target image.

[0043] Referring to step 304 in Figure 3B , after receiving the image data of the target image through step 302, the feature data of the target image can be obtained according to the image data. The feature data is the data obtained by calculating the image data through N-layer neural network, and N is greater than 0 and less than a preset threshold. In the embodiment of the present application, the feature data of the target image can include the shallow feature data of the target image. The shallow feature data can include data used to indicate the color, structure, texture and other features of the image. Specifically, the feature data of the target image can be obtained by performing N-layer neural network calculation by the chip 115 in the neural network circuit 110. That is, the feature data can be the feature map output by the first N-layer neural network calculation performed by the neural network circuit 110. The value of N can be set according to actual needs, for example, N can be less than 5 or N can be less than 10, etc., which is not limited herein. For the convenience of illustration, the N-layer neural network calculation performed by the chip 115 is referred to as the feature acquisition module 403 in Figure 4 . For example, in actual application, the feature data output by any one of the first 5 layers in the neural network calculation performed by the neural network circuit 110 can be used as the feature data of the target image. Taking the n-layer neural network calculation performed by the neural network circuit 110 as shown in Figure 2 , the feature data output by the first layer 202, the second layer 204, the third layer 206, the fourth layer 208 or the fifth layer 210 can be used as the feature data of the target image. It should be noted that Figure 4 the feature acquisition module 403, the feature drift module 405, the enhancement processing module 407 and the next layer neural network 409 are logical concepts, which are used to indicate the neural network calculation performed by the neural network circuit 110 in Figure 1 .

[0044] Referring to Figure 3B In step 306, the feature data and the image data are calculated based on the set network parameters to obtain residual data. The residual data is used to indicate the deviation between the feature data of the target image and the feature data of the clear image. The network parameters are used to indicate the corresponding relationship between the feature data of the low-quality image and the feature data of the clear image. The feature data of the clear image includes the feature data of the clear image used in the training process. The feature data of the clear image includes the shallow feature data of the clear image. As Figure 4 As shown, after obtaining the feature data 404, the feature data 404 and the image data of the target image 402 can be calculated by a feature de-drifting module 405 to obtain residual data 406. It should be noted that the feature de-drifting module 405 is also a neural network calculation performed in the neural network circuit 110. In the embodiment of the present application, the feature de-drifting module 405 is a neural network calculation module based on a non-classical receptive field of adaptive antagonistic convolution.

[0045] In the process of implementing the present application, it is found through research that in different images, the feature representation of the clear image block with similar structure and the feature representation of the corresponding low-quality image block have the same feature drift rule, and this rule is irrelevant to the image content (semantic information). Specifically, the shallow features of all structure-similar clear image blocks are gathered together, and similarly, the shallow features of all low-quality image blocks corresponding to the clear image blocks are also gathered together. Moreover, this gathering effect is irrelevant to the content of the image. Based on this finding, the embodiment of the present application establishes the corresponding relationship between the feature data of the low-quality image (which can be referred to as low-quality feature) and the feature data of the clear image (which can be referred to as clear feature), and improves the processing accuracy of the low-quality image according to this corresponding relationship.

[0046] In order to better learn the corresponding relationship between low-quality features and clear features, an embodiment of the present application proposes a feature drift network based on a non-classical receptive field (nCRF) mechanism in the human eye retina, and a "center-peripheral convolution mechanism" is proposed based on the photoreceptor principle of bipolar cells. The receptive field is the basic structure and functional unit of the visual system information processing. The retinal ganglion has a concentrically antagonistic classical receptive field (CRF), and the spatial integration characteristic is to process the image area brightness contrast information and extract the image edge information. The non-classical receptive field is a large area outside the classical receptive field, and the stimulation of the area alone cannot directly cause the response of the cell, but has a modulation effect on the response caused by the stimulation in the classical receptive field. The non-classical receptive field of the retinal ganglion cell is mainly disinhibitory, so it can compensate for the loss of low-frequency information caused by the classical receptive field to a certain extent, while maintaining the boundary enhancement function, and transmit the image area brightness gradient information, and display the slow change of brightness on a large area surface. As can be seen, the non-classical receptive field greatly widens the range of visual cell processing, and provides a neural basis for integrating and detecting complex patterns in a large range. The non-classical receptive field in the human eye retina includes multiple mutually antagonistic sub-areas, and each sub-area realizes the function of high-frequency enhancement and low-frequency maintenance through mutual cooperation, thereby helping the human eye to better distinguish external things. The antagonism refers to the phenomenon that one substance (or process) is inhibited by another substance (or process).

[0047] Figure 5 A structure diagram of the non-classical receptive field in the human eye retina provided by an embodiment of the present application is shown in Figure 5 As shown in the figure, the non-classical receptive field in the human eye retina can include three areas: a center area 502, a peripheral area 504 and an edge area 506. The mathematical expression of the non-classical receptive field in the human eye retina can be:

[0048] f=A1(I*G(σ1))+A2(I*G(σ2))+A3(I*G(σ3)) formula (1)

[0049] In formula (1), G1-G3 represent three different bandwidth Gaussian convolution kernels:

[0050]

[0051] A1-A3 represent the weighting coefficients of the center area, the peripheral area and the edge area respectively, and the variances σ1-σ3 determine the bandwidths of the three Gaussian functions. In formula (1), formula (1) can be written as follows:

[0052] f = A1(I*G1) + A2((I*G1)*G'2) + A3(((I*G1)*G'2)*G'3) Equation (3)

[0053] wherein, Through Equation (3), we can see that the output of the first convolution can be used as the input of the second convolution, and the output of the second convolution can be used as the input of the third convolution.

[0054] In order to simulate the antagonistic mechanism in the above-mentioned non-classical receptive field and enhance the low-quality features, the application provides a feature drift module. The module includes a multi-level structure, and each substructure in each level includes one or more convolution layers. A set of low-quality feature maps is input, each sub-network in the feature drift module can output a set of results, and finally a plurality of sets of output results are weighted and fused to obtain final residual data.

[0055] Figure 6 A structural diagram of a feature drift module 405 provided for an embodiment of the application is shown. Taking the case where the feature drift module 405 includes three convolution modules as an example. As shown in Figure 6 The feature drift module 405 can include three convolution modules: G1 4051, G2 4052 and G3 4053, and the feature drift module 405 can also include a processing module 4054. Among them, the convolution module G1 4051 is used to simulate the function of the central region 502 of the non-classical receptive field in the human eye retina as shown in Figure 5 The convolution module G2 4052 is used to simulate the function of the peripheral region 504 as shown in Figure 5 The convolution module G3 4053 is used to simulate the function of the edge region 506 as shown in Figure 5 Among them, each convolution module can be implemented by one or more convolution layers. Figure 6 Taking the case where a level of convolution includes two convolution layers as an example. For example, G1 4051 includes convolution layers G1_1 and G1_2, G2 4052 includes convolution layers G2_1 and G2_2, and G3 4053 includes convolution layers G3_1 and G3_2. The processing module 4054 is used to realize the superposition processing of the results output by the three convolution modules G1 4051, G2 4052 and G3 4053 after weighting.

[0056] To better enhance high-frequency information in low-quality images, embodiments of the present application propose a "center-surround convolution" based on the principle of bipolar cell photosensing. Bipolar neuron is a neuron cell that sends out one protrusion from each end of the cell body. One protrusion is distributed to the peripheral sensory receptor (also known as peripheral protrusion or dendrite), and the other protrusion enters the central part (also known as central protrusion or axon). In the retina, bipolar neurons connect photoreceptor cells and ganglion cells, and play a vertical liaison role. Bipolar neurons can be divided into on-center and off-center types. On-center means that the center is excited by light, and is inhibited when the peripheral light is stimulated. Off-center means that the center cell is excited when the central region stops stimulating, and is inhibited when the peripheral light is stopped.

[0057] In embodiments of the present application, based on the principle of bipolar cell photosensing of human eye retina, Figure 6 Each convolution module needs to perform calculation according to the "center-surround convolution mechanism". Specifically, each convolution layer in each convolution module needs to perform center convolution and surround convolution in parallel. For example, for the convolution modules G1 4051, G24052 and G3 4053 shown in Figure 6 Each convolution module includes two convolution layers, and each convolution layer needs to perform center convolution and surround convolution. It can be understood that when a convolution module includes multiple convolution layers, the input data of the next convolution layer includes the calculation result of the previous convolution layer, and the calculation result of the last convolution layer in a convolution module is the result of the convolution module. For example, taking the convolution module G1 as an example, the convolution layer G1_1 performs surround convolution and center convolution on the input data to obtain the calculation result, and then inputs the calculation result into the convolution layer G1_2. The convolution layer G1_2 continues to perform surround convolution and center convolution on the calculation result of the convolution layer G1_1 based on the set network parameters. The calculation result of the convolution layer G1_2 is the calculation result of the convolution module G1.

[0058] The following will take the convolution layer G1_1 in the convolution module G1 4051 as an example to describe how the convolution module shown in Figure 6 performs center-surround convolution. Figure 7 A schematic diagram of a center-surround convolution mechanism provided by embodiments of the present application. It should be noted that, Figure 7The following diagram illustrates how convolutional layer G1_1 in convolutional module G1 4051 implements center-periphery convolution. In practical applications, the working principle of convolutional layer G1_2 in convolutional module G1 4051, as well as the convolutional layers in convolutional modules G2 4052 and G3 4053, is similar to that of convolutional layer G1_1. (See reference...) Figure 7 The description.

[0059] like Figure 7 As shown, convolutional layer G1_1 performs peripheral convolution operations on input data 702 based on the first convolutional kernel 7022 set in the convolutional network G1_1, simulating on-center bipolar neurons. Simultaneously, convolutional layer G1_1 also performs central convolution operations on input data 702 based on the second convolutional kernel 7024, simulating off-center bipolar neurons. It can be understood that in practical applications, the first part of the computational resources (or computational nodes) executing convolutional layer G1_1 can perform peripheral convolution calculations, while the second part of the computational resources executing convolutional layer G1_1 can perform central convolution calculations in parallel. For example... Figure 7 As shown, convolutional layer G1_1 can perform a first convolution operation on the input data based on the first convolution kernel 7022 to obtain a first intermediate result 704. The weights in the central region of the first convolution kernel are 0, indicating that no convolution calculation is performed on the values ​​corresponding to the weights in the central region of the first convolution kernel. For example, as... Figure 7 As shown, the first convolution kernel 7022 can be a hollow 3*3 convolution kernel. The first convolution operation can be called a peripheral convolution operation.

[0060] Furthermore, while performing peripheral convolution, convolutional layer G1_1 can also perform a second convolution operation on the input data based on the second convolution kernel 7024 to obtain a second intermediate result 706. Here, the weights in the central region of the second convolution kernel 7024 are valid, while the weights in the peripheral region of the second convolution kernel 7024 can be 0. For example, as... Figure 7 As shown, the second convolution kernel can be a 1*1 kernel with a center. The second convolution operation can be called a center convolution operation. When the center convolution operation is performed in convolutional layer G1_1, it means that convolution calculation is performed only on the values ​​corresponding to the weights in the center region of the second convolution kernel. In this embodiment of the invention, the first convolution kernel 7022 and the second convolution kernel 7024 are the same size, for example, both are 3*3. It is understood that 3*3 is merely one example of the size of the convolution kernel; in practical applications, the size of the convolution kernel can also be 4*4, 5*5, 9*9, etc., and the size of the convolution kernel is not limited here. Furthermore, in practical applications, the second convolution kernel can also have only one value at the center. Figure 7The center of the second convolution kernel 7024 is exemplified as 3*3 in size for the purpose of illustrating the difference between the first convolution kernel 7022 and the second convolution kernel 7024. It can be understood that the side length of the central region of a convolution kernel needs to be smaller than the side length of the convolution kernel. For example, when the convolution kernel is 3*3 in size, the central region of the convolution kernel can be 1 weight located at the center of the convolution kernel. When the convolution kernel is 4*4 in size, the central region of the convolution kernel can be 4 weights surrounding the center of the convolution kernel.

[0061] Continuing to refer to FIG. 7, Figure 7 After the 3*3 hollow convolution is performed by the convolution layer G1_1, a first intermediate result 704 can be obtained. Meanwhile, after the 1*1 central convolution is performed by the convolution layer G1_1 in parallel, a second intermediate result 706 can be obtained. Further, the first intermediate result 704 and the second intermediate result 706 can be superimposed to obtain the calculation result of the convolution layer G1_1. In the implementation of the present application, the calculation result of the convolution layer G1_1 can include two types: a first calculation result 708 and a second calculation result 710. The first calculation result 708 is the calculation result of strengthening the central peripheral effect. The second calculation result 710 is equivalent to the calculation result of performing a normal 3*3 convolution and does not strengthen the central peripheral effect. Specifically, when the first intermediate result 704 is greater than 0 and the second intermediate result 706 is less than 0, or when the first intermediate result 704 is less than 0 and the second intermediate result 706 is greater than 0, the first calculation result 708 is output. When the first intermediate result 704 is greater than 0 and the second intermediate result 706 is greater than 0, or when the first intermediate result 704 is less than 0 and the second intermediate result 706 is less than 0, the second calculation result 710 is output.

[0062] Referring to FIG. 7, Figure 6 When the convolution module G1 4051 includes two convolution layers, after the calculation result of the convolution layer G1_1 is obtained, the calculation result of the convolution layer G1_1 can be sent to the convolution layer G1_2 for further convolution calculation. Similar to the convolution layer G1_1, after the convolution layer G1_2 simultaneously performs the peripheral convolution and the central convolution based on the set convolution kernel, the calculation result of the convolution layer G1_2 can be output. It can be understood that when the convolution module G1 4051 only includes the convolution layer G1_1 as shown in FIG. 7, Figure 6 When the convolution module G1 4051 includes the convolution layers G1_1 and G1_2 as shown in FIG. 7, Figure 6 When the convolution module G1 4051 includes the convolution layers G1_1 and G1_2 as shown in FIG. 7, the calculation result of the convolution layer G1_2 is the calculation result of the convolution module G1 4051. It can be understood that when the convolution module G1 4051 further includes other convolution layers, the calculation result of the last convolution layer is the calculation result of the convolution module G1 4051.

[0063] In combination Figure 6 Since the convolution modules G1 4051, G2 4052 and G3 4053 are three-level convolution, the input data of the convolution modules G1 4051, G2 4052 and G3 4053 are not the same. Specifically, the input data of the convolution module G1 4051 is the feature data 404 obtained in step 304 and the image data of the target image received in step 302. The input data of the convolution module G2 4052 is the output data of the convolution module G1 4051 (i.e., the calculation result of the convolution module G1 4051), and the input data of the convolution module G3 4053 is the output data of the convolution module G2 4052 (i.e., the calculation result of the convolution module G2 4052).

[0064] As mentioned above, the working principle of the convolution modules G2 4052 and G3 4053 is similar to that of the convolution module G1 4051, and the convolution layers in the convolution modules G2 4052 and G3 4053 can be referred to as Figure 7 The data processing flow is shown in the figure. Specifically, after obtaining the calculation result of the convolution module G1 4051, the convolution module G2 4052 can perform a second-level center_periphery convolution calculation on the calculation result of the convolution module G1 4051 based on the set network parameters. Specifically, the convolution module G2 4052 can perform a third convolution operation on the calculation result of the first-level center_periphery convolution based on a third convolution kernel to obtain a third intermediate result. Wherein, the value of the center region of the third convolution kernel is 0. At the same time of performing the third convolution operation, the convolution module G2 4052 can also perform a fourth convolution operation on the calculation result of the first-level center_periphery convolution based on a fourth convolution kernel to obtain a fourth intermediate result. Wherein, the fourth convolution kernel only includes the weight value of the center region, and the third convolution kernel and the fourth convolution kernel have the same size. The calculation result of the second-level center_periphery convolution is obtained based on the third intermediate result and the fourth intermediate result.

[0065] Similarly, after obtaining the calculation result of the convolution module G2 4052, the convolution module G3 4053 can perform a third level center_periphery convolution calculation on the calculation result of the convolution module G2 4052 based on the set network parameters. Specifically, the convolution module G3 4053 can perform a fifth convolution operation on the calculation result of the second level center_periphery convolution based on a fifth convolution kernel to obtain a fifth intermediate result. The fifth convolution kernel has a center region with a weight value of 0. At the same time of performing the fifth convolution operation, the convolution module G3 4053 can also perform a sixth convolution operation on the calculation result of the second level center_periphery convolution based on a sixth convolution kernel to obtain a sixth intermediate result. The sixth convolution kernel only includes a center region with a weight value, and the fifth convolution kernel and the sixth convolution kernel have the same size. The calculation result of the third level center_periphery convolution is obtained based on the fifth intermediate result and the sixth intermediate result.

[0066] Figure 7 For example, the convolution modules G2 4052 and G3 4053 each include two convolution layers. As described above, the convolution modules G2 4052 and G3 4053 can each include one or more convolution layers, and each convolution layer can perform the center convolution and the periphery convolution operation based on different convolution kernels. Similar to the convolution module G1 4051, the convolution module G2 4052 can obtain the calculation result of the convolution module G2 4052 based on the calculation result of one or more convolution layers in the convolution module G2 4052, and the convolution module G3 4053 can obtain the calculation result of the convolution module G3 4053 based on the calculation result of one or more convolution layers in the convolution module G3 4053.

[0067] In the embodiments of the present application, the convolution kernels set in each convolution layer in the convolution modules G1 4051, G2 4052 and G3 4053 can also be collectively referred to as the network parameters of the feature drift module 405. The network parameters are obtained after training on a plurality of low-quality images, and can be used to indicate the corresponding relationship between the low-quality image feature data and the clear image feature data. The convolution kernels of different convolution layers can be different, and the convolution kernels performing the periphery convolution and the center convolution in the same convolution layer have the same size. It should be noted that, since the feature drift module 405 provided in the embodiments of the present application is implemented based on the feature drift rule of the image, the corresponding relationship between the low-quality image feature data and the clear image feature data indicated by the network parameters of the feature drift module 405 obtained by training is irrelevant to the specific content of the image.

[0068] Again referring to Figure 6 When each convolution layer in the convolution modules G1 4051, G2 4052 and G3 4053 respectively performs the center convolution and the periphery convolution operation as described above, the feature drift module 405 can obtain the calculation result of the convolution module G1 4051 based on the calculation result of the first convolution layer in the convolution module G1 4051, and obtain the calculation result of the convolution module G2 4052 based on the calculation result of the first convolution layer in the convolution module G2 4052. Figure 7After the center-periphery convolution shown, the output results of the convolution modules G1 4051, G2 4052 and G3 4053 can be input into the processing module 4054 for accumulation processing, so as to obtain residual data 406 corresponding to the target image. The residual data 406 is used to indicate the deviation between the feature data of the target image and the clear image feature data. In actual application, the processing module 4054 can also be a convolution layer, and a 1*1 convolution operation can be performed on the output results of the convolution modules G1 4051, G2 4052 and G3 4053 based on the weights set in the processing module 4054. The weights in the processing module 4054 can be set as the weights A1, A2 and A3 in the above formula (3).

[0069] In step 308, enhanced image feature data is obtained according to the residual data and the shallow layer feature data. Specifically, the residual data 406 and the shallow layer feature data 404 can be superimposed by the enhancement processing module 407, so as to obtain the enhanced image feature data 408. It can be understood that the enhancement processing module 407 can be implemented by an adder or a convolution layer, and the implementation manner of the enhancement processing module 407 is not limited here.

[0070] In step 310, the target image is processed based on the enhanced image feature data, and a processing result is obtained. Specifically, after obtaining the enhanced image feature data 408, the enhanced image feature data 408 can be input into the next layer neural network 409, so as to process the target image 402 based on the enhanced image feature data 408, so as to obtain the final processing result of the target image. For example, the target image can be recognized, classified, detected and the like based on the enhanced image feature data 408.

[0071] From the description of the image processing method provided by the embodiment of the present application, it can be known that in the embodiment of the present application, the low-quality image itself is not processed in advance, but in the image processing process, the network parameters are used to process the image data of the low-quality image, obtain the enhanced image feature data of the low-quality image, and process the low-quality image based on the enhanced image feature data. Since the network parameters reflect the corresponding relationship between the feature data of the low-quality image and the feature data of the clear image, that is, the relationship between the low-quality image feature and the clear image feature is used to process the feature of the low-quality target image in the processing process, the network feature can be improved, the processing effect of the low-quality image can be improved, and for example, the recognition accuracy of the low-quality image can be improved.

[0072] Further, the image processing method provided by the embodiment of the present application utilizes the feature drift attribute of the image in the image processing process, and processes the shallow features of the low-quality image based on the center-peripheral convolution mechanism constructed according to the non-classical receptive field structure of the retina and the photosensitive principle of the bipolar cell to obtain enhanced image feature data, and then processes the low-quality image based on the enhanced image feature data. Since the processing process refers to the non-classical receptive field structure of the retina and simulates the photosensitive principle of the bipolar cell of the retina, the function of enhancing the high-frequency information in the target image and maintaining the low-frequency information in the target image is achieved, so that the enhanced image feature data is easier to be recognized or extracted, and the processing effect is better, for example, the recognition accuracy of the low-quality image can be improved. And the robustness (or stability) of the network can be stronger. Further, since the embodiment of the present application utilizes the feature drift attribute of the image to process the low-quality image, the processing process does not need the supervision of the semantic signal (used to indicate the image content), and the network parameters are less.

[0073] As described above, the network parameters of the feature drift module 405 in the embodiment of the present application are obtained according to training. The training process of the network parameters of the feature drift module 405 will be briefly introduced below. Figure 8 A training schematic diagram of a neural network system provided by the embodiment of the present application is shown in FIG. 8. Similar to FIG. 4, Figure 8 The first shallow feature calculation module 803, the feature drift module 405, the enhancement processing module 807, the second shallow feature calculation module 809, and the error calculation module 811 shown in FIG. 8 are logical concepts and can be neural network calculations performed by a neural network device, wherein the feature drift module 405 is a neural network to be trained. It should be noted that, Figure 8 The training process provided by the embodiment of the present application can be directly trained in the neural network circuit as shown in Figure 1 FIG. 4, or can be trained on a central processing unit (CPU), a graphic processing unit (GPU), a tensor processing unit (TPU), or the like. The training scenario is not limited herein.

[0074] In practical applications, multiple clear images can be selected based on their resolution before training; for example, more than 15 clear images with rich content can be chosen. A degraded image imaging model then generates multiple low-quality images from the selected clear images to obtain a training set. The generated low-quality images can include various types and degrees of degradation. For example, 15 degradation types can be considered, each of which can include at least 5 degrees of degradation. That is, 15 types of low-quality images of different degradation types can be generated for each clear image, and each degradation type can include at least 5 degrees of degradation.

[0075] like Figure 8 As shown, during training, a low-quality image 802 can be input into a first feature acquisition module 803 to obtain first feature data 804 of the low-quality image 802. The first feature data 804 may include shallow feature data of the low-quality image 802. Furthermore, a clear image 810 can be input into a second feature acquisition module 809 to obtain second feature data 812 of the clear image 810. The second feature data 812 may include shallow feature data of the clear image 810. Figure 4 Similar to the feature acquisition module 403 shown, both the first feature acquisition module 803 and the second feature acquisition module 809 can be the first N layers of a neural network system, where N is less than the preset threshold. For example, both the first feature acquisition module 803 and the second feature acquisition module 809 can be the first N layers of a neural network on a VGG16 or AlexNet network. VGG16 and AlexNet are two network models. For example, in a VGG16 or AlexNet network, the feature data output from the first pooling layer ("pooling1") and the first convolutional layer ("Conv1") can be selected as the first feature data 804, respectively. In this embodiment of the invention, the type of neural network used to extract feature data from low-quality images is not limited. Furthermore, it should be noted that in this embodiment of the invention, the size of the input image and the size of the output feature map are not limited and can be set according to the network and user needs.

[0076] After obtaining the first shallow feature data 804, the first shallow feature data 804 and the image data of the low-quality image 802 can be input into the feature drift module 405. The feature drift module 405 can then... Figure 6 The network structure shown and Figure 7 The convolution process shown obtains the residual data of the trained image data. For a description of the feature shift module 405, please refer to step 306 above. Figure 6 and Figure 7 The description is as follows. It is understandable that the computational process of the training process is similar to the computational process of the aforementioned image processing process, except that the training process...Figure 6 The network parameters (or convolution kernels) set in each convolution module and the values of the network parameters set after training are different. The purpose of training is to obtain appropriate network parameters. In the training process, the feature drift module 405 can first process the input low-quality image according to the network parameters initially set in each convolution module. It can be understood that the network parameters initially set in the feature drift module 405 can be obtained by Gaussian initialization or other initialization methods (such as Xavier).

[0077] In reference to the calculation process shown in Figure 6 and Figure 7 After obtaining the residual data 806 of the low-quality image 802, the obtained residual data 802 and the first feature data 804 can be input into the enhancement processing module 807 for accumulation processing to obtain enhanced image feature data 808. The enhancement processing module 807 can be implemented by an adder or a convolution layer.

[0078] Further, the error calculation module 811 can compare the enhanced image feature data 808 and the second shallow feature data 812 of the clear image 810 to obtain the error between the enhanced image feature data 808 and the second shallow feature data 812 of the clear image 810. In actual application, the error calculation module 811 can use the mean square error (MSE) function to calculate the error between the enhanced image feature data 808 and the second shallow feature data 812 of the clear image 810. After calculating the error, the network parameters in the feature drift module 405 can be optimized by gradient backpropagation according to the calculated error. It should be noted that in the process of adjusting the network parameters according to the error, the weights in the first shallow feature calculation module 803 can be kept unchanged, and only the network parameters in the feature drift module 405 can be optimized. That is, the weights in each convolution module in the feature drift module 405 can be adjusted according to the error.

[0079] In actual application, after multiple training and learning of multiple low-quality images in the training set, the error calculation module 811 can obtain an error less than a preset threshold, so as to obtain the trained network parameters of the feature drift module 405. That is, the network parameters of the feature drift module 405 after the error converges can be used as the network parameters of the feature drift module 405 applied in the image processing process.

[0080] The training method provided in this embodiment of the invention utilizes the feature drift properties of images during training, and no semantic signals are used for supervision during training. Therefore, the trained feature drift module 405 can be applied to any low-quality image of the same type as the training data. In other words, the network parameters obtained through training in this embodiment of the invention can be embedded into an existing neural network to process input degraded images without needing to be retrained according to the actual application scenario. Furthermore, since the low-quality image is identified by utilizing the relationship between low-quality image features and clear image features, the identifiability of network features can be improved, thereby improving the processing effect of low-quality images, for example, improving the recognition accuracy of low-quality images.

[0081] Figure 9 This is a schematic diagram of another image processing apparatus provided in an embodiment of the present invention. Figure 9 As shown, the image processing apparatus 900 may include a receiving module 902, a feature enhancement module 904, and a processing module 906. The receiving module 902 receives image data of a target image, which is a low-quality image. The feature enhancement module 904 processes the image data based on network parameters to obtain enhanced image feature data of the target image, wherein the network parameters indicate the correspondence between feature data of the low-quality image and feature data of a clear image. The processing module 906 processes the target image based on the enhanced image feature data.

[0082] Specifically, the feature enhancement module 904 can obtain feature data of the target image based on the image data. After obtaining the feature data, it can perform neural network calculations on the feature data and the image data based on the network parameters to obtain residual data, and obtain enhanced image feature data of the target image based on the residual data and the feature data. The feature data is obtained by calculating the image data through an N-layer neural network, where N is greater than 0 and less than a preset threshold. The residual data is used to indicate the deviation between the feature data of the target image and the feature data of the clear image.

[0083] In the process of obtaining the enhanced image feature data of the target image, the feature enhancement module 904 is configured to perform center-surround convolution calculation on the feature data and the image data based on the network parameters. In an embodiment, the feature enhancement module 904 can perform at least a first level center-surround convolution calculation, a second level center-surround convolution calculation and a third level center-surround convolution calculation on the feature data and the image data based on the set network parameters. The input data of the first level center-surround convolution calculation includes the feature data and the image data, the input data of the second level center-surround convolution calculation includes the calculation result of the first level center-surround convolution calculation, and the input data of the third level center-surround convolution calculation includes the calculation result of the second level center-surround convolution calculation. The feature enhancement module 904 can obtain the residual data based on the calculation result of the first level center-surround convolution calculation, the calculation result of the second level center-surround convolution calculation and the calculation result of the third level center-surround convolution calculation.

[0084] In an embodiment, the feature enhancement module 904 can perform a first convolution operation on the feature data and the image data based on a first convolution kernel to obtain a first intermediate result, wherein the weight value of the center region of the first convolution kernel is 0. And the feature enhancement module 904 can perform a second convolution operation on the feature data and the image data based on a second convolution kernel to obtain a second intermediate result, wherein the second convolution kernel only includes the weight value of the center region, and the size of the first convolution kernel and the second convolution kernel is the same. Further, the feature enhancement module 904 can obtain the calculation result of the first level center-surround convolution based on the first intermediate result and the second intermediate result.

[0085] In an embodiment, the feature enhancement module 904 can further perform a third convolution operation on the calculation result of the first level center-surround convolution based on a third convolution kernel to obtain a third intermediate result, wherein the value of the center region of the third convolution kernel is 0. And perform a fourth convolution operation on the calculation result of the first level center-surround convolution based on a fourth convolution kernel to obtain a fourth intermediate result, wherein the fourth convolution kernel only includes the weight value of the center region, and the size of the third convolution kernel and the fourth convolution kernel is the same. Thus, the calculation result of the second level center-surround convolution can be obtained according to the third intermediate result and the fourth intermediate result.

[0086] In an implementation manner, the feature enhancement module 904 can further perform a fifth convolution operation on the calculation result of the second center_periphery convolution based on a fifth convolution kernel, to obtain a fifth intermediate result, where the fifth convolution kernel has a center region with a weight value of 0. In addition, the feature enhancement module 904 can further perform a sixth convolution operation on the calculation result of the second center_periphery convolution based on a sixth convolution kernel, to obtain a sixth intermediate result, where the sixth convolution kernel only includes a weight value of the center region, and the fifth convolution kernel and the sixth convolution kernel have the same size. Further, the feature enhancement module 904 can obtain the calculation result of the third center_periphery convolution based on the fifth intermediate result and the sixth intermediate result.

[0087] Figure 9 The image processing apparatus shown does not pre-process the low-quality image itself, but in the image processing process, the image data of the low-quality image is processed by using the set network parameters, to obtain enhanced image feature data of the low-quality image, and the low-quality image is processed based on the enhanced image feature data. Since the network parameters reflect the corresponding relationship between the feature data of the low-quality image and the feature data of the clear image, the processing effect of the low-quality target image is better. Specifically, the image processing apparatus provided in the embodiment of the present application uses the feature drift attribute of the image, and processes the shallow features of the low-quality image by using the center_periphery convolution mechanism constructed according to the non-classical receptive field structure of the retina and the photosensitive principle of the bipolar cells, to obtain enhanced image feature data, and then processes the low-quality image based on the enhanced image feature data, so that the processing effect of the image is better, and the recognition accuracy is higher.

[0088] It can be understood that, Figure 9 The modules in the image processing apparatus 900 shown can be located in Figure 1 one or more devices in the image processing apparatus. In the embodiment of the present application, the modules in the image processing apparatus can be selected according to actual needs. Figure 9 Part or all of the modules in the embodiment shown can be used to achieve the purpose of the embodiment. Figure 9 The parts not described in detail in the embodiment can refer to the related description in Figure 1-8 the embodiment shown.

[0089] It can be understood that the above-described apparatus embodiments are merely illustrative, for example, the division of the modules is merely a logical function division, and another division manner can be used in actual implementation. For example, a plurality of modules or components can be combined or integrated into another system, or some features can be omitted or not executed. In addition, the connection between the modules discussed in the above embodiments can be electrical, mechanical or other forms. The modules described as separate components can or can not be physically separated. The components shown as modules can or can not be physical modules. In addition, each function module in each embodiment of the application embodiments can exist independently or can be integrated into a processing module. For example, Figure 9 The function modules shown can be integrated in Figure 1 The neural network circuit or the processor shown can be implemented by corresponding devices.

[0090] The application embodiments further provide a computer program product for data processing, including a computer readable storage medium storing program codes, and the program codes include instructions for executing the method flow of any one of the above method embodiments. Those skilled in the art can understand that the above storage medium includes a U disk, a mobile hard disk, a magnetic disk, an optical disk, a random access memory (RAM), a solid state disk (SSD) or a non-volatile memory, and various non-transitory machine readable media that can store program codes.

[0091] It should be noted that the embodiments provided in the present application are merely illustrative. Those skilled in the art can clearly understand that, for the convenience and brevity of description, the description of each embodiment in the above embodiments is focused on, and the parts not described in detail in a certain embodiment can be referred to the relevant description of other embodiments. The features disclosed in the embodiments of the present application, claims and drawings can exist independently or in combination. The features described in the embodiments of the present application in the form of hardware can be executed by software, and vice versa. This is not limited.

Claims

1. An image processing method, characterized by, The method comprises: receiving image data of a target image, the target image being a low-quality image; obtaining feature data of the target image according to the image data; performing at least first, second and third center-periphery convolution calculations on the feature data and the image data based on set network parameters to obtain enhanced image feature data of the target image, wherein the network parameters are used to indicate a corresponding relationship between the feature data of the low-quality image and the feature data of a clear image, the first center-periphery convolution calculation is used to simulate a response of a central region of a human eye retina to the target image, the second center-periphery convolution calculation is used to simulate a response of a peripheral region of the human eye retina to the target image, and the third center-periphery convolution calculation is used to simulate a response of an edge region of the human eye retina to the target image; processing the target image based on the enhanced image feature data.

2. The image processing method of claim 1, wherein, The feature data is feature data obtained by performing N-layer neural network calculation on the image data, N being greater than 0 and less than a preset threshold; The performing at least first, second and third center-periphery convolution calculations on the feature data and the image data based on the set network parameters comprises: performing at least first, second and third center-periphery convolution calculations on the feature data and the image data based on the network parameters to obtain residual data, the residual data being used to indicate a deviation between the feature data of the target image and the feature data of a clear image; obtaining enhanced image feature data of the target image according to the residual data and the feature data.

3. The image processing method of claim 1, wherein: Input data of the first center-periphery convolution calculation comprises the feature data and the image data, input data of the second center-periphery convolution calculation comprises a calculation result of the first center-periphery convolution calculation, and input data of the third center-periphery convolution calculation comprises a calculation result of the second center-periphery convolution calculation.

4. The image processing method of claim 1 or 2, characterized by, The first center-periphery convolution calculation comprises: performing a first convolution operation on the feature data and the image data based on a first convolution kernel to obtain a first intermediate result, wherein a weight value of a central region of the first convolution kernel is 0; performing a second convolution operation on the feature data and the image data based on a second convolution kernel to obtain a second intermediate result, wherein the second convolution kernel only comprises a weight value of a central region, and the first convolution kernel and the second convolution kernel have the same size; obtaining a calculation result of the first center-periphery convolution based on the first intermediate result and the second intermediate result.

5. The image processing method of claim 1 or 2, characterized by, The second center-periphery convolution calculation comprises: performing a third convolution operation on the calculation result of the first center-periphery convolution based on a third convolution kernel to obtain a third intermediate result, wherein a numerical value of a central region of the third convolution kernel is 0; performing a fourth convolution operation on the calculation result of the first center-periphery convolution based on a fourth convolution kernel, to obtain a fourth intermediate result, wherein the fourth convolution kernel only includes weight values of the center region, and the third convolution kernel and the fourth convolution kernel have the same size; obtaining the calculation result of the second center-periphery convolution based on the third intermediate result and the fourth intermediate result.

6. The image processing method of claim 1 or 2, wherein, The third center-periphery convolution calculation includes: performing a fifth convolution operation on the calculation result of the second center-periphery convolution based on a fifth convolution kernel, to obtain a fifth intermediate result, wherein the weight value of the center region of the fifth convolution kernel is 0; performing a sixth convolution operation on the calculation result of the second center-periphery convolution based on a sixth convolution kernel, to obtain a sixth intermediate result, wherein the sixth convolution kernel only includes weight values of the center region, and the fifth convolution kernel and the sixth convolution kernel have the same size; obtaining the calculation result of the third center-periphery convolution based on the fifth intermediate result and the sixth intermediate result.

7. The image processing method of claim 1 or 2, characterized by, The method is performed by a neural network device, and the network parameters are obtained after training.

8. An image processing apparatus characterized by comprising: comprise: a receiving module configured to receive image data of a target image, the target image being a low-quality image; a feature enhancement module configured to obtain feature data of the target image according to the image data, and perform at least first center-periphery convolution calculation, second center-periphery convolution calculation, and third center-periphery convolution calculation on the feature data and the image data based on set network parameters, to obtain enhanced image feature data of the target image, wherein the network parameters are used to indicate a corresponding relationship between the feature data of the low-quality image and the feature data of a clear image, the first center-periphery convolution calculation is used to simulate a response of a center region of a human eye retina to the target image, the second center-periphery convolution calculation is used to simulate a response of a peripheral region of the human eye retina to the target image, and the third center-periphery convolution calculation is used to simulate a response of an edge region of the human eye retina to the target image; a processing module configured to process the target image based on the enhanced image feature data.

9. The image processing apparatus according to claim 8, characterized by, The feature data is feature data obtained by performing N-layer neural network calculation on the image data, N being greater than 0 and less than a preset threshold; The feature enhancement module is specifically configured to perform at least first center-periphery convolution calculation, second center-periphery convolution calculation, and third center-periphery convolution calculation on the feature data and the image data based on the network parameters, to obtain residual error data, the residual error data being used to indicate a deviation between the feature data of the target image and the feature data of a clear image; obtaining enhanced image feature data of the target image according to the residual error data and the feature data.

10. The image processing apparatus according to claim 8, characterized by: The input data of the first center-periphery convolution calculation includes the feature data and the image data, the input data of the second center-periphery convolution calculation includes the calculation result of the first center-periphery convolution calculation, and the input data of the third center-periphery convolution calculation includes the calculation result of the second center-periphery convolution calculation.

11. The image processing apparatus according to claim 8 or 9, characterized by The feature enhancement module is configured to: perform a first convolution operation on the feature data and the image data based on a first convolution kernel, to obtain a first intermediate result, wherein the center region of the first convolution kernel has a weight value of 0; perform a second convolution operation on the feature data and the image data based on a second convolution kernel, to obtain a second intermediate result, wherein the second convolution kernel only includes a weight value of the center region, and the first convolution kernel and the second convolution kernel have the same size; obtain the calculation result of the first center-periphery convolution based on the first intermediate result and the second intermediate result.

12. The image processing apparatus according to claim 8 or 9, characterized by The feature enhancement module is configured to: perform a third convolution operation on the calculation result of the first center-periphery convolution based on a third convolution kernel, to obtain a third intermediate result, wherein the center region of the third convolution kernel has a weight value of 0; perform a fourth convolution operation on the calculation result of the first center-periphery convolution based on a fourth convolution kernel, to obtain a fourth intermediate result, wherein the fourth convolution kernel only includes a weight value of the center region, and the third convolution kernel and the fourth convolution kernel have the same size; obtain the calculation result of the second center-periphery convolution based on the third intermediate result and the fourth intermediate result.

13. The image processing apparatus according to claim 8 or 9, characterized by The feature enhancement module is configured to: perform a fifth convolution operation on the calculation result of the second center-periphery convolution based on a fifth convolution kernel, to obtain a fifth intermediate result, wherein the center region of the fifth convolution kernel has a weight value of 0; perform a sixth convolution operation on the calculation result of the second center-periphery convolution based on a sixth convolution kernel, to obtain a sixth intermediate result, wherein the sixth convolution kernel only includes a weight value of the center region, and the fifth convolution kernel and the sixth convolution kernel have the same size; obtain the calculation result of the third center-periphery convolution based on the fifth intermediate result and the sixth intermediate result.

14. The image processing apparatus according to claim 8 or 9, characterized by The image processing apparatus is applied to a neural network device, and the network parameters are obtained after training.

15. An image processing apparatus characterized by comprising: A neural network for implementing the image processing method according to any one of claims 1-7.

16. A computer-readable storage medium, characterized in that, The computer readable storage medium is configured to store program codes including instructions executed by a computer to implement the image processing method according to any one of claims 1-7.

Images