Image processing method and device, and vehicle

Abstract

This application provides an image processing method and device, and a vehicle, and relates to fields such as artificial intelligence, autonomous driving and computer vision. The method is mainly applied to a convolutional neural network model, and the method includes: obtaining a current RAW image frame; determining a first feature of the current RAW image frame; analyzing context feature information of the current RAW image frame, where the context feature information includes local feature information and global feature information; determining local features and global features in the first feature based on the local feature information and the global feature information; and fusing the global feature and the local feature to obtain a feature fusion map of the current RAW image frame. According to the solution of this application, the algorithm complexity in the image processing procedure can be reduced.

Description

[Technical Field]

[0001] The present application relates to the field of artificial intelligence, and more particularly to an image processing method and apparatus, and a vehicle. [Background technology]

[0002] With the continuous development of autonomous driving technology, on-board cameras are becoming more and more widely used. The working principle of on-board cameras is as follows: an optical image of an object is generated by using a lens, which is projected onto an image sensor to obtain a Bayer image, and then an analog electrical signal is obtained through optical-electrical conversion, and a digital image signal (i.e., a raw image) is obtained through noise reduction and analog-to-digital conversion. After obtaining the raw image, it needs to be processed, and then an image of a specific format is generated for the autonomous driving system to perform perception processing.

[0003] In existing solutions, RAW images are usually processed by image signal processing (ISP). However, ISP is a set of complex image processing algorithms (including black level compensation, bad pixel correction, color interpolation, Bayer noise reduction, white balance correction, color reproduction and correction, gamma correction, color noise reduction and edge enhancement in color space, color and contrast enhancement, automatic exposure control, etc.). In addition, each independent algorithm needs to adapt to different weather and scenarios, resulting in high algorithm complexity.

[0004] Therefore, how to reduce the algorithm complexity in image processing procedures is an urgent technical problem to be solved. Summary of the Invention [Problem to be solved by the invention]

[0005] The present application provides an image processing method and apparatus, and a vehicle, that reduces the algorithmic complexity in image processing procedures. [Means for solving the problem]

[0006] According to a first aspect, an image processing method is applied to a convolutional neural network model, the method including the steps of: obtaining a current RAW image frame; determining first features of the current RAW image frame; analyzing context feature information of the current RAW image frame, where the context feature information includes local feature information and global feature information; determining local features and global features in the first features based on the local feature information and the global feature information; and fusing the local features and the global features to obtain a feature fusion map for the current RAW image frame.

[0007] In this embodiment of the present application, the convolutional neural network model is mainly used to replace the ISP in the existing solution for processing the current RAW image frame, thereby reducing the algorithm complexity in the image processing procedure. In addition, in the image processing procedure, the local features and global features of the current RAW image frame are determined based on the context feature information, thereby improving the accuracy of determining the local features and global features, thereby improving the quality of the final image.

[0008] In relation to the first aspect, in some implementations of the first aspect, the method further includes a step of acquiring a previous RAW image frame and a step of extracting a second feature of the previous RAW image frame, and the step of determining the first feature of the current RAW image frame includes a step of extracting a third feature of the current RAW image frame and a step of determining the first feature of the current RAW image frame based on the second feature and the third feature.

[0009] In this embodiment of the present application, the first feature of the current RAW image frame can be determined based on features extracted from two consecutive frames of the image, which can ensure that the determined first feature of the current RAW image frame is robust, thereby ensuring the time-series consistency in the brightness and color of the video and avoiding the flicker phenomenon.

[0010] In relation to the first aspect, in some implementations of the first aspect, the step of determining a first feature of the current RAW image frame based on the second feature and the third feature includes the steps of: determining a coincident-scene area between the previous RAW image frame and the current RAW image frame; determining a fourth feature that is within the second feature and corresponds to the coincident-scene area based on the coincident-scene area; determining a fifth feature that is within the third feature and corresponds to the coincident-scene area based on the coincident-scene area; and determining the first feature of the current RAW image frame based on a difference between the fourth feature and the fifth feature.

[0011] In this embodiment of the present application, the first feature of the current RAW image frame can be determined based on the difference between the features corresponding to the matching scene areas of two consecutive frames of the image, so that the difference in the same scene area of ​​the two consecutive frames is within a suitable range, thereby ensuring that the determined first feature of the current RAW image frame is robust, thereby ensuring the time-series consistency in the brightness and color of the video and avoiding the flicker phenomenon.

[0012] In relation to the first aspect, in some implementations of the first aspect, determining a first feature of the current RAW image frame based on a difference between the fourth feature and the fifth feature includes: determining a first feature of the current RAW image frame based on the fifth feature if the difference between the fourth feature and the fifth feature is less than or equal to a first threshold; determining a first feature of the current RAW image frame based on the fourth feature and the fifth feature if the difference between the fourth feature and the fifth feature is greater than the first threshold and less than a second threshold; or determining a first feature of the current RAW image frame based on the fourth feature if the difference between the fourth feature and the fifth feature is greater than or equal to the second threshold.

[0013] In relation to the first aspect, in some implementations of the first aspect, the resolution of the previous RAW image frame is half the resolution of the current RAW image frame.

[0014] In this embodiment of the present application, the resolution of the previous RAW image frame is set to half of the resolution of the current RAW image frame, which can reduce the processing amount of the convolutional neural network model and further reduce the requirements on computational power.

[0015] In relation to the first aspect, in some implementations of the first aspect, the same weighting factor is used to extract the third feature of the current RAW image frame and the second feature of the previous RAW image frame.

[0016] In this embodiment of the present application, the third feature of the current RAW image frame and the second feature of the previous RAW image frame are extracted using the same weight coefficient. In this way, the convolutional neural network model can extract features of two frames of images by sharing weight values. In other words, the features of two frames of images are extracted by using a weight matrix. In this case, features corresponding to matching scene areas in two frames of images can be compared more intuitively, thereby improving the accuracy of data processing. In addition, the amount of network parameters that need to be stored in the convolutional neural network model can be further reduced.

[0017] In relation to the first aspect, in some implementations of the first aspect, the method further includes generating an RGB image based on the feature fusion map.

[0018] With respect to the first aspect, in some implementations of the first aspect, the method further includes transmitting the feature fusion map or the RGB image to a perceptual processing module.

[0019] In this embodiment of the present application, the obtained feature fusion map can be directly transmitted to the perceptual processing module, which can reduce the requirements of the perceptual processing module in terms of computational power.

[0020] In relation to the first aspect, in some implementations of the first aspect, the method further includes evaluating the generated RGB image and optimizing the convolutional neural network model based on the evaluation results.

[0021] Optionally, the generated RGB image may be evaluated by using an evaluation model.

[0022] Optionally, the evaluation model used for evaluation may be pre-trained by using a sample RGB image training set. The evaluation model may be a neural network model. The model may be used to evaluate RGB images generated by using a convolutional neural network model. In addition, the evaluation results can facilitate continuous optimization of the convolutional neural network model. In this way, the images output by the convolutional neural network model are more useful for the perception algorithm, resulting in improved accuracy of perception processing.

[0023] In relation to the first aspect, in some implementations of the first aspect, the global features include at least one of overall brightness, white balance, color, and contrast, and the local features include at least one of local texture detail and local contrast.

[0024] According to a second aspect, there is provided an image processing apparatus for use in a convolutional neural network model, the apparatus including: an acquisition module configured to acquire a current RAW image frame; and a processing module configured to determine first features of the current RAW image frame, analyze context feature information of the current RAW image frame, where the context feature information includes local feature information and global feature information, determine local features and global features within the first features based on the local feature information and the global feature information, and fuse the local features and the global features to obtain a feature fusion map for the current RAW image frame.

[0025] In relation to the second aspect, in some implementations of the second aspect, the acquisition module is further configured to acquire a previous RAW image frame, and the processing module is further configured to extract a second feature of the previous RAW image frame, extract a third feature of the current RAW image frame, and determine a first feature of the current RAW image frame based on the second feature and the third feature.

[0026] In relation to the second aspect, in some implementations of the second aspect, the processing module is further configured to determine a matching scene area between the previous RAW image frame and the current RAW image frame, determine a fourth feature that is within the second feature and corresponds to the matching scene area based on the matching scene area, determine a fifth feature that is within the third feature and corresponds to the matching scene area based on the matching scene area, and determine a first feature of the current RAW image frame based on a difference between the fourth feature and the fifth feature.

[0027] In relation to the second aspect, in some implementations of the second aspect, the processing module is further configured to determine a first feature of the current RAW image frame based on the fifth feature if a difference between the fourth feature and the fifth feature is less than or equal to a first threshold, determine a first feature of the current RAW image frame based on the fourth feature and the fifth feature if a difference between the fourth feature and the fifth feature is greater than the first threshold and less than a second threshold, or determine a first feature of the current RAW image frame based on the fourth feature if a difference between the fourth feature and the fifth feature is greater than or equal to the second threshold.

[0028] In relation to the second aspect, in some implementations of the second aspect, the resolution of the previous RAW image frame is half the resolution of the current RAW image frame.

[0029] In relation to the second aspect, in some implementations of the second aspect, the same weighting factor is used to extract the third feature of the current RAW image frame and the second feature of the previous RAW image frame.

[0030] With respect to the second aspect, in some implementations of the second aspect, the processing module is further configured to generate an RGB image based on the feature fusion map.

[0031] With respect to the second aspect, in some implementations of the second aspect, the apparatus further includes a transmission module configured to transmit the feature fusion map or the RGB image to the perceptual processing module.

[0032] With respect to the second aspect, in some implementations of the second aspect, the processing module is further configured to evaluate the generated RGB image and optimize the convolutional neural network model based on the evaluation results.

[0033] In relation to the second aspect, in some implementations of the second aspect, the global features include at least one of overall brightness, white balance, color, and contrast, and the local features include at least one of local texture detail and local contrast.

[0034] According to a third aspect, there is provided an image processing device, comprising an input / output interface, a processor, and a memory, wherein the processor is configured to control the input / output interface to receive and send signals or information, the memory is configured to store a computer program, and the processor is configured to call the computer program from the memory and execute the computer program so that the image processing device performs the image processing method according to the first aspect or any one of the possible implementations of the first aspect.

[0035] According to a fourth aspect, a vehicle is provided, comprising a module configured to perform an image processing method according to the first aspect or any one of the possible implementations of the first aspect.

[0036] According to a fifth aspect, a terminal device is provided, comprising a module configured to perform an image processing method according to the first aspect or any one of the possible implementations of the first aspect.

[0037] Optionally, the terminal device may be any device related to the visual field, such as a mobile phone, a tablet computer, a vehicle, a computer, a machine vision recognition device or an intelligent security device.

[0038] According to a sixth aspect, there is provided a computing device including at least one processor and a memory, wherein the at least one processor is coupled to the memory and configured to read and execute instructions in the memory to perform an image processing method according to the first aspect or any one of the possible implementations of the first aspect.

[0039] According to a seventh aspect, there is provided a computer program product comprising instructions which, when executed on a computer, enable the computer to perform an image processing method according to the first aspect or any one of the possible implementations of the first aspect.

[0040] According to an eighth aspect, there is provided a computer-readable medium storing program code which, when executed on a computer, enables the computer to perform an image processing method according to the first aspect or any one of the possible implementations of the first aspect.

[0041] According to a ninth aspect, there is provided a chip, the chip including a processor and a data interface, the processor reading instructions stored in a memory via the data interface to perform the image processing method according to the first aspect or any one of the possible implementations of the first aspect.

[0042] Optionally, in one implementation, the chip may further include a memory, the memory storing instructions, and the processor configured to execute the instructions stored in the memory, such that when the instructions are executed, the processor is configured to perform an image processing method according to the first aspect or any one of the possible implementations of the first aspect.

[0043] According to a tenth aspect, there is provided a chip system, the chip system including at least one processor configured to support implementation of functionality according to the first aspect or possible implementations thereof, such as receiving or processing data and / or information in the manner described above.

[0044] In a possible design, the chip system further includes a memory configured to store program instructions and data, the memory being located internal or external to the processor, and the chip system may include the chip or may include separate components apart from the chip. [Brief explanation of the drawings]

[0045] [Figure 1] FIG. 1 is an exemplary diagram of an ISP's processing procedure.

[0046] [Figure 2] FIG. 1 is an exemplary diagram of a system architecture configuration according to an embodiment of the present application.

[0047] [Figure 3] FIG. 1 is an exemplary diagram of a convolutional neural network configuration, according to an embodiment of the present application.

[0048] [Figure 4] FIG. 1 is an exemplary diagram of another convolutional neural network configuration according to an embodiment of the present application.

[0049] [Figure 5] 2 is an exemplary diagram of a hardware structure of a chip according to an embodiment of the present application;

[0050] [Figure 6] FIG. 1 is an exemplary diagram of a system architecture for image processing, according to an embodiment of the present application.

[0051] [Figure 7]1 is an exemplary diagram of an image processing method according to an embodiment of the present application;

[0052] [Figure 8] FIG. 1 is an exemplary diagram of a system architecture of a convolutional neural network model, according to an embodiment of the present application.

[0053] [Figure 9] 1 is an exemplary diagram of an image processing device according to an embodiment of the present application;

[0054] [Figure 10] 1 is a block diagram of an example of a hardware structure of an apparatus according to an embodiment of the present application; DETAILED DESCRIPTION OF THE INVENTION

[0055] The technical solutions of the present application will be described below with reference to the accompanying drawings.

[0056] 1 is an exemplary diagram of the processing procedure of an ISP. As shown in FIG. 1, an optical image of a natural scene 11 is generated by using a lens 12 and projected onto an image sensor 13 to obtain a Bayer image, which then undergoes optical-to-electrical conversion 14 to obtain an analog electrical signal 15, and further undergoes noise reduction and analog-to-digital conversion (A / D conversion) 16 to obtain a digital image signal (i.e., a raw image) 17. The digital image signal may then enter an image signal processor (ISP) (i.e., a digital signal processing chip) 10.

[0057] The ISP 10 generally includes functions such as black level compensation (BLC) 18, lens shading correction 19, bad pixel correction (BPC) 20, demosaic 21, Bayer denoising 22, auto white balance (AWB) 23, Ga The image processing system includes modules such as a gamma correction (GC) 24, an auto exposure (AE) 25, an auto focus (AF) (not shown in FIG. 1), a color correction (CC) 26, a gamma correction (G) 27, a color gamut conversion (GA) 28, a color noise reduction / detail enhancement 29, a color enhancement (CE) 30, an encoder (formatter) 31, and an input / output (I / O) control 32.

[0058] It is clear that ISP10 is a complex set of image processing algorithms. In fact, each independent algorithm needs to adapt to different weather and different scenarios, resulting in a very complex algorithm and a large number of debugging parameters. Generally, multiple algorithms need to interact with each other to implement a function, which requires advanced skills from the optimizer.

[0059] Based on the above problem, this application provides an image processing method. A convolutional neural network model is mainly used to replace the ISP in existing solutions for processing a current RAW image frame, thereby reducing the algorithm complexity in the image processing procedure. In addition, in the image processing procedure, local features and global features of the current RAW image frame are determined based on context feature information, thereby improving the accuracy of determining the local features and global features. This can improve the quality of the final image.

[0060] The image processing method provided in the embodiments of the present application can be applied to camera vision related fields, such as autonomous driving, intelligent security, machine recognition and mobile phones.

[0061] A large number of neural networks are used in the embodiments of the present application, and therefore, for ease of understanding, the following first describes terms and concepts related to neural networks that may be used in the embodiments of the present application.

[0062] (1) Neural Networks

[0063] A neural network may include neurons, where x s and an intercept of 1 as input. The output of the operation unit may be:

number

[0064] where s is 1, 2, ..., n, where n is a natural number greater than 1, and W s x s where σ is the weight of the neuron, b is the bias of the neuron, and f is the activation function of the neuron. The activation function is used to introduce nonlinear features into the neural network to convert the neuron's input signal into an output signal. The output signal of the activation function may be used as the input of the next convolutional layer. The activation function may be a sigmoid function. A neural network is a network formed by connecting multiple single neurons together. Specifically, the output of one neuron may be the input of another neuron. The input of each neuron may be connected to the local receptive field of the previous layer to extract features of the local receptive field. The local receptive field may be an area containing several neurons.

[0065] (2) Deep Neural Networks

[0066] A deep neural network (DNN), also known as a multi-layer neural network, can be understood as a neural network with multiple hidden layers. DNNs are divided based on the location of different layers, and the neural network within a DNN can be divided into three types: input layer, hidden layer, and output layer. Generally, the first layer is the input layer, the last layer is the output layer, and the intermediate layers are hidden layers. The layers are fully connected. Specifically, any neuron in the i-th layer is always connected to any neuron in the i+1-th layer.

[0067] Although DNNs may seem complex, the behavior of each layer is not. Simply put, DNNs are expressed by the following linear relationships:

number

number

number

number

number

number

number

number

[0068] In conclusion, the coefficients from the kth neuron in the (L-1)th layer to the jth neuron in the Lth layer are

number

[0069] Note that there is no parameter W in the input layer. In a deep neural network, the more hidden layers there are, the better the network's ability to describe complex real-world cases. Theoretically, a model with more parameters has higher complexity and greater "capacity," which indicates that the model can complete more complex learning tasks. The process of training a deep neural network is the process of learning a weight matrix, and the ultimate goal of training is to obtain the weight matrix (the weight matrix formed by the vector W in many layers) in all layers of the trained deep neural network.

[0070] (3) Convolutional Neural Networks

[0071] A convolutional neural network (CNN) is a deep neural network with a convolutional structure. A convolutional neural network includes a feature extractor, which includes a convolutional layer and a subsampling layer. The feature extractor may be considered a filter. The convolution process may be considered as performing convolution on an input image or a convolutional feature map by using a trainable filter. A convolutional layer is a neuron layer in a convolutional neural network where a convolutional process is performed on an input signal. In a convolutional layer of a convolutional neural network, one neuron may only be connected to neurons in several adjacent layers. A convolutional layer usually includes several feature maps, and each feature map may include several neurons in a rectangular array. Neurons in the same feature plane share weights, and in this specification, the shared weights are convolution kernels. Weight sharing may be understood to mean that the method of extracting image information is location-independent. The principle implied in this specification is that the statistical information of one part of an image is the same as the statistical information of another part. This means that image information learned in one part can be used in another part. Therefore, the same image information obtained through learning can be used for all positions on the image. In the same convolution layer, multiple convolution kernels can be used to extract different image information. Generally, the more convolution kernels there are, the richer the image information reflected in the convolution operation.

[0072] The convolution kernels can be initialized in the form of a matrix of random size. In the process of training a convolutional neural network, appropriate weights for the convolution kernels can be obtained through learning. In addition, a direct benefit of weight sharing is that it reduces the connections between layers of the convolutional neural network, thereby reducing the risk of overfitting.

[0073] (4) Loss function

[0074] In the training process of a deep neural network, the output of the deep neural network is expected to be as close as possible to the actual expected predicted value. Therefore, the current predicted value of the network may be compared with the actual expected target value. Then, the weight vector of each layer of the neural network is updated based on the difference between the predicted value and the target value. (Indeed, there is usually an initialization process before the first update; specifically, parameters are pre-configured for all layers of the deep neural network.) For example, if the predicted value of the network is large, the weight vector is adjusted to lower the predicted value until the deep neural network can predict a value closer to the actual expected or desired target value. Therefore, it is necessary to predefine how to obtain the difference between the predicted value and the target value through comparison. This is the loss function or objective function. The loss function and objective function are important equations that measure the difference between the predicted value and the target value. We use the loss function as an example. The larger the output value (loss) of the loss function, the greater the difference. Therefore, training a deep neural network is a process of minimizing the loss as much as possible. In general, a smaller loss indicates a higher quality of deep neural network training, and a larger loss indicates a lower quality of deep neural network training. Similarly, a smaller loss variability indicates a more stable training, and a larger loss variability indicates an unstable training.

[0075] (5) Backpropagation Algorithm

[0076] The neural network may correct the values ​​of the parameters of the initial model in the training process according to a backpropagation (BP) algorithm so that the reconstruction error loss of the model is reduced. Specifically, the input signal is forward-transferred until an error loss occurs at the output, and the parameters of the initial super-resolution model are updated based on the backpropagation error loss information to converge the error loss. The backpropagation algorithm is an error-loss-centered backpropagation movement designed to obtain optimal super-resolution model parameters, such as a weight matrix.

[0077] 2 shows a system architecture 100 according to an embodiment of the present application. In FIG. 2, a data collection device 160 is configured to collect training data. In the present application, the training data is mainly image data.

[0078] After collecting the training data, the data collection device 160 stores the training data in the database 130, and the training device 120 obtains the target model / rules 101 through training based on the training data maintained in the database 130.

[0079] The following describes the process by which the training device 120 obtains the target model / rule 101 based on the training data. The training device 120 processes the input raw images and compares the output value with the target value until the difference between the value output by the training device 120 and the target value is less than a certain threshold. In this way, the training of the target model / rule 101 is completed.

[0080] The target model / rules 101 may be used to implement the image processing method in the embodiment of the present application. The target model / rules 101 in this embodiment of the present application may specifically be a neural network model, specifically a convolutional neural network model, etc. It should be noted that in practical applications, the training data maintained in the database 130 may not necessarily all be collected by the data collection device 160, but may be received from another device. It should further be noted that the training device 120 does not necessarily need to train the target model / rules 101 entirely based on the training data maintained in the database 130, but may obtain training data from a cloud or another location to perform model training. The foregoing description should not be construed as limiting the embodiment of the present application.

[0081] The target model / rule 101 acquired through training by the training device 120 may be applied to a different system or device, such as the execution device 110 shown in FIG. 2 . The execution device 110 may be a terminal, such as a mobile phone terminal, a tablet computer, a notebook computer, an augmented reality (AR) / virtual reality (VR) terminal, or an in-vehicle terminal; or may be a server, a cloud, or the like. In FIG. 2 , an input / output (I / O) interface 112 is configured within the execution device 110 and is configured to exchange data with an external device. A user may input data to the I / O interface 112 by using the client device 140. The input data in this embodiment of the present application may include an image to be processed (i.e., a RAW image) captured by a camera.

[0082] In the process where the execution device 110 preprocesses input data or the computing module 111 of the execution device 110 performs processing related to calculations, etc., the execution device 110 may call data, code, etc. in the data storage system 150 for the corresponding processing, and may further store the data, instructions, etc. obtained through the corresponding processing in the data storage system 150.

[0083] Finally, the I / O interface 112 returns the processing results, for example the above-mentioned acquired image processing results, to the client device 140, and provides the processing results to the user.

[0084] It should be noted that the training device 120 may generate corresponding target models / rules 101 for different targets or different tasks based on different training data, which may be used to implement said targets or complete said tasks and provide the user with the required results.

[0085] In the case shown in FIG. 2 , a user may manually specify input data, and the manual specification may be performed on an interface provided by I / O interface 112. In another case, client device 140 may automatically transmit input data to I / O interface 112. If client device 140 needs permission from the user to automatically transmit input data, the user may set corresponding permission in client device 140. The user may view the results output by execution device 110 on client device 140. The results may be specifically presented in a specific manner, such as a display, sound, or action. Client device 140 may alternatively function as a data collection end, collecting input data input to I / O interface 112 shown in the drawing and output results output from I / O interface 112 as new sample data and storing the new sample data in database 130. Of course, client device 140 may alternatively not perform collection. Instead, the I / O interface 112 directly stores the input data input to the I / O interface 112 shown in the drawing and the output results output from the I / O interface 112 in the database 130 as new sample data.

[0086] It should be noted that Figure 2 is merely a schematic diagram of a system architecture according to this embodiment of the present application. The location relationships between devices, components, modules, etc. shown in the drawing are not limited. For example, in Figure 2, data storage system 150 is an external memory associated with execution device 110. In another case, data storage system 150 may alternatively be located in execution device 110.

[0087] 2, the target model / rules 101 are obtained through training by a training device 120. The target model / rules 101 in this embodiment of the present application may be the image processing device in the present application. Specifically, the image processing device provided in the embodiment of the present application may be a CNN or the like.

[0088] Since CNN is a very common neural network, the structure of CNN will be described in detail below with reference to Figure 3. As explained in the introduction to basic concepts above, a convolutional neural network is a deep neural network with a convolutional structure and is a deep learning architecture. In a deep learning architecture, multi-layer learning is performed at different abstraction levels according to a neural network model update algorithm. As a deep learning architecture, CNN is a feed-forward artificial neural network. Neurons in a feed-forward artificial neural network can respond to images input to the feed-forward artificial neural network.

[0089] The structure of a neural network specifically used in the image processing method in the embodiment of the present application may be shown in Fig. 3. In Fig. 3, a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (where the pooling layer is optional), and a fully connected layer 230. The input layer 210 may acquire an image to be processed, and send the acquired image to the convolutional layer / pooling layer 220 and the subsequent fully connected layer 230 for processing, thereby obtaining a processed image result. The layer structure of the CNN 200 in Fig. 3 will be described in detail below.

[0090] Convolutional / Pooling Layer 220:

[0091] Convolutional layer:

[0092] 3, convolutional / pooling layer 220 may include, for example, layers 221-226. For example, in one implementation, layer 221 is a convolutional layer, layer 222 is a pooling layer, layer 223 is a convolutional layer, layer 224 is a pooling layer, layer 225 is a convolutional layer, and layer 226 is a pooling layer. In another embodiment, layers 221 and 222 are convolutional layers, layer 223 is a pooling layer, layers 224 and 225 are convolutional layers, and layer 226 is a pooling layer. That is, the output of a convolutional layer may be used as the input of a subsequent pooling layer or may be used as the input of another convolutional layer to continue the convolution operation.

[0093] Hereinafter, the internal working principle of the convolution layer will be described using the convolution layer 221 as an example.

[0094] The convolution layer 221 may include multiple convolution operators. A convolution operator is also called a kernel. In image processing, a convolution operator functions as a filter that extracts specific information from a matrix of an input image. A convolution operator may essentially be a weight matrix, which is usually predefined. In the process of performing a convolution operation on an image, a weight matrix is ​​usually used to process pixels at a granularity of one pixel (or two pixels, depending on the stride value) in the horizontal direction of the input image and extract specific features from the image. The size of the weight matrix should be related to the size of the image. Note that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the convolution operation process, the weight matrix spans the entire depth of the input image. Therefore, a convolution output of a single depth dimension is generated through convolution with a single weight matrix. However, in many cases, a single weight matrix is ​​not used, but multiple weight matrices of the same size (row × column), i.e., multiple matrices of the same type, are applied. The outputs of the weight matrices are stacked to form the depth dimension of the convolved image. The dimension here can be understood to be determined based on the "multiple" mentioned above. Different weight matrices may be used to extract different features from the image. For example, one weight matrix is ​​used to extract edge information of the image, another weight matrix is ​​used to extract specific colors of the image, and a further weight matrix is ​​used to blur unwanted noise in the image. The multiple weight matrices have the same size (rows x columns), and the convolved feature maps extracted from the multiple weight matrices of the same size have the same size. The extracted multiple convolved feature maps of the same size are then combined to form the output of the convolution operation.

[0095] In practical applications, the weight values ​​in these weight matrices need to be obtained through extensive training. Each weight matrix formed using the weight values ​​obtained through training can be used to extract information from an input image, so that the convolutional neural network 200 performs accurate predictions.

[0096] When the convolutional neural network 200 has multiple convolutional layers, a relatively large amount of general features are typically extracted in shallow convolutional layers (e.g., 221). General features are sometimes referred to as low-level features. As the depth of the convolutional neural network 200 increases, the features extracted in subsequent convolutional layers (e.g., 226) become more complex, e.g., high-level semantic features. Features with higher semantics are more applicable to the problem to be solved.

[0097] Pooling layer:

[0098] Typically, it is necessary to reduce the amount of training parameters. Therefore, pooling layers usually need to be introduced periodically after convolutional layers. In layers 221-226 in layer 220 shown in FIG. 3, one convolutional layer may be followed by one pooling layer, or multiple convolutional layers may be followed by one or more pooling layers. In image processing procedures, the sole purpose of pooling layers is to reduce the spatial size of the image. Pooling layers may include average pooling operators and / or max pooling operators to perform sampling on input images to obtain smaller images. The average pooling operator may be used to calculate pixel values ​​in an image within a specific range and generate an average value as the average pooling result. The max pooling operator may be used to select the pixel with the maximum value within a specific range as the max pooling result. In addition, just as the size of the weight matrix in a convolutional layer needs to be related to the size of the image, the operators in a pooling layer should also be related to the size of the image. The size of the processed image output from a pooling layer may be smaller than the size of the image input to the pooling layer. Each pixel of the image output from the pooling layer represents the average or maximum value of the corresponding sub-area of ​​the image input to the pooling layer.

[0099] Fully connected layer 230:

[0100] After the processing performed in the convolutional / pooling layer 220, the convolutional neural network 200 still cannot output the required output information. As described above, the convolutional / pooling layer 220 only performs feature extraction and reduces the parameters provided by the input image. However, to generate the final output information (required class information or other related information), the convolutional neural network 200 needs to use the fully connected layer 230 to generate an output of one required class or a group of required classes. Therefore, the fully connected layer 230 may include multiple hidden layers (231 and 232 to 23n as shown in FIG. 3) and an output layer 240. The parameters included in the multiple hidden layers may be obtained through pre-training based on training data related to a specific task type. For example, the task type may include image processing, image recognition, image classification, super-resolution image reconstruction, etc.

[0101] The fully connected layer 230 includes multiple hidden layers followed by an output layer 240, i.e., the final layer of the entire convolutional neural network 200. The output layer 240 has a loss function similar to categorical cross entropy, which is used, among other things, to calculate prediction error. Once forward propagation through the entire convolutional neural network 200 (e.g., propagation from 210 to 240 in FIG. 3 ) is completed, backward propagation (e.g., propagation from 240 to 210 in FIG. 3 ) begins to update the weights and deviations of each of the aforementioned layers to reduce the loss of the convolutional neural network 200 and the error between the results output by the convolutional neural network 200 through the convolutional layers and the ideal results.

[0102] The structure of a neural network specifically used in the image processing method in the embodiment of the present application may be shown in Fig. 4. In Fig. 4, a convolutional neural network (CNN) 200 may include an input layer 210, a convolutional layer / pooling layer 220 (wherein the pooling layer is optional), and a fully connected layer 230. Compared with Fig. 3, in Fig. 4, multiple convolutional layers / pooling layers are arranged in parallel in the convolutional layer / pooling layer 220, and the extracted features are all input to the fully connected layer 230 for processing.

[0103] It should be noted that the convolutional neural network shown in Figure 3 and the convolutional neural network shown in Figure 4 are merely examples of two possible convolutional neural networks that may be used in the image processing methods of the embodiments of the present application. In certain applications, the convolutional neural network used in the image processing methods of the embodiments of the present application may alternatively exist in the form of another network model.

[0104] In addition, the structure of the convolutional neural network obtained by using the neural network structure searching method in this embodiment of the present application can be shown in the structure of the convolutional neural network in FIG. 3 and the structure of the convolutional neural network in FIG. 4.

[0105] FIG. 5 shows a hardware structure of a chip according to an embodiment of the present application. The chip includes a neural network processing unit 50. The chip may be disposed in the execution device 110 shown in FIG. 2 and configured to complete the computing operations of the computing module 111. Alternatively, the chip may be disposed in the training device 120 shown in FIG. 2 and complete the training operations of the training device 120 and output the target model / rules 101. The algorithms in all layers of the convolutional neural network shown in FIG. 3 or FIG. 4 may be implemented in the chip shown in FIG. 5.

[0106] The neural network processing unit NPU 50 functions as a co-processor and is mounted on a host central processing unit CPU (host CPU). The host CPU assigns tasks to it. The core of the NPU is an arithmetic circuit 303. A controller 304 controls the arithmetic circuit 303 to retrieve data from memory (weight memory or input memory) and perform arithmetic operations.

[0107] In some implementations, the arithmetic circuitry 303 includes multiple processing elements (process engines, PEs). In some implementations, the arithmetic circuitry 303 is a two-dimensional systolic array. Alternatively, the arithmetic circuitry 303 may be a one-dimensional systolic array or another electronic circuit capable of performing mathematical operations such as multiplication and addition. In some implementations, the arithmetic circuitry 303 is a general-purpose matrix processor.

[0108] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The arithmetic circuit fetches data corresponding to matrix B from the weight memory 302 and caches the data in each PE in the arithmetic circuit. The arithmetic circuit fetches data corresponding to matrix A from the input memory 301, performs a matrix operation on matrix B, and stores the obtained partial result or the obtained final result of the matrix in an accumulator 308.

[0109] The vector computing unit 307 may further process the output of the arithmetic circuitry, such as vector multiplication, vector addition, exponentiation, logarithm operation, value comparison, etc. For example, the vector computing unit 307 may be configured to perform network computing such as pooling, batch normalization (BN), or local response normalization in non-convolutional / non-FC layers of a neural network.

[0110] In some implementations, the vector computing unit 307 can store the processed output vector in the unified memory 306. For example, the vector computing unit 307 can apply a nonlinear function to the output of the arithmetic circuitry 303, e.g., a vector of accumulated values, to generate activation values. In some implementations, the vector computing unit 307 generates normalized values, combined values, or both normalized values ​​and combined values. In some implementations, the processed output vector can be used as an activation input for the arithmetic circuitry 303, e.g., for use in subsequent layers of a neural network.

[0111] The unified memory 306 is configured to store input data and output data.

[0112] For weight data, a direct memory access controller (DMAC) 305 transfers input data in external memory to input memory 301 and / or unified memory 306, stores weight data in external memory to weight memory 302, and stores data in unified memory 306 in external memory.

[0113] A bus interface unit (BIU) 310 is configured to implement the interaction between the host CPU, the DMAC, and the instruction fetch buffer 309 over the bus.

[0114] An instruction fetch buffer 309 coupled to the controller 304 is configured to store instructions used by the controller 304 .

[0115] The controller 304 is configured to access instructions cached in the instruction fetch buffer 309 and control the operation process of the computation accelerator.

[0116] Typically, unified memory 306, input memory 301, weight memory 302, and instruction fetch buffer 309 are all on-chip memories. External memory is memory external to the NPU, and can be double data rate synchronous dynamic random access memory (DDR SDRAM), high bandwidth memory (HBM), or another readable and writable memory.

[0117] The operations in each layer of the convolutional neural network shown in FIG. 3 or FIG. 4 may be performed by the arithmetic circuitry 303 or the vector computing unit 307.

[0118] The execution device 110 of Fig. 2 described above can perform the steps of the image processing method in the embodiment of the present application. The CNN model shown in Fig. 3, the CNN model shown in Fig. 4, and the chip shown in Fig. 5 can also be configured to perform the steps of the image processing method in the embodiment of the present application.

[0119] 6 is an exemplary diagram of a system architecture for image processing according to an embodiment of the present application. As shown in FIG. 6, the system architecture 600 includes an image sensor 610 and an image processing model 620.

[0120] The image sensor 610 is configured to convert the acquired optical image into a digital image signal, ie, a RAW image.

[0121] The image processing model 620 is used to process the RAW image acquired by the image sensor 610 according to the image processing method 700 provided below, and output a feature fusion map or an RGB image. The image processing model 620 can be a neural network model, for example, a convolutional neural network model, which is not limited in this application. For ease of explanation, the following will provide an explanation by using a convolutional neural network model as an example.

[0122] Optionally, system architecture 600 may further include a perceptual processing module 630. Perceptual processing module 630 is configured to perform perceptual processing, such as image recognition, detection, and segmentation, on the feature fusion map or RGB image output by image processing model 620.

[0123] FIG. 7 is an exemplary diagram of an image processing method according to an embodiment of the present application. It should be understood that the image processing method 700 may be applied to the image processing model 620, and in particular, may be applied to a convolutional neural network model. As shown in FIG. 7, the image processing method 700 includes steps S710 to S750. It should be understood that the order of the steps is not limited in the embodiment of the present application, and any solution of the present application that can be implemented in any order of steps should fall within the scope of protection of the present application. These steps will be described in detail below.

[0124] S710: The current RAW image frame is obtained.

[0125] It should be understood that a RAW image is raw data obtained by converting a captured light source signal into a digital signal using a complementary metal-oxide-semiconductor (CMOS) image sensor or a charge-coupled device (CCD) image sensor, and is lossless. Therefore, the RAW image contains the original information of the subject. Since the input of the neural network model is the RAW image, image information can be maximized. In practical applications, the RAW image may be acquired by an image capture device (e.g., a lens and a sensor) or may be received from another device. This is not particularly limited in the embodiments of the present application. In addition, in practice, the input of the neural network may alternatively be other image data, image signals, etc. This is not limited in the present application.

[0126] S720: Determine a first feature of the current RAW image frame.

[0127] Before the first feature of the current RAW image frame is determined, features may first be extracted from the current RAW image frame. It should be understood that the process of extracting features in the present application may be implemented by using a convolutional neural network model, and more specifically, may be implemented by using a convolutional layer (e.g., convolutional layer 221) of the convolutional neural network model.

[0128] Optionally, the feature may be at least one of a color feature, a brightness feature, a texture feature, an edge feature, a shape feature, a spatial relationship feature, etc. of the image. It should be understood that the feature is typically represented by a feature vector. If the image has only one feature, the feature vector is a one-dimensional vector, and if the image is a combination of n features, the feature vector is an n-dimensional feature vector.

[0129] Optionally, in one implementation, the feature extracted from the current RAW image frame (ie, the third feature below) may be directly used as the first feature of the current RAW image frame.

[0130] In another implementation, the features extracted from the current RAW image frame may alternatively be corrected to obtain the first feature of the current RAW image frame, and it should be understood that the method of correcting the features extracted from the current RAW image frame is not limited in this application.

[0131] Optionally, the features extracted from the current RAW image frame may be corrected based on the features extracted from the previous frame of the image to obtain a first feature of the current RAW image frame, thereby avoiding the flicker problem caused by an inconsistent image sequence.

[0132] Optionally, the previous frame of the image may be a RAW image, or may be an image on which correction or other processing is performed, which is not limited in this application, however, for ease of explanation, the following description will be provided by using the previous RAW image frame as an example.

[0133] Optionally, the features extracted from the current frame may be corrected based on the features extracted from another frame of the image, which is not limited thereto. For ease of explanation, the following description will be provided by using the previous frame as an example.

[0134] Optionally, a previous RAW image frame may be acquired first, and a second feature of the previous RAW image frame is extracted. In addition, a third feature of the current RAW image frame is extracted. Then, a first feature of the current RAW image frame is determined based on the second feature and the third feature extracted from the previous RAW image frame and the current RAW image frame, respectively. In other words, the third feature extracted from the current RAW image frame is corrected based on the second feature extracted from the previous RAW image frame to determine the first feature of the current RAW image frame. The method of correcting the third feature extracted from the current RAW image frame based on the second feature extracted from the previous RAW image frame is not limited in this application.

[0135] It should be noted that since the frame rate of the camera is high, typically 33 frames per second, there will be large matching scene areas between two consecutive frames of an image, or even between multiple consecutive frames of an image. Therefore, optionally, in practice, the current RAW image frame can be corrected based on one or more other frames of the RAW image that have matching scene areas. It should be understood that matching scene areas means that the same area location in different images corresponds to the same scene. The scene can be a natural scene, an object, a person, an animal, etc. This is not limited.

[0136] Based on this, determining a first feature of the current RAW image frame based on the second feature and the third feature may include: determining a matching scene area of ​​the previous RAW image frame and the current RAW image frame; determining a fourth feature that is within the second feature and corresponds to the matching scene area based on the matching scene area; determining a fifth feature that is within the third feature and corresponds to the matching scene area based on the matching scene area; and determining a first feature of the current RAW image frame based on a difference between the fourth feature and the fifth feature.

[0137] It should be understood that determining a fourth feature within the second feature and corresponding to the matching scene area is determining a feature located within the matching scene area and within the features extracted from the previous RAW image frame, and similarly, determining a fifth feature within the third feature and corresponding to the matching scene area is determining a feature located within the matching scene area and within the features extracted from the current RAW image frame.

[0138] Optionally, in practice, the matching scene area of ​​the previous RAW image frame and the current RAW image frame may alternatively be determined first, and then only the features in the matching scene area of ​​the previous RAW image frame are extracted. Furthermore, the features in the matching scene area of ​​the current RAW image frame are corrected based on the feature amount extracted from the matching scene area of ​​the previous RAW image frame. This is not limited in the present application.

[0139] It should be noted that determining the first feature of the current RAW image frame based on the difference between the fourth feature and the fifth feature specifically includes: correcting features in the coincident scene area of ​​the current RAW image frame based on the difference between the fourth feature and the fifth feature; and then determining the first feature of the current RAW image frame based on the corrected features in the coincident scene area of ​​the current RAW image frame and features in another non-coincided scene area.

[0140] Optionally, if the difference between the fourth feature and the fifth feature is less than or equal to the first threshold, the first feature of the current RAW image frame may be directly determined based on the fifth feature, in other words, the features in the matching scene area of ​​the current RAW image frame do not need to be corrected.

[0141] Optionally, if the difference between the fourth feature and the fifth feature is greater than a first threshold and less than a second threshold, the first feature of the current RAW image frame may be determined based on the fourth feature and the fifth feature. Specifically, the fifth feature in the matching scene area of ​​the current RAW image frame may be corrected based on the fourth feature and the fifth feature, and then the first feature of the current RAW image frame may be determined based on the corrected feature in the matching scene area and a feature in another non-matching scene area. Optionally, the fifth feature in the matching scene area of ​​the current RAW image frame may be corrected based on an average value of the fourth feature and the fifth feature, or may be determined based on another ratio. This is not limited thereto.

[0142] Optionally, if the difference between the fourth feature and the fifth feature is equal to or greater than a second threshold, the first feature of the current RAW image frame may be determined based on the fourth feature. Specifically, the fifth feature in the matching scene area of ​​the current RAW image frame is corrected based on the fourth feature (in other words, the fifth feature in the matching scene area of ​​the current RAW image frame is replaced with the fourth feature), and the first feature of the current RAW image frame is determined based on the corrected feature in the matching scene area and a feature in another non-matching scene area.

[0143] In this embodiment of the present application, the first feature of the current RAW image frame is determined based on the difference between the features corresponding to the matching scene areas of two consecutive frames of the image, so that the difference between the same scene areas of the two consecutive frames of the image is within a suitable range, thereby ensuring that the determined first feature of the current RAW image frame is robust, thereby ensuring the time-series consistency in the brightness and color of the video and avoiding the flicker phenomenon.

[0144] Optionally, the resolution of the previous RAW image frame may be half the resolution of the current RAW image frame, or may be the same as the resolution of the current RAW image frame, or may be another ratio of the resolution of the current RAW image frame, which is not limited in this application.

[0145] In this embodiment of the present application, the resolution of the previous RAW image frame is set to half of the resolution of the current RAW image frame, which can reduce the amount of processing in the image processing model and further reduce the requirements on computing power.

[0146] Optionally, the same weight coefficient is used to extract the third feature of the current RAW image frame and the second feature of the previous RAW image frame. In this way, the convolutional neural network model can extract features of two frames of an image by sharing weight values. In other words, the features of the two frames of an image are extracted by using a weight matrix. In this case, features corresponding to matching scene areas in the two frames of an image can be compared more intuitively, thereby improving the accuracy of data processing. In addition, the amount of network parameters that need to be stored in the convolutional neural network model can be further reduced. For the related concept of weights, please refer to the description of the convolutional layer 221 above. Details will not be described again.

[0147] S730: Analyze context feature information of the current RAW image frame, where the context feature information includes local feature information and global feature information.

[0148] Optionally, the context feature information of the current RAW image frame may be analyzed by the context encoder.

[0149] Optionally, the context encoder may be trained by using pixel reconstruction loss (i.e., L2 loss) and generative adversarial network (GAN) adversarial loss. In this way, the context encoder can learn the appearance and semantic information of the visual structure of the image. The appearance of the visual structure mainly corresponds to global features, including overall brightness, white balance, color, and contrast. The semantic information mainly corresponds to local features, including texture details (such as sharpness and noise), local contrast, etc. For training the context encoder, please refer to related prior art. Details will not be described again.

[0150] It should be understood that the context encoder is trained to generate context feature information for any image area based on the surrounding environment. Specifically, the context encoder receives data about the surroundings of the image area and attempts to generate data appropriate for the image area.

[0151] Based on this, the context feature information of the current RAW image frame is analyzed, and then the local feature information and global feature information corresponding to the current RAW image frame can be determined.

[0152] S740: Determine local features and global features within the first feature based on the local feature information and the global feature information.

[0153] As described above, the global feature information and local feature information of the current RAW image frame can be learned by analyzing the context feature information, and then the global feature and the local feature in the first feature can be distinguished based on the information. In other words, the context feature information can guide the generation of the global feature and the local feature.

[0154] Optionally, the global features may include at least one of overall brightness, white balance, color and contrast, and the local features may include at least one of local texture detail and local contrast.

[0155] In this embodiment of the present application, the local features and global features within the first feature are determined based on the context feature information, so that the accuracy of determining the local features and global features is improved.

[0156] S750: Fuse the global features and the local features to obtain a feature fusion map of the current RAW image frame.

[0157] Optionally, in this application, feature fusion may be implemented by using an add and concat fusion policy. Specifically, feature maps (i.e., global feature maps and local feature maps) with different sizes (i.e., different resolutions) may first be fused by a concatenation operation, and then, after performing the concatenation operation, information about the global feature map and information about the local feature map are superimposed by performing an add operation.

[0158] Note that a concatenation operation is an increase in the amount of channels. Specifically, the amount of features (amount of channels) used to describe an image increases, but the information about the features does not increase, i.e., it is a horizontal or vertical spatial overlap. An addition operation is a simple pixel overlap. This means that the amount of information about the features used to describe an image increases, but the dimension that describes the image does not increase, only the amount of information in each dimension increases.

[0159] In practice, it should be understood that feature fusion may alternatively be implemented by using another fusion policy, which is not a limitation in this application.

[0160] Optionally, the method 700 may further include generating an RGB image based on the feature fusion map.

[0161] In this embodiment of the present application, the convolutional neural network model is mainly used to replace the ISP in the existing solution for processing the current RAW image frame, thereby reducing the algorithm complexity in the image processing procedure. In addition, in the image processing procedure, the local features and global features of the current RAW image frame are determined based on the context feature information, thereby improving the accuracy of determining the local features and global features. In this way, the local features and global features can be highlighted separately in the decoding process, which can improve the quality of the final image.

[0162] Optionally, after the feature fusion map of the current RAW image frame is obtained, the feature fusion map may be sent directly to the perceptual processing module, or after the RGB image is generated, the RGB image may be sent to the perceptual processing module.

[0163] In this embodiment of the present application, the obtained feature fusion map may be directly transmitted to the perceptual processing module, which can reduce the requirements of the perceptual processing module in terms of computational power.

[0164] Optionally, the generated RGB image may be evaluated, and based on the evaluation results, the convolutional neural network model described above is optimized.

[0165] Optionally, the generated RGB image may be evaluated by using an evaluation model.

[0166] Optionally, the evaluation model used for evaluation may be pre-trained by using a sample RGB image training set. The evaluation model may be a neural network model. The model may be used to evaluate the RGB images generated by using the convolutional neural network model. In addition, the evaluation results can facilitate continuous optimization of the convolutional neural network model. In this way, the images output by the convolutional neural network model are more useful for the perception algorithm, improving the accuracy of perception processing.

[0167] 8 is an exemplary diagram of a system architecture of a convolutional neural network model according to an embodiment of the present application. As shown in FIG. 8, the convolutional neural network model 800 includes a feature encoding module 810, a context encoding module 820, a feature segmentation module 830, and a feature fusion module 840.

[0168] The feature encoding module 810 is configured to perform feature encoding, i.e., feature extraction, on the current RAW image frame and the previous RAW image frame. The feature encoding module 810 may extract feature information of two consecutive frames by using a multi-resolution framework with weight value sharing. Specifically, the feature encoding module 810 may extract features of two consecutive RAW image frames with different resolutions, and the same weight value may be used for the two consecutive frames of the RAW image in the extraction process.

[0169] Optionally, in the training process, the feature encoding module 810 may use a form of implicit regularization, such as a batch stochastic gradient descent algorithm, in which multiple samples are used to update parameters at each iteration, thereby accelerating the speed at which the model is trained.

[0170] Optionally, the convolutional neural network model 800 may further include a matching and correction module configured to perform matching on features in matching scene areas of the two consecutive frames after the feature encoding module 810 extracts the features of the two consecutive frames, and correct the features of the current frame based on the matching result to determine a first feature of the current frame.

[0171] The context encoding module 820 is configured to analyze the context feature information of the current RAW image frame to determine global feature information and local feature information of the current RAW image frame, see the above description for details.

[0172] The feature splitting module 830 is configured to split the determined first feature of the current frame into global features and local features based on the context feature information.

[0173] The feature fusion module 840 is configured to fuse the global features and the local features to generate a feature fusion map. It should be understood that the feature fusion map may be used directly as an input to the perceptual processing module.

[0174] Optionally, the convolutional neural network model 800 may further include a feature decoding module 850 configured to decode the feature fusion map output by the feature fusion module 840 to generate an RGB image. It should be understood that the RGB image may also be used as an input to the perceptual processing module.

[0175] Optionally, the RGB image generated by the feature decoding module 850 may alternatively be input to an evaluation model to evaluate the generated RGB image, and the optimization direction of the convolutional neural network model 800 is controlled based on the evaluation result. In this way, the RGB image output by the convolutional neural network model 800 is more useful for the perception algorithm, and the accuracy of the perception algorithm is improved.

[0176] Optionally, the evaluation model used for the evaluation may be pre-trained by using a sample RGB image training set, and the evaluation model may be a neural network model.

[0177] It should be understood that the evaluation model may be a module within the convolutional neural network model 800 or may exist independently of the convolutional neural network model 800. This is not a limitation in the present application.

[0178] 9 is an exemplary diagram of an image processing apparatus according to an embodiment of the present application. The apparatus 900 is used in a convolutional neural network model. The image processing apparatus 900 includes an acquisition module 910 and a processing module 920.

[0179] The acquisition module 910 is configured to acquire a current RAW image frame.

[0180] The processing module 920 is configured to determine a first feature of the current RAW image frame, analyze context feature information of the current RAW image frame, where the context feature information includes local feature information and global feature information, determine local features and global features in the first feature based on the local feature information and the global feature information, and fuse the local features and the global features to obtain a feature fusion map of the current RAW image frame.

[0181] Optionally, the acquisition module 910 may be further configured to acquire a previous RAW image frame, and the processing module 920 may be further configured to extract a second feature of the previous RAW image frame, extract a third feature of the current RAW image frame, and determine a first feature of the current RAW image frame based on the second feature and the third feature.

[0182] Optionally, the processing module 920 may be further configured to determine a matching scene area between the previous RAW image frame and the current RAW image frame, determine a fourth feature that is within the second feature and corresponds to the matching scene area based on the matching scene area, determine a fifth feature that is within the third feature and corresponds to the matching scene area based on the matching scene area, and determine a first feature of the current RAW image frame based on a difference between the fourth feature and the fifth feature.

[0183] Optionally, the processing module may be further configured to determine a first feature of the current RAW image frame based on the fifth feature if a difference between the fourth feature and the fifth feature is less than or equal to a first threshold, or determine a first feature of the current RAW image frame based on the fourth feature and the fifth feature if a difference between the fourth feature and the fifth feature is greater than the first threshold and less than a second threshold, or determine a first feature of the current RAW image frame based on the fourth feature if a difference between the fourth feature and the fifth feature is greater than or equal to the second threshold.

[0184] Optionally, the resolution of the previous RAW image frame may be half the resolution of the current RAW image frame.

[0185] Optionally, the same weighting factor is used to extract the third feature of the current RAW image frame and the second feature of the previous RAW image frame.

[0186] Optionally, the processing module 920 may be further configured to generate an RGB image based on the feature fusion map.

[0187] Optionally, the device 900 may further include a transmission module configured to transmit the feature fusion map or the RGB image to the perceptual processing module.

[0188] Optionally, the processing module 920 may be further configured to evaluate the generated RGB image and optimize the convolutional neural network model based on the evaluation result.

[0189] Optionally, the global features may include at least one of overall brightness, white balance, color and contrast, and the local features may include at least one of local texture detail and local contrast.

[0190] 10 is a block diagram of an example of a hardware structure of an apparatus according to an embodiment of the present application. Optionally, the apparatus 1000 may be specifically a computer device. The apparatus 1000 includes a memory 1010, a processor 1020, a communication interface 1030, and a bus 1040 The memory 1010, the processor 1020, and the communication interface 1030 are communicatively connected to each other via a bus 1040.

[0191] The memory 1010 may be a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 1010 may store a program. When the program stored in the memory 1010 is executed by the processor 1020, the processor 1020 is configured to perform steps of the image processing method in the embodiment of the present application.

[0192] The processor 1020 may be a general-purpose central processing unit (CPU), a microprocessor, an application-specific integrated circuit (ASIC), a graphics processing unit (GPU), or one or more integrated circuits, and is configured to execute associated programs to implement the image processing methods in the method embodiments of the present application.

[0193] Alternatively, the processor 1020 may be an integrated circuit chip and have signal processing capabilities. In the implementation process, the image processing method in the present application may be completed by using an integrated logic circuit of hardware in the processor 1020 or instructions in the form of software.

[0194] The processor 1020 may alternatively be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor may implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of the present application. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor, etc. The steps of the methods disclosed in connection with the embodiments of the present application may be directly executed and completed by a hardware decoding processor, or may be executed and completed by a combination of hardware and software modules in the decoding processor. The software modules may be located in a storage medium well-established in the art, such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, an electrically erasable programmable memory, or a register. The storage medium is located in the memory 1010. The processor 1020 reads the information in the memory 1010 and, in combination with the hardware in the processor 1020, completes the functions that need to be performed by the modules included in the device in this embodiment of the present application, or performs the image processing method in the method embodiment of the present application.

[0195] The communication interface 1030 facilitates communication between the apparatus 1000 and another device or a communication network using a transceiver device, such as but not limited to a transceiver.

[0196] Bus 1040 may include a path for transmitting information between components of device 1000 (eg, memory 1010, processor 1020, and communication interface 1030).

[0197] An embodiment of the present application further provides a vehicle including a module configured to perform the image processing method of the method embodiment of the present application.

[0198] An embodiment of the present application further provides a terminal device including a module configured to perform the image processing method in the method embodiment of the present application.

[0199] Optionally, the terminal device may be any device related to the visual field, such as a mobile phone, a tablet computer, a vehicle, a computer, a machine vision recognition device or an intelligent security device.

[0200] An embodiment of the present application further provides a computing device including at least one processor and a memory, wherein the at least one processor is coupled to the memory and configured to read and execute instructions in the memory to perform an image processing method in a method embodiment of the present application.

[0201] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enable the computer to perform an image processing method in a method embodiment of the present application.

[0202] An embodiment of the present application further provides a computer-readable medium, which stores program code, which, when executed on a computer, enables the computer to perform the image processing method in the method embodiment of the present application.

[0203] An embodiment of the present application further provides a chip, which includes a processor and a data interface, wherein the processor reads instructions stored in the memory via the data interface and executes the image processing method in the method embodiment of the present application.

[0204] Optionally, in one implementation, the chip may further include a memory, the memory storing instructions, and the processor configured to execute the instructions stored in the memory, which, when executed, configures the processor to perform the image processing method in the method embodiment of the present application.

[0205] Those skilled in the art may recognize that, in combination with the examples described in the embodiments disclosed herein, the units and algorithm steps may be implemented by electronic hardware or a combination of computer software and electronic hardware. Whether a function is performed by hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered as going beyond the scope of this application.

[0206] For the sake of convenience and concise description, for those skilled in the art to clearly understand, the detailed operation processes of the above systems, devices and units may be referred to the corresponding processes in the above method embodiments, and the details will not be described again in this specification.

[0207] In some embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other ways. For example, the described device embodiments are merely examples. For example, the division into units is merely a logical division of function, and other divisions may be used in actual implementation. For example, multiple units or components may be combined or integrated into another system, or some features may be omitted or not implemented. In addition, the shown or discussed mutual couplings or direct couplings or communication connections may be implemented by using some interfaces. Indirect couplings or communication connections between devices or units may be implemented in electronic, mechanical, or other forms.

[0208] Units described as separate parts may or may not be physically separate, and parts shown as units may or may not be physical units, and may be located in one location or distributed across multiple network units. Some or all of the units may be selected based on actual requirements to achieve the objectives of the solutions in the embodiments.

[0209] In addition, the functional units in the embodiments of the present application may be integrated into one processing unit, or each of the units may exist physically alone, or two or more units may be integrated into one unit.

[0210] When a function is implemented in the form of a software functional unit and sold or used as an independent product, the function may be stored in a computer-readable storage medium. Based on this understanding, the technical solutions of the present application may be essentially implemented in the form of a software product, or a portion of the technical solutions may be implemented in the form of a software product. The computer software product is stored in a storage medium and includes some instructions for instructing a computer device (which may be a personal computer, a server, a network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. The storage medium includes any medium that can store program code, such as a USB flash drive, a removable disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk, or an optical disk.

[0211] The above description is merely a specific implementation of the present application, but is not intended to limit the scope of protection of the present application. Any modifications or replacements that are easily understood by those skilled in the art within the technical scope disclosed in the present application shall fall within the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the scope of protection of the claims.

Claims

1. 1. An image processing method, the method being applied to a convolutional neural network model, the method comprising: obtaining a current RAW image frame; determining a first characteristic of the current RAW image frame; analyzing the current RAW image frame to generate context feature information for the current RAW image frame, wherein the context feature information includes local feature information and global feature information, the local feature information includes semantic information for the current RAW image frame, the semantic information being local features including at least one of local texture detail and local contrast, and the global feature information includes visual structure appearance for the current RAW image frame, the visual structure appearance corresponding to global features including at least one of overall brightness, white balance, color, and contrast; determining the local features and the global features within the first feature based on the local feature information and the global feature information included in the context feature information; fusing the local features and the global features to obtain a feature fusion map of the current RAW image frame; A method comprising:

2. The method comprises: acquiring a previous RAW image frame; extracting a second feature of the previous raw image frame; Further comprising: The step of determining a first characteristic of the current RAW image frame comprises: extracting a third feature of the current raw image frame; determining the first feature of the current raw image frame based on the second feature and the third feature; The method of claim 1 , comprising:

3. determining the first feature of the current RAW image frame based on the second feature and the third feature, determining a matching scene area between the previous raw image frame and the current raw image frame; determining a fourth feature based on the matching scene area, the fourth feature being within the second feature and corresponding to the matching scene area; determining a fifth feature based on the matching scene area, the fifth feature being within the third feature and corresponding to the matching scene area; determining the first feature of the current raw image frame based on a difference between the fourth feature and the fifth feature; The method of claim 2 , comprising:

4. determining the first feature of the current RAW image frame based on a difference between the fourth feature and the fifth feature, determining the first feature of the current raw image frame based on the fifth feature if the difference between the fourth feature and the fifth feature is less than or equal to a first threshold; determining the first feature of the current RAW image frame based on the fourth feature and the fifth feature if the difference between the fourth feature and the fifth feature is greater than a first threshold and less than a second threshold; or determining the first feature of the current raw image frame based on the fourth feature if the difference between the fourth feature and the fifth feature is greater than or equal to a second threshold; The method of claim 3, comprising:

5. the resolution of the previous RAW image frame is half the resolution of the current RAW image frame; 5. The method according to any one of claims 2 to 4.

6. the same weighting coefficient is used to extract the third feature of the current RAW image frame and the second feature of the previous RAW image frame; 6. The method according to any one of claims 2 to 5.

7. The method comprises: generating an RGB image based on the feature fusion map; The method of claim 1 , further comprising:

8. The method comprises: sending the feature fusion map or the RGB image to a perceptual processing module; The method of claim 7 further comprising:

9. The method comprises: evaluating the generated RGB image and optimizing the convolutional neural network model based on the evaluation results; The method of claim 7 further comprising:

10. 1. An image processing apparatus for use in a convolutional neural network model, the apparatus comprising: an acquisition module configured to acquire a current RAW image frame; a processing module configured to determine first features of the current RAW image frame, and analyze the current RAW image frame to generate context feature information for the current RAW image frame, wherein the context feature information includes local feature information and global feature information, the local feature information includes semantic information of the current RAW image frame, the semantic information being local features including at least one of local texture detail and local contrast, and the global feature information includes visual structure appearance of the current RAW image frame, the visual structure appearance corresponding to global features including at least one of overall brightness, white balance, color, and contrast; determine the local features and the global features within the first features based on the local feature information and the global feature information, and fuse the local features and the global features to obtain a feature fusion map for the current RAW image frame; An apparatus comprising:

11. The acquisition module: further configured to acquire a previous RAW image frame; the processing module is further configured to extract a second feature of the previous raw image frame, extract a third feature of the current raw image frame, and determine the first feature of the current raw image frame based on the second feature and the third feature.

11. The apparatus of claim 10.

12. The processing module includes: and determining a matching scene area between the previous raw image frame and the current raw image frame; determining a fourth feature based on the matching scene area, the fourth feature being within the second feature and corresponding to the matching scene area; determining a fifth feature based on the matching scene area, the fifth feature being within the third feature and corresponding to the matching scene area; and determining the first feature of the current raw image frame based on a difference between the fourth feature and the fifth feature.

12. The apparatus of claim 11.

13. The processing module includes: and determining the first feature of the current RAW image frame based on the fifth feature if the difference between the fourth feature and the fifth feature is less than or equal to a first threshold, or determining the first feature of the current RAW image frame based on the fourth feature and the fifth feature if the difference between the fourth feature and the fifth feature is greater than a first threshold and less than a second threshold, or determining the first feature of the current RAW image frame based on the fourth feature if the difference between the fourth feature and the fifth feature is greater than or equal to a second threshold.

13. The apparatus of claim 12.

14. A vehicle comprising the image processing device according to any one of claims 10 to 13.

15. 10. A computer readable medium storing program code that, when executed on a computer, enables the computer to perform the image processing method of any one of claims 1 to 9.

16. 10. A computing device comprising at least one processor and a memory, the at least one processor coupled to the memory and configured to read and execute instructions in the memory to perform the image processing method of any one of claims 1 to 9.

Images