Image processing method and related device

Abstract

An image processing method and related device can be applied to the field of image processing in the field of artificial intelligence. The method can include: obtaining first feature information of at least one first frame in a first neural network calculation process, and performing image processing on a second frame by using a second neural network, the first feature information being used as input information in a second neural network calculation process, the first neural network and the second neural network both being neural networks used for implementing image processing, and the first frame and the second frame having a correlation. When the second frame is processed by using the second neural network, the feature information of the first frame in the first neural network calculation process is introduced, the continuity between the processing results of the first frame and the second frame having the correlation is improved, and the accuracy of the processing result of the second frame is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to an image processing method and related equipment. BACKGROUND

[0002] Artificial intelligence (AI) is the use of digital computers or digital computer-controlled machines to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a branch of computer science that aims to understand the essence of intelligence and produce a new intelligent machine that can react in a similar way to human intelligence. Artificial intelligence is the design principle and implementation method of various intelligent machines, so that machines have the functions of perception, reasoning and decision-making.

[0003] Video processing is a mainstream application in the field of artificial intelligence. At present, after obtaining a plurality of continuous video frames, each video frame is independently processed by using a neural network (NN) to obtain the processing result of each video frame.

[0004] Since the processing mode is to independently process each video frame, it is easy to cause the processing results of different video frames to be not coherent, and therefore a more coherent video frame processing scheme is urgently needed. SUMMARY

[0005] Therefore, the embodiments of the present application provide an image processing method and related equipment, which introduce the feature information of the first frame in the first neural network calculation process when processing the second frame by using the second neural network, improve the coherence between the processing results of the first frame and the second frame which have relevance, and improve the accuracy of the processing result of the second frame.

[0006] A first aspect of this application provides an image processing method applied to the field of image processing in the field of artificial intelligence. The method may include: an execution device acquiring first feature information of at least one first frame during the computation of a first neural network, the first feature information specifically representing a one-dimensional tensor, a two-dimensional tensor, a three-dimensional tensor, or a higher-dimensional tensor. The execution device then uses a second neural network to process the image of a second frame, wherein the first feature information serves as input information during the computation of the second neural network. Both the first and second neural networks are neural networks used to implement image processing, and there is a correlation between the first and second frames, where the correlation refers to a similarity between the first and second frames being greater than or equal to a first threshold. Further, the similarity algorithm is any one of the following: cosine similarity, Euclidean distance, Mahalanobis distance, Manhattan distance, and Hamming distance. In this implementation, when processing the second frame using the second neural network, the feature information of the first frame during the computation of the first neural network is introduced, thereby improving the coherence between the processing results of the correlated first and second frames; furthermore, by introducing the feature information of the first frame, the second neural network can integrate more feature information when processing the second frame, thereby improving the accuracy of the processing result of the second frame.

[0007] In one possible implementation of the first aspect, the first neural network includes a first number of neural network layers, and the second neural network includes a second number of neural network layers, where the number of second layers is less than the number of first layers. Further, by removing one or more neural network layers from the first neural network, a second neural network can be obtained. In this implementation, because the first neural network has more neural network layers than the second neural network, the processing result output by the first neural network has higher accuracy. The time taken for the second neural network to generate the second processing result is less than the time taken for the first neural network to generate the first processing result. However, because the first feature information corresponding to the first frame is introduced during the generation of the second processing result—that is, more precise feature information is introduced—the accuracy of the second processing result is improved. Therefore, the total processing time for multiple consecutive video frames is shortened, and the accuracy of the processing results for multiple consecutive video frames is guaranteed.

[0008] In one possible implementation of the first aspect, the first feature information includes feature information of each first frame, where the feature information of each first frame is generated during the computation of the first neural network. In this implementation, the set of feature information generated during the computation of the first neural network for each first frame is directly determined as the first feature information, which is simple to operate.

[0009] In an implementation form of the first aspect, the first feature information is information generated by merging the feature information of each first frame. The feature information of each first frame is feature information generated by the first neural network during the calculation process, and the merging includes one or more of splicing, adding, fusing, and multiplying. In this implementation form, another generation manner of the first feature information is provided, and the implementation flexibility of the scheme is improved.

[0010] In an implementation form of the first aspect, the feature information of each first frame is a feature map generated by the first neural network during the calculation process. In this implementation form, the feature information of each first frame is concretized, and the combination degree with the actual application scenario is improved.

[0011] In an implementation form of the first aspect, the feature information of each first frame is calculated and output by an i-th layer of the first neural network, and the i-th layer is any layer of the first neural network. The second neural network is obtained by pruning the first layer to the i-th layer of the first neural network. In this implementation form, the first layer to the i-th layer of the second neural network are pruned, which greatly reduces the processing time of the second neural network; and the size adjustment of the first feature information is avoided, so as to further shorten the total processing time.

[0012] In an implementation form of the first aspect, the execution device performs image processing on the second frame by using the second neural network, including: the execution device generates second feature information corresponding to the second frame by using the second neural network, and obtains the second feature information generated in the image processing process of the second frame by using the second neural network. The second feature information refers to the feature information generated in the processing process of the second frame by using the second neural network; the execution device combines the first feature information and the second feature information to obtain combined feature information, and the combination includes one or more of splicing, adding, fusing, and multiplying; and the execution device processes the combined feature information by using the second neural network.

[0013] In an implementation form of the first aspect, the first frame and the second frame are continuous video frames in the same video, or the first frame and the second frame are two images with the same background, or the first frame and the second frame are two images with the same moving track, or the motion vectors of the objects in the first frame and the second frame are the same. In this implementation form, various specific forms of the first frame and the second frame are provided, and the implementation flexibility of the scheme is improved.

[0014] In an implementation form of the first aspect, the first frame is a single frame in the video, and the second frame is a double frame in the video; or, the first frame is a double frame in the video, and the second frame is a single frame in the video. In this implementation form, the first feature information corresponding to the first frame is introduced in the process of processing the second frame by the second neural network, the adjacent video frame of the second frame is determined as the first frame, so that the second frame has a high similarity with the first frame, which is beneficial to improve the accuracy of the second processing result.

[0015] In an implementation form of the first aspect, the second frame and the first frame belong to a group of frames, the group of frames includes three continuous frames; the second frame is the third frame in the three continuous frames, and the at least one first frame includes the first frame and the second frame in the three continuous frames; or, the second frame is the second frame in the three continuous frames, and the at least one first frame includes the first frame and the third frame in the three continuous frames; or, the second frame is the first frame in the three continuous frames, and the at least one first frame includes the second frame and the third frame in the three continuous frames. In this implementation form, the three continuous video frames are determined as a group of video frames, two video frames in the three continuous video frames are determined as the first frame, and one video frame in the three continuous video frames is determined as the second frame. Since the similarity of the three continuous video frames is also high, it is beneficial to improve the accuracy of the processing result of the second neural network; in addition, the feature information corresponding to the two first frames is introduced in the process of processing the second frame, so as to further improve the accuracy of the processing result of the second neural network.

[0016] In a second aspect, the embodiments of the present application provide an image processing device, which can be used in the field of image processing in the field of artificial intelligence. The device can include an acquisition module and a processing module. The acquisition module is configured to acquire first feature information of at least one first frame in a first neural network calculation process. The processing module is configured to perform image processing on a second frame by using a second neural network, wherein the first feature information is used as input information in a second neural network calculation process, the first neural network and the second neural network are both neural networks for realizing image processing, and the first frame and the second frame have a correlation.

[0017] The modules in the image processing device provided in the second aspect of the embodiments of the present application are also configured to perform the steps in the various possible implementation forms of the first aspect. For the specific implementation steps of the second aspect of the embodiments of the present application and the various possible implementation forms of the second aspect, and the beneficial effects brought by each possible implementation form, reference can be made to the description in the various possible implementation forms of the first aspect, which will not be repeated here.

[0018] In a third aspect, an embodiment of the present application provides an execution device, which can include a processor, the processor and a memory are coupled, and the memory stores program instructions, when the program instructions stored in the memory are executed by the processor, the image processing method in the first aspect is implemented. For the steps executed by the execution device in each possible implementation manner of the processor, the specific implementation manners can be referred to the first aspect, and will not be described here.

[0019] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which stores a computer program, when the computer program is executed on a computer, the computer executes the image processing method in the first aspect.

[0020] In a fifth aspect, an embodiment of the present application provides a circuit system, which includes a processing circuit, and the processing circuit is configured to execute the image processing method in the first aspect.

[0021] In a sixth aspect, an embodiment of the present application provides a computer program, when the computer program is executed on a computer, the computer executes the image processing method in the first aspect.

[0022] In a seventh aspect, an embodiment of the present application provides a chip system, which includes a processor, and is configured to support the execution device to implement the functions involved in the above aspects, for example, sending or processing the data and / or information involved in the above method. In a possible design, the chip system further includes a memory, and the memory is configured to store the necessary program instructions and data of the server or the communication device. The chip system can be composed of a chip, or can include the chip and other discrete devices. BRIEF DESCRIPTION OF DRAWINGS

[0023] Figure 1 A structural schematic diagram of an image processing system provided by an embodiment of the present application;

[0024] Figure 2 A schematic diagram of an application scenario of an image processing method provided by an embodiment of the present application;

[0025] Figure 3 A flowchart of an image processing method provided by an embodiment of the present application;

[0026] Figure 4 A schematic diagram of N groups of video frames in an image processing method provided by an embodiment of the present application;

[0027] Figure 5 Another schematic diagram of N groups of video frames in an image processing method provided by an embodiment of the present application;

[0028] Figure 6A flowchart provided by an embodiment of the present application for a first neural network to process a first frame;

[0029] Figure 7 A flowchart provided by an embodiment of the present application for a second neural network to process a second frame;

[0030] Figure 8 A flowchart provided by an embodiment of the present application for processing consecutive video frames in an image processing method;

[0031] Figure 9 A flowchart provided by an embodiment of the present application for a network training method;

[0032] Figure 10 A structural diagram of a video frame processing apparatus provided by an embodiment of the present application;

[0033] Figure 11 A structural diagram of an execution device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0034] Embodiments of the present application provide an image processing method and related devices, which are used to introduce feature information of a first frame in a first neural network calculation process when processing a second frame by using a second neural network, to improve the continuity between the processing results of the first frame and the second frame with correlation, and to improve the accuracy of the processing result of the second frame.

[0035] The terms "first", "second", etc. in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily have to be used to describe a specific order or sequence. It should be understood that the terms used in this way can be interchanged under appropriate circumstances, which is only a distinguishing way used in the description of the embodiments of the present application to describe the objects with the same attributes. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, so that the processes, methods, systems, products or devices containing a series of units do not have to be limited to those units, but can include other units that are not clearly listed or inherent to these processes, methods, products or devices.

[0036] The embodiments of the present application can be applied to various scenarios of processing continuous video frames. The embodiments of the present application can be applied to scenarios of processing continuous video frames as a whole, in which the last output is the processed video frame. As an example, in a scenario of performing image signal processing (ISP) on continuous video frames, the ISP processing on continuous video frames includes but is not limited to color correction on continuous video frames, black level compensation on continuous video frames, green balance adjustment on continuous video frames, bad pixel correction on continuous video frames, demosaic processing on continuous video frames, denoise processing on continuous video frames, auto white balance processing on continuous video frames, gamma correction on continuous video frames, or denoise and sharpness on continuous video frames, etc. As another example, in a scenario of performing super-resolution processing on continuous video frames. The embodiments of the present application can also be applied to scenarios of processing objects in video frames, as an example, performing image recognition operation on objects in continuous video frames, as another example, performing image segmentation on continuous video frames, as another example, performing image classification operation on objects in continuous video frames, etc. In the foregoing scenarios, the image processing method provided by the embodiments of the present application can also be used to improve the continuity between the processing results of multiple continuous video frames. It should be understood that the examples herein are only for the convenience of understanding the application scenarios of the embodiments of the present application, and do not exhaust the application scenarios of the embodiments of the present application. It can be known by those skilled in the art that, as technology develops and new scenarios appear, the technical solutions provided by the embodiments of the present application are also applicable to similar technical problems.

[0037] In order to facilitate the understanding of the present application, the embodiments of the present application first combine Figure 1 The image processing system provided by the embodiments of the present application is introduced, please refer to Figure 1 , Figure 1 A structural schematic diagram of the image processing system provided by the embodiments of the present application. In Figure 1 , the image processing system 100 includes an execution device 110, a training device 120, a database 130 and a data storage system 140, and the execution device 110 includes a calculation module 111.

[0038] In the training stage, the training data set is stored in the database 130, which can include a plurality of groups of continuous video frames, and each group of continuous video frames includes at least two video frames. The training device 120 generates a target model / rule 101 for image processing, and iteratively trains the target model / rule 101 by using the training data set in the database to obtain a mature target model / rule 101, which can be specifically an image processing network. The image processing network obtained by the training device 120 can be applied in different systems or devices.

[0039] In the inference stage, the execution device 110 can call data, code, etc. in the data storage system 140, or store data, instructions, etc. in the data storage system 140. The data storage system 140 can be configured in the execution device 110, or the data storage system 140 can be an external storage relative to the execution device 110. The computing module 111 can process the continuous video frames obtained by the execution device 110 by using the image processing network to obtain a processing result, and the specific form of the processing result is related to the function of the image processing network.

[0040] In some embodiments of the present application, for example Figure 1 In the above description, the "user" can directly interact with the execution device 110, that is, the execution device 110 and the client device are integrated in the same device. However Figure 1 The position relationship between the devices, components, modules, etc. shown in the figure is not intended to constitute any limitation. In another embodiment of the present application, the execution device 110 and the client device can be separate devices, and the execution device 110 is configured with an input / output interface to interact with the client device. The "user" can input the collected image to the input / output interface through the client device, and the execution device 110 returns the processing result to the client device through the input / output interface.

[0041] In combination with the above description, please refer to Figure 2 , Figure 2An application scenario of the image processing method provided in the embodiment of the present application is shown in the figure. After the execution device obtains the video stream, the execution device needs to perform ISP processing on each frame of image in the video stream through an ISP network (i.e., a neural network for ISP processing), as shown in the figure, the ISP processing includes but is not limited to adjusting black balance, adjusting green balance, bad point correction, demosaicing processing, noise reduction processing, automatic white balance processing, gamma correction, and noise reduction and sharpening. The frame set corresponding to the video stream includes a first frame and a second frame, the first frame and the second frame are consecutive video frames. In the process of calculating the first frame by the execution device using the ISP network, the feature information of each first frame can be obtained, and the feature information of each first frame is stored in a double data rate synchronous dynamic random access memory (DDR). In the process of calculating the second frame by the execution device using the ISP network, the feature information of each first frame is obtained from the DDR as input information in the process of calculating the second neural network. Further, the embedded neural-network processing unit (NPU) performs a visual task on the video stream after ISP processing through a neural network, and the aforementioned visual task includes but is not limited to image classification, image segmentation, image recognition, etc. The execution device introduces the first feature information obtained according to the first frame in the process of processing the second frame, and the second frame and the first frame are consecutive video frames in the same video, thereby improving the continuity between the processing results of consecutive video frames. The processes of the second neural network in the inference stage and the training stage will be introduced respectively as follows.

[0042] I. Inference stage

[0043] Please refer to Figure 3 , Figure 3 A flowchart of the image processing method provided in the embodiment of the present application, the image processing method provided in the embodiment of the present application can include:

[0044] 301. The execution device obtains the first frame.

[0045] In the embodiments of the present application, the execution device obtains the first frame. The first frame refers to the source frame of the first feature information, that is, in the process of image processing of the second frame by the execution device using the second neural network, the first feature information will be used as input information in the calculation process of the second neural network to obtain the processing result of the second frame, therefore, the first feature information will affect the processing result of the second frame. The correlation between the first frame and the second frame refers to the similarity between the first frame and the second frame being greater than or equal to the first threshold. The similarity between the first frame and the second frame can be calculated according to the first frame, the second frame and a similarity algorithm, which includes but is not limited to cosine similarity, Euclidean distance, Mahalanobis distance, Manhattan distance, Hamming distance or other similarity algorithms, etc., which are not exhaustively listed here. As an example, for example, the first frame and the second frame are consecutive video frames in the same video, then in the above-mentioned scenario, the similarity between the first frame and the second frame is considered to be greater than or equal to the first threshold. As another example, for example, the first frame and the second frame are two images with the same background, or the first frame and the second frame are two images with the same moving track, or the motion vectors of the objects in the first frame and the second frame are the same, etc., which are not exhaustively listed here. Further, the first frame and the second frame can be RAW images. The first feature information can be a one-dimensional tensor (also referred to as a one-dimensional vector), a two-dimensional tensor (also referred to as a two-dimensional matrix or a feature map), a three-dimensional tensor (also referred to as a three-dimensional feature tensor) or a higher-dimensional tensor, etc., and the specific form is not limited here. Further, the three-dimensional tensor can also be regarded as the superposition of three two-dimensional matrices, and the higher-dimensional tensor can be regarded as the superposition of multiple two-dimensional matrices.

[0046] In the case where the first frame and the second frame are consecutive video frames in the same video, after obtaining the video to be processed, the execution device can obtain a video frame set corresponding to the video to be processed, the video frame set including a plurality of consecutive video frames, the sorting positions of the plurality of consecutive video frames being consistent with the appearance positions of the video frames in the video, that is, the video frame appearing earlier in the video is sorted earlier in the video frame set, and the video frame appearing later in the video is sorted later in the video frame set. The execution device obtains the first frame from the video frame set.

[0047] Specifically, after obtaining the video frame set, that is, obtaining a plurality of continuous video frames arranged in sequence, the execution device can perform grouping processing every n video frames, that is, each group of continuous video frames includes n video frames, so that the video frame set can be divided into N groups of video frames, n is an integer greater than or equal to 2, and N is an integer greater than or equal to 1. A group of video frames is composed of at least one second frame and at least one first frame. During the subsequent process of processing the video frames by the execution device using the image processing network, a group of video frames can be used as a processing unit, that is, when the execution device processes the second frame using the second neural network, the first feature information introduced by the execution device comes from the first frame in the same group, so that the execution device can obtain the first frame from a group of video frames.

[0048] More specifically, in an implementation manner, a group of video frames can include 2 continuous video frames (that is, the value of n is 2), the second frame is an even frame in a group of video frames, and the first frame is an odd frame in a group of video frames; the execution device obtains the odd frame from a group of video frames, and determines the obtained odd frame as the first frame.

[0049] For further understanding of the scheme, please refer to Figure 4 , Figure 4 An example of N groups of video frames in the image processing method provided by the embodiment of the present application. Figure 4 In the embodiment, 30 continuous video frames in the video frame set corresponding to the video to be processed are taken as an example, Figure 4 1, 2, 3, 4, …, 29, 30 in the embodiment represent the positions of each video frame in the video to be processed, and also represent the positions of each video frame in the video frame set. Figure 4 In the embodiment, the first three groups of video frames are described, each group of video frames includes two adjacent video frames, the even frame in a group of video frames is the second frame, and the odd frame in a group of video frames is the first frame. For example, video frame 2 is the second frame, video frame 1 is the first frame, video frame 4 is the second frame, video frame 3 is the first frame, video frame 6 is the second frame, and video frame 5 is the first frame. For the combination mode of video frame 7 to video frame 30, please refer to the combination mode of the first three groups of video frames, which will not be described one by one here, and it should be understood that Figure 4 The example in the embodiment is only for the convenience of understanding the scheme, and is not used to limit the scheme.

[0050] In another implementation, the set of video frames includes 2 consecutive video frames (i.e., n is 2), the second frame is an odd frame in the set of video frames, and the first frame is an even frame in the set of video frames. The execution device obtains the even frame from the set of video frames and determines the obtained even frame as the first frame. In the embodiment of the present application, the first feature information is introduced in the process of processing the second frame by using the second neural network, the adjacent video frame of the second frame is determined as the first frame, so that the second frame and the first frame have a high similarity, which is beneficial to improve the accuracy of the second processing result.

[0051] In another implementation, the set of video frames includes 3 consecutive video frames (i.e., n is 3), the second frame is the third video frame in the three consecutive video frames, and the at least one first frame includes the first video frame and the second video frame in the three consecutive video frames. The first video frame appears earlier than the second video frame in the video, and the second video frame appears earlier than the third video frame in the video. The execution device selects one video frame from the first video frame and the second video frame in the set of video frames and determines the selected video frame as a first frame.

[0052] For further understanding of the present scheme, please refer to Figure 5 , Figure 5 A schematic diagram of N sets of video frames in the image processing method provided by the embodiment of the present application. Figure 5 In the embodiment, the set of video frames corresponding to the video to be processed includes 30 consecutive video frames, Figure 5 1, 2, 3, 4, …, 29, and 30 in the embodiment represent the positions of each video frame in the video to be processed and the positions of each video frame in the set of video frames. Figure 5 In the embodiment, the first three sets of video frames are also described, each set of video frames includes three consecutive video frames, the first video frame and the second video frame in a set of consecutive video frames are first frames, and the third video frame in a set of consecutive video frames is a second frame. For example, video frame 1 and video frame 2 are first frames, and video frame 3 is a second frame; video frame 4 and video frame 5 are first frames, and video frame 6 is a second frame; video frame 7 and video frame 8 are first frames, and video frame 9 is a second frame. For the combination mode of video frame 10 to video frame 30, please refer to the combination mode of the first three sets of video frames, which will not be described here again. It should be understood that Figure 5 The examples in the embodiment are only for the convenience of understanding the present scheme and are not used to limit the present scheme.

[0053] In another implementation, the group of video frames includes 3 consecutive video frames (i.e., n is 3), the second frame is the first one of the three consecutive video frames, and the at least one first frame includes the second and third ones of the three consecutive video frames. The execution device selects one of the second and third ones of the group of video frames as a first frame.

[0054] In another implementation, the group of video frames includes 3 consecutive video frames (i.e., n is 3), the second frame is the first one of the three consecutive video frames, and the at least one first frame includes the second and third ones of the three consecutive video frames. The execution device selects one of the second and third ones of the group of video frames as a first frame.

[0055] In the embodiments of the present application, the three consecutive video frames are determined as a group of video frames, two of the three consecutive video frames are determined as first frames, and one of the three consecutive video frames is determined as a second frame. Since the three consecutive video frames have high similarity, the accuracy of the processing result of the second neural network is improved. In addition, the feature information corresponding to the two first frames is introduced in the processing of the second frame, so as to further improve the accuracy of the processing result of the second neural network.

[0056] In another implementation, the group of video frames includes 3 consecutive video frames (i.e., n is 3), the at least one second frame includes the second and third ones of the three consecutive video frames, and the first frame is the first one of the three consecutive video frames. The execution device selects the first one of the group of video frames as a first frame.

[0057] In another implementation, the group of video frames includes 3 consecutive video frames (i.e., n is 3), the at least one second frame includes the first and third ones of the three consecutive video frames, and the first frame is the second one of the three consecutive video frames. The execution device selects the second one of the group of video frames as a first frame.

[0058] In another implementation, the group of video frames includes 3 consecutive video frames (i.e., n is 3), the at least one second frame includes the first and second ones of the three consecutive video frames, and the first frame is the third one of the three consecutive video frames. The execution device selects the third one of the group of video frames as a first frame.

[0059] In other implementations, n can also be 4, 5, 6, 7, 8, 9, 10, or other values, and the value of n can be flexibly determined according to a specific implementation scenario. The position of the first frame in the group of video frames can also be flexibly set according to actual conditions, which is not limited here.

[0060] 302. The execution device performs image processing on the first frame by using the first neural network to obtain a first processing result output by the first neural network.

[0061] In some embodiments of the present application, after obtaining a first frame, the execution device only inputs the first frame into the first neural network, performs image processing on the first frame by using the first neural network, and obtains a first processing result output by the first neural network. The processing method of the first frame is different from that of the second frame. When the execution device processes the second frame by using the second neural network, the second frame and the first feature information are input into the second neural network. When the execution device processes the first frame by using the first neural network, only the first frame is input into the first neural network, that is, no feature information corresponding to other video frames is introduced in the process of processing the first frame.

[0062] The first processing result is the processing result of the first frame. The specific form of the first processing result can be determined according to the function of the first neural network. For example, the first processing result can be a video frame that has undergone color correction, a video frame that has undergone black level adjustment, a video frame that has undergone demosaicing, a video frame that has undergone noise reduction, a video frame that has undergone gamma correction, a video frame that has undergone super-resolution, and the like. Further, the first processing result can be in RAW format, RGB format, or other formats. As another example, the first processing result can be a recognition result in the first frame, such as indicating that the object in the first frame is a cat, and the like, which is not exhaustive here.

[0063] The second neural network and the first neural network can each include a feature extraction module and a feature processing module, and each of the feature extraction module and the feature processing module can include multiple neural network layers. For example, in the case of applying the second neural network and the first neural network to ISP, the feature extraction module can specifically be a decoder, and the feature processing module can specifically be an encoder. As another example, in the case of applying the second neural network and the first neural network to image classification, the feature extraction module can specifically be a convolutional layer, and the feature processing module can specifically be a perception machine, and the specific form of the second neural network and the second image network can be determined according to the specific application scenario and the specific function of the image processing network, which is not limited here.

[0064] Further, the second neural network and the first neural network are functionally identical, and are both neural networks for implementing image processing. The second neural network includes a first number of neural network layers, and the first neural network includes a second number of neural network layers, the first number being less than the second number. The first neural network can be an encoder-decoder type network, a convolutional network, a U-shaped neural network, or another type of neural network, etc. The specific form of the first neural network is determined by the specific application scenario of the first neural network, and is not exhaustively listed here.

[0065] On the basis of the first neural network, one or more neural network layers are pruned to obtain the second neural network. The first feature information includes feature information of each first frame. Since the first feature information is input information in the calculation process of the second neural network, the neural network layer for generating the first feature information in the second neural network can be pruned. Optionally, the feature information of each first frame is calculated and output by an i-th layer of the first neural network, and the i-th layer is any layer of the first neural network. The second neural network can be obtained by pruning the first layer to the i-th layer of the first neural network. In the embodiment of the present application, the first layer to the i-th layer of the second neural network are pruned, which greatly reduces the processing time of the second neural network; and avoids the size adjustment of the first feature information, so as to further shorten the total processing time.

[0066] To more clearly understand the relationship between the second neural network and the first neural network, please refer to Figure 6 and Figure 7 , Figure 6 The flowchart provided by the embodiment of the present application intends to process the first frame by the first neural network, Figure 7 The flowchart provided by the embodiment of the present application intends to process the second frame by the second neural network. Figure 6 and Figure 7 Both of them take the application of the second neural network and the first neural network in the ISP scenario as an example for introduction, and Figure 6 and Figure 7 Each rectangle in the figure represents a feature map. The wider the rectangle, the higher the dimension of the corresponding feature map. Please refer to Figure 6 , Figure 6 The feature map indicated by C1 is a boundary. The feature maps on the left side of C1 (not including the feature map represented by C1) are all feature maps obtained by the decoder of the first neural network for feature extraction on the input first frame. The feature maps on the right side of C1 (including the feature map represented by C1) include feature maps generated by the encoder of the first neural network in the ISP processing process. Figure 6The arrows in the figure represent the relationship between the feature maps, C2 represents the feature map generated after cropping C3, C4 represents the feature map generated after cropping C5, C6 represents the feature map generated after convolution of C7, and C8 represents the feature map obtained by convolution of C9. After the execution device obtains C2 and C6 through the encoder of the first neural network, Figure 6 The generation methods of other feature maps in the figure can be referred to the foregoing description and will not be described one by one.

[0067] In combination Figure 6 The description Figure 7 , the execution device can obtain C8 generated in the process of processing the first frame, so that the execution device directly obtains C8 when performing ISP processing by using the encoder of the second neural network. Since C8 is input in the operation process of the second neural network, the neural network layer for generating C8 in the second neural network can be pruned, that is, the feature map of D1 part will not be generated in the calculation process of the second neural network. Compared with Figure 6 the same, Figure 7 The arrows in the figure represent the relationship between the feature maps, Figure 7 D2 in the figure represents the feature map generated after cropping C8, and D4 represents the feature map obtained by convolution of D5, Figure 7 The generation methods of other feature maps in the figure can be referred to the foregoing description and will not be described one by one. Compared with Figure 6 and Figure 7 It can be seen that since the number of neural network layers of the second neural network is less than that of the first neural network, the number of feature maps generated by the first neural network in the process of processing the first frame is more than the number of feature maps generated by the second neural network in the process of processing the second frame. The second neural network introduces the feature map corresponding to the first frame, which not only improves the continuity of the output results between the continuous video frames, but also improves the accuracy of the output results of the second frame.

[0068] It should be noted that Figure 6 and Figure 7 The diagram in the figure is only for the convenience of understanding the scheme, and in other embodiments, the second neural network can also introduce more feature maps in the process of processing the second frame, for example, the second neural network can introduce the feature maps represented by C5 and C4 in the process of ISP processing of the second frame. Figure 6 and Figure 7 In the example shown in the figure, the second neural network introduces the feature map corresponding to one first frame in the process of ISP processing of the second frame, and in other embodiments, the feature map corresponding to multiple first frames can also be introduced, and the like, which are not limited here.

[0069] 303. The execution device obtains first feature information of the first frames in the first neural network calculation process.

[0070] In the embodiments of the present application, the execution device needs to obtain first feature information of at least one first frame in the first neural network calculation process, and the first feature information includes feature information of each first frame. The feature information of each first frame is feature information generated by each first frame in the first neural network calculation process. Further, the feature information of each first frame is a characteristic map generated by each first frame in the first neural network calculation process. In the embodiments of the present application, the feature information of each first frame is concretized to improve the degree of combination with the actual application scenario. In combination Figure 6 For example, the first feature information can only include the feature information represented by C8, or can include the feature information represented by C3 and C8, or can include the feature information represented by C3, C5 and C8, etc. The specific feature information at which position can be flexibly determined in combination with the actual application environment, which is not limited here.

[0071] Specifically, the execution device can store the obtained feature information into a preset storage device during the processing of the first frame by the first neural network, or after the processing of the first frame by the first neural network, and then the execution device obtains the first feature information from the preset storage device. Further, since a group of video frames can include one or at least two first frames, if the group of video frames includes one first frame, steps 301 to 303 only need to be executed once to obtain the first feature information of the one first frame in the first neural network calculation process. If the group of video frames includes at least two first frames, the execution device needs to execute steps 301 to 303 multiple times to obtain the first feature information of the at least two first frames in the first neural network calculation process.

[0072] More specifically, in an implementation manner, step 303 can include: after the execution device obtains the feature information generated by each first frame in the first neural network calculation process, determining a set composed of the feature information generated by all the first frames in the first neural network calculation process as the first feature information. In the embodiments of the present application, the set composed of the feature information generated by each first frame in the first neural network calculation process is directly determined as the first feature information, which is simple to operate.

[0073] In another implementation, step 303 can include: after the execution device obtains the feature information generated in the first neural network calculation process for each first frame, the execution device can perform merging processing on the feature information of the at least two first frames to generate first feature information. The merging manner includes but is not limited to conact, add, fusion, multiplication, etc. Further, since the feature information of each first frame can include at least two feature information, the execution device can merge the feature information at the corresponding positions. In the embodiment of the present application, another generation manner of the first feature information is provided, which improves the implementation flexibility of the present scheme.

[0074] 304. The execution device obtains a second frame.

[0075] In the embodiment of the present application, the execution device obtains a second frame. Steps 301 and 302 are optional steps. If steps 301 and 302 are not performed, in the case that the first frame and the second frame are consecutive video frames in the same video, the execution device can process the first frame in the same manner as processing the second frame, and the second frame and the first frame are both any one of the video frame set corresponding to the video to be processed. If steps 301 and 302 are performed, the second frame can be a video frame at a preset position, which can be referred to in the introduction of the second frame in step 301, and will not be described here. The execution device can obtain the second frame from the group of video frames where the first frame is located, and the format of the second frame can be the same as that of the first frame.

[0076] 305. The execution device performs image processing on the second frame by using a second neural network, and the first feature information is used as input information in the calculation process of the second neural network.

[0077] In the embodiment of the present application, after obtaining the second frame, the execution device performs image processing on the second frame by using a second neural network, and the first feature information is used as input information in the calculation process of the second neural network. The concept of the second neural network can be referred to in the description of step 302, and will not be described here.

[0078] Specifically, the training device can first input the second frame into the second neural network, and input the first feature information into the second neural network in the process of processing the second frame by the second neural network. Since the first feature information can include one or more feature information, the step of inputting the first feature information into the second neural network by the execution device can be performed one or more times. It should be noted that if the first feature information is processed by the jth layer of the second neural network, but the size of the first feature information does not meet the requirement of the jth layer, the size of the first feature information needs to be adjusted before inputting the first feature information into the jth layer, specifically, the size of the first feature information is adjusted to be larger or smaller. The foregoing size adjustment manner includes but is not limited to convolution or other processing manners.

[0079] For further understanding of the present scheme, please refer to Figure 8 , Figure 8 A flowchart for processing a continuous video frame in the image processing method provided by the embodiment of the present application is shown. Figure 8 In the embodiment, two first frames in a group of continuous video frames are taken as an example, and two flowcharts F1 and F2 respectively show the process of processing the two first frames by the execution device through the first neural network, and F3 shows the process of processing the second frame by the execution device through the second neural network. Among them, F4 and F5 represent the feature information generated by the processing layer 2 of the first neural network, and F6 and F7 represent the feature information generated by the processing layer 4 of the first neural network. As shown in Figure 8 , the execution device will perform the operation of inputting the first feature information into the second neural network twice in the process of processing the second frame by the second neural network. The first time is to input the feature information represented by F4 and F5 into the processing layer 3 of the second neural network, and after the processing layer 3 of the second neural network obtains the feature information generated by the input layer, F4 and F5, the processing layer 3 of the second neural network combines the feature information generated by the input layer, F4 and F5, and generates new feature information according to the combined feature information. The second time is to input the feature information represented by F6 and F7 into the processing layer 5 of the second neural network, and after the processing layer 5 of the second neural network obtains the feature information generated by the processing layer 3, F6 and F7, the processing layer 5 of the second neural network combines the feature information generated by the processing layer 3, F6 and F7, and performs classification operation on the object in the second frame according to the combined feature information. In addition, by Figure 8 It can also be seen that the number of layers of the neural network in the second neural network is less than the number of layers of the neural network in the first neural network, and since the feature information of the first frame is introduced in the process of processing the second frame, the accuracy of the second processing result is ensured on the premise of reducing the processing time of the second frame. It should be understood that Figure 8 the examples in the embodiment are only for the convenience of understanding the present scheme, and are not used to limit the present scheme.

[0080] In the embodiments of the present application, after the execution device inputs the second frame into the second neural network, the second neural network generates the second feature information corresponding to the second frame, and obtains the second feature information generated in the image processing process of the second frame by the second neural network; the first feature information and the second feature information are combined to obtain the combined feature; and the second neural network is used to process the combined feature information. The second feature information refers to the feature information generated in the processing process of the second frame by the second neural network. The combination methods include but are not limited to splicing, addition, fusion, multiplication and pooling.

[0081] Specifically, in an application scenario, if the feature extraction module and the feature processing module of the second neural network both generate feature information; as an example, for example, when the second neural network is applied to the ISP scene of color correction of continuous video frames, bad point correction of continuous video frames, etc., the feature extraction module and the feature processing module of the second neural network both generate feature information. Therefore, in the foregoing scenario, the feature extraction module and the feature processing module of the second neural network can both introduce the feature information corresponding to the first frame.

[0082] More specifically, the execution device can only perform the combination operation of the first feature information and the second feature information by the feature extraction module of the second neural network, in combination with Figure 7 For illustration, the second feature information corresponding to the first frame can be introduced only in the left part of the Figure 7 For illustration, the second feature information corresponding to the first frame can be introduced only in the right part of the Figure 7 For illustration, the second feature information corresponding to the first frame can be introduced only in the right part of the Figure 7 For illustration, the second feature information corresponding to the first frame can be introduced only in the right part of the Figure 7 For illustration, the second feature information corresponding to the first frame can be introduced only in the right part of the Figure 7 For illustration, the second feature information corresponding to the first frame can be introduced only in the right part of the Figure 7 For illustration, the second feature information corresponding to the first frame can be introduced only in the right part of the

[0083] In another application scenario, the feature extraction module of the second neural network generates feature information, and the feature processing module of the second neural network does not generate feature information; as an example, when the second neural network is applied to image recognition of an object in a continuous video frame, the feature processing module of the second neural network does not generate feature information again. In the foregoing scenario, only the feature information corresponding to the first frame can be introduced into the feature extraction module of the second neural network.

[0084] 306. The execution device obtains a second processing result output by the second neural network.

[0085] In the embodiment of the application, after the execution device generates the second processing result according to the second frame and the first feature information through the second neural network, the execution device outputs the second processing result, and the second processing result is a processing result of the second frame. The specific form of the second processing result is consistent with the specific form of the first processing result, and the specific form can be referred to the description in the step 302, which will not be described here.

[0086] In the embodiment of the application, the first frame is input into the first neural network to obtain the first processing result of the first frame output by the first neural network, and the neural network layer of the first neural network is more than that of the second neural network, so that the accuracy of the first processing result output by the first neural network is higher. In addition, the time length for the second neural network to generate the second processing result is less than the time length for the first neural network to generate the first processing result, but because the first feature information corresponding to the first frame is introduced in the process of generating the second processing result, that is, the feature information with higher accuracy is introduced to improve the accuracy of the second processing result, the total processing time length of the plurality of continuous video frames is shortened, and the accuracy of the processing result of the plurality of continuous video frames is guaranteed.

[0087] In the embodiment of the application, when the second frame is processed by the second neural network, the feature information of the first frame in the calculation process of the first neural network is introduced, so that the continuity between the processing results of the first frame and the second frame with correlation can be improved. In addition, the feature information of the first frame is introduced, that is, the image processing network can integrate more feature information when processing the second frame, so that the accuracy of the processing result of the first frame can be improved.

[0088] II. Training phase

[0089] Please refer to Figure 9 , Figure 9 A flowchart of a network training method provided in the embodiment of the application, the network training method provided in the embodiment of the application can include:

[0090] 901. The training device obtains a first training frame.

[0091] In the embodiments of the present application, the training device can be preconfigured with a training frame set corresponding to the training video, the training frame set includes a plurality of groups of continuous training frames, and the training device can also process the training frames in units of a group of training frames by using the image processing network. A group of continuous training frames includes n training frames, and the n training frames include a first training frame and a second training frame. The concept of the first training frame is similar to that of the first frame, and the first training frame refers to a source frame of the third feature information, that is, the training device inputs at least a group of third feature information corresponding to the first training frame and the second training frame into the second neural network to obtain a processing result of the second training frame, and therefore the third feature information corresponding to the first training frame will affect the processing result of the second training frame. The specific form of the first training frame and the position in the group of continuous training frames can be referred to the description of the first frame, and the specific form of the third feature information can be referred to the description of the first feature information, which will not be repeated here.

[0092] The training device obtains a first training frame from a group of continuous training frames. The specific implementation mode can be referred to the description of the first frame Figure 3 The description of step 301 in the corresponding embodiments will not be repeated here.

[0093] The training device can also be preconfigured with a first target result corresponding to the first training frame and a second target result corresponding to the second training frame. The first target result and the second target result have the same format, the first target result is an ideal result of the first training frame after the first neural network, and the second target result is an ideal result of the second training frame after the second neural network. In the case that the first neural network is applied to the ISP scene, the first target result and the second target result can specifically be clear video frames without mosaics, without noise and with high resolution. In the case that the first neural network is applied to the image super-resolution scene, the first target result and the second target result can be video frames with high resolution. In the case that the first neural network is applied to the image recognition scene, the first target result and the second target result can be accurate recognition results of objects in the video frames, etc. The specific forms of the first target result and the second target result depend on the functions of the first neural network and the second neural network, which will not be repeated here. Figure 3 The specific forms of the first processing result in the corresponding embodiments are the same, and all depend on the functions of the first neural network, which will not be repeated here.

[0094] In the embodiments of the present application, the specific implementation mode of step 902 can be referred to the description of step 301 Figure 3 The description of step 302 in the corresponding embodiments will not be repeated here, and the concepts of the second neural network and the first neural network have been described in the description of step 301. Figure 3The meaning of the third processing result is described in step 302 of the corresponding embodiment, and is similar to that of the first processing result Figure 3 The meaning of the first processing result in the corresponding embodiment is similar, and will not be described here.

[0095] 903、The training device trains the first neural network according to the third processing result and the first loss function.

[0096] In the embodiment of the present application, after obtaining the third processing result, the training device will calculate the function value of the first loss function according to the third processing result and the first target result, and the first loss function is used to indicate the similarity between the third processing result and the first target result. The first loss function can be L1 loss function (loss), mean squared error (MSE) loss function, structural similarity index (SSIM) loss function or other types of loss function, etc., which is not limited this time.

[0097] After obtaining the function value of the first loss function, the training device will generate a first gradient value according to the function value of the first loss function, and use the first gradient value to perform back propagation on the first neural network to update the value of the weight parameter in each neural network layer in the first neural network, thereby completing a training of the first neural network. The training device can repeatedly execute steps 901 to 903 to realize the iterative training of the first neural network.

[0098] 904、The training device obtains third feature information of the first training frame in the first neural network calculation process.

[0099] 905、The training device obtains a second training frame.

[0100] 906、The training device uses the second neural network to perform image processing on the second training frame, and the third feature information is used as input information in the second neural network calculation process.

[0101] 907、The training device obtains a fourth processing result output by the second neural network.

[0102] In the embodiment of the present application, the specific implementation of steps 904 to 907 can refer to Figure 3The description of steps 303 to 306 in the corresponding embodiment is not repeated here. Among them, the second training frame and the first training frame are consecutive video frames in the same training video, the meaning of the second training frame is similar to the meaning of the second frame, the position of the second training frame in a set of training frames and the specific form of the second training frame can be referred to the introduction of the second frame, the difference is that the second training frame is used in the training stage, and the second frame is used in the inference stage. The specific form of the fourth processing result is the same as that of the third processing result, and is not repeated here.

[0103] 908、The training device trains the second neural network according to the fourth processing result and the second loss function.

[0104] In the embodiment of the application, after obtaining the fourth processing result, the training device will calculate the function value of the second loss function according to the fourth processing result and the second target result, and the second loss function is used to indicate the similarity between the fourth processing result and the second target result. The second loss function can be L1 loss function (loss), mean squared error (MSE) loss function, structural similarity index (SSIM) loss function or other types of loss functions, etc., which is not limited this time.

[0105] After obtaining the function value of the second loss function, the training device will generate a second gradient value according to the function value of the second loss function, and use the second gradient value to back propagate the first neural network to update the value of the weight parameter in each neural network layer in the first neural network, thereby completing a training of the first neural network. The training device can realize iterative training of the first neural network by repeatedly executing steps 904 to 908.

[0106] It should be noted that the number of times of executing steps 901 to 903 and steps 904 to 908 can be a multiple-to-one relationship, a one-to-one relationship, or a one-to-many relationship. If the relationship between the number of first training frames and the number of first training videos in a set of consecutive training frames is multiple-to-one, then the relationship between the number of times of executing steps 901 to 903 and steps 904 to 908 is multiple-to-one. If the relationship between the number of first training frames and the number of first training videos in a set of consecutive training frames is one-to-one, then the relationship between the number of times of executing steps 901 to 903 and steps 904 to 908 is one-to-one. If the relationship between the number of first training frames and the number of first training videos in a set of consecutive training frames is one-to-many, then the relationship between the number of times of executing steps 901 to 903 and steps 904 to 908 is one-to-many.

[0107] 909、The training device outputs the trained first neural network and the trained second neural network.

[0108] In the embodiments of the present application, the training device repeatedly performs steps 901 to 903 to realize iterative training of the first neural network until the convergence condition of the first loss function is met, and the trained first neural network is obtained. The training device repeatedly performs steps 904 to 908 to realize iterative training of the second neural network until the convergence condition of the second loss function is met, and the trained second neural network is obtained. Then the trained first neural network and the second neural network can be output. The convergence condition of the first loss function can be that the function value of the first loss function is less than or greater than a first preset threshold, and the convergence condition of the second loss function can be that the function value of the second loss function is less than or greater than a second preset threshold.

[0109] In the embodiments of the present application, the time length of processing a video frame by the trained second neural network is less than the time length of processing a video frame by the trained first neural network, but the feature information obtained by the trained first neural network is introduced in the processing of the trained second neural network, that is, the feature information with higher accuracy is introduced, which is conducive to improving the accuracy of the processing result of the trained second neural network, shortening the total processing time length of multiple continuous video frames, and ensuring the accuracy of the processing result of multiple continuous video frames.

[0110] In the embodiments of the present application, the trained second neural network introduces the feature information of the first frame when processing the second frame, which can improve the continuity between the processing results of the first frame and the second frame with correlation. In addition, by introducing the feature information of the first frame, that is, the trained second neural network can integrate more feature information when processing the second frame, thereby improving the accuracy of the processing result of the second frame.

[0111] In order to have a more intuitive understanding of the beneficial effects brought by the embodiments of the present application, the beneficial effects brought by the embodiments of the present application are further introduced as follows in combination with the implementation data. Taking a group of video frames including two video frames as an example, the second neural network and the first neural network are both neural networks for demosaicking, and the first neural network is a neural network for demosaicking in the prior art, which includes an input layer, a convolution layer 1, five residual modules, a convolution layer 2 and an output layer. In the second neural network, the convolution layer 1 and the five residual modules are removed on the basis of the neural network for demosaicking in the prior art, only the convolution layer 2 is retained, and a convolution layer 3 is added before the output layer as a rejection filter. The total time length for processing a group of continuous video frames by using the image processing network in the prior art is 10.694 ms, and the total time length for processing a group of continuous video frames by using the second neural network and the first neural network in the embodiments of the present application is 6.271 ms, the processing time length of each video frame is reduced by 4.446 ms on average. Obviously, the image processing method provided by the embodiments of the present application shortens the total processing time length of continuous video frames.

[0112] In Figures 1 to 9 On the basis of the corresponding embodiments, in order to better implement the above-mentioned scheme of the embodiments of the present application, the related equipment for implementing the above-mentioned scheme is further provided below. For details, please refer to Figure 10 , Figure 10 A structural schematic diagram of a video frame processing device provided by the embodiments of the present application is shown. The image processing device 1000 includes an acquisition module 1001 and a processing module 1002. The acquisition module 1001 is configured to acquire first feature information of at least one first frame in a first neural network calculation process; and the processing module 1002 is configured to perform image processing on a second frame by using a second neural network, wherein the first feature information is used as input information in a second neural network calculation process, the first neural network and the second neural network are both neural networks for implementing image processing, and the first frame and the second frame have a correlation.

[0113] In a possible design, the first neural network includes a first number of neural network layers, and the second neural network includes a second number of neural network layers, and the second number is less than the first number.

[0114] In a possible design, the first feature information includes feature information of each first frame, and the feature information of each first frame is feature information generated in the first neural network calculation process of each first frame.

[0115] In a possible design, the first feature information is information generated after merging feature information of each first frame, where the feature information of each first frame is feature information generated in a first neural network calculation process of each first frame, and the merging includes one or more of the following: splicing, adding, fusing, and multiplying.

[0116] In a possible design, the feature information of each first frame is a feature map generated in the first neural network calculation process of each first frame.

[0117] In a possible design, the feature information of each first frame is calculated and output by an i-th layer of the first neural network, the i-th layer is any layer in the first neural network, and the second neural network is obtained after pruning the first layer to the i-th layer in the first neural network.

[0118] In a possible design, the processing module 1002 is specifically configured to: generate second feature information corresponding to the second frame by using the second neural network, to obtain second feature information generated in an image processing process of the second frame by using the second neural network; combine the first feature information and the second feature information to obtain combined feature information; and process the combined feature information by using the second neural network.

[0119] In a possible design, the first frame and the second frame are consecutive video frames in a same video.

[0120] In a possible design, the first frame is an odd frame in the video, and the second frame is an even frame in the video; or, the first frame is an even frame in the video, and the second frame is an odd frame in the video.

[0121] In a possible design, the second frame and the first frame belong to a group of frames, the group of frames includes three consecutive frames; the second frame is a third frame in the three consecutive frames, and the at least one first frame includes a first frame and a second frame in the three consecutive frames; or, the second frame is a second frame in the three consecutive frames, and the at least one first frame includes the first frame and a third frame in the three consecutive frames; or, the second frame is a first frame in the three consecutive frames, and the at least one first frame includes a second frame and the third frame in the three consecutive frames.

[0122] It should be noted that the information interaction and execution process between the modules / units in the image processing apparatus 1000, and the beneficial effects brought by the image processing apparatus 1000, are the same as those of the image processing apparatus 1000 described in the preceding embodiments of the present application. Figures 3 to 9 The corresponding method embodiments are based on the same concept, and specific contents can be referred to the descriptions in the method embodiments described in the preceding embodiments of the present application, which will not be described herein again.

[0123] The present application also provides an execution device, which can be referred to Figure 11 ,Figure 11 A structural schematic diagram of an execution device provided by an embodiment of the present application is shown in FIG. 11. The execution device 1100 can be deployed with Figure 10 The image processing apparatus 1000 described in the corresponding embodiments is used to implement Figures 3 to 9 the functions of the execution device in the corresponding embodiments. Specifically, the execution device 1100 includes a receiver 1101, a transmitter 1102, a processor 1103 and a memory 1104 (wherein the number of processors 1103 in the execution device 1100 can be one or more, and the processor 1103 in the embodiment is exemplified by one processor), wherein the processor 1103 can include an application processor 11031 and a communication processor 11032. In some embodiments of the present application, the receiver 1101, the transmitter 1102, the processor 1103 and the memory 1104 can be connected through a bus or other means. Figure 11

[0124] The memory 1104 can include a read-only memory and a random access memory, and provide the processor 1103 with instructions and data. A part of the memory 1104 can also include a non-volatile random access memory (NVRAM). The memory 1104 stores processor and operation instructions, executable modules or data structures, or subsets thereof, or expanded sets thereof, wherein the operation instructions can include various operation instructions for implementing various operations.

[0125] The processor 1103 controls the operation of the execution device. In specific applications, various components of the execution device are coupled together through a bus system, which can include a data bus in addition to a power bus, a control bus and a status signal bus, etc. However, in order to clearly illustrate, various buses are referred to as a bus system in the figure.

[0126] ​The method disclosed in the embodiments of the present application can be applied to the processor 1103 or implemented by the processor 1103. The processor 1103 can be an integrated circuit chip with a signal processing capability. In the implementation process, the steps of the method can be completed by using an integrated logic circuit or an instruction in the form of software in the processor 1103. The processor 1103 can be a general-purpose processor, a digital signal processor (DSP), a microprocessor or a microcontroller. The processor 1103 can further include an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components. The processor 1103 can implement or execute the methods, steps and logical block diagrams disclosed in the embodiments of the present application. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor or the like. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware code processor for execution, or a combination of hardware and software modules in the code processor for execution. The software module can be located in a random access memory, a flash memory, a read-only memory, a programmable read-only memory or an electrically erasable programmable memory, a register or other mature storage medium in the art. The storage medium is located in the storage 1104, and the processor 1103 reads the information in the storage 1104 and combines the hardware to complete the steps of the method.

[0127] The receiver 1101 can be configured to receive input digital or character information, and generate signal input related to the relevant settings and function control of the execution device. The transmitter 1102 can be configured to output digital or character information through the first interface; the transmitter 1102 can also be configured to send instructions to the disk group through the first interface to modify the data in the disk group; the transmitter 1102 can also include a display device such as a display screen.

[0128] In the embodiments of the present application, the application processor 11031 is configured to execute Figures 3 to 9 the functions of the execution device. For the specific implementation manner of the application processor 11031 executing the image processing method and the beneficial effects brought by the specific implementation manner, reference can be made to the description in the corresponding method embodiments, which will not be repeated here. Figures 3 to 9

[0129] In the embodiments of the present application, a computer readable storage medium is also provided, and the computer readable storage medium stores a program for generating a vehicle driving speed. When the program is executed on a computer, the computer is caused to execute the method as described above. Figures 3 to 9 ​The method described in the illustrated embodiment describes steps performed by the device performing the method.

[0130] The embodiment of the present application further provides a computer program product comprising instructions which, when executed on a computer, cause the computer to carry out the method as described above. Figures 3 to 9 The method described in the illustrated embodiment describes steps performed by the device performing the method.

[0131] The embodiment of the present application further provides a circuit system comprising a processing circuit configured to perform the method as described above. Figures 3 to 9 The method described in the illustrated embodiment describes steps performed by the device performing the method.

[0132] The execution device provided by the embodiment of the present application can be a chip, which comprises a processing unit and a communication unit. The processing unit can be a processor, and the communication unit can be an input / output interface, a pin, a circuit, etc. The processing unit can execute computer execution instructions stored in a storage unit, so that the chip in the execution device performs the image processing method as described above. Figures 3 to 9 Optionally, the storage unit is a storage unit in the chip, such as a register, a cache, etc. The storage unit can also be a storage unit outside the chip in the wireless access device, such as a read-only memory (ROM) or other types of static storage devices that can store static information and instructions, a random access memory (RAM), etc.

[0133] The processor mentioned in any of the above can be a general central processor, a microprocessor, an ASIC, or one or more integrated circuits for controlling the execution of the program of the method of the first aspect.

[0134] It should be noted that the apparatus embodiments described above are merely illustrative, and the units described as separate components can or can not be physically separated, and the components shown as units can or can not be physical units, i.e. they can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiment. In addition, the connection relationship between the modules in the apparatus embodiment provided by the present application indicates that there is a communication connection between them, which can be implemented as one or more communication buses or signal lines.

[0135] Those skilled in the art can clearly understand that the application can be implemented by means of software plus necessary universal hardware, and of course can also be implemented by means of dedicated hardware including special integrated circuit, special CLU, special memory, special component, etc. Generally, any function completed by computer program can be easily implemented by corresponding hardware, and the specific hardware structure for implementing the same function can also be various, such as analog circuit, digital circuit or special circuit, etc. However, for the application, the software program implementation is a better embodiment. Based on such understanding, the technical solution of the application or the part of the application which makes contribution to the prior art can be embodied in the form of software product, which is stored in readable storage medium, such as floppy disk, U disk, mobile hard disk, ROM, RAM, magnetic disk or optical disk, etc., and includes a plurality of instructions for making a computer device (which can be personal computer, server or network device, etc.) execute the method described in various embodiments of the application.

[0136] In the above embodiments, the implementation can be achieved by software, hardware, firmware or any combination thereof, entirely or partially. When implemented by software, the implementation can be in the form of computer program product entirely or partially.

[0137] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on the computer, the flow or function described in the embodiments of the application is entirely or partially generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another, for example, the computer instructions can be transferred from one website, computer, server or data center to another website, computer, server or data center through wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) mode. The computer-readable storage medium can be any available medium that can be stored by the computer or a data storage device such as server, data center, etc. integrated with one or more available media. The available medium can be magnetic medium (such as floppy disk, hard disk, magnetic tape), optical medium (such as DVD) or semiconductor medium (such as solid state disk (SSD)) etc.

Claims

1. An image processing method, characterized by, The method comprises: obtaining first feature information of at least one first frame in a first neural network calculation process; performing image processing on a second frame by using a second neural network, wherein the first feature information is used as input information in a calculation process of the second neural network, the first neural network and the second neural network are neural networks used for image processing, the first neural network and the second neural network have the same function, the second neural network is obtained by performing pruning processing on one or more neural network layers in the first neural network, and the first frame and the second frame are related.

2. The method of claim 1, wherein: the first neural network comprises a first number of neural network layers, and the second neural network comprises a second number of neural network layers, and the second number is less than the first number.

3. The method of claim 1 or 2, wherein: the first feature information comprises feature information of each first frame, and the feature information of each first frame is generated in the first neural network calculation process.

4. The method according to claim 1 or 2, characterized in that, the first feature information is obtained by merging feature information of each first frame, wherein the feature information of each first frame is generated in the first neural network calculation process, and the merging comprises one or more of splicing, adding, fusing, and multiplying.

5. The method of claim 3 or 4, wherein: the feature information of each first frame is a feature map generated in the first neural network calculation process.

6. The method of any one of claims 3 to 5, wherein: the feature information of each first frame is calculated and output by an i-th layer of the first neural network, the i-th layer is any layer in the first neural network, and the second neural network is obtained by performing pruning processing on the first layer to the i-th layer in the first neural network.

7. The method according to any one of claims 1 to 6, characterized in that, the image processing on the second frame by using the second neural network comprises: generating second feature information corresponding to the second frame by using the second neural network, obtaining second feature information generated in the image processing on the second frame by using the second neural network; combining the first feature information and the second feature information to obtain combined feature information; and processing the combined feature information by using the second neural network.

8. The method according to any one of claims 1 to 6, characterized in that, the first frame and the second frame are consecutive video frames in the same video.

9. The method of claim 8, wherein, the first frame is a single frame in the video, and the second frame is a double frame in the video; or the first frame is a double frame in the video, and the second frame is a single frame in the video.

10. The method of claim 8, wherein, the second frame and the first frame belong to a group of frames, and the group of frames comprises three consecutive frames; the second frame is the third frame in the three consecutive frames, and the at least one first frame comprises the first frame and the second frame in the three consecutive frames; or the second frame is the third frame in the three consecutive frames, and the at least one first frame comprises the first frame and the second frame in the three consecutive frames. The second frame is the second frame in the three continuous frames, and the at least one first frame comprises the first frame and the third frame in the three continuous frames; or The second frame is the first frame in the three continuous frames, and the at least one first frame comprises the second frame and the third frame in the three continuous frames.

11. An image processing apparatus characterized by comprising: Comprise: The acquisition module is configured to acquire first feature information of the at least one first frame in a first neural network calculation process; The processing module is configured to perform image processing on the second frame by using a second neural network, wherein the first feature information is used as input information in a second neural network calculation process, the first neural network and the second neural network are neural networks for realizing image processing, the first neural network and the second neural network have the same function, the second neural network is obtained by performing pruning processing on one or more neural network layers in the first neural network, and the first frame and the second frame have correlation.

12. The apparatus of claim 11, wherein The first neural network comprises a first number of neural network layers, and the second neural network comprises a second number of neural network layers, and the second number is less than the first number.

13. The apparatus of claim 11 or 12, wherein The first feature information comprises feature information of each first frame, and the feature information of each first frame is feature information generated by each first frame in the first neural network calculation process.

14. The apparatus of claim 11 or 12, wherein, The first feature information is information obtained by merging feature information of each first frame, wherein the feature information of each first frame is feature information generated by each first frame in the first neural network calculation process, and the merging comprises one or more of splicing, adding, fusing, and multiplying.

15. The apparatus of claim 13 or 14, wherein The feature information of each first frame is a feature map generated by each first frame in the first neural network calculation process.

16. The apparatus of any one of claims 13 to 15, wherein The feature information of each first frame is calculated and output by an i-th layer of the first neural network, the i-th layer is any layer of the first neural network, and the second neural network is obtained by performing pruning processing on the first layer to the i-th layer in the first neural network.

17. The apparatus of any one of claims 11 to 16, wherein The processing module is specifically configured to: generate second feature information corresponding to the second frame by using the second neural network, to obtain second feature information generated in the image processing process of the second frame by using the second neural network; combine the first feature information and the second feature information to obtain combined feature information; and perform processing on the combined feature information by using the second neural network.

18. The apparatus of any one of claims 11 to 16, wherein, The first frame and the second frame are continuous video frames in the same video.

19. The apparatus of claim 18, wherein, The first frame is a single frame in the video, and the second frame is a double frame in the video; or The first frame is a single frame in the video, and the second frame is a double frame in the video; or The first frame is an even frame in the video, and the second frame is an odd frame in the video.

20. The apparatus of claim 18, wherein, The second frame and the first frame belong to a group of frames, and the group of frames includes three consecutive frames; The second frame is a third frame in the three consecutive frames, and the first frame includes a first frame and a second frame in the three consecutive frames; or, The second frame is a second frame in the three consecutive frames, and the first frame includes a first frame and a third frame in the three consecutive frames; or, The second frame is a first frame in the three consecutive frames, and the first frame includes a second frame and a third frame in the three consecutive frames.

21. A computer-readable storage medium, characterized in that, A computer program product, which, when running on a computer, causes the computer to perform the method of any one of claims 1 to 10.

22. An execution device, comprising: A processor, the processor and a memory are coupled, and the memory stores program instructions, when the program instructions stored in the memory are executed by the processor, the method of any one of claims 1 to 10 is realized.

Images