Image processing method, device, system and electronic equipment

Abstract

The present application provides an image processing method, device, system and electronic device, which relates to the field of artificial intelligence technology, and in particular to the field of image processing. In this solution, a convolutional neural network is used to integrate an RGB image into a depth map corresponding to the RGB image in stages, and the depth map is reconstructed to obtain a high-resolution depth map, thereby improving image blur, object modeling, gesture recognition accuracy and the number of instructions, as well as the effects in 3D dynamic pictures. Among them, the first stage is to integrate the RGB image into the depth map corresponding to the RGB image, and the second stage is to integrate the edge image of the RGB image into the image obtained in the first stage, thereby obtaining a high-resolution depth map.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence and image processing technology, and in particular to an image processing method, device, system and electronic equipment. Background Art

[0002] At present, more and more electronic devices are able to process images, for example, to perform operations such as image blurring, object modeling, gesture recognition, and image 3D perspective conversion. Electronic devices often rely on the depth information of the image when processing images. In order to obtain the depth information of the image, a depth camera is often configured on the electronic device, for example, a time-of-flights (ToF) depth camera. However, due to the power consumption of the device, the size of the device, and other reasons, the depth cameras on electronic devices generally have low device specifications and low available power, which leads to low resolution of the depth map, inaccurate edges, large gaps, and failure to meet subsequent applications or causing pain points. Therefore, how to improve the resolution and accuracy of the depth map is a technical problem that needs to be solved urgently. Summary of the invention

[0003] The embodiments of the present application provide an image processing method, device, system and electronic device, which reconstructs a low-resolution depth map by integrating the image features of an RGB image into a depth map corresponding to the RGB image in stages to obtain a high-resolution depth map, thereby improving the resolution and accuracy of the depth map.

[0004] In a first aspect, an embodiment of the present application provides an image processing system, the system comprising a processor, a first convolutional neural network, and a second convolutional neural network;

[0005] The processor is used to determine a first image to be processed and a second image related to the first image, and input the first image to a first input channel of a first convolutional neural network, and input the second image to a second input channel of the first convolutional neural network, wherein an object represented by the first image is the same as an object represented by the second image, and a resolution of the first image is lower than or equal to a resolution of the second image;

[0006] a first convolutional neural network, used for performing image fusion on the first image and the second image to obtain a third image, wherein a resolution of the third image is higher than a resolution of the first image and lower than or equal to a resolution of the second image;

[0007] The processor is further configured to perform edge extraction on the second image to obtain a fourth image, and input the third image into a third input channel of the second convolutional neural network, and input the fourth image into a fourth input channel of the second convolutional neural network;

[0008] The second convolutional neural network is used to perform image fusion on the third image and the fourth image to obtain a fifth image, where the resolution of the fifth image is lower than or equal to the resolution of the third image, and lower than or equal to the resolution of the second image.

[0009] Therefore, a convolutional neural network is used to integrate the RGB image into the depth map corresponding to the RGB image in stages, and the depth map is reconstructed to obtain a high-resolution depth map, thereby improving the image blur, object modeling, gesture recognition accuracy and instruction count, and the effects of 3D dynamic pictures. Among them, the first stage is to integrate the RGB image into the depth map corresponding to the RGB image, and the second stage is to integrate the edge image of the RGB image into the image obtained in the first stage, thereby obtaining a high-resolution depth map.

[0010] In a possible implementation, the target convolutional neural network includes a first network, a second network, and a third network, the first network is used to extract features of an image entering a first target input channel to obtain first target image features, the second network is used to extract features of an image entering a second target input channel to obtain second target image features, and the third network is used to fuse the first target image features, the second target image features, and the image entering the first target input channel to obtain a third target image;

[0011] Wherein, the target convolutional neural network is the first convolutional neural network, the first target input channel is the first input channel, the second target input channel is the second input channel, and the third target image is the third image;

[0012] or

[0013] The target convolutional neural network is the second convolutional neural network, the first target input channel is the third input channel, the second target input channel is the fourth input channel, and the third target image is the fifth image.

[0014] In a possible implementation manner, the first network includes at least one first subnetwork, and the first subnetwork includes:

[0015] A first convolutional layer, for extracting image features of an image entering a first target input channel;

[0016] The first activation layer is used to add nonlinear factors to the data output by the first convolutional layer;

[0017] The first fusion layer is used to extract image features from the data output by the first activation layer using multiple first convolution kernels of different sizes configured in parallel, and to integrate the image features extracted by the multiple first convolution kernels using at least one second convolution kernel configured in series.

[0018] In a possible implementation manner, the first network includes a plurality of first sub-networks, and the plurality of first sub-networks are configured in series.

[0019] In a possible implementation manner, the second network includes at least one second sub-network, and the second sub-network includes:

[0020] A second convolutional layer, for extracting image features of the image entering the second target input channel;

[0021] The second activation layer is used to add nonlinear factors to the data output by the first convolutional layer.

[0022] In a possible implementation manner, the second network includes multiple second sub-networks, and the multiple second sub-networks are configured in series.

[0023] In a possible implementation, the first network includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series;

[0024] The input of at least one network among the plurality of third sub-networks is the output of the network in the previous layer of at least one network and the output of any one network among the plurality of fourth sub-networks.

[0025] In a possible implementation manner, the third network includes:

[0026] A second fusion layer is used to extract image features from the data output by the first network and the second network using a plurality of third convolution kernels of different sizes configured in parallel, and to integrate the image features extracted by the plurality of third convolution kernels using at least one second convolution kernel configured in series;

[0027] The first deconvolution layer is used to perform a deconvolution operation on the data output by the second fusion layer;

[0028] The third activation layer is used to add nonlinear factors to the data output by the first deconvolution layer;

[0029] The fourth convolutional layer is used to extract features from the data output by the third activation layer;

[0030] The output layer is used to fuse the data output by the fourth convolutional layer with the image entering the first target input channel to obtain a third target image.

[0031] In a second aspect, an embodiment of the present application provides an image processing method, the method comprising:

[0032] Determine a first image to be processed and a second image related to the first image, wherein a resolution of the first image is lower than a resolution of the second image, wherein an object represented by the first image is the same as an object represented by the second image;

[0033] Inputting the first image to a first input channel of a first convolutional neural network, and inputting the second image to a second input channel of the first convolutional neural network;

[0034] Using a first convolutional neural network to fuse the first image and the second image to obtain a third image, where a resolution of the third image is higher than a resolution of the first image and lower than or equal to a resolution of the second image;

[0035] Performing edge extraction on the second image to obtain a fourth image;

[0036] Inputting the third image to a third input channel of the second convolutional neural network, and inputting the fourth image to a fourth input channel of the second convolutional neural network;

[0037] The third image and the fourth image are fused using a second convolutional neural network to obtain a fifth image, where a resolution of the fifth image is lower than or equal to a resolution of the third image and lower than or equal to a resolution of the second image.

[0038] In a possible implementation, using a target convolutional neural network to perform image fusion on a first target image and a second target image to obtain a third target image includes:

[0039] extracting a first target image feature of an image entering a first target input channel using a first network in a target convolutional neural network;

[0040] extracting second target image features of the image entering the second target input channel using a second network in the target convolutional neural network;

[0041] Using a third network in the target convolutional neural network to fuse the first target image features, the second target image features, and the image entering the first target input channel to obtain a third target image;

[0042] Wherein, the target convolutional neural network is the first convolutional neural network, the first target input channel is the first input channel, the second target input channel is the second input channel, and the third target image is the third image;

[0043] or

[0044] The target convolutional neural network is the second convolutional neural network, the first target input channel is the third input channel, the second target input channel is the fourth input channel, and the third target image is the fifth image.

[0045] In a possible implementation, the first network includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series;

[0046] The method also includes:

[0047] Using at least one network among the plurality of third sub-networks to receive first data output by a network in an upper layer of at least one network, and second data output by any network among the plurality of fourth sub-networks;

[0048] At least one network is used to perform feature extraction based on the first data and the second data.

[0049] In a third aspect, an embodiment of the present application provides an image processing device, the device comprising:

[0050] A communication module, configured to determine a first image to be processed and a second image related to the first image, wherein a resolution of the first image is lower than a resolution of the second image, and an object represented by the first image is the same as an object represented by the second image;

[0051] A processing module, configured to input the first image into a first input channel of the first convolutional neural network, and input the second image into a second input channel of the first convolutional neural network;

[0052] The processing module is further used to use the first convolutional neural network to perform image fusion on the first image and the second image to obtain a third image, where the resolution of the third image is higher than the resolution of the first image and lower than or equal to the resolution of the second image;

[0053] The processing module is further used to perform edge extraction on the second image to obtain a fourth image;

[0054] The processing module is further used to input the third image into a third input channel of the second convolutional neural network, and input the fourth image into a fourth input channel of the second convolutional neural network;

[0055] The third image and the fourth image are fused using a second convolutional neural network to obtain a fifth image, where a resolution of the fifth image is lower than or equal to a resolution of the third image and lower than or equal to a resolution of the second image.

[0056] In a possible implementation, the processing module is further configured to:

[0057] extracting a first target image feature of an image entering a first target input channel using a first network in a target convolutional neural network;

[0058] extracting second target image features of the image entering the second target input channel using a second network in the target convolutional neural network;

[0059] Using a third network in the target convolutional neural network to fuse the first target image features, the second target image features, and the image entering the first target input channel to obtain a third target image;

[0060] Wherein, the target convolutional neural network is the first convolutional neural network, the first target input channel is the first input channel, the second target input channel is the second input channel, and the third target image is the third image;

[0061] or

[0062] The target convolutional neural network is the second convolutional neural network, the first target input channel is the third input channel, the second target input channel is the fourth input channel, and the third target image is the fifth image.

[0063] In a possible implementation, the first network includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series;

[0064] The processing module is also used to:

[0065] Using at least one network among the plurality of third sub-networks to receive first data output by a network in an upper layer of at least one network, and second data output by any network among the plurality of fourth sub-networks;

[0066] At least one network is used to perform feature extraction based on the first data and the second data.

[0067] In a fourth aspect, an embodiment of the present application provides an electronic device, including:

[0068] at least one memory for storing a program;

[0069] At least one processor is used to call the program stored in the memory to execute the method provided in the second aspect.

[0070] In a fifth aspect, an embodiment of the present application provides a computer storage medium, in which instructions are stored. When the instructions are executed on a computer, the computer executes the method provided in the first and second aspects.

[0071] In a sixth aspect, an embodiment of the present application provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the method provided in the second aspect.

[0072] In a seventh aspect, an embodiment of the present application provides a chip, comprising at least one processor and an interface;

[0073] At least one processor obtains program instructions or data through the interface;

[0074] At least one processor is used to execute program line instructions to implement the method provided in the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0075] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0076] Figure 1 This is a schematic diagram of an artificial intelligence main framework provided by an embodiment of the present application;

[0077] Figure 2 It is a schematic diagram of a system architecture provided by an embodiment of the present application;

[0078] Figure 3 It is another system architecture schematic diagram provided by an embodiment of the present application;

[0079] Figure 4 It is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0080] Figure 5 yes Figure 4 The schematic diagram of the structure of the fusion layer in the convolutional neural network shown;

[0081] Figure 6 It is a schematic diagram of the structure of a convolutional neural network provided in an embodiment of the present application;

[0082] Figure 7 is a schematic diagram of an image processing process provided by an embodiment of the present application;

[0083] Figure 8a is an original depth map provided in an embodiment of the present application;

[0084] Figure 8b is an RGB image provided in an embodiment of the present application;

[0085] Figure 8c The embodiment of this application provides Figure 4 The high-resolution depth map obtained after the processing process shown;

[0086] Figure 8d It is a high-resolution depth map obtained by processing the original depth map in the related art;

[0087] Fig. 9 It is a schematic diagram comparing the present solution and other solutions provided in the embodiments of the present application;

[0088] Fig.10 is a structural schematic diagram of an image processing system provided in an embodiment of the present application;

[0089] Fig.11 It is a flowchart of an image processing method provided in an embodiment of the present application;

[0090] Fig.12 is a schematic diagram of a step of determining a third image or a fifth image provided by an embodiment of the present application;

[0091] Fig.13 This is a schematic diagram of a step of pre-fusing image features of each image provided by an embodiment of the present application;

[0092] Fig.14 is a structural schematic diagram of an image processing device provided in an embodiment of the present application;

[0093] Fig.15 It is a schematic diagram of the structure of a chip provided in an embodiment of the present application. DETAILED DESCRIPTION

[0094] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application are described. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0095] Figure 1 A schematic diagram of an artificial intelligence main framework is shown, which describes the overall workflow of the artificial intelligence system and is applicable to general artificial intelligence field requirements.

[0096] The following is an explanation of the above artificial intelligence theme framework from two dimensions: "intelligent information chain" (horizontal axis) and "IT value chain" (vertical axis).

[0097] The "intelligent information chain" reflects a series of processes from data acquisition to processing. For example, it can be a general process of intelligent information perception, intelligent information representation and formation, intelligent reasoning, intelligent decision-making, intelligent execution and output. In this process, data undergoes a condensed process of "data-information-knowledge-wisdom".

[0098] The "IT value chain" reflects the value that artificial intelligence brings to the information technology industry, from the underlying infrastructure of human intelligence, information (providing and processing technology implementation) to the system's industrial ecological process.

[0099] (1) Infrastructure:

[0100] The infrastructure provides computing power support for the artificial intelligence system, enables communication with the outside world, and supports it through the basic platform. It communicates with the outside world through sensors; computing power is provided by smart chips, among which smart chips can be central processing units (CPU), network processors (neural-network processing units, NPU), graphics processing units (graphics processing units, GPU), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) and other hardware acceleration chips; the basic platform includes distributed computing frameworks and networks and other related platform guarantees and support, which can include cloud storage and computing, interconnected networks, etc. For example, sensors communicate with the outside world to obtain data, and these data are provided to the smart chips in the distributed computing system provided by the basic platform for calculation.

[0101] (2) Data

[0102] The data on the upper layer of the infrastructure is used to represent the data source in the field of artificial intelligence. The data involves graphics, images, voice, text, and IoT data of traditional devices, including business data of existing systems and sensor data such as force, displacement, liquid level, temperature, and humidity. In this solution, the data can be images, such as low-resolution depth maps and high-resolution RGB images.

[0103] (3) Data processing

[0104] Data processing usually includes data training, machine learning, deep learning, search, reasoning, decision-making and other methods.

[0105] Among them, machine learning and deep learning can symbolize and formalize data for intelligent information modeling, extraction, preprocessing, and training.

[0106] Reasoning refers to the process of simulating human intelligent reasoning in computers or intelligent systems, using formalized information to perform machine thinking and solve problems based on reasoning control strategies. Typical functions are search and matching.

[0107] Decision-making refers to the process of making decisions after intelligent information is reasoned, usually providing functions such as classification, sorting, and prediction.

[0108] In this solution, data processing can be a process of integrating the image features of the RGB image into the depth map in stages, reconstructing the low-resolution depth map, and obtaining a high-resolution depth map.

[0109] (4) General ability

[0110] After the data is processed as mentioned above, some general capabilities can be formed based on the results of the data processing, such as an algorithm or a general system, for example, translation, text analysis, computer vision processing, speech recognition, image recognition, etc. In this solution, the general capabilities can be computer vision processing, image recognition, etc.

[0111] (5) Smart products and industry applications

[0112] Smart products and industry applications refer to the products and applications of artificial intelligence systems in various fields. They are the encapsulation of the overall artificial intelligence solution, which productizes intelligent information decision-making and realizes practical application. Its application areas mainly include: smart manufacturing, smart transportation, smart home, smart medical care, smart security, autonomous driving, safe cities, smart terminals, etc.

[0113] Next, based on the above-mentioned artificial intelligence main framework, a system architecture in an embodiment of the present application is introduced.

[0114] See attached Figure 2 The embodiment of the present application provides a system architecture 200. The data acquisition device 260 is used to collect low-resolution depth maps, high-resolution RGB images and other data and store them in the database 230. The training device 220 generates a target model / rule 201 based on the low-resolution depth maps, high-resolution RGB images and other data maintained in the database 230.

[0115] The training device 220 can train the neural network based on the data in the database 230 to obtain the target model / rule 201. Exemplarily, the neural network can be a convolutional neural network, and in this case, the target model / rule 201 is a convolutional neural network.

[0116] The target model / rule 201 obtained by the training device 220 can be applied to different systems or devices. Figure 2 In the embodiment, the execution device 210 is configured with an I / O interface 212 for data interaction with external devices, and a “user” can input data into the I / O interface 212 through a client device 240 .

[0117] The execution device 210 can call data, codes, etc. in the data storage system 250 , and can also store data, instructions, etc. in the data storage system 250 .

[0118] The calculation module 211 processes the input data using the target model / rule 201. For example, the image features of the RGB image are integrated into the depth map in stages, and the low-resolution depth map is reconstructed to obtain a high-resolution depth map, thereby improving the image blur, object modeling, gesture recognition accuracy and instruction count, and the effects in 3D dynamic pictures.

[0119] Finally, the I / O interface 212 returns the processing result to the client device 240 and provides it to the user. For example, a high-resolution depth map is provided to the user.

[0120] In the attached Figure 2 In the case shown in , the user can manually specify the data to be input into the execution device 210, for example, by operating in the interface provided by the I / O interface 212. In another case, the client device 240 can automatically input data into the I / O interface 212 and obtain the result. If the client device 240 needs to obtain the user's authorization to automatically input data, the user can set the corresponding authority in the client device 240. The user can view the results output by the execution device 210 on the client device 240, and the specific presentation form can be a specific method such as display, sound, action, etc. The client device 240 can also be used as a data acquisition terminal to store the collected low-resolution depth map, high-resolution RGB image and other data into the database 230.

[0121] It is worth noting that Figure 2 This is only a schematic diagram of a system architecture provided by an embodiment of the present application. The positional relationship between the devices, components, modules, etc. shown in the figure does not constitute any limitation. Figure 2 In the embodiment, the data storage system 250 is an external memory relative to the execution device 210. In other cases, the data storage system 250 may also be placed in the execution device 210.

[0122] Next, based on Figure 2 The system architecture shown introduces another system architecture provided by this solution.

[0123] See attached Figure 3 , an embodiment of the present application provides a system architecture 300. The execution device 210 is implemented by one or more servers, and optionally cooperates with other computing devices, such as data storage, routers, load balancers and other devices; the execution device 210 can be arranged at one physical site, or distributed at multiple physical sites. The execution device 210 can use the data in the data storage system 250, or call the program code in the data storage system 250 to integrate the image features of the RGB image into the depth map in stages, reconstruct the low-resolution depth map, and obtain a high-resolution depth map, thereby improving the image blur, object modeling, gesture recognition accuracy and the number of instructions, and the effects in 3D dynamic pictures.

[0124] Users can operate their respective user devices (e.g., local device 301 and local device 302) to interact with execution device 210. Each local device can represent any computing device, such as a personal computer, a computer workstation, a smart phone, a tablet computer, a smart camera, a smart car or other type of cellular phone, a media consumption device, a wearable device, a set-top box, a game console, etc.

[0125] Each user's local device can interact with the execution device 210 through a communication network of any communication mechanism / communication standard. The communication network can be a wide area network, a local area network, a point-to-point connection, etc., or any combination thereof.

[0126] In another implementation, one or more aspects of the execution device 210 may be implemented by each local device. For example, the local device 301 may provide local data or feedback calculation results to the execution device 210 .

[0127] It should be noted that all functions of the execution device 210 can also be implemented by the local device. For example, the local device 301 implements the functions of the execution device 210 and provides services to its own users, or provides services to the users of the local device 302.

[0128] Next, based on the system architecture described above, the convolutional neural network used in this solution is introduced.

[0129] Convolutional neural network (CNN) is a deep neural network with a convolutional structure. It is a deep learning architecture. A deep learning architecture refers to multiple levels of learning at different levels of abstraction through machine learning algorithms. As a deep learning architecture, CNN is a feed-forward artificial neural network in which each neuron responds to overlapping areas in the image input to it.

[0130] like Figure 4 As shown, a convolutional neural network (CNN) 100 may include an input layer 110 , a convolutional layer / activation layer / fusion layer 120 , wherein the fusion layer is optional, and a neural network layer 130 .

[0131] Convolution layer / activation layer / fusion layer 120:

[0132] like Figure 4As shown, the convolution layer / activation layer / fusion layer 120 may include layers 121-125, where layer 121 is a convolution layer, layer 122 is an activation layer, layer 123 is a convolution layer, layer 124 is an activation layer, and layer 125 is a fusion layer. In this solution, layers 121-122 are used to process RGB images or process edge images extracted from RGB images (i.e., Figure 4 The image to be processed 1) is shown in FIG. 1 , and layers 123-125 are used to process a low-resolution depth map or a high-resolution depth map obtained in the first stage (i.e. Figure 4 The image to be processed 2 is shown. In this solution, the low-resolution depth map can be understood as the original depth map.

[0133] Taking the convolution layer 121 as an example, the convolution layer 121 may include a plurality of convolution operators, which are also called kernels. The convolution operator is equivalent to a filter that extracts specific information from the input image matrix in image processing. The convolution operator can essentially be a weight matrix, which is usually predefined. In the process of performing convolution operations on the image, the weight matrix is ​​usually processed one pixel after another (or two pixels after two pixels... depending on the value of the step length stride) in the horizontal direction on the input image, thereby completing the work of extracting specific features from the image. The size of the weight matrix should be related to the size of the image. It should be noted that the depth dimension of the weight matrix is ​​the same as the depth dimension of the input image. In the process of performing convolution operations, the weight matrix will extend to the entire depth of the input image. Therefore, convolution with a single weight matrix will produce a convolution output with a single depth dimension, but in most cases, a single weight matrix is ​​not used, but multiple weight matrices of the same dimension are applied. The output of each weight matrix is ​​stacked to form the depth dimension of the convolved image. Different weight matrices can be used to extract different features in the image. For example, one weight matrix is ​​used to extract image edge information, another weight matrix is ​​used to extract specific colors of the image, and another weight matrix is ​​used to blur unnecessary noise in the image... The multiple weight matrices have the same dimensions, and the feature maps extracted by the multiple weight matrices with the same dimensions are also of the same dimension. The extracted feature maps with the same dimensions are then merged to form the output of the convolution operation.

[0134] The weight values ​​in these weight matrices need to be obtained through a lot of training in practical applications. The weight matrices formed by the weight values ​​obtained through training can extract information from the input image, thereby helping the convolutional neural network 100 to make correct predictions.

[0135] When the convolutional neural network 100 has multiple convolutional layers, the initial convolutional layer (for example, 121) often extracts more general features, which can also be called low-level features. As the depth of the convolutional neural network 100 increases, the features extracted by the later convolutional layers become more and more complex, such as high-level semantic features. Features with higher semantics are more suitable for the problem to be solved.

[0136] Taking activation layer 122 as an example, activation layer 121 may include an activation function, which can be used to add nonlinear factors to solve the problem of insufficient expression of linear models. For images, convolution is mainly used to process, that is, a weight is assigned to each pixel, and this operation is obviously linear. However, for image samples, they are not necessarily linearly separable. In order to solve this problem, nonlinear factors can be introduced to solve problems that linear models cannot solve. Exemplarily, the activation function can use the ReLU function.

[0137] The fusion layer 125 may include a plurality of convolution kernels of different sizes, wherein the layer is similar to the convolution layer, but the structure of the layer is different from that of the convolution layer. Figure 5 As shown, the fusion layer 125 includes convolution kernels 11-1n and convolution kernel 21, wherein convolution kernels 11-1n are convolution kernels of multiple different sizes, for example, the size of convolution kernel 11 is 1x1, the size of convolution kernel 12 is 3x3, the size of convolution kernel 13 is 5x5; the size of convolution kernel 21 is 1x1. In this solution, convolution kernels 11-1n are mainly used to extract features from the input data respectively, so as to obtain feature representations of different scales; convolution kernel 21 is mainly used to integrate the features extracted by convolution kernels 11-1n, and then input them to the next layer. Exemplarily, the process performed by the fusion layer 125 can be expressed as: out =F′ 1×1 (F 1×1 (F in )+F 3×3 (F in )+F 5×5 (F in )); where F 1×1 (·),F 3×3 (·),F 5×5 (·) represents the context information under different receptive fields obtained by three different convolution kernels in the fusion layer, and F′ 1×1 (·) for information integration, F in Represents the input features of the fusion layer, F outRepresents the output features of the fusion layer. It is understandable that if the same size of convolution kernel is used for feature extraction, the extracted features will be fixed, that is, the context information obtained under the receptive field is consistent, which makes it difficult to find the optimal context information under the receptive field; while in this solution, multiple convolution kernels of different sizes are used for feature extraction, so that the extracted features are variable, that is, the context information obtained under the receptive field is inconsistent, so that the optimal context information under the receptive field can be found from the context information under different receptive fields, thereby improving the accuracy of image processing and the resolution of the reconstructed depth map. In addition, by setting the fusion layer 125 to multiple convolution kernels of different sizes, the number of convolution layers is also reduced, thereby reducing the number of convolution operations and reducing system power consumption.

[0138] Neural network layer 130:

[0139] After being processed by the convolution layer / activation layer / fusion layer 120, the convolution neural network 100 is not sufficient to output the required output information. As mentioned above, the convolution layer / activation layer / fusion layer 120 only extracts features and reduces the parameters brought by the input image. However, in order to generate the final output information (the required class information or other related information), the convolution neural network 100 needs to use the neural network layer 130 to generate one or a group of outputs of the required number of classes. Therefore, the neural network layer 130 may include multiple hidden layers (such as Figure 4 131, 132 to 137) and output layer 140 shown, the parameters contained in the multi-layer hidden layer can be pre-trained according to the relevant training data of the specific task type, for example, the task type can include image super-resolution reconstruction. It can be understood that the number of hidden layers can be adaptively increased or decreased, and is not limited here. In this scheme, hidden layers 131 and 134 can be the fusion layer 125 described above. Exemplarily, after receiving data related to the RGB image and data related to the original depth map, the hidden layer 131 can integrate the received data and input it to the next layer. Hidden layers 132 and 135 can be deconvolution layers. Among them, the forward propagation process of the deconvolution layer is the back propagation process of the convolution layer. Hidden layers 133 and 136 can be the activation layers described above. Hidden layer 137 can be the convolution layer described above.

[0140] After the multiple hidden layers in the neural network layer 130, that is, the last layer of the entire convolutional neural network 100 is the output layer 140. After receiving the data transmitted by the previous layer (such as the hidden layer 137), the output layer 140 can fuse the received data with the data processed by the input layer 112 contained in the input layer 110, and then reconstruct the image 2 to be processed to obtain the depth map required for the first stage or the second stage.

[0141] In one example, the data transmitted by the previous layer (such as the hidden layer 137) received by the output layer 140 can be understood as the residual feature between the image to be processed 1 and the image to be processed 2, and the residual feature can be considered as the feature of the data difference between the two images. Further, the output layer 140 fuses the residual feature with the image to be processed 2, that is, the image to be processed 2 can be reconstructed to obtain a reconstructed high-resolution image.

[0142] It should be noted that if Figure 4 The convolutional neural network 100 shown is only an example of a convolutional neural network. In specific applications, the convolutional neural network can also exist in the form of other network models, for example, Figure 6 As shown, the convolution layer / activation layer / fusion layer 120 may include multiple groups of processing layers consisting of convolution layers 121 and activation layers 122, and multiple groups of processing layers consisting of convolution layers 121, activation layers 122 and fusion layers; wherein the data output by the activation layer 122 may be input to the convolution layer 121 of the next layer, and to the convolution layer 123; the convolution layer 123 may combine the data input by the convolution layer 122 with the data input by the fusion layer 125 for processing, thereby improving the accuracy of subsequent data processing.

[0143] It can be understood that in this solution, a convolutional neural network is used to integrate the RGB image into the depth map corresponding to the RGB image in stages, and the depth map is reconstructed to obtain a high-resolution depth map. Among them, the first stage is to extract features from the low-resolution depth map and the high-resolution RGB image respectively, and then fuse the image features of the two (i.e., reconstruct the depth map) to obtain a high-resolution depth map; the second stage is to first extract the edge image of the high-resolution RGB image, and then extract features from the edge image and the high-resolution depth map reconstructed in the first stage respectively, and then fuse the image features of the two (i.e., reconstruct the depth map) to obtain a high-resolution depth map. The high-resolution depth map obtained in the second stage is the final depth map. In an example, an RGB image can be understood as an MxNx3 color pixel array, where each color pixel is a three-value group, and these three values ​​correspond to the red, green, and blue components of the RGB image at a specific spatial position.

[0144] Next, based on the convolutional neural network described above, combined with Figure 7 The image processing process in this scheme is introduced in stages.

[0145] Phase 1

[0146] (1) Obtain a low-resolution depth map and an RGB image

[0147] Generally, a TOF camera can be used to obtain a low-resolution depth map. The principle of the TOF camera to obtain a depth image is: by emitting continuous near-infrared pulses to the target scene, and then using a sensor to receive the light pulses reflected by the object; then, by comparing the phase difference between the emitted light pulse and the light pulse reflected by the object, the transmission delay between the light pulses can be calculated, and then the distance of the object relative to the transmitter can be obtained, and finally a depth image can be obtained. In one example, a depth map can also be obtained based on structured light. The principle of obtaining a depth image based on structured light is: projecting structured light to the scene, and capturing the corresponding pattern with structured light by the image sensor. Since the pattern of structured light will be deformed due to the shape of the object, the depth information of each point in the scene can be obtained by calculating the position of the pattern image in the captured image and the degree of deformation using the triangulation principle. In one example, a depth map can also be obtained based on a laser radar. The basic principle of the laser radar to obtain a depth map is: emitting lasers into space at a certain time interval, and recording the time interval between the signal of each scanning point from the laser radar to the object in the measured scene, and then reflecting back to the laser radar through the object, and the distance between the surface of the object and the laser radar is calculated accordingly. It is understandable that there are many ways to obtain a low-resolution depth map, which can be determined according to actual conditions and are not limited here.

[0148] The RGB image can be obtained by shooting with a camera or other image acquisition device. In one example, the RGB image corresponds to a low-resolution depth map, or it can be said that the RGB image is another form of expression of the corresponding low-resolution depth map, or it can be said that the object represented by the RGB image is the same as the object represented by the depth map.

[0149] (2) Input the depth map and RGB image into the convolutional neural network

[0150] In this solution, before the depth map is input into the convolutional neural network, the original depth map (i.e., the low-resolution depth map) can be processed by the Bicubic upsampling method to obtain a high-resolution depth map. After that, the high-resolution depth map is input into the convolutional neural network together with the RGB image. For example, the high-resolution depth map can be Figure 4 Or the image to be processed 2 in 6, the RGB image can be Figure 4 Or the image to be processed 1 in 6.

[0151] (3) Image feature extraction

[0152] In this scheme, when the convolutional neural network extracts features of the depth map and the RGB image respectively, it can extract the image features of the depth map based on the encoder (i.e., the convolution method), and extract the image features of the RGB image based on the RGB-guided method (i.e., the method of convolution with red, green and yellow as input).

[0153] (4) Progressively fusion the image features of the depth map and the image features of the RGB image

[0154] In this scheme, we can Figure 5 The convolutional neural network shown in Figure 1 progressively fuses the image features of the depth map with the image features of the RGB image. For progressive fusion, continue to refer to Figure 6 ,exist Figure 6 After the second activation layer 122 from bottom to top is processed, the processed data is input into the convolutional layers 121 and 123 respectively; wherein, the data entering the convolutional layer 123 (i.e., the data related to the image 1 to be processed and the data related to the image 2 to be processed) passes through the convolutional layer 123, the activation layer 124 and the fusion layer 125 in sequence, and can be fused with each other. Figure 6 ,exist Figure 6 After the third activation layer 122 from bottom to top is processed, the processed data is input to the convolution layers 121 and 123 respectively; similarly, the data entering the convolution layer 123 (i.e., the data related to the image 1 to be processed and the data related to the image 2 to be processed) can be fused with each other after passing through the convolution layer 123, the activation layer 124 and the fusion layer 125 in sequence. Through this progressive fusion, the image features of the two images can be fused with each other.

[0155] It is understandable that in this step, it is also possible to Figure 4 The convolutional neural network shown fuses the image features of the depth map and the image features of the RGB image, which is not limited here.

[0156] (5) Reconstructing the depth map

[0157] Continue reading Figure 6 After the image to be processed 1 (i.e., RGB image) and the image to be processed 2 (i.e., original depth map) are processed by the convolution layer / activation layer / fusion layer 120 in the convolutional neural network, the processed data can enter the neural network layer 130. After being processed by the neural network layer 130, the depth map can be reconstructed, thereby obtaining a reconstructed high-resolution depth map.

[0158] At this point, the first stage ends and enters the second stage.

[0159] Phase II

[0160] (6) Obtaining the edge image of the RGB image

[0161] In this solution, edge extraction can be performed on the RGB image to obtain an edge image. For example, edge extraction is performed based on gradient, wherein the gradient film operator that can be used can be Roberts operator, Sobel operator, Prewitt operator and Krisch operator.

[0162] (7) Input the edge image and the depth map obtained in the first stage into the convolutional neural network

[0163] In this solution, the edge image and the depth map obtained in the first stage can be input into the convolutional neural network together. For example, the edge image can be Figure 4 Or the image to be processed 1 in 6, the depth map obtained in the first stage can be Figure 4 Or the image to be processed 2 in 6.

[0164] (8) Image feature extraction

[0165] In this scheme, when the convolutional neural network extracts features of the edge image and the depth map obtained in the first stage respectively, it can extract the image features of the depth map obtained in the first stage based on the encoding method (i.e., the convolution method), and extract the image features of the edge image based on the edge-guided method (i.e., the method of convolution with grayscale as input).

[0166] (9) Image features of the progressive edge image and the image features of the depth map obtained in the first stage

[0167] In this scheme, we can Figure 6 The convolutional neural network shown in FIG. 1 progressively fuses the image features of the edge image and the image features of the depth map obtained in the first stage. For the progressive fusion, please refer to the description in the first stage above, which will not be repeated here.

[0168] (10) Reconstructing the depth map

[0169] Continue reading Figure 6 After the image to be processed 1 (i.e., the edge image) and the image to be processed 2 (i.e., the depth map obtained in the first stage) are processed by the convolution layer / activation layer / fusion layer 120 in the convolutional neural network, the processed data can enter the neural network layer 130. After being processed by the neural network layer 130, the depth map can be reconstructed, thereby obtaining a reconstructed high-resolution depth map.

[0170] The second stage ends here, and the reconstructed high-resolution depth map obtained in the second stage is the high-resolution depth map finally obtained. In one example, the second stage can be understood as a refinement of the image obtained in the first stage.

[0171] It can be understood that in this solution, the structures of the convolutional neural network used in the first stage and the second stage can be the same or different, which can be determined according to the actual situation and is not limited here.

[0172] Next, based on the image processing process described above, the depth map obtained in this solution is compared with the depth maps obtained in other solutions.

[0173] like Figure 8a As shown in , this figure is the original depth map (i.e., the low-resolution depth map). Figure 8b As shown in the figure, Figure 8a The RGB image corresponding to the depth map in . Figure 8c As shown in FIG. , this figure is a high-resolution depth map obtained by the image processing method in this solution. Figure 8d As shown in FIG. , this figure is a high-resolution depth map obtained by image processing in other schemes. Figure 8a and 8c By comparison, it can be seen that the resolution of the depth map obtained in this solution is significantly higher than that of the original depth map. Figure 8c and 8d By comparison, it can be seen that the resolution of the depth map obtained in this scheme is significantly higher than that obtained in other schemes. In order to better illustrate that the resolution of the depth map obtained in this scheme is significantly higher than that obtained in other schemes, a comparison is made through numbers below. As shown in Figure 8, RMSE in the figure is the abbreviation of Root Mean Squared Error. It can be seen from the figure that under 4 times super resolution, the root mean square error of this scheme is 1.39, while the root mean square error of other schemes is 1.62. This scheme is significantly better than other schemes; under 8 times super resolution, the root mean square error of this scheme is 2.86, while the root mean square error of other schemes is 3.26. This scheme is significantly better than other schemes; under 16 times super resolution, the root mean square error of this scheme is 5.62, while the root mean square error of other schemes is 6.51. This scheme is significantly better than other schemes; under 32 times super resolution, the root mean square error of this scheme is 8.76, while the root mean square error of other schemes is 12.8. This scheme is significantly better than other schemes. In addition, continue to refer to Fig. 9When the same graphics card (i.e., TITAN Xp) runs the image processing process in this scheme and other schemes, the running time of this scheme is 39 milliseconds, while the running time of other schemes is 170 milliseconds. It can be seen that the running time of this scheme is also significantly lower than that of other schemes, that is, in this scheme, the number of convolutions is reduced by setting a fusion layer in the convolutional neural network, thereby significantly reducing the running time of image processing.

[0174] Next, based on the artificial intelligence main framework, system architecture, convolutional neural network, etc. described above, an image processing system provided by an embodiment of the present application is introduced. The system is proposed based on the image processing process described above, and part or all of the steps performed by each component of the system can be found in the description of the image processing process above.

[0175] See also Fig.10 , Fig.10 Schematic diagram of the structure of an image processing system provided in an embodiment of the present application. Fig.10 As shown, the image processing system 1000 includes: a processor 1011, a first convolutional neural network 1012 and a second convolutional neural network 1013.

[0176] The processor 1011 may be used to determine a first image to be processed and a second image related to the first image, and input the first image to a first input channel of the first convolutional neural network 1012, and input the second image to a second input channel of the first convolutional neural network 1012, wherein the object represented by the first image is the same as the object represented by the second image, and the resolution of the first image is lower than the resolution of the second image. In one example, the processor 1011 may be Figure 2 In the execution device 210, the first convolutional neural network 1012 and the second convolutional neural network 1013 can be Figure 2 In one example, the first image may be Figure 7 The low-resolution depth map described above, the second image can be Figure 7 The described RGB image.

[0177] The first convolutional neural network 1012 can be used to fuse the first image and the second image to obtain a third image, where the resolution of the third image is higher than the resolution of the first image and lower than or equal to the resolution of the second image. In one example,

[0178] The first convolutional neural network 1012 can be Figure 4 or Figure 6 In one example, the third image can be Figure 7 The high-resolution depth map obtained in the first stage described.

[0179] The processor 1011 is further configured to perform edge extraction on the second image to obtain a fourth image, and to input the third image into a third input channel of the second convolutional neural network, and to input the fourth image into a fourth input channel of the second convolutional neural network. In one example, the fourth image may be Figure 7 The edge image obtained in the second stage is described.

[0180] The second convolutional neural network 1013 can be used to fuse the third image and the fourth image to obtain a fifth image, wherein the resolution of the fifth image is lower than or equal to the resolution of the third image, and lower than or equal to the resolution of the second image. Figure 4 or Figure 6 In one example, the fifth image may be Figure 7 The high-resolution depth map obtained in the second stage described.

[0181] In one implementation, the target convolutional neural network includes a first network, a second network, and a third network. The first network is used to extract features of an image entering a first target input channel to obtain features of a first target image. The second network is used to extract features of an image entering a second target input channel to obtain features of a second target image. The third network is used to fuse the features of the first target image, the features of the second target image, and the image entering the first target input channel to obtain a third target image. Among them, the target convolutional neural network can be the first convolutional neural network 1012. In this case, the first target input channel is the first input channel, the second target input channel is the second input channel, and the third target image is the third image. In addition, the target convolutional neural network can also be the second convolutional neural network 1013. In this case, the first target input channel is the third input channel, the second target input channel is the fourth input channel, and the third target image is the fifth image. In one example, the first network can be Figure 4 The network consisting of convolutional layer 123, activation layer 124 and fusion layer 125 described in, or Figure 6 The network described in FIG. 1 is composed of multiple convolutional layers 123, multiple activation layers 124 and multiple fusion layers 125. The second network can be Figure 4 The network consisting of convolutional layer 121 and activation layer 122 described in, or Figure 6 The third network may be a network composed of multiple convolutional layers 121 and multiple activation layers 122 as described in Figure 4 Or the neural network 130 described in 6.

[0182] In one implementation, the first network includes at least one first sub-network, and the first sub-network includes: a first convolution layer, which is used to extract image features of an image entering a first target input channel; a first activation layer, which is used to add nonlinear factors to the data output by the first convolution layer; a first fusion layer, which is used to extract image features from the data output by the first activation layer using multiple first convolution kernels of different sizes configured in parallel, and to integrate the image features extracted by multiple first convolution kernels using at least one second convolution kernel configured in series. In one example, the first convolution layer can be Figure 4 Or the convolution layer 123 described in 6, the first activation layer can be Figure 4 or the activation layer 124 described in 6, the first fusion layer can be Figure 4 Or the fusion layer 125 described in 6.

[0183] In one implementation, the first network includes a plurality of first sub-networks, and the plurality of first sub-networks are configured in series. In one example, the first sub-network can be understood as Figure 6 The network consists of a convolutional layer 123, an activation layer 124 and a fusion layer 125.

[0184] In one implementation, the second network includes at least one second sub-network, and the second sub-network includes: a second convolutional layer for extracting image features of an image entering a second target input channel; and a second activation layer for adding nonlinear factors to the data output by the first convolutional layer. In one example, the second convolutional layer can be Figure 4 Or the convolution layer 121 described in 6, the second activation layer can be Figure 4 Or the activation layer 122 described in 6.

[0185] In one implementation, the second network includes a plurality of second sub-networks, and the plurality of second sub-networks are configured in series. In one example, the second sub-network can be understood as Figure 6 The network consists of a convolutional layer 121 and an activation layer 122.

[0186] In one implementation, the first network includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series. The input of at least one of the plurality of third sub-networks is the output of the upper layer network of at least one network, and the output of any one of the plurality of fourth sub-networks. In one example, the third sub-network can be understood as Figure 6 The fourth sub-network consists of a convolutional layer 123, an activation layer 124 and a fusion layer 125. Figure 6In one example, the input of at least one network in the plurality of third sub-networks is the output of the previous network of at least one network, and the output of any network in the plurality of fourth sub-networks, which can be understood as: Figure 6 From bottom to top, the input of the second convolution layer 123 is the input of the second activation layer 122 and the first fusion layer 125.

[0187] In one implementation, the third network includes: a second fusion layer, which is used to extract image features from the data output by the first network and the second network using multiple third convolution kernels of different sizes configured in parallel, and to integrate the image features extracted by multiple third convolution kernels using at least one second convolution kernel configured in series; a first deconvolution layer, which is used to perform deconvolution operations on the data output by the second fusion layer; a third activation layer, which is used to add nonlinear factors to the data output by the first deconvolution layer; a fourth convolution layer, which is used to extract features from the data output by the third activation layer; and an output layer, which is used to fuse the data output by the fourth convolution layer with the image entering the first target input channel to obtain a third target image. In one example, the third network can be Figure 4 or the neural network 130 described in 6. Exemplarily, the second fusion layer can be Figure 6 The hidden layer 131 described in the above, the first deconvolution layer can be Figure 6 The hidden layer 132 described in the above, the third activation layer can be Figure 6 The hidden layer 133 described in the fourth convolutional layer can be Figure 6 The output of the hidden layer 137 described in Figure 6 The output layer 140 described in .

[0188] Therefore, this solution uses a convolutional neural network to integrate the RGB image into the depth map corresponding to the RGB image in stages, reconstruct the depth map, and thus obtain a high-resolution depth map, thereby improving image blur, object modeling, gesture recognition accuracy and instruction count, and the effects of 3D dynamic pictures.

[0189] Next, based on the artificial intelligence main framework, system architecture, convolutional neural network, etc. described above, an image processing method provided by an embodiment of the present application is introduced. It can be understood that this method is another way of expressing part or all of the process in the image processing process described above, and the two are combined. This method is proposed based on the image processing process described above, and some or all of the content of this method can be found in the description of the image processing process above.

[0190] See also Fig.11 , Fig.11is a flowchart of an image processing method provided in an embodiment of the present application. It can be understood that the method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. Fig.11 As shown, the image processing method includes:

[0191] Step S101: determine a first image to be processed and a second image related to the first image, wherein a resolution of the first image is lower than a resolution of the second image, and an object represented by the first image is the same as an object represented by the second image.

[0192] In this solution, two images may be acquired first, and the objects represented by the two images are the same, for example, the two images are images corresponding to the same object. In one example, the first image may be Figure 7 The low-resolution depth map described above, the second image can be Figure 7 The described RGB image.

[0193] Step S102: input the first image to a first input channel of a first convolutional neural network, and input the second image to a second input channel of the first convolutional neural network.

[0194] In this solution, after obtaining the first image and the second image, they can be respectively input into the input channel of the first convolutional neural network, so that the first convolutional neural network processes the first image and the second image respectively. In one example, the first convolutional neural network 1012 can be Figure 4 or Figure 6 The convolutional neural network described.

[0195] Step S103: Use the first convolutional neural network to perform image fusion on the first image and the second image to obtain a third image, where the resolution of the third image is higher than the resolution of the first image and lower than or equal to the resolution of the second image.

[0196] In this solution, the first convolutional neural network can be used to extract features from the first image and the second image, and perform image fusion to obtain a third image. In one example, the third image can be Figure 7 The high-resolution depth map obtained in the first stage described.

[0197] Step S104: extract edges of the second image to obtain a fourth image.

[0198] In this solution, the edge of the second image can be extracted based on the gradient edge method to obtain the fourth image. In one example, the fourth image can be Figure 7 The edge image obtained in the second stage is described.

[0199] Step S105: input the third image to the third input channel of the second convolutional neural network, and input the fourth image to the fourth input channel of the second convolutional neural network.

[0200] In this solution, after the third image and the fourth image are acquired, they can be respectively input into the input channel of the second convolutional neural network, so that the second convolutional neural network processes the third image and the fourth image respectively. In one example, the second convolutional neural network 1012 can be Figure 4 or Figure 6 The convolutional neural network described.

[0201] Step S106: Use the second convolutional neural network to perform image fusion on the third image and the fourth image to obtain a fifth image, where the resolution of the fifth image is lower than or equal to the resolution of the third image, and lower than or equal to the resolution of the second image.

[0202] In this solution, the second convolutional neural network can be used to extract features from the third image and the fourth image, and perform image fusion to obtain the fifth image. In one example, the fifth image can be Figure 7 The high-resolution depth map obtained in the second stage described.

[0203] It is understandable that part or all of the description of the method provided in this solution can be referred to the above description, and will not be repeated here.

[0204] Therefore, this solution uses a convolutional neural network to integrate the RGB image into the depth map corresponding to the RGB image in stages, reconstruct the depth map, and thus obtain a high-resolution depth map, thereby improving image blur, object modeling, gesture recognition accuracy and instruction count, and the effects of 3D dynamic pictures.

[0205] In one example, the above Fig.11 In steps S103 and S106 shown in FIG. Fig.12 The image fusion is performed in the manner shown. Fig.12 As shown, using a target convolutional neural network to fuse the first target image and the second target image to obtain a third target image includes the following steps:

[0206] Step S201, using the first network in the target convolutional neural network to extract the first target image features of the image entering the first target input channel. Wherein, when the target convolutional neural network is the first convolutional neural network, the first target input channel is the first input channel. When the target convolutional neural network is the second convolutional neural network, the first target input channel is the third input channel.

[0207] Step S202: Use the second network in the target convolutional neural network to extract the second target image features of the image entering the second target input channel. When the target convolutional neural network is the first convolutional neural network, the second target input channel is the second input channel. When the target convolutional neural network is the second convolutional neural network, the second target input channel is the fourth input channel.

[0208] Step S203, using the third network in the target convolutional neural network to fuse the first target image features, the second target image features, and the image entering the first target input channel to obtain a third target image. Wherein, when the target convolutional neural network is the first convolutional neural network, the third target image is the third image. When the target convolutional neural network is the second convolutional neural network, the third target image is the fifth image.

[0209] Therefore, the image features of each image are extracted in the convolutional neural network respectively, and the extracted image features are fused with the image to be processed input into the convolutional neural network to obtain a high-resolution depth map.

[0210] In one example, if Fig.11 The first convolutional neural network and the second convolutional neural network described in are Figure 6 The convolutional neural network shown in FIG. 1 includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series, wherein the third sub-network can be understood as Figure 6 The fourth sub-network consists of a convolutional layer 123, an activation layer 124 and a fusion layer 125. Figure 6 The network consists of a convolutional layer 121 and an activation layer 122; Fig.13 As shown, the image processing may further include the following steps:

[0211] Step S301: Use at least one network among the plurality of third subnetworks to receive first data output by an upper layer network of at least one network and second data output by any one network among the plurality of fourth subnetworks.

[0212] Step S302: Use at least one network to perform feature extraction based on the first data and the second data.

[0213] Therefore, the image features of different images are fused in advance in the image feature extraction stage to improve the fusion effect of the later images.

[0214] Based on the method in the above embodiment, the present application embodiment provides an image processing device. Fig.14 , Fig.14 Schematic diagram of the structure of an image processing device provided in an embodiment of the present application. Fig.14 As shown, the image processing device 1400 may include: a communication module 1401 and a processing module 1402 .

[0215] The communication module 1401 may determine a first image to be processed and a second image related to the first image, wherein the resolution of the first image is lower than that of the second image, and wherein the object represented by the first image is the same as the object represented by the second image.

[0216] The processing module 1402 may input the first image into a first input channel of the first convolutional neural network, and input the second image into a second input channel of the first convolutional neural network.

[0217] The processing module 1402 can also be used to use the first convolutional neural network to perform image fusion on the first image and the second image to obtain a third image, where the resolution of the third image is higher than the resolution of the first image and lower than or equal to the resolution of the second image.

[0218] The processing module 1402 may also perform edge extraction on the second image to obtain a fourth image.

[0219] The processing module 1402 may also input the third image into a third input channel of the second convolutional neural network, and input the fourth image into a fourth input channel of the second convolutional neural network.

[0220] The processing module 1402 can also use the second convolutional neural network to perform image fusion on the third image and the fourth image to obtain a fifth image, where the resolution of the fifth image is lower than or equal to the resolution of the third image, and lower than or equal to the resolution of the second image.

[0221] In one example, the communication module 1401 may be Figure 2 The data acquisition device 260 described above, the processing module 1402 may be Figure 2 The execution device 210 described. The first convolutional neural network and the second convolutional neural network can be Figure 2 The target model / rules 201 are described.

[0222] In one implementation, the processing module 1402 can also use the first network in the target convolutional neural network to extract the first target image features of the image entering the first target input channel; use the second network in the target convolutional neural network to extract the second target image features of the image entering the second target input channel; use the third network in the target convolutional neural network to fuse the first target image features, the second target image features, and the image entering the first target input channel to obtain a third target image.

[0223] Among them, the target convolutional neural network is the first convolutional neural network, the first target input channel is the first input channel, the second target input channel is the second input channel, and the third target image is the third image; or the target convolutional neural network is the second convolutional neural network, the first target input channel is the third input channel, the second target input channel is the fourth input channel, and the third target image is the fifth image.

[0224] In one implementation, the first network includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series;

[0225] The processing module 1402 can also use at least one network among the multiple third sub-networks to receive the first data output by the upper layer network of at least one network, and the second data output by any network among the multiple fourth sub-networks; and use at least one network to perform feature extraction based on the first data and the second data.

[0226] It should be understood that the above-mentioned device is used to execute the method in the above-mentioned embodiment. The implementation principle and technical effect of the corresponding program module in the device are similar to those described in the above-mentioned method. The working process of the device can refer to the corresponding process in the above-mentioned method, which will not be repeated here.

[0227] Based on the method in the above embodiment, the present application embodiment also provides a chip. Fig.15 , Fig.15 A schematic diagram of the structure of a chip provided in an embodiment of the present application. Figure 4 and Figure 6 The convolutional neural network based algorithm shown can be found in Fig.15 The neural network processor NPU 150 is mounted on the host CPU as a coprocessor, and the host CPU assigns tasks. The core part of the NPU is the operation circuit 150, which controls the operation circuit 1503 through the controller 1504 to extract the matrix data in the memory and perform multiplication operations.

[0228] In some implementations, the operation circuit 1503 includes multiple processing units (Process Engine, PE) inside. In some implementations, the operation circuit 1503 is a two-dimensional systolic array. The operation circuit 1503 can also be a one-dimensional systolic array or other electronic circuits capable of performing mathematical operations such as multiplication and addition. In some implementations, the operation circuit 1503 is a general-purpose matrix processor.

[0229] For example, suppose there is an input matrix A, a weight matrix B, and an output matrix C. The operation circuit takes the corresponding data of matrix B from the weight memory 1502 and caches it on each PE in the operation circuit. The operation circuit takes the matrix A data from the input memory 1501 and performs matrix operation with matrix B, and the partial result or final result of the matrix is ​​stored in the accumulator 1508.

[0230] The unified memory 1506 is used to store input data and output data. The weight data is directly transferred to the weight memory 1502 through the storage unit access controller 1505 Direct Memory Access Controller, DMAC. The input data is also transferred to the unified memory 1506 through the DMAC.

[0231] BIU stands for Bus Interface Unit, i.e., bus interface unit 1510, which is used for the interaction between A15I bus and DMAC and instruction fetch buffer 1509.

[0232] The bus interface unit 1510 (Bus Interface Unit, BIU for short) is used for the instruction fetch memory 1509 to obtain instructions from the external memory, and is also used for the storage unit access controller 1505 to obtain the original data of the input matrix A or the weight matrix B from the external memory.

[0233] DMAC is mainly used to transfer input data in the external memory DDR to the unified memory 1506 or to transfer weight data to the weight memory 1502 or to transfer input data to the input memory 1501.

[0234] The vector calculation unit 1507 may include multiple operation processing units, and further process the output of the operation circuit, such as vector multiplication, vector addition, exponential operation, logarithmic operation, size comparison, etc., if necessary. In some implementations, the vector calculation unit 1507 can store the processed output vector to the unified buffer 1506. For example, the vector calculation unit 1507 can apply a nonlinear function to the output of the operation circuit 1503, such as a vector of accumulated values, to generate an activation value. In some implementations, the vector calculation unit 1507 generates a normalized value, a merged value, or both. In some implementations, the processed output vector can be used as an activation input to the operation circuit 1503, for example, for use in a subsequent layer in a neural network.

[0235] An instruction fetch buffer 1509 connected to the controller 1504 is used to store instructions used by the controller 1504;

[0236] Unified memory 1506, input memory 1501, weight memory 1502 and instruction fetch memory 1509 are all on-chip memories. External memories are private to the NPU hardware architecture.

[0237] in, Figure 4 and Figure 6 The operations of each layer in the convolutional neural network shown can be performed by the matrix calculation unit 212 or the vector calculation unit 1507.

[0238] It is to be understood that, in the description of the embodiments of the present application, words such as "exemplary", "for example" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary", "for example" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary", "for example" or "for example" is intended to present related concepts in a concrete manner.

[0239] In the description of the embodiments of the present application, the term "and / or" is merely a description of the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B may represent: A exists alone, B exists alone, and A and B exist at the same time. In addition, unless otherwise specified, the term "multiple" means two or more. For example, multiple systems refers to two or more systems, multiple terminals refers to two or more terminals, and multiple video streams refers to two or more video streams.

[0240] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "include", "comprises", "has" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0241] It is understood that the processor in the embodiments of the present application may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. The general-purpose processor may be a microprocessor or any conventional processor.

[0242] The method steps in the embodiments of the present application can be implemented by hardware or by a processor executing software instructions. The software instructions can be composed of corresponding software modules, and the software modules can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, mobile hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor so that the processor can read information from the storage medium and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can be located in an ASIC.

[0243] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid state disk (SSD)), etc.

[0244] It should be understood that the various numerical numbers involved in the embodiments of the present application are only used for the convenience of description and are not used to limit the scope of the embodiments of the present application.

Claims

1. An image processing system, characterized in that: The system includes a processor, a first convolutional neural network, and a second convolutional neural network; The processor is used to determine a first image to be processed and a second image related to the first image, and input the first image into a first input channel of the first convolutional neural network, and input the second image into a second input channel of the first convolutional neural network, wherein the object represented by the first image is the same as the object represented by the second image, and the resolution of the first image is lower than the resolution of the second image; The first convolutional neural network is used to perform image fusion on the first image and the second image to obtain a third image, where a resolution of the third image is higher than a resolution of the first image and lower than or equal to a resolution of the second image; The processor is further configured to perform edge extraction on the second image to obtain a fourth image, and input the third image into a third input channel of the second convolutional neural network, and input the fourth image into a fourth input channel of the second convolutional neural network; The second convolutional neural network is used to perform image fusion on the third image and the fourth image to obtain a fifth image, wherein the resolution of the fifth image is lower than or equal to the resolution of the third image, and lower than or equal to the resolution of the second image.

2. The system according to claim 1, characterized in that The target convolutional neural network includes a first network, a second network, and a third network, wherein the first network is used to extract features of an image entering a first target input channel to obtain features of a first target image, the second network is used to extract features of an image entering a second target input channel to obtain features of a second target image, and the third network is used to fuse the features of the first target image, the features of the second target image, and the image entering the first target input channel to obtain a third target image; Wherein, the target convolutional neural network is the first convolutional neural network, the first target input channel is the first input channel, the second target input channel is the second input channel, and the third target image is the third image; or The target convolutional neural network is the second convolutional neural network, the first target input channel is the third input channel, the second target input channel is the fourth input channel, and the third target image is the fifth image.

3. The system according to claim 2, characterized in that The first network includes at least one first sub-network, and the first sub-network includes: A first convolutional layer, used to extract image features of the image entering the first target input channel; A first activation layer, used to add nonlinear factors to the data output by the first convolutional layer; The first fusion layer is used to extract image features from the data output by the first activation layer using multiple first convolution kernels of different sizes configured in parallel, and to integrate the image features extracted by the multiple first convolution kernels using at least one second convolution kernel configured in series.

4. The system according to claim 3, characterized in that The first network includes a plurality of the first sub-networks, and the plurality of the first sub-networks are configured in series.

5. The system according to any one of claims 2 to 4, characterized in that: The second network includes at least one second sub-network, and the second sub-network includes: A second convolutional layer, used to extract image features of the image entering the second target input channel; The second activation layer is used to add nonlinear factors to the data output by the first convolutional layer.

6. The system according to claim 5, characterized in that The second network includes a plurality of the second sub-networks, and the plurality of the second sub-networks are configured in series.

7. The system according to any one of claims 2-4 and 6, characterized in that: The first network includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series; The input of at least one network among the plurality of the third sub-networks is the output of the network in the previous layer of the at least one network and the output of any one network among the plurality of the fourth sub-networks.

8. The system according to any one of claims 2-4 and 6, characterized in that: The third network includes: a second fusion layer, configured to extract image features from the data output by the first network and the second network using a plurality of third convolution kernels of different sizes configured in parallel, and to integrate the image features extracted by the plurality of the third convolution kernels using at least one second convolution kernel configured in series; A first deconvolution layer, used to perform a deconvolution operation on the data output by the second fusion layer; A third activation layer, used for adding nonlinear factors to the data output by the first deconvolution layer; A fourth convolutional layer, used for extracting features from the data output by the third activation layer; The output layer is used to fuse the data output by the fourth convolutional layer with the image entering the first target input channel to obtain the third target image.

9. An image processing method, characterized in that: The method comprises: Determine a first image to be processed and a second image related to the first image, wherein a resolution of the first image is lower than a resolution of the second image, and an object represented by the first image is the same as an object represented by the second image; Inputting the first image to a first input channel of a first convolutional neural network, and inputting the second image to a second input channel of the first convolutional neural network; Using the first convolutional neural network to perform image fusion on the first image and the second image to obtain a third image, where a resolution of the third image is higher than a resolution of the first image and lower than or equal to a resolution of the second image; performing edge extraction on the second image to obtain a fourth image; Inputting the third image to a third input channel of a second convolutional neural network, and inputting the fourth image to a fourth input channel of the second convolutional neural network; The third image and the fourth image are fused using the second convolutional neural network to obtain a fifth image, wherein a resolution of the fifth image is lower than or equal to a resolution of the third image, and lower than or equal to a resolution of the second image.

10. The method according to claim 9, characterized in that Using a target convolutional neural network to fuse the first target image and the second target image to obtain a third target image, including: Extracting a first target image feature of an image entering a first target input channel using a first network in the target convolutional neural network; Extracting second target image features of the image entering the second target input channel using a second network in the target convolutional neural network; Using a third network in the target convolutional neural network to fuse the first target image features, the second target image features, and the image entering the first target input channel to obtain the third target image; Wherein, the target convolutional neural network is the first convolutional neural network, the first target input channel is the first input channel, the second target input channel is the second input channel, and the third target image is the third image; or The target convolutional neural network is the second convolutional neural network, the first target input channel is the third input channel, the second target input channel is the fourth input channel, and the third target image is the fifth image.

11. The method according to claim 10, characterized in that The first network includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series; The method further comprises: Using at least one network among the plurality of the third sub-networks to receive first data output by a network in an upper layer of the at least one network, and second data output by any one network among the plurality of the fourth sub-networks; Feature extraction is performed based on the first data and the second data using the at least one network.

12. An image processing device, characterized in that: The device comprises: a communication module, configured to determine a first image to be processed and a second image related to the first image, wherein a resolution of the first image is lower than a resolution of the second image, and an object represented by the first image is the same as an object represented by the second image; A processing module, configured to input the first image into a first input channel of a first convolutional neural network, and input the second image into a second input channel of the first convolutional neural network; The processing module is further configured to use the first convolutional neural network to perform image fusion on the first image and the second image to obtain a third image, wherein a resolution of the third image is higher than a resolution of the first image and lower than or equal to a resolution of the second image; The processing module is further used to perform edge extraction on the second image to obtain a fourth image; The processing module is further configured to input the third image into a third input channel of the second convolutional neural network, and input the fourth image into a fourth input channel of the second convolutional neural network; The third image and the fourth image are fused using the second convolutional neural network to obtain a fifth image, wherein a resolution of the fifth image is lower than or equal to a resolution of the third image, and lower than or equal to a resolution of the second image.

13. The device according to claim 12, characterized in that The processing module is further used for: extracting a first target image feature of an image entering a first target input channel using a first network in a target convolutional neural network; Extracting second target image features of the image entering the second target input channel using a second network in the target convolutional neural network; Using a third network in the target convolutional neural network, the first target image feature, the second target image feature, and the image entering the first target input channel are fused to obtain a third target image; Wherein, the target convolutional neural network is the first convolutional neural network, the first target input channel is the first input channel, the second target input channel is the second input channel, and the third target image is the third image; or The target convolutional neural network is the second convolutional neural network, the first target input channel is the third input channel, the second target input channel is the fourth input channel, and the third target image is the fifth image.

14. The device according to claim 13, characterized in that The first network includes a plurality of third sub-networks configured in series, and the second network includes a plurality of fourth sub-networks configured in series; The processing module is further used for: Using at least one network among the plurality of the third sub-networks to receive first data output by a network in an upper layer of the at least one network, and second data output by any one network among the plurality of the fourth sub-networks; Feature extraction is performed based on the first data and the second data using the at least one network.

15. An electronic device, characterized in that: include: at least one memory for storing a program; At least one processor, configured to call the program stored in the memory to execute the method according to any one of claims 9 to 11.

16. A computer storage medium, wherein instructions are stored in the computer storage medium, and when the instructions are executed on a computer, the computer executes the method according to any one of claims 9 to 11.

17. A computer program product comprising instructions, which, when executed on a computer, causes the computer to execute the method according to any one of claims 9 to 11.

18. A chip, characterized in that: comprising at least one processor and an interface; The at least one processor obtains program instructions or data through the interface; The at least one processor is used to execute the program line instructions to implement the method according to any one of claims 9-11.

Images