Image processing method and device, electronic device and storage medium

Abstract

The present disclosure relates to an image processing method and device, an electronic device and a storage medium, the method comprising: extracting features from a color image and a corresponding depth image respectively, obtaining an N-level first feature map of the color image and an N-level second feature map of the depth image, wherein the scales of the first feature maps of each level in the N-level first feature map are different, and the scales of the second feature maps of each level in the N-level second feature map are different, N is an integer and N≥2; fusing the n-th level first feature map of the N-level first feature map with the n-th level second feature map of the N-level second feature map to obtain the n-th level third feature map of the N-level third feature map, n is an integer and 1≤n≤N; fusing the N-level third feature map, and obtaining a fused feature map according to the fused processing result; determining the processing results of the color image and the corresponding depth image according to the fused feature map. The embodiments of the present disclosure can improve the accuracy of image processing.

Description

Technical Field

[0001] The present disclosure relates to the field of computer technology, and in particular to an image processing method and device, an electronic device, and a storage medium. Background Art

[0002] In the field of computer vision, it is usually necessary to process images, such as extracting feature maps of images, analyzing object information in the image scene based on the feature maps, and then obtaining image processing results.

[0003] In the related art, in the scheme of using color images and corresponding depth images (such as RGB-D images) for semantic segmentation and instance segmentation, feature maps extracted by deeper networks are adopted. As the semantic level increases, the resolution decreases, which may cause excessive loss of image details and affect the accuracy of the segmented image. Summary of the invention

[0004] The present disclosure proposes an image processing technology solution.

[0005] According to one aspect of the present disclosure, an image processing method is provided, including: performing feature extraction on a color image and a corresponding depth image respectively to obtain an N-level first feature map of the color image and an N-level second feature map of the depth image, wherein the scales of the first feature maps of each level in the N-level first feature map are different, and the scales of the second feature maps of each level in the N-level second feature map are different, where N is an integer and N≥2; fusing the n-th level first feature map of the N-level first feature map with the n-th level second feature map of the N-level second feature map to obtain an n-th level third feature map of the N-level third feature map, where n is an integer and 1≤n≤N; performing fusion processing on the N-level third feature map, and obtaining a fused feature map according to the fusion processing result; and determining the processing results of the color image and the corresponding depth image according to the fused feature map.

[0006] In a possible implementation, the fusing process of the N-level third feature maps and obtaining a fused feature map according to the fusion process result includes: taking the first-level third feature map as the first-level first-fourth feature map; fusing the k-th-level third feature map to obtain the k-th-level first-fourth feature map, where k is an integer and 1<k≤N; fusing the first fourth feature maps of each level and obtaining a fused feature map according to the fusion process result.

[0007] In a possible implementation, the fusing process of the k-th level third feature map to obtain the k-th level first fourth feature map includes: scaling down the k-1-th level first fourth feature map to obtain the k-th level first fifth feature map; and fusing the k-th level third feature map with the k-th level first fifth feature map to obtain the k-th level first fourth feature map.

[0008] In a possible implementation, the fusion processing of the first fourth feature maps at each level includes: performing the (m - 1)-th fusion processing on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level, where m is an integer and 1 < m ≤ M - n + 2, and M is the number of times of performing the fusion processing on the first fourth feature map at the first level.

[0009] In a possible implementation, in response to the case of n = 1, the performing the (m - 1)-th fusion processing on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level includes: performing scale-up on the (m - 1)-th fourth feature map at the (n + 1)-th level to obtain the (m - 1)-th fifth feature map at the n-th level; performing fusion processing on the first (m - 1) fourth feature maps at the n-th level and the (m - 1)-th fifth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level.

[0010] In a possible implementation, in response to the case of 1 < n < N, the performing the (m - 1)-th fusion processing on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level includes: performing scale-down on the m-th fourth feature map at the (n - 1)-th level to obtain the m-th sixth feature map at the n-th level; performing scale-up on the (m - 1)-th fourth feature map at the (n + 1)-th level to obtain the (m - 1)-th fifth feature map at the n-th level; performing fusion processing on the first (m - 1) fourth feature maps at the n-th level, the (m - 1)-th fifth feature map at the n-th level, and the m-th sixth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level.

[0011] In a possible implementation, in response to the case of n = N, the performing the (m - 1)-th fusion processing on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level includes: performing scale-down on the m-th fourth feature map at the (n - 1)-th level to obtain the m-th sixth feature map at the n-th level; performing fusion processing on the first (m - 1) fourth feature maps at the n-th level and the m-th sixth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level.

[0012] In a possible implementation, the obtaining the fused feature map according to the fusion processing result includes: using the (M + 1)-th fourth feature map at the first level as the fused feature map.

[0013] In a possible implementation, the determining the processing results of the color image and the corresponding depth image according to the fused feature map includes: performing scale-up on the fused feature map to obtain a segmentation feature map, where the height and width of the segmentation feature map are the same as those of the color image; performing segmentation on the color image and the corresponding depth image according to the segmentation feature map to obtain the processing results of the color image and the corresponding depth image.

[0014] According to one aspect of the present disclosure, there is provided an image processing apparatus, including: a feature extraction module configured to respectively perform feature extraction on a color image and a corresponding depth image to obtain N-level first feature maps of the color image and N-level second feature maps of the depth image, where scales of the first feature maps at each level in the N-level first feature maps are different, scales of the second feature maps at each level in the N-level second feature maps are different, N is an integer and N≥2; a first fusion module configured to fuse the n-th level first feature map of the N-level first feature maps with the n-th level second feature map of the N-level second feature maps to obtain the n-th level third feature map of the N-level third feature maps, where n is an integer and 1≤n≤N; a second fusion module configured to perform fusion processing on the N-level third feature maps and obtain a fused feature map according to a result of the fusion processing; and a processing module configured to determine a processing result of the color image and the corresponding depth image according to the fused feature map.

[0015] In a possible implementation, the second fusion module includes: a first processing sub-module configured to use the first level third feature map as the first first fourth feature map at the first level; a first fusion sub-module configured to perform fusion processing on the k-th level third feature maps to obtain the first first fourth feature map at the k-th level, where k is an integer and 1<k≤N; and a second fusion sub-module configured to perform fusion processing on the first fourth feature maps at each level and obtain a fused feature map according to a result of the fusion processing.

[0016] In a possible implementation, the first fusion sub-module includes: a first downscaling unit configured to downscale the first first fourth feature map at the k-1-th level to obtain the first first fifth feature map at the k-th level; and a first fusion unit configured to fuse the k-th level third feature map with the first first fifth feature map at the k-th level to obtain the first first fourth feature map at the k-th level.

[0017] In a possible implementation, the second fusion sub-module includes: a second fusion unit configured to perform the (m-1)-th fusion processing on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level, where m is an integer and 1<m≤M-n+2, and M is the number of times of performing fusion processing on the first first fourth feature map at the first level.

[0018] In a possible implementation, in response to the case where n = 1, the second fusion unit includes: a first upscaling sub-unit configured to upscale the (m-1)-th fourth feature map at the n+1-th level to obtain the (m-1)-th fifth feature map at the n-th level; and a first fusion sub-unit configured to fuse the first (m-1) fourth feature maps at the n-th level with the (m-1)-th fifth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level.

[0019] In a possible implementation, in response to the case where 1 < n < N, the second fusion unit includes: a first downscaling subunit, configured to downscale the m-th fourth feature map of the (n - 1)-th level to obtain the m-th sixth feature map of the n-th level; a second upscaling subunit, configured to upscale the (m - 1)-th fourth feature map of the (n + 1)-th level to obtain the (m - 1)-th fifth feature map of the n-th level; and a second fusion subunit, configured to fuse the first m - 1 fourth feature maps of the n-th level, the (m - 1)-th fifth feature map of the n-th level, and the m-th sixth feature map of the n-th level to obtain the m-th fourth feature map of the n-th level.

[0020] In a possible implementation, in response to the case where n = N, the second fusion sub-module includes: a second downscaling subunit, configured to downscale the m-th fourth feature map of the (n - 1)-th level to obtain the m-th sixth feature map of the n-th level; and a third fusion subunit, configured to fuse the first m - 1 fourth feature maps of the n-th level and the m-th sixth feature map of the n-th level to obtain the m-th fourth feature map of the n-th level.

[0021] In a possible implementation, the second fusion sub-module includes: a first processing unit, configured to use the (M + 1)-th fourth feature map of the first level as the fusion feature map.

[0022] In a possible implementation, the processing module includes: a second processing sub-module, configured to upscale the fusion feature map to obtain a segmentation feature map, where the height and width of the segmentation feature map are the same as those of the color image; and a segmentation sub-module, configured to segment the color image and the corresponding depth image according to the segmentation feature map to obtain a processing result of the color image and the corresponding depth image.

[0023] According to an aspect of the present disclosure, there is provided an electronic device, including: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0024] According to an aspect of the present disclosure, there is provided a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented.

[0025] In the disclosed embodiments, by performing feature extraction on a color image and a corresponding depth image respectively, an N-level first feature map of the color image and an N-level second feature map of the depth image are obtained, so that the semantic information of the color image and the semantic information of the depth image at a certain depth can be extracted respectively; the n-th level first feature map of the N-level first feature map is fused with the n-th level second feature map of the N-level second feature map to obtain the n-th level third feature map of the N-level third feature map, so that the semantic information contained in the obtained N-level third feature map can be made more accurate and critical; the N-level third feature map is then fused, and a fused feature map is obtained according to the fusion process, so that the fused feature map obtained by the fusion process contains richer and more accurate semantic information without excessive loss of resolution; the processing results of the color image and the depth image are determined according to the fused feature map, so that the accuracy of image processing can be improved.

[0026] It should be understood that the above general description and the following detailed description are exemplary and explanatory only and do not limit the present disclosure. Other features and aspects of the present disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0028] Figure 1 A flowchart showing an image processing method according to an embodiment of the present disclosure;

[0029] Figure 2 A schematic diagram showing the structure of a neural network according to an embodiment of the present disclosure is shown;

[0030] Figure 3 A block diagram showing an image processing apparatus according to an embodiment of the present disclosure;

[0031] Figure 4 A block diagram of an electronic device according to an embodiment of the present disclosure is shown;

[0032] Figure 5 A block diagram of an electronic device according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0033] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0034] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0035] The term "and / or" herein is only a description of the association relationship of the associated objects, indicating that there may be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the term "at least one" herein represents any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can represent including any one or more elements selected from the set consisting of A, B, and C.

[0036] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0037] Figure 1 A flowchart of an image processing method according to an embodiment of the present disclosure is shown. Figure 1 As shown, the image processing method includes:

[0038] In step S11, feature extraction is performed on the color image and the corresponding depth image respectively to obtain N-level first feature maps of the color image and N-level second feature maps of the depth image, wherein the scales of the first feature maps of each level in the N-level first feature maps are different, and the scales of the second feature maps of each level in the N-level second feature maps are different, and N is an integer and N≥2;

[0039] In step S12, the nth level first feature map of the N-level first feature map is fused with the nth level second feature map of the N-level second feature map to obtain the nth level third feature map of the N-level third feature map, where n is an integer and 1≤n≤N;

[0040] In step S13, the N-level third feature maps are fused, and a fused feature map is obtained according to the fusion processing result;

[0041] In step S14, the processing results of the color image and the corresponding depth image are determined according to the fused feature map.

[0042] In a possible implementation, the image processing method may be executed by an electronic device such as a terminal device or a server, and the terminal device may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, a vehicle-mounted device, a wearable device, etc. The method may be implemented by a processor calling a computer-readable instruction stored in a memory. Alternatively, the method may be executed by a server.

[0043] In a possible implementation, in step S11, the color image and the corresponding depth image may be images of a real scene acquired by an image acquisition device (e.g., a depth camera). The color image may be a multi-channel image, for example, an RGB (Red, Red; Green, Green; Blue) image, and the depth image may be a depth map (DepthMap) of the color image.

[0044] In a possible implementation, the color image and the corresponding depth image may be acquired by an image acquisition device, or may be acquired from a database or other device storing the color image and the corresponding depth image. The disclosed embodiment does not limit the source of the color image and the corresponding depth image.

[0045] In a possible implementation, in step S11, a multi-level convolution operation may be performed on the color image and the corresponding depth image, or a multi-level convolution operation and a multi-level batch normalization operation may be performed on the color image and the corresponding depth image, respectively, to perform multi-level feature extraction on the color image and the corresponding depth image, so as to obtain an N-level first feature map of the color image and an N-level second feature map of the depth image. The multi-level batch normalization operation may be performed after each level of convolution operation.

[0046] In a possible implementation, a multi-level convolution operation can be performed on the color image and the corresponding depth image respectively through a multi-level convolution layer of a neural network, that is, multi-level feature extraction. Among them, the multi-level convolution layer can include two groups of multi-level convolution layers, and batch normalization can be used after each convolution layer. Each first feature map and each second feature map can be the output of different levels of two groups of multi-level convolution layers, and the scale of the feature map output by the convolution layers of different levels (that is, the resolution of the feature map, or the height and width of the feature map) is gradually reduced. The scale of the feature map output at the same level in the two groups of multi-level convolution layers is the same, so that the feature map output at the same level can be fused in step S12 later.

[0047] For example, two groups of three-level convolutional layers can be used to perform multi-level feature extraction on the RGB image and the corresponding depth image, that is, one group of three-level convolutional layers is used for feature extraction on the RGB image, and another group of three-level convolutional layers is used for feature extraction on the corresponding depth image. The scale of the feature map output by each level of the convolutional layer in the group of three-level convolutional layers can be, for example, 1 / 2, 1 / 4, and 1 / 8 of the color image, respectively; the scale of the feature map output by each level of the convolutional layer in the other group of three-level convolutional layers can be, for example, 1 / 2, 1 / 4, and 1 / 8 of the depth image, respectively; the scale of the feature map output by the same level of the convolutional layer in the two groups of three-level convolutional layers can be the same, for example, the scale of the first level first feature map output by the first level of the convolutional layer in the group of three-level convolutional layers can be the same. The first-level second feature map output by the first-level convolution layer of the other set of three-level convolution layers The same scale, and so on.

[0048] In one possible manner, as described above, a multi-level convolutional layer can be used to perform multi-level feature extraction on a color image and a corresponding depth image, respectively, and the scales of the feature maps output at the same level in the multi-level convolutional layer can be the same. In step S12, the n-th level first feature map of the N-level first feature map is fused with the n-th level second feature map of the N-level second feature map, so that the first feature map and the second feature map of the same scale can be fused to obtain the n-th level third feature map of the N-level third feature map. It can be understood that the N-level first feature map is fused with the N-level second feature map to obtain the N-level third feature map. For example, the first feature map of level 1 can be fused with the second feature map of level 2. With the second feature map of level 1 Fusion is performed to obtain the third feature map X of the first level 0,0 , and so on, the 3rd level first feature map is fused with the 3rd level second feature map to obtain the 3rd level third feature map.

[0049] In a possible implementation, the fusion between feature maps of the same scale may be achieved by connecting feature maps, that is, connecting feature maps in the channel direction to generate a fused feature map, for example, connecting a first feature map of length × width × channel: 64 × 64 × 3 and a second feature map of 64 × 64 × 3 in the channel direction to obtain a third feature map of 64 × 64 × 9; or by adding feature maps, that is, adding pixel values ​​of corresponding channels in each feature map to generate a fused feature map, for example, adding pixel values ​​of a first feature map of 64 × 64 × 3 and a second feature map of 64 × 64 × 3 in each corresponding channel to obtain a third feature map of 64 × 64 × 3. The disclosed embodiment does not limit the fusion method between feature maps.

[0050] In a possible implementation, in step S13, the N-level third feature map is fused, and the multi-scale feature map of the N-level third feature map is extracted through the N-level fusion layer of the neural network, and then the extracted multi-scale feature maps are feature fused. The N-level third feature maps can be the input of the N-level fusion layer of the neural network respectively. The scale of the feature map output by different fusion layers is gradually reduced. Each fusion layer may include a multi-level convolution layer to perform a multi-level convolution operation on the third feature map input to the layer to extract multi-level features. The convolution layer included in the same level of fusion layer does not change the scale of the output feature map, that is, the scale (that is, resolution) of the feature map output by the convolution layer included in the same level of fusion layer is the same. The scale of the feature map output by each level of fusion layer is the same as the scale of the third feature map corresponding to the level. Among them, batch normalization can be used after each level of convolution layer.

[0051] In a possible implementation, operations such as scale up, scale down, skip connection, and lateral connection may be used in the N-level fusion layer to implement fusion processing of the N-level third feature maps.

[0052] In a possible implementation, downscaling can be achieved through downsampling and convolution operations, and downscaling can be used to reduce the feature map to 1 / 2 of the feature map, and then the reduced feature map is used as the input of a fusion process in the next fusion layer; upscaling can be achieved through upsampling and convolution operations, and upscaling can be used to enlarge the feature map to twice the feature map, and then the enlarged feature map is used as the input of a fusion process in the previous fusion layer. Feature maps of different scales can be fused through upscaling and downscaling.

[0053] In a possible implementation, the skip connection can be used to fuse the feature maps output by non-adjacent convolutional layers in the same fusion layer; the lateral connection can be used to fuse the feature maps output by adjacent convolutional layers in the same fusion layer. The lateral connection can be implemented by a convolution operation that does not change the scale of the feature map, that is, the feature map obtained by the previous fusion process can be subjected to a convolution operation that does not change the scale of the feature map, and then the feature map output by the convolution operation that does not change the scale of the feature map can be used as the input of this fusion process. The fusion of different feature maps of the same scale can be achieved through skip connections and lateral connections.

[0054] In a possible implementation, the fusion processing result may include a feature map obtained by the fusion processing. The feature map obtained according to the fusion processing result may be a feature map obtained by a certain fusion processing as a fused feature map, for example, a feature map obtained by the last fusion processing of the third feature map of level N may be used as a fused feature map.

[0055] In a possible implementation, the stochastic gradient descent method can be used to train the neural network in the embodiment of the present disclosure, the batch size can be 64, and all parameters of the neural network can be randomly initialized. The embodiment of the present disclosure does not limit the training method of the neural network.

[0056] In a possible implementation, in step S14, the processing results of the color image and the corresponding depth image are determined according to the fused feature map, and the fused feature map may be directly segmented to obtain the segmentation result; or the fused feature map may be further processed, and the segmentation process is performed according to the processed fused feature map to obtain the segmentation result. The segmentation process may include semantic segmentation, instance segmentation, etc., and the embodiment of the present disclosure does not limit the segmentation process method.

[0057] In a possible implementation, the processing results of the color image and the corresponding depth image may be the above-mentioned segmentation results, or may be the results obtained by further processing the above-mentioned segmentation results according to the actual image processing task. For example, in image editing tasks, the foreground area and the background area may be distinguished according to the segmentation results, and the foreground area and / or the background area may be processed accordingly, such as blurring the background area, etc., to obtain the final image processing result. The embodiments of the present disclosure do not limit the segmentation method of the feature map and the specific contents included in the processing results.

[0058] It should be noted that although the above-mentioned scenario of segmentation processing based on the fused feature map is introduced as an example, those skilled in the art will understand that the present disclosure should not be limited to this. In fact, users can use the fused feature map for various image processing tasks according to actual application scenarios. For example, image classification tasks, target detection tasks, etc. can be performed based on the fused feature map; for image classification tasks, the classification results of the color image and the corresponding depth image can be determined based on the fused feature map; for target detection tasks, the target detection results of the color image and the corresponding depth image can be determined based on the fused feature map.

[0059] In the disclosed embodiments, by performing feature extraction on a color image and a corresponding depth image respectively, an N-level first feature map of the color image and an N-level second feature map of the depth image are obtained, so that the semantic information of the color image and the semantic information of the depth image at a certain depth can be extracted respectively; the n-th level first feature map of the N-level first feature map is fused with the n-th level second feature map of the N-level second feature map to obtain the n-th level third feature map of the N-level third feature map, so that the semantic information contained in the obtained N-level third feature map can be made more accurate and critical; the N-level third feature map is then fused, and a fused feature map is obtained according to the fusion process, so that the fused feature map obtained through the fusion process contains richer and more accurate semantic information without excessive loss of resolution; the processing results of the color image and the depth image are determined according to the fused feature map, so that the accuracy of image processing can be improved.

[0060] In a possible implementation, in step S13, performing fusion processing on the N-level third feature maps and obtaining a fused feature map according to the fusion processing result may include:

[0061] The third characteristic map of the first level is used as the first fourth characteristic map of the first level;

[0062] The third feature map of the kth level is fused to obtain the first fourth feature map of the kth level, where k is an integer and 1<k≤N;

[0063] The first and fourth feature maps of each level are fused, and a fused feature map is obtained according to the fusion processing result.

[0064] In a possible implementation, the k-th level third feature map is fused to obtain the k-th level first fourth feature map, which may be a feature map obtained by fusing the k-1-th level first fourth feature map with the k-th level third feature map. The k-1-th level first fourth feature map may be scaled down to be processed into a feature map of the same scale as the k-th level third feature map, thereby achieving fusion between feature maps of different scales.

[0065] In a possible implementation, the fusion processing is performed on the first fourth feature map of each level, and the fourth feature maps of the first to Nth levels are fused multiple times. As described above, the fusion processing result may be a feature map obtained by the fusion processing. For multiple fusion processing, multiple feature maps may be obtained, and then a fused feature map is determined based on the multiple feature maps.

[0066] As described above, the fusion processing of the N-level third feature maps can be realized through a neural network. Figure 2 FIG. 2 shows a schematic diagram of a neural network structure according to an embodiment of the present disclosure. Figure 2The left network in the neural network shown can be used to obtain a 4-level third feature map based on the RGB image and the depth image. The right network is used to fuse the 4-level third feature maps; the right network uses connection methods such as scale enlargement, scale reduction, skip connection and lateral connection to achieve the fusion between the third feature maps; represents the first feature map, represents the second feature map, represents the third characteristic graph, X 0,0 ~X 3,4 Both represent the fourth feature map. The X in the fourth feature map 7,0 It can be a fused feature map.

[0067] For ease of understanding, Figure 2 The neural network shown in the figure is taken as an example to illustrate the above fusion processing process. Among them, the first-level third feature map is used as the first-level fourth feature map, which can be As X 0,0 , that is With X 0,0 The k-th level third feature map is fused to obtain the k-th level first fourth feature map, for example, After fusion processing, we get X 1,0 ,right Fusion processing to obtain X 2,0 ,right Fusion processing to obtain X 3,0 ; The first and fourth feature maps of each level are fused, which can be X 0,0 ~X 3,0 Multiple fusion processes are performed to obtain multiple fourth feature maps, and then a fusion feature map is obtained according to the multiple fourth feature maps.

[0068] In the disclosed embodiment, the third feature maps of N levels are fused to obtain the first fourth feature maps of N levels, and then the first fourth feature maps of each level are fused, and a fused feature map is obtained based on the fusion processing results, so that the fused feature map can be effectively obtained.

[0069] In a possible implementation, performing fusion processing on the k-th level third feature map to obtain the k-th level first fourth feature map may include:

[0070] The first fourth feature map of the k-1th level is scaled down to obtain the first fifth feature map of the kth level;

[0071] The k-th level third feature map and the k-th level first fifth feature map are fused to obtain the k-th level first fourth feature map.

[0072] For ease of understanding the above fusion process, take Figure 2 the neural network shown as an example. Among them, the first fourth feature map of the (k - 1)-th level is scaled down to obtain the first fifth feature map of the k-th level. For example, the first fourth feature map X 1,0 of the second level can be scaled down to obtain the first fifth feature map of the third level; the first fifth feature map of the third level is fused with the third feature map of the third level to obtain the first fourth feature map X 2,0 of the third level. The process of obtaining the first fourth feature map of other levels will not be elaborated here.

[0073] In the embodiments of the present disclosure, the first fourth feature map of the k-th level can be effectively obtained based on the third feature map of the k-th level.

[0074] In a possible implementation manner, the fusion process for the first fourth feature map of each level may include:

[0075] Performing the (m - 1)-th fusion process on the first fourth feature map of the n-th level to obtain the m-th fourth feature map of the n-th level, where m is an integer and 1 < m ≤ M - n + 2, and M is the number of fusion processes performed by the first fusion layer of the neural network set in advance, that is, the number of fusion processes performed on the first fourth feature map of the first level. The number of fusion processes of the entire neural network is related to the number of fusion processes of the first fusion layer.

[0076] In a possible implementation manner, M may be the number of fusion processes performed by the first fusion layer of the preset neural network, that is, the number of fusion processes performed on the first fourth feature map of the first level. The number of fusion processes of the entire neural network is related to the number of fusion processes of the first fusion layer.

[0077] In a possible implementation manner, it can be set that the number of fusion processes performed by the first to N-th fusion layers on the first fourth feature map of this level decreases gradually, so that the neural network can use less downsampling and learn more on high-resolution feature maps.

[0078] In a possible implementation manner, after setting the number of fusion processes M performed by the first fusion layer on the first fourth feature map of the first level, the number of fusion processes performed by the n-th fusion layer on the first fourth feature map of the n-th level can be determined according to M, and may include: the number of fusion processes performed by the n-th fusion layer on the first fourth feature map of the n-th level is M - n + 1. For example, for a three-level fusion layer, if the first fusion layer performs 7 fusion processes on the first fourth feature map of the first level, that is, M = 7, then the second fusion layer performs 6 fusion processes, and the third fusion layer performs 5 fusion processes.

[0079] For ease of understanding the above fusion process, take Figure 2 the neural network shown as an example. For example, for the first fourth feature map X of the second level1,0 Perform the first fusion process to obtain the second fourth feature map X at the second level 1,1 ; then perform the second fusion process to obtain the third fourth feature map X at the second level 1,2 , and so on, until the sixth fusion process is performed to obtain the seventh fourth feature map X at the second level 1,6 .

[0080] In a possible implementation, considering that when determining the processing result of the corresponding image based on the feature map, the high-resolution feature map contains more image details. In step S13, obtaining the fused feature map according to the fusion processing result may include using the (M + 1)-th fourth feature map at the first level (such as Figure 2 X in 0,7 ) as the fused feature map, that is, using the last fourth feature map output by the first-level fusion layer as the fused feature map, which can realize determining the processing result based on the high-resolution feature map. At the same time, the (M + 1)-th fourth feature map at the first level obtained by fusing through multiple fusion layers of the neural network also contains semantic information at different levels, so that the fused feature map contains deeper semantic features and is a high-resolution feature map.

[0081] In the embodiments of the present disclosure, since the process of performing the (m - 1)-th fusion process on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level is relatively complex, for the sake of simplicity, it will be elaborated in detail below.

[0082] In the embodiments of the present disclosure, by performing the (m - 1)-th fusion process on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level, where m is an integer and 1 < m ≤ M - n + 2, it can effectively enable the neural network to use less downsampling and learn more on the high-resolution feature map, and thus, without losing resolution, the obtained fused feature map contains deeper semantic features.

[0083] The following describes the fusion process of performing the (m - 1)-th fusion process on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level in different cases.

[0084] In a possible implementation, in response to the case where n = 1, the (m - 1)-th fusion process on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level includes:

[0085] Scale up the (m - 1)-th fourth feature map at the (n + 1)-th level to obtain the (m - 1)-th fifth feature map at the n-th level;

[0086] Fuse the first (m - 1) fourth feature maps at the n-th level with the (m - 1)-th fifth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level.

[0087] In a possible implementation, n = 1 represents the first fusion layer among the N-level fusion layers of the neural network.

[0088] Figure 2 A schematic structural diagram of a neural network according to an embodiment of the present disclosure is shown. For ease of understanding the above process of obtaining the m-th fourth feature map of the n-th level, taking Figure 2 the shown neural network as an example, the description is as follows:

[0089] For example, in the case of m = 2, that is, obtaining the second fourth feature map X of the first level 0,1 (that is, the first fusion process for the first fourth feature map of the first level), scale up the first fourth feature map X of the second level 1,0 to the first fifth feature map of the first level with the same scale as the fourth feature map X 0,1 ; fuse the first fifth feature map of the first level with the previous fourth feature map X of the first level 0,0 to obtain the second fourth feature map X of the first level 0,1 .

[0090] It can be understood that the above takes the fusion to obtain the fourth feature map X 0,1 as an example to illustrate the first fusion process for the first fourth feature map of the first level when n = 1. The other fusion processes for the first fourth feature map are the same as the above example and will not be elaborated here.

[0091] In the embodiments of the present disclosure, by performing multiple fusion processes on the first fourth feature map of the first level, multiple fusions between the fourth feature map of the first level and feature maps of different scales are achieved, enabling sufficient fusion between the feature maps, and thus obtaining a fusion feature map with high resolution and rich semantics.

[0092] In a possible implementation, in response to the situation of 1 < n < N, performing the (m - 1)-th fusion process on the first fourth feature map of the n-th level to obtain the m-th fourth feature map of the n-th level includes:

[0093] Scale down the m-th fourth feature map of the (n - 1)-th level to obtain the m-th sixth feature map of the n-th level;

[0094] Scale up the (m - 1)-th fourth feature map of the (n + 1)-th level to obtain the (m - 1)-th fifth feature map of the n-th level;

[0095] Fuse the previous (m - 1) fourth feature maps of the n-th level, the (m - 1)-th fifth feature map of the n-th level, and the m-th sixth feature map of the n-th level to obtain the m-th fourth feature map of the n-th level.

[0096] In a possible implementation, 1 < n < N represents other hierarchical fusion layers in the N-level fusion layer of the neural network except the first-level fusion layer and the N-level fusion layer.

[0097] To facilitate understanding of the process of obtaining the m-th fourth feature map of the n-th level in the above embodiments of the present disclosure in the case of 1 < n < N, the following is an example of the neural network shown: Figure 2 The illustration is as follows:

[0098] For example, in the case of n = 3 and m = 3, that is, obtaining the third fourth feature map X 2,2 (that is, the second fusion process for the first fourth feature map of the third level). Scale down the third fourth feature map X of the second level 1,2 to the third sixth feature map of the third level with the same scale as the fourth feature map X 2,2 ; scale up the second fourth feature map X of the fourth level 3,1 to the second sixth feature map of the third level with the same scale as the fourth feature map X 2,2 ; fuse the third sixth feature map of the third level, the second sixth feature map of the third level, and the first two fourth feature maps of the third level (that is, X 2,0 and X 2,1 ) to obtain the third fourth feature map X of the third level 2,2 .

[0099] It can be understood that the above takes the fusion to obtain the fourth feature map X 2,2 as an example to illustrate the second fusion process for the first fourth feature map of the third level in the case of 1 < n < N. The other fusion processes for the first fourth feature map of the n-th level are the same as the above example and will not be elaborated here.

[0100] In the embodiments of the present disclosure, by performing multiple fusion processes on the first fourth feature map of the n-th level, feature maps of different scales can be fused, enabling full fusion between feature maps, and thus obtaining a high-resolution and semantically rich fused feature map.

[0101] In a possible implementation, in response to the case of n = N, the process of performing the (m - 1)-th fusion process on the first fourth feature map of the n-th level to obtain the m-th fourth feature map of the n-th level includes:

[0102] Scale down the m-th fourth feature map of the (n - 1)-th level to obtain the m-th sixth feature map of the n-th level;

[0103] The first m-1 fourth feature maps of the nth level are fused with the mth sixth feature map of the nth level to obtain the mth fourth feature map of the nth level.

[0104] In a possible implementation, n=N represents the Nth fusion layer among the N-level fusion layers of the neural network.

[0105] To facilitate understanding of the above process of obtaining the mth fourth characteristic map of the nth level in response to the case of n=N, Figure 2 The neural network structure shown is used as an example to illustrate as follows:

[0106] For example, when m=3, the fourth feature map X of the fourth level is obtained. 3,2 (That is, the second fusion process for the first fourth feature map of the fourth level). 2,2 Reduce the scale to the same size as the fourth feature map X 3,2 The third sixth feature map of the fourth level of the same scale; the third sixth feature map of the fourth level and the first two fourth feature maps of the fourth level (ie, X 3,0 and X 3,1 ) to obtain the third fourth feature map X of the fourth level 3,2 .

[0107] It can be understood that the fourth feature map X is obtained by fusion 3,2 As an example, when n=N, the second fusion process is for the first fourth feature map of the 4th level, and the other fusion processes for the first fourth feature map of the Nth level are the same as the above example, which will not be repeated here.

[0108] In the disclosed embodiment, by performing multiple fusion processing on the first fourth feature map of the Nth level, the fusion between the deep semantic information and the shallow semantic information can be achieved, thereby obtaining a high-resolution and semantically rich fused feature map.

[0109] It should be noted that, in the embodiment of the present disclosure, the fifth feature map may refer to a feature map obtained by scaling up the fourth feature map; the sixth feature map may refer to a feature map obtained by scaling down the fourth feature map.

[0110] In a possible implementation, as described above, the fused feature map may be further processed, and the processed fused feature map may be segmented to obtain a segmentation result. In step S14, the processing results of the color image and the corresponding depth image are determined according to the fused feature map, which may include:

[0111] The fusion feature map is scaled up to obtain a segmentation feature map, and the height and width of the segmentation feature map are the same as those of the color image;

[0112] According to the segmentation feature map, the color image and the corresponding depth image are segmented to obtain processing results of the color image and the corresponding depth image.

[0113] In one possible implementation, the scale enlargement can be achieved through upsampling and convolution operations. Through scale enlargement, the fused feature map can be enlarged to the same resolution (i.e., height and width) as the color image and the corresponding depth image. The enlarged fused feature map is the segmentation feature map.

[0114] In a possible implementation, the color image and the corresponding depth image are segmented according to the segmentation feature map to obtain the processing result of the color image and the corresponding depth image. The segmentation feature map can be directly segmented to obtain the segmentation result; or the segmentation result can be processed again according to the actual image processing task. The segmentation processing may include semantic segmentation, instance segmentation, etc. The embodiments of the present disclosure do not limit the segmentation method of the segmentation feature map and the specific content included in the processing result.

[0115] In the disclosed embodiment, by scaling up the fused feature map to obtain a segmentation feature map, and determining the image processing result based on the segmentation feature map, image processing can be performed based on the high-resolution and semantically rich fused feature map, thereby improving the accuracy of image processing.

[0116] At present, the research on semantic segmentation and instance segmentation using RGB-D images has become a development trend. Most of the RGB-D semantic segmentation solutions in related technologies have shortcomings such as inaccurate segmentation, poor real-time performance, complex network structure, and poor reproducibility.

[0117] In view of the above problems, according to the image processing method of the embodiment of the present disclosure, high-precision real-time RGB-D semantic segmentation can be achieved.

[0118] Figure 2 FIG. 2 shows a schematic diagram of a neural network structure according to an embodiment of the present disclosure. Figure 2 A neural network structure is shown, and the neural network of the embodiment of the present disclosure can be a fully convolutional neural network. The input of the network is an RGB image and a depth image. The RGB image and the depth image are independently extracted through several layers of convolution layers in the right network, and then fused at multiple different levels.

[0119] There are a large number of connections between the feature layers (fusion network layers) of different levels on the left side of the neural network to achieve multi-scale fusion and feature extraction. The deeper the feature layer, the higher the semantic level and the lower the resolution; the shallower the feature layer, the lower the semantic level and the higher the resolution, which can be used for more accurate positioning.

[0120] The disclosed embodiment can generate fine-resolution feature maps with strong semantic information by connecting feature layers to build a network with fine resolution and high resolution. These connections include: 1. Horizontal connection: convolution operation that does not change the size of the feature map; 2. Downward connection: reducing the feature map to 1 / 2 of the original through convolution and downsampling; 3. Upward connection: enlarging the feature map to twice the original through convolution and upsampling; 4. Jump connection: connecting non-adjacent feature maps. Compared with the feature fusion method in the related art, the disclosed embodiment adopts a richer connection method, increases the connection between different feature maps, and achieves a more complete fusion between feature maps.

[0121] In one possible implementation, Batch Normalization is used after each convolutional layer. During the training process, stochastic gradient descent is used for training, the batch size is 64, and all parameters are randomly initialized.

[0122] In the embodiment of the present disclosure, learning can be performed through multi-layer convolution on a higher-resolution feature map to avoid irreversible resolution loss caused by over-downsampling, which leads to a decrease in positioning accuracy. Compared with the image fusion method in the related art, the image processing method of the embodiment of the present disclosure can use less downsampling and more high-resolution feature map learning, effectively improving the accuracy of the final semantic segmentation.

[0123] In the disclosed embodiments, by performing feature extraction and learning on higher-resolution feature maps, the high-resolution accuracy of the final output segmentation result is ensured; by fully fusing feature maps of different levels, the network's joint learning and mutual complementation of high-level semantic information and low-level positioning information is ensured.

[0124] In the disclosed embodiment, a richer connection method is adopted to increase the connection between different feature maps, achieve more complete fusion, generate feature maps with strong semantic information and fine resolution, and thus build a network with fine resolution and high resolution capability.

[0125] It can be understood that the above-mentioned various method embodiments mentioned in the present disclosure can be combined with each other to form a combined embodiment without violating the principle logic. Due to space limitations, the present disclosure will not repeat them. It can be understood by those skilled in the art that in the above-mentioned method of the specific implementation method, the specific execution order of each step should be determined according to its function and possible internal logic.

[0126] In addition, the present disclosure also provides an image processing device, an electronic device, a computer-readable storage medium, and a program, all of which can be used to implement any image processing method provided by the present disclosure. The corresponding technical solutions and descriptions are referred to the corresponding records in the method part and will not be repeated here.

[0127] Figure 3 A block diagram of an image processing device according to an embodiment of the present disclosure is shown. Figure 3 As shown, the device comprises:

[0128] A feature extraction module 101 is used to perform feature extraction on a color image and a corresponding depth image, respectively, to obtain an N-level first feature map of the color image and an N-level second feature map of the depth image, wherein the scales of the first feature maps of each level in the N-level first feature map are different, and the scales of the second feature maps of each level in the N-level second feature map are different, and N is an integer and N≥2;

[0129] A first fusion module 102 is used to fuse the nth level first feature map of the N-level first feature map with the nth level second feature map of the N-level second feature map to obtain the nth level third feature map of the N-level third feature map, where n is an integer and 1≤n≤N;

[0130] The second fusion module 103 is used to perform fusion processing on the N-level third feature maps and obtain a fused feature map according to the fusion processing result;

[0131] The processing module 104 is used to determine the processing results of the color image and the corresponding depth image according to the fusion feature map.

[0132] In a possible implementation, the second fusion module 103 includes: a first processing submodule, used to use the first-level third feature map as the first-level first fourth feature map; a first fusion submodule, used to perform fusion processing on the k-th level third feature map to obtain the k-th level first fourth feature map, k is an integer, and 1<k≤N; a second fusion submodule, used to perform fusion processing on the first fourth feature maps of each level, and obtain a fused feature map according to the fusion processing result.

[0133] In a possible implementation, the first fusion sub-module includes: a first reduction unit configured to reduce the scale of the first fourth feature map at the (k - 1)-th level to obtain the first fifth feature map at the k-th level; a first fusion unit configured to fuse the third feature map at the k-th level with the first fifth feature map at the k-th level to obtain the first fourth feature map at the k-th level.

[0134] In a possible implementation, the second fusion sub-module includes: a second fusion unit configured to perform the (m - 1)-th fusion process on the first fourth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level, where m is an integer and 1 < m ≤ M - n + 2, and M is the number of times of performing the fusion process on the first fourth feature map at the first level.

[0135] In a possible implementation, in response to the case where n = 1, the second fusion unit includes: a first magnification sub-unit configured to magnify the scale of the (m - 1)-th fourth feature map at the (n + 1)-th level to obtain the (m - 1)-th fifth feature map at the n-th level; a first fusion sub-unit configured to fuse the first (m - 1) fourth feature maps at the n-th level with the (m - 1)-th fifth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level.

[0136] In a possible implementation, in response to the case where 1 < n < N, the second fusion unit includes: a first reduction sub-unit configured to reduce the scale of the m-th fourth feature map at the (n - 1)-th level to obtain the m-th sixth feature map at the n-th level; a second magnification sub-unit configured to magnify the scale of the (m - 1)-th fourth feature map at the (n + 1)-th level to obtain the (m - 1)-th fifth feature map at the n-th level; a second fusion sub-unit configured to fuse the first (m - 1) fourth feature maps at the n-th level, the (m - 1)-th fifth feature map at the n-th level, and the m-th sixth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level.

[0137] In a possible implementation, in response to the case where n = N, the second fusion sub-module includes: a second reduction sub-unit configured to reduce the scale of the m-th fourth feature map at the (n - 1)-th level to obtain the m-th sixth feature map at the n-th level; a third fusion sub-unit configured to fuse the first (m - 1) fourth feature maps at the n-th level with the m-th sixth feature map at the n-th level to obtain the m-th fourth feature map at the n-th level.

[0138] In a possible implementation, the second fusion sub-module includes: a first processing unit configured to use the (M + 1)-th fourth feature map at the first level as the fusion feature map.

[0139] In one possible implementation, the processing module 104 includes: a second processing sub-module, used to scale up the fusion feature map to obtain a segmentation feature map, the height and width of the segmentation feature map are the same as the height and width of the color image; a segmentation sub-module, used to segment the color image and the corresponding depth image according to the segmentation feature map to obtain processing results of the color image and the corresponding depth image.

[0140] In the disclosed embodiments, by performing feature extraction on a color image and a corresponding depth image respectively, an N-level first feature map of the color image and an N-level second feature map of the depth image are obtained, so that the semantic information of the color image and the semantic information of the depth image at a certain depth can be extracted respectively; the n-th level first feature map of the N-level first feature map is fused with the n-th level second feature map of the N-level second feature map to obtain the n-th level third feature map of the N-level third feature map, so that the semantic information contained in the obtained N-level third feature map can be made more accurate and critical; the N-level third feature map is then fused, and a fused feature map is obtained according to the fusion process, so that the fused feature map obtained by the fusion process contains richer and more accurate semantic information without excessive loss of resolution; the processing results of the color image and the depth image are determined according to the fused feature map, so that the accuracy of image processing can be improved.

[0141] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0142] The embodiment of the present disclosure also provides a computer-readable storage medium, on which computer program instructions are stored, and when the computer program instructions are executed by a processor, the above method is implemented. The computer-readable storage medium may be a non-volatile computer-readable storage medium.

[0143] An embodiment of the present disclosure further proposes an electronic device, comprising: a processor; and a memory for storing instructions executable by the processor; wherein the processor is configured to call the instructions stored in the memory to execute the above method.

[0144] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code. When the computer-readable code is executed on a device, a processor in the device executes instructions for implementing the image processing method provided in any of the above embodiments.

[0145] The embodiment of the present disclosure also provides another computer program product for storing computer-readable instructions, which, when executed, enables a computer to perform the operations of the image processing method provided by any of the above embodiments.

[0146] The electronic device may be provided as a terminal, a server, or a device in other forms.

[0147] Figure 4 A block diagram of an electronic device 800 according to an embodiment of the present disclosure is shown. For example, the electronic device 800 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, or the like.

[0148] Reference Figure 4 , the electronic device 800 may include one or more of the following components: a processing component 802 , a memory 804 , a power component 806 , a multimedia component 808 , an audio component 810 , an input / output (I / O) interface 812 , a sensor component 814 , and a communication component 816 .

[0149] The processing component 802 generally controls the overall operation of the electronic device 800, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the above-mentioned method. In addition, the processing component 802 may include one or more modules to facilitate the interaction between the processing component 802 and other components. For example, the processing component 802 may include a multimedia module to facilitate the interaction between the multimedia component 808 and the processing component 802.

[0150] The memory 804 is configured to store various types of data to support operations on the electronic device 800. Examples of such data include instructions for any application or method operating on the electronic device 800, contact data, phone book data, messages, pictures, videos, etc. The memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0151] The power supply component 806 provides power to the various components of the electronic device 800. The power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the electronic device 800.

[0152] The multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 808 includes a front camera and / or a rear camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0153] The audio component 810 is configured to output and / or input audio signals. For example, the audio component 810 includes a microphone (MIC), and when the electronic device 800 is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 804 or sent via the communication component 816. In some embodiments, the audio component 810 also includes a speaker for outputting audio signals.

[0154] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0155] The sensor assembly 814 includes one or more sensors for providing various aspects of status assessment for the electronic device 800. For example, the sensor assembly 814 can detect the open / closed state of the electronic device 800, the relative positioning of the components, such as the display and keypad of the electronic device 800, and the sensor assembly 814 can also detect the position change of the electronic device 800 or a component of the electronic device 800, the presence or absence of contact between the user and the electronic device 800, the orientation or acceleration / deceleration of the electronic device 800, and the temperature change of the electronic device 800. The sensor assembly 814 may include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 814 may also include an optical sensor, such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, the sensor assembly 814 may also include an acceleration sensor, a gyroscope sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

[0156] The communication component 816 is configured to facilitate wired or wireless communication between the electronic device 800 and other devices. The electronic device 800 can access a wireless network based on a communication standard, such as a wireless network (WiFi), a second generation mobile communication technology (2G) or a third generation mobile communication technology (3G), or a combination thereof. In an exemplary embodiment, the communication component 816 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 816 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0157] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above methods.

[0158] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions, which can be executed by a processor 820 of an electronic device 800 to perform the above method.

[0159] Figure 5 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server. Figure 5 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0160] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output (I / O) interface 1958. The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as a Microsoft Server operating system (Windows Server 2003). TM), a graphical user interface operating system launched by Apple (Mac OSX TM ), a multi-user, multi-process computer operating system (Unix TM ), a free and open source Unix-like operating system (Linux TM ), an open source Unix-like operating system (FreeBSD TM ) or similar.

[0161] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0162] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0163] Computer readable storage medium can be a tangible device that can hold and store instructions used by an instruction execution device. Computer readable storage medium can be, for example, (but not limited to) an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination thereof. More specific examples (non-exhaustive list) of computer readable storage medium include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, for example, a punch card or a convex structure in a groove on which instructions are stored, and any suitable combination thereof. The computer readable storage medium used here is not interpreted as a transient signal itself, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagated by a waveguide or other transmission medium (for example, a light pulse by an optical fiber cable), or an electrical signal transmitted by a wire.

[0164] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0165] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be personalized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0166] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0167] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0168] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0169] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0170] The computer program product may be implemented in hardware, software or a combination thereof. In one optional embodiment, the computer program product is embodied as a computer storage medium, and in another optional embodiment, the computer program product is embodied as a software product, such as a software development kit (SDK) and the like.

[0171] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or improvements to the technology in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. An image processing method, characterized in that: Including: Performing feature extraction on a color image and a corresponding depth image respectively to obtain N-level first feature maps of the color image and N-level second feature maps of the depth image. The scales of the first feature maps at each level in the N-level first feature maps are different, and the scales of the second feature maps at each level in the N-level second feature maps are different. N is an integer and N≥2; Fusing the nth-level first feature map of the N-level first feature maps with the nth-level second feature map of the N-level second feature maps to obtain the nth-level third feature map of the N-level third feature maps, where n is an integer and 1≤n≤N; Performing fusion processing on the N-level third feature maps and obtaining a fused feature map according to the result of the fusion processing; Determining the processing result of the color image and the corresponding depth image according to the fused feature map; Wherein, the performing fusion processing on the N-level third feature maps and obtaining a fused feature map according to the result of the fusion processing includes: Taking the first-level third feature map as the first first-level fourth feature map; Performing fusion processing on the kth-level third feature map to obtain the first kth-level fourth feature map, where k is an integer and 1<k≤N; Performing fusion processing on the first fourth feature maps at each level and obtaining a fused feature map according to the result of the fusion processing.

2. The method according to claim 1, characterized in that The performing fusion processing on the kth-level third feature map to obtain the first kth-level fourth feature map includes: Reducing the scale of the first (k - 1)th-level fourth feature map to obtain the first kth-level fifth feature map; Fusing the kth-level third feature map with the first kth-level fifth feature map to obtain the first kth-level fourth feature map.

3. The method according to claim 1, characterized in that The performing fusion processing on the first fourth feature maps at each level includes: Performing the (m - 1)th fusion processing on the first fourth feature map at the nth level to obtain the mth fourth feature map at the nth level, where m is an integer and 1<m≤M - n + 2, and M is the number of times of performing fusion processing on the first fourth feature map at the first level.

4. The method according to claim 3, characterized in that In response to the case of n = 1, the performing the (m - 1)th fusion processing on the first fourth feature map at the nth level to obtain the mth fourth feature map at the nth level includes: Enlarging the scale of the (m - 1)th fourth feature map at the (n + 1)th level to obtain the (m - 1)th fifth feature map at the nth level; Fusing the first (m - 1) fourth feature maps at the nth level with the (m - 1)th fifth feature map at the nth level to obtain the mth fourth feature map at the nth level.

5. The method according to claim 3, characterized in that: In response to the case of 1<n<N, the performing the (m - 1)th fusion processing on the first fourth feature map at the nth level to obtain the mth fourth feature map at the nth level includes: Reducing the scale of the mth fourth feature map at the (n - 1)th level to obtain the mth sixth feature map at the nth level; Enlarging the scale of the (m - 1)th fourth feature map at the (n + 1)th level to obtain the (m - 1)th fifth feature map at the nth level; Fusing the first (m - 1) fourth feature maps at the nth level, the (m - 1)th fifth feature map at the nth level, and the mth sixth feature map at the nth level to obtain the mth fourth feature map at the nth level.

6. The method according to any one of claims 3 to 5, characterized in that: In response to the case where n=N, the performing the m-1th fusion process on the first fourth feature map of the nth level to obtain the mth fourth feature map of the nth level includes: Reduce the scale of the m-th fourth feature map of the n-1th level to obtain the m-th sixth feature map of the nth level; The first m-1 fourth feature maps of the nth level are fused with the mth sixth feature map of the nth level to obtain the mth fourth feature map of the nth level.

7. The method according to any one of claims 3 to 5, characterized in that: The step of obtaining a fusion feature map according to the fusion processing result includes: The M+1th fourth feature map of the 1st level is used as the fused feature map.

8. The method according to claim 1, characterized in that Determining the processing results of the color image and the corresponding depth image according to the fusion feature map includes: Enlarging the scale of the fusion feature map to obtain a segmentation feature map, wherein the height and width of the segmentation feature map are the same as the height and width of the color image; The color image and the corresponding depth image are segmented according to the segmentation feature map to obtain processing results of the color image and the corresponding depth image.

9. An image processing device, characterized in that: include: A feature extraction module is used to perform feature extraction on the color image and the corresponding depth image respectively, to obtain N-level first feature maps of the color image and N-level second feature maps of the depth image, wherein the scales of the first feature maps of each level in the N-level first feature maps are different, and the scales of the second feature maps of each level in the N-level second feature maps are different, and N is an integer and N≥2; A first fusion module is used to fuse the nth level first feature map of the N-level first feature map with the nth level second feature map of the N-level second feature map to obtain the nth level third feature map of the N-level third feature map, where n is an integer and 1≤n≤N; A second fusion module is used to perform fusion processing on the N-level third feature maps, and obtain a fused feature map according to the fusion processing result; A processing module, used to determine the processing results of the color image and the corresponding depth image according to the fusion feature map; Among them, the second fusion module includes: a first processing submodule, used to use the first-level third feature map as the first-level first fourth feature map; a first fusion submodule, used to fuse the k-th level third feature map to obtain the k-th level first fourth feature map, k is an integer, and 1<k≤N; a second fusion submodule, used to fuse the first fourth feature maps of each level, and obtain a fused feature map according to the fusion processing results.

10. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; The processor is configured to call the instructions stored in the memory to execute the method described in any one of claims 1 to 8.

11. A computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images