Image processing method, device, electronic device and storage medium

Abstract

The present application provides an image processing method, device, electronic device and storage medium; the method includes: obtaining an image to be processed and a sparse depth image corresponding to the image to be processed; based on the sparse depth image, determining a semi-dense depth image corresponding to the sparse depth image; based on the semi-dense depth image, determining a three-dimensional pose image corresponding to the semi-dense depth image; based on a dual-branch convolutional neural network, processing the three-dimensional pose image, the semi-dense depth image and the image to be processed to determine the dense depth image corresponding to the image to be processed. In this way, the image can be processed intelligently to improve the accuracy of image processing and the image quality of the dense depth map obtained after image processing.

Description

Technical Field

[0001] The present application relates to computer vision technology, and in particular to an image processing method, device, electronic device and storage medium. Background Art

[0002] With the rapid development and widespread use of computer vision technology, people use computer vision technology to process sparse depth maps to obtain dense depth maps, which has gradually become the mainstream of computer vision technology applications. However, in the process of processing sparse depth maps to obtain dense depth maps, the dense depth maps obtained by existing image processing methods have blurred edges, low image quality, and dense depth maps are prone to artifacts and geometric distortions. The image processing accuracy is low. People hope to improve the image quality of dense depth maps obtained through image processing and improve the accuracy of image processing.

[0003] Therefore, how to intelligently process images to improve the accuracy of image processing and improve the image quality of dense depth maps obtained after image processing has always been a goal. Summary of the invention

[0004] Embodiments of the present application provide an image processing method, device, electronic device, and storage medium.

[0005] According to the first aspect of the present application, an image processing method is provided, the method comprising: obtaining an image to be processed and a sparse depth image corresponding to the image to be processed; based on the sparse depth image, determining a semi-dense depth image corresponding to the sparse depth image; based on the semi-dense depth image, determining a three-dimensional pose image corresponding to the semi-dense depth image; based on a dual-branch convolutional neural network, processing the three-dimensional pose image, the semi-dense depth image and the image to be processed to determine the dense depth image corresponding to the image to be processed.

[0006] According to an embodiment of the present application, determining a semi-dense depth image corresponding to the sparse depth image based on the sparse depth image includes: determining positions of invalid value pixels in the sparse depth image; determining a square area to be supplemented with the position of the invalid value pixel as the center and a side length of 5 pixels; in response to the number of valid value pixels in the area to be supplemented satisfying a preset number threshold, supplementing the area to be supplemented based on a weighted kernel function, and determining that the supplemented sparse depth image is the semi-dense depth image.

[0007] According to one embodiment of the present application, determining the three-dimensional pose image corresponding to the semi-dense depth image based on the semi-dense depth image includes: determining a camera intrinsic parameter matrix; performing three-dimensional pose projection on the semi-dense depth image based on the camera intrinsic parameter matrix to determine the three-dimensional pose coordinates corresponding to the pixel points in the semi-dense depth image; and determining the three-dimensional pose image corresponding to the semi-dense depth image based on the three-dimensional pose coordinates.

[0008] According to one embodiment of the present application, the three-dimensional pose image, the semi-dense depth image and the image to be processed are processed based on a dual-branch convolutional neural network to determine the dense depth image corresponding to the image to be processed, including: the dual-branch convolutional neural network includes: a semantic guidance branch and a depth refinement branch; based on the semantic guidance branch, the semi-dense depth image and the image to be processed are semantically guided to determine the semantic depth image corresponding to the image to be processed; based on the depth refinement branch, the semantic depth image and the three-dimensional pose image are depth refined to determine the dense depth image corresponding to the image to be processed.

[0009] According to one embodiment of the present application, based on the semantic guidance branch, semantic guidance is performed on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed, including: the semantic guidance branch includes a dense jump connection structure; based on the dense jump connection structure, multi-scale semantic fusion is performed on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed.

[0010] According to the second aspect of the present application, an image processing device is provided, which includes: an acquisition module for acquiring an image to be processed and a sparse depth image corresponding to the image to be processed; an adaptive dense module for determining a semi-dense depth image corresponding to the sparse depth image based on the sparse depth image; a coordinate projection module for determining a three-dimensional pose image corresponding to the semi-dense depth image based on the semi-dense depth image; and a determination module for processing the three-dimensional pose image, the semi-dense depth image and the image to be processed based on a dual-branch convolutional neural network to determine the dense depth image corresponding to the image to be processed.

[0011] According to one embodiment of the present application, the adaptive dense module is used to: determine the position of invalid value pixels in the sparse depth image; determine a square area to be supplemented with the position of the invalid value pixel as the center and a side length of 5 pixels; in response to the number of valid value pixels in the area to be supplemented meeting a preset number threshold, supplement the area to be supplemented based on a weighted kernel function, and determine that the sparse depth image after supplementation is the semi-dense depth image.

[0012] According to one embodiment of the present application, the coordinate projection module is used to: determine a camera intrinsic parameter matrix; based on the camera intrinsic parameter matrix, perform three-dimensional pose projection on the semi-dense depth image to determine the three-dimensional pose coordinates corresponding to the pixel points in the semi-dense depth image; based on the three-dimensional pose coordinates, determine the three-dimensional pose image corresponding to the semi-dense depth image.

[0013] According to one embodiment of the present application, the dual-branch convolutional neural network includes: a semantic guidance branch and a depth refinement branch, and the determination module is used to: based on the semantic guidance branch, perform semantic guidance on the semi-dense depth image and the image to be processed, and determine the semantic depth image corresponding to the image to be processed; based on the depth refinement branch, perform depth refinement on the semantic depth image and the three-dimensional pose image, and determine the dense depth image corresponding to the image to be processed.

[0014] According to one embodiment of the present application, the semantic guidance branch includes a dense jump connection structure, and the determination module is used to: based on the dense jump connection structure, perform multi-scale semantic fusion on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed.

[0015] According to a third aspect of the present application, an electronic device is provided, including:

[0016] at least one processor; and

[0017] a memory communicatively connected to the at least one processor; wherein,

[0018] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method described in the present application.

[0019] According to a fourth aspect of the present application, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method described in the present application.

[0020] The method of the embodiment of the present application obtains an image to be processed and a sparse depth image corresponding to the image to be processed; based on the sparse depth image, determines a semi-dense depth image corresponding to the sparse depth image; based on the semi-dense depth image, determines a three-dimensional pose image corresponding to the semi-dense depth image; based on a dual-branch convolutional neural network, processes the three-dimensional pose image, the semi-dense depth image and the image to be processed to determine a dense depth image corresponding to the image to be processed. In this way, the image can be processed intelligently to improve the accuracy of image processing and the image quality of the dense depth map obtained after image processing.

[0021] It should be understood that the teachings of the present application are not required to achieve all of the beneficial effects described above, but specific technical solutions can achieve specific technical effects, and other embodiments of the present application can also achieve beneficial effects not mentioned above. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] By reading the detailed description below with reference to the accompanying drawings, the above and other purposes, features and advantages of the exemplary embodiments of the present application will become readily understood. In the accompanying drawings, several embodiments of the present application are shown in an exemplary and non-limiting manner, wherein:

[0023] In the drawings, the same or corresponding reference numerals represent the same or corresponding parts.

[0024] Figure 1 The processing flow diagram of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 1 ;

[0025] Figure 2 The processing flow diagram of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 2 ;

[0026] Figure 3 The processing flow diagram of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 3 ;

[0027] Figure 4 The processing flow diagram of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 4 ;

[0028] Figure 5 The processing flow diagram of the image processing method provided in the embodiment of the present application is shown in FIG. Figure 5 ;

[0029] Figure 6 An application scenario diagram of the image processing method provided by an embodiment of the present application is shown;

[0030] Figure 7 Another application scenario diagram of the image processing method provided in the embodiment of the present application is shown;

[0031] Figure 8 Another application scenario diagram of the image processing method provided in the embodiment of the present application is shown;

[0032] Fig. 9 Another application scenario diagram of the image processing method provided in the embodiment of the present application is shown;

[0033] Fig.10 An optional schematic diagram of an image processing device provided in an embodiment of the present application is shown;

[0034] Fig.11 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0035] In order to make the purpose, features, and advantages of the present application more obvious and easy to understand, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present application.

[0036] In the following description, reference is made to “some embodiments”, which describe a subset of all possible embodiments, but it will be understood that “some embodiments” may be the same subset or different subsets of all possible embodiments and may be combined with each other without conflict.

[0037] In the following description, the terms "first\second" involved are merely used to distinguish similar objects and do not represent a specific ordering of the objects. It can be understood that "first\second" can be interchanged with a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0038] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0039] In the related art, the currently known technical solutions for image processing have blurred edges in the dense depth map obtained by the existing image processing method and are prone to artifacts and geometric distortion. The related art has the problem of low image quality and low image processing accuracy after image processing.

[0040] In view of the above-mentioned image processing method provided by the related art, the image quality obtained after image processing is low and the image processing accuracy is low. The method of the embodiment of the present application obtains the image to be processed and the sparse depth image corresponding to the image to be processed; based on the sparse depth image, the semi-dense depth image corresponding to the sparse depth image is determined; based on the semi-dense depth image, the three-dimensional pose image corresponding to the semi-dense depth image is determined; based on the double-branch convolutional neural network, the three-dimensional pose image, the semi-dense depth image and the image to be processed are processed to determine the dense depth image corresponding to the image to be processed. In this way, the method that can integrate the traditional image algorithm with the convolutional neural network uses the traditional image algorithm to adaptively complete the planar area in the depth image, and uses the convolutional neural network to complete the geometrically complex area in the depth image. It effectively overcomes the problem of non-robust processing of geometric boundaries in the existing image processing methods, and the problem that the convolutional neural network is prone to cause artifacts and geometric inconsistencies. In this way, the image can be processed intelligently to improve the accuracy of image processing and improve the image quality of the dense depth map obtained after image processing. Therefore, compared with the related art in which the image quality obtained after image processing is low and the image processing accuracy is low, the image processing method of the present application can improve the accuracy of image processing and improve the image quality of the dense depth map obtained after image processing.

[0041] The processing flow of the image processing method provided in the embodiment of the present application is described. Figure 1 , Figure 1 The processing flow diagram of the image processing method provided in the embodiment of the present application is as follows: Figure 1 , will combine Figure 1 Steps S101-S104 are shown for explanation.

[0042] Step S101, obtaining an image to be processed and a sparse depth image corresponding to the image to be processed.

[0043] In some embodiments, the image to be processed may include: a color image, and the embodiments of the present application do not limit the specific image to be processed. The sparse depth image corresponding to the image to be processed may include: a sparse depth image obtained by projecting a three-dimensional point cloud on the image to be processed. The three-dimensional point cloud is collected outdoors by a three-dimensional imaging sensor.

[0044] Step S102: Based on the sparse depth image, determine a semi-dense depth image corresponding to the sparse depth image.

[0045] In some embodiments, based on the sparse depth image, determining a semi-dense depth image corresponding to the sparse depth image may include: determining the positions of invalid value pixels in the sparse depth image; determining a square area to be supplemented with the position of the invalid value pixel as the center and a side length of 5 pixels; in response to the number of valid value pixels in the area to be supplemented satisfying a preset number threshold, supplementing the area to be supplemented based on a weighted kernel function, and determining that the supplemented sparse depth image is a semi-dense depth image.

[0046] In some embodiments, invalid value pixels may include: pixels with a pixel value of 0 in a sparse depth image. Valid value pixels may include: pixels with a pixel value that is not 0 in a sparse depth image. In particular, invalid value pixels do not contain depth information, and valid value pixels contain depth information. A square area to be supplemented may include: a square area of ​​5 pixels * 5 pixels, wherein the area to be supplemented may include 25 pixels, and the pixels in the area to be supplemented may include: valid value pixels and invalid value pixels. The preset quantity threshold may include: the minimum value of the number of valid value pixels in the preset area to be supplemented, and the embodiments of the present application do not limit the specific quantity threshold. The weighted kernel function may include: a 5 pixel * 5 pixel Euclidean distance weighted kernel function. The semi-dense depth image may include: a depth image after all invalid value pixels in the sparse depth image are supplemented.

[0047] In a specific implementation, the position of an invalid pixel with a pixel value of 0 in the sparse depth image is determined, and then a 5-pixel*5-pixel square area to be supplemented with the position of the invalid pixel as the center is determined, the number of valid pixel points in the area to be supplemented is determined, and it is determined whether the number of valid pixel points in the area to be supplemented is greater than or equal to a preset number threshold. In response to the number of valid pixel points in the area to be supplemented being greater than or equal to the preset number threshold, the weight of the 5-pixel*5-pixel Euclidean distance weighted kernel function is multiplied by the pixel value of the corresponding pixel point in the 5-pixel*5-pixel square area to be supplemented to obtain a weighted product, the weighted product of the corresponding pixel point in the 5-pixel*5-pixel square area to be supplemented is summed to obtain a summed result, and the summed result is divided by the sum of the weights of the 5-pixel*5-pixel Euclidean distance weighted kernel function to obtain a supplemented result, and the supplemented result is used as the pixel value of the invalid pixel point corresponding to the area to be supplemented, and the invalid pixel point corresponding to the area to be supplemented is supplemented, and it is determined that the sparse depth image after all the invalid pixel points in the sparse depth image are supplemented is a semi-dense depth image.

[0048] As an example, the preset number threshold is 3, the position of the invalid value pixel A is determined, a 5 pixel*5 pixel square area to be supplemented with the position of the invalid value pixel as the center is determined, the number of valid value pixels in the area to be supplemented is determined to be 3, and the valid value pixels in the area to be supplemented include: valid value pixel B, valid value pixel C, and valid value pixel D. The pixel value of the valid value pixel B is 25, the pixel value of the valid value pixel C is 30, and the pixel value of the valid value pixel D is 20. The weight of the Euclidean distance weighted kernel function corresponding to the valid value pixel B is 1, the weight of the Euclidean distance weighted kernel function corresponding to the valid value pixel C is 0.5, and the weight of the Euclidean distance weighted kernel function corresponding to the valid value pixel D is 0.5. The summation result is 25*1+30*0.5+20*0.5=40; the supplementation result is 40 / (1+0.5+0.5)=20, and the supplementation result 20 is used as the pixel value of the invalid value pixel point A corresponding to the area to be supplemented, and the invalid value pixel point A corresponding to the area to be supplemented is supplemented, and it is determined that the sparse depth image after all the invalid value pixels in the sparse depth image are supplemented is a semi-dense depth image.

[0049] Step S103: determining a three-dimensional pose image corresponding to the semi-dense depth image based on the semi-dense depth image.

[0050] In some embodiments, based on the semi-dense depth image, determining the three-dimensional pose image corresponding to the semi-dense depth image may include: determining a camera intrinsic parameter matrix; based on the camera intrinsic parameter matrix, performing three-dimensional pose projection on the semi-dense depth image to determine the three-dimensional pose coordinates corresponding to the pixel points in the semi-dense depth image; based on the three-dimensional pose coordinates, determining the three-dimensional pose image corresponding to the semi-dense depth image.

[0051] In some embodiments, the camera intrinsic parameter matrix may include: the intrinsic parameter matrix represents the characteristic parameters of the depth camera, wherein the characteristic parameters are the horizontal focal length, the vertical focal length, the offset of the camera pixel in the horizontal direction, and the offset of the camera pixel in the vertical direction. The three-dimensional pose projection may include: through the depth camera, projecting the pixel points of the semi-dense depth image to the camera coordinate system to obtain the three-dimensional pose coordinates. Among them, the three-dimensional pose coordinates may include: the two-dimensional coordinate points of the pixel points of the semi-dense depth image correspond to the three-dimensional coordinate points projected in the camera coordinate system. The three-dimensional pose image may include: the three-dimensional pose image corresponding to the three-dimensional pose coordinates of all the pixel points in the semi-dense depth image projected in the camera coordinate system. The camera coordinate system may include: the three-dimensional coordinate system includes the X-axis (horizontal axis), the Y-axis (vertical axis), and the Z-axis (vertical axis).

[0052] In the specific implementation, the camera intrinsic parameter matrix is ​​determined; based on the camera intrinsic parameter matrix, the pixel points of the semi-dense depth image are projected onto the X-axis, Y-axis and Z-axis through the depth camera to determine the three-dimensional pose coordinates corresponding to all the pixel points in the semi-dense depth image; the three-dimensional pose image corresponding to the three-dimensional pose coordinates of all the pixel points in the semi-dense depth image projected onto the X-axis, Y-axis and Z-axis is determined.

[0053] Step S104, based on a dual-branch convolutional neural network, the three-dimensional pose image, the semi-dense depth image and the image to be processed are processed to determine a dense depth image corresponding to the image to be processed.

[0054] In some embodiments, based on a dual-branch convolutional neural network, a three-dimensional pose image, a semi-dense depth image and an image to be processed are processed to determine a dense depth image corresponding to the image to be processed, which may include: based on a semantic guidance branch, semantically guiding the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed; based on a depth refinement branch, depth refinement is performed on the semantic depth image and the three-dimensional pose image to determine the dense depth image corresponding to the image to be processed.

[0055] In some embodiments, a dual-branch convolutional neural network may include a semantic guidance branch and a depth refinement branch. The semantic guidance branch may include: a semantic encoder, a semantic decoder, and a dense jump connection structure. Among them, the semantic encoder, the semantic decoder, and the dense jump connection structure all include semantic guidance nodes. The depth refinement branch may include: a refinement encoder and a refinement decoder. Among them, the refinement encoder and the refinement decoder include depth refinement nodes. The semantic depth image may include: a dense depth map with semantic guidance information corresponding to the image to be processed. The dense depth image may include: a dense depth map with semantic guidance information and depth information corresponding to the image to be processed.

[0056] In the specific implementation, the semi-dense depth image and the image to be processed are input into the semantic encoder, and through a series of dense jump connection structures, the semi-dense depth image and the image to be processed are semantically fused at multiple scales to determine the semantic depth image corresponding to the image to be processed, and the semantic depth image is used as the output of the semantic guidance branch through the semantic decoder. The semantic depth image and the three-dimensional pose image are input into the refinement encoder, and through the refinement encoder and refinement decoder, the depth information of the semantic depth image and the three-dimensional pose image is fused, the depth of the semantic depth image and the three-dimensional pose image is refined, and the dense depth image corresponding to the image to be processed is determined, and the dense depth image is used as the output of the depth refinement branch through the refinement decoder.

[0057] Based on the semantic guidance branch, semantic guidance is performed on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed. In specific implementation, it can include: based on a dense skip connection structure, multi-scale semantic fusion is performed on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed.

[0058] In some embodiments, the processing flow of the image processing method is shown as follows: Figure 2 ,like Figure 2 As shown, including:

[0059] Step S201, determining the positions of invalid value pixels in the sparse depth image.

[0060] Step S202, determining a square area to be supplemented with the position of the invalid value pixel as the center and a side length of 5 pixels.

[0061] Step S203 , in response to the number of effective value pixels in the area to be supplemented meeting a preset number threshold, the area to be supplemented is supplemented based on a weighted kernel function, and the supplemented sparse depth image is determined to be a semi-dense depth image.

[0062] In the specific implementation of steps S201 to S203, the position of the invalid pixel with a pixel value of 0 in the sparse depth image is determined, and then a 5-pixel*5-pixel square area to be supplemented with the position of the invalid pixel as the center is determined, the number of valid pixel points in the area to be supplemented is determined, and whether the number of valid pixel points in the area to be supplemented is greater than or equal to a preset number threshold is determined. In response to the number of valid pixel points in the area to be supplemented being greater than or equal to the preset number threshold, the weight of the 5-pixel*5-pixel Euclidean distance weighted kernel function is compared with the 5-pixel*5-pixel positive The pixel values ​​of the corresponding pixels in the square area to be supplemented are multiplied to obtain a weighted product, the weighted products of the corresponding pixels in the 5 pixel * 5 pixel square area to be supplemented are summed to obtain a sum result, and then the sum result is divided by the sum of the weights of the 5 pixel * 5 pixel Euclidean distance weighted kernel function to obtain a supplement result, and the supplement result is used as the pixel value of the invalid value pixel corresponding to the area to be supplemented, and the invalid value pixel corresponding to the area to be supplemented is supplemented, and it is determined that the sparse depth image after all the invalid value pixels in the sparse depth image are supplemented is a semi-dense depth image.

[0063] The method of the embodiment of the present application determines the semi-dense depth image corresponding to the sparse depth image based on the sparse depth image, and generates the semi-dense depth image without significantly increasing the error of the sparse depth image. The semi-dense depth image has more depth geometry information. In this way, the sparse depth image can be processed intelligently to improve the accuracy of image processing and the image quality of the dense depth map obtained after image processing.

[0064] In some embodiments, the processing flow of the image processing method is shown as follows: Figure 3 ,like Figure 3 As shown, including:

[0065] Step S301, determining the camera intrinsic parameter matrix.

[0066] In specific implementation, through step S301, the camera intrinsic parameter matrix is ​​determined, which can be expressed by the following formula (1):

[0067]

[0068] Among them, f x represents the horizontal focal length of the depth camera, f y represents the vertical focal length of the depth camera, c x Indicates the horizontal camera pixel offset, c y Indicates the vertical offset of the camera pixels.

[0069] Step S302: Perform three-dimensional pose projection on the semi-dense depth image based on the camera intrinsic parameter matrix to determine the three-dimensional pose coordinates corresponding to the pixel points in the semi-dense depth image.

[0070] In the specific implementation, through step S302, based on the camera intrinsic parameter matrix, the semi-dense depth image is projected with a three-dimensional pose to determine the three-dimensional pose coordinates corresponding to the pixel points in the semi-dense depth image, which can be expressed by the following formula (2):

[0071]

[0072] Among them, f x represents the horizontal focal length of the depth camera, f y represents the vertical focal length of the depth camera, c x Indicates the horizontal camera pixel offset, c y Indicates the offset of the camera pixel in the vertical direction. The two-dimensional coordinates of the pixel points of the semi-dense depth image correspond to the three-dimensional coordinate system projected in the camera coordinate system, including the X axis, Y axis, Z axis, X c The two-dimensional coordinates of the pixel points in the semi-dense depth image correspond to the horizontal coordinates projected in the camera coordinate system, Y c The two-dimensional coordinates of the pixel points in the semi-dense depth image correspond to the vertical coordinates projected in the camera coordinate system, Z c The two-dimensional coordinates of the pixel points of the semi-dense depth image correspond to the vertical coordinates projected in the camera coordinate system, D represents the depth value of the pixel points of the semi-dense depth image, u represents the horizontal coordinates of the pixel points of the semi-dense depth image, and v represents the vertical coordinates of the pixel points of the semi-dense depth image.

[0073] Step S303: determining a three-dimensional pose image corresponding to the semi-dense depth image based on the three-dimensional pose coordinates.

[0074] In some embodiments, the processing flow of the image processing method is shown as follows: Figure 4 ,like Figure 4 As shown, including:

[0075] Step S401 : Based on the semantic guidance branch, semantic guidance is performed on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed.

[0076] Step S402: Based on the depth refinement branch, depth refinement is performed on the semantic depth image and the three-dimensional pose image to determine a dense depth image corresponding to the image to be processed.

[0077] Steps S401 to S402, in specific implementation, input the semi-dense depth image and the image to be processed into the semantic encoder, perform multi-scale semantic fusion on the semi-dense depth image and the image to be processed through a series of dense jump structures, determine the semantic depth image corresponding to the image to be processed, and use the semantic depth image as the output of the semantic guidance branch through the semantic decoder. Input the semantic depth image and the three-dimensional pose image into the refinement encoder, perform depth information fusion on the semantic depth image and the three-dimensional pose image through the refinement encoder and the refinement decoder, perform depth refinement on the semantic depth image and the three-dimensional pose image, determine the dense depth image corresponding to the image to be processed, and use the dense depth image as the output of the depth refinement branch through the refinement decoder.

[0078] The method of the embodiment of the present application inputs the semi-dense depth image and the image to be processed into the semantic encoder, and through a series of dense jump structures, performs multi-scale semantic fusion on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed. In this way, the semantic gap between the feature information of the semantic encoder and the semantic decoder can be reduced to improve the accuracy of image processing and the image quality of the dense depth map obtained after image processing. Based on a two-branch convolutional neural network, the three-dimensional pose image, the semi-dense depth image and the image to be processed are processed to determine the dense depth image corresponding to the image to be processed. In this way, the convolutional neural network can be used to complete the geometrically complex areas in the depth image, improve the accuracy of image processing and the image quality of the dense depth map obtained after image processing.

[0079] In some embodiments, the processing flow of the image processing method is shown as follows: Figure 5 ,like Figure 5 As shown, including:

[0080] Step S501 : Based on the dense skip connection structure, multi-scale semantic fusion is performed on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed.

[0081] In the specific implementation, the semi-dense depth image and the image to be processed are input into the semantic encoder, and multi-scale semantic fusion is performed on the semi-dense depth image and the image to be processed through a series of dense jump connection structures to determine the semantic depth image corresponding to the image to be processed, and the semantic depth image is used as the output of the semantic guidance branch through the semantic decoder.

[0082] Step S502: Based on the depth refinement branch, depth refinement is performed on the semantic depth image and the three-dimensional pose image to determine a dense depth image corresponding to the image to be processed.

[0083] The method of the embodiment of the present application inputs the semi-dense depth image and the image to be processed into the semantic encoder, and through a series of dense jump structures, multi-scale semantic fusion is performed on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed. Based on the depth refinement branch, the semantic depth image and the three-dimensional pose image are deeply refined to determine the dense depth image corresponding to the image to be processed. In this way, the color-dominant information and depth-dominant information of the semantic guidance branch and the depth refinement branch can be fully utilized to reduce the semantic gap between the feature information of the semantic encoder and the semantic decoder, so as to improve the accuracy of image processing and the image quality of the dense depth map obtained after image processing. Based on a dual-branch convolutional neural network, the three-dimensional pose image, the semi-dense depth image and the image to be processed are processed to determine the dense depth image corresponding to the image to be processed. In this way, the convolutional neural network can be used to complete the geometrically complex areas in the depth image, improve the accuracy of image processing and the image quality of the dense depth map obtained after image processing.

[0084] Figure 6 An application scenario diagram of the image processing method provided in an embodiment of the present application is shown.

[0085] refer to Figure 6 An application scenario of the image processing method provided in the embodiment of the present application is used to obtain a dense depth map based on a sparse depth image and a color image. First, a color image is obtained, and a three-dimensional point cloud is projected onto the color image to obtain a sparse depth map. The three-dimensional point cloud is collected outdoors by a three-dimensional imaging sensor.

[0086] The position of the invalid pixel with a pixel value of 0 in the sparse depth map is determined by an adaptive densification module, and then a 5-pixel*5-pixel square area to be supplemented with the position of the invalid pixel as the center is determined through a 5-pixel*5-pixel filter, and the number of valid pixel points in the area to be supplemented is determined. It is judged whether the number of valid pixel points in the area to be supplemented is greater than or equal to a preset number threshold. In response to the number of valid pixel points in the area to be supplemented being greater than or equal to the preset number threshold, the weight of the 5-pixel*5-pixel Euclidean distance weighted kernel function is compared with the 5-pixel*5-pixel square area to be supplemented. The pixel values ​​of the corresponding pixels in the to-be-complemented area of ​​the square of pixels are multiplied to obtain a weighted product, the weighted products of the corresponding pixels in the to-be-complemented area of ​​the square of 5 pixels * 5 pixels are summed to obtain the sum result, and then the sum result is divided by the sum of the weights of the Euclidean distance weighted kernel function of 5 pixels * 5 pixels to obtain the supplementation result, and the supplementation result is used as the pixel value of the invalid value pixel points corresponding to the to-be-complemented area, and the invalid value pixel points corresponding to the to-be-complemented area are supplemented, and it is determined that the sparse depth map after all the invalid value pixels in the sparse depth map are supplemented is a semi-dense depth map.

[0087] Determine the camera intrinsic parameter matrix of the coordinate projection module. Based on the camera intrinsic parameter matrix, the coordinate projection module projects the pixels of the semi-dense depth map onto the X-axis, Y-axis, and Z-axis to determine the three-dimensional pose coordinates corresponding to all the pixels in the semi-dense depth map, and finally determine the three-dimensional pose graph corresponding to the three-dimensional pose coordinates of all the pixels in the semi-dense depth map projected onto the X-axis, Y-axis, and Z-axis.

[0088] The semi-dense depth map and color image are used as the input of the semantic guidance branch, and the semi-dense depth map and color image are input to the semantic encoder. Through a series of dense jump structures, the semi-dense depth map and color image are semantically fused at multiple scales to determine the semantic depth map and semantic fusion features corresponding to the color image. The semantic depth map is output as the semantic guidance branch through the semantic decoder, and the semantic fusion features of each convolutional layer in (1) to (4) of the semantic guidance branch are transmitted to the corresponding convolutional layer in (1) to (4) of the depth refinement branch. The semantic depth map and 3D pose map output by the semantic guidance branch are used as the input of the depth refinement branch, and the semantic depth map and 3D pose map output by the semantic guidance branch are input to the refinement encoder. Through the refinement encoder and refinement decoder, the semantic depth map and 3D pose image are fused with depth information, and the semantic depth map and 3D pose image are deeply refined to determine the dense depth map corresponding to the color image. The dense depth map is output as the depth refinement branch through the refinement decoder.

[0089] Understandably, Figure 6 The application scenarios of the image processing method are only some exemplary implementations in the embodiments of the present application. The application scenarios of the image processing method in the embodiments of the present application include but are not limited to Figure 6 Application scenarios of the image processing method shown.

[0090] Figure 7 Another application scenario diagram of the image processing method provided in an embodiment of the present application is shown.

[0091] refer to Figure 7 Another application scenario of the image processing method provided in the embodiment of the present application is to process a three-dimensional pose graph, a semi-dense depth graph and a color image based on a dual-branch convolutional neural network to determine a dense depth graph corresponding to the color image. The semantic guidance branch may include: a semantic encoder, a semantic decoder and a dense jump connection structure. Among them, X i,j , i is greater than or equal to 0 and less than or equal to 4, j is greater than or equal to 0 and less than or equal to 4. X i,j A semantic guidance node representing a semantic guidance branch. The semantic guidance node may include a residual block. The semantic encoder may include: 0,0 , X 1,0 , X 2,0 , X 3,0 and X 4,0 The semantic decoder may include: X 0,4 , X 1,3 , X 2,0 and X 4,0 The dense jump connection structure may include: 0,0 and X 1,0 Jumping can determine X 0,1 Semantic feature information of X. 0,4 , X 1,3 , X 2,0 , X 4,0 , X 2,1 , X 1,2 , X 0,3 , X 1,1 and X 0,2 Jumping can be performed in a similar manner to determine the semantic feature information of each semantic guide node.

[0092] The deep refinement branch may include: a refinement encoder and a refinement decoder. i and D j , i is greater than or equal to 0 and less than or equal to 4, j is greater than or equal to 0 and less than or equal to 4. i and D j A depth refinement node representing a depth refinement branch. The depth refinement node may include a residual block. The refinement encoder may include: 0 、E 1 、E 2 、E 3 and E 4 . The refinement decoder may include: D0 , D 1 , D 2 , D 3 and D 4 . Figure 7 All solid arrows in the figure represent convolutions, dashed arrows between semantic guidance nodes represent skip connections, and dashed arrows between semantic guidance nodes and depth refinement nodes represent superpositions. Figure 7 All dash-dot arrows in represent deconvolution.

[0093] First, the semi-dense depth map and color image are used as the input of the semantic guidance branch. The semi-dense depth map and color image are input to the semantic encoder through convolution. Through a series of dense skip structures, multi-scale semantic fusion is performed on the semi-dense depth map and color image to determine the semantic depth map and semantic fusion features corresponding to the color image. The semantic depth map is output as the semantic guidance branch through the semantic decoder through convolution, and X 4,0 The semantic fusion features are superimposed and transmitted to E 4 ; X 3,0 and X 3,1 The semantic fusion features are superimposed and transmitted to E 3 ; X 2,0 , X 2,1 and X 2,2 The semantic fusion features are superimposed and transmitted to E 2 ; X 1,0 , X 1,1 , X 1,2 and X 1,3 The semantic fusion features are superimposed and transmitted to E 1 The semantic depth map and the three-dimensional pose map output by the semantic guidance branch are used as the input of the depth refinement branch, and the semantic depth map and the three-dimensional pose map output by the semantic guidance branch are input to the refinement encoder through convolution. Through the refinement encoder, the semantic depth map and the three-dimensional pose image are convolved to fuse the depth information with the semantic information. Through the refinement decoder, the depth information and the semantic information are fused to perform depth refinement in the form of replacing convolution with deconvolution, and the dense depth map corresponding to the color image is determined. The dense depth map is used as the output of the depth refinement branch, and the refinement decoder outputs the dense depth map through convolution.

[0094] Understandably, Figure 7 The application scenarios of the image processing method are only some exemplary implementations in the embodiments of the present application. The application scenarios of the image processing method in the embodiments of the present application include but are not limited to Figure 7 Application scenarios of the image processing method shown.

[0095] Figure 8 Another application scenario diagram of the image processing method provided in an embodiment of the present application is shown.

[0096] refer to Figure 8 Another application scenario of the image processing method provided in the embodiment of the present application is to supplement the area to be supplemented based on the weighted kernel function, and determine that the supplemented sparse depth image is a semi-dense depth image. Figure 8 The 5 pixel * 5 pixel Euclidean distance weighted kernel function is shown, where 0 represents the weight corresponding to the center pixel of the 5 pixel * 5 pixel Euclidean distance weighted kernel function. The center pixel of the Euclidean distance weighted kernel function corresponds to the center of the 5 pixel * 5 pixel square area to be supplemented, where the center of the area to be supplemented is an invalid value pixel. The other values ​​in the 5 pixel * 5 pixel Euclidean distance weighted kernel function also represent the weights corresponding to the pixel points at that position, and the other values ​​in the 5 pixel * 5 pixel Euclidean distance weighted kernel function correspond to the pixel points at the corresponding positions in the 5 pixel * 5 pixel square area to be supplemented.

[0097] As an example, the position of the invalid value pixel point A is determined, and a 5 pixel*5 pixel square area to be supplemented is determined with the position of the invalid value pixel point as the center. The valid value pixels in the area to be supplemented include: valid value pixel point B, valid value pixel point C, and valid value pixel point D. The pixel value of the valid value pixel point B is 25, the pixel value of the valid value pixel point C is 30, and the pixel value of the valid value pixel point D is 20. The weight of the Euclidean distance weighted kernel function corresponding to the valid value pixel point B is 1, the weight of the Euclidean distance weighted kernel function corresponding to the valid value pixel point C is 0.5, and the weight of the Euclidean distance weighted kernel function corresponding to the valid value pixel point D is 0.5. The summation result is 25*1+30*0.5+20*0.5=40; the supplementation result is 40 / (1+0.5+0.5)=20, and the supplementation result 20 is used as the pixel value of the invalid value pixel point A corresponding to the area to be supplemented, and the invalid value pixel point A corresponding to the area to be supplemented is supplemented, and it is determined that the sparse depth image after all the invalid value pixels in the sparse depth image are supplemented is a semi-dense depth image.

[0098] Understandably, Figure 8 The application scenarios of the image processing method are only some exemplary implementations in the embodiments of the present application. The application scenarios of the image processing method in the embodiments of the present application include but are not limited to Figure 8 Application scenarios of the image processing method shown.

[0099] Fig. 9 Another application scenario diagram of the image processing method provided in an embodiment of the present application is shown.

[0100] refer to Fig. 9Another application scenario of the image processing method provided in the embodiment of the present application is applied to estimate a dense depth image from a sparse depth image of a lidar. The sparse depth image may include: a sparse depth image obtained by projecting a three-dimensional point cloud onto the image to be processed. The image to be processed may include: a color image, and the embodiment of the present application does not limit the specific image to be processed. The dense depth image may include: a dense depth map with semantic guidance information and depth information corresponding to the image to be processed. Based on the sparse depth image and the image to be processed, the image processing method of the present application can estimate the dense depth image corresponding to the image to be processed.

[0101] Understandably, Fig. 9 The application scenarios of the image processing method are only some exemplary implementations in the embodiments of the present application. The application scenarios of the image processing method in the embodiments of the present application include but are not limited to Fig. 9 Application scenarios of the image processing method shown.

[0102] The method of the embodiment of the present application is based on a dual-branch convolutional neural network to process a three-dimensional pose image, a semi-dense depth image and an image to be processed, and determine the dense depth image corresponding to the image to be processed. In this way, the color-dominant information and depth-dominant information of the semantic guidance branch and the depth refinement branch can be fully utilized, and the method of fusing the traditional image algorithm with the convolutional neural network is used to adaptively complete the planar area in the depth image, and the dual-branch convolutional neural network is used to complete the geometrically complex area in the depth image. It effectively overcomes the problem of non-robust processing of geometric boundaries in existing image processing methods, and the problem that convolutional neural networks are prone to artifacts and geometric inconsistencies. In this way, the image can be processed intelligently to improve the accuracy of image processing and improve the image quality of the dense depth map obtained after image processing.

[0103] Therefore, compared with the related art in which the image quality obtained after image processing is low and the image processing accuracy is low, the image processing method of the present application can improve the accuracy of image processing and improve the image quality of the dense depth map obtained after image processing.

[0104] The following is a description of an exemplary structure of the image processing device 70 provided in the embodiment of the present application implemented as a software module. In some embodiments, Fig.10As shown, the software modules in the image processing device 70 may include: an acquisition module 701, used to acquire the image to be processed and the sparse depth image corresponding to the image to be processed; an adaptive dense module 702, used to determine the semi-dense depth image corresponding to the sparse depth image based on the sparse depth image; a coordinate projection module 703, used to determine the three-dimensional pose image corresponding to the semi-dense depth image based on the semi-dense depth image; a determination module 704, used to process the three-dimensional pose image, the semi-dense depth image and the image to be processed based on a dual-branch convolutional neural network, and determine the dense depth image corresponding to the image to be processed.

[0105] In some embodiments, the adaptive dense module 702 is used, in the process of determining a semi-dense depth image corresponding to a sparse depth image based on a sparse depth image, to: determine the positions of invalid value pixels in the sparse depth image; determine a square area to be supplemented with a side length of 5 pixels and the position of the invalid value pixel as the center; in response to the number of valid value pixels in the area to be supplemented satisfying a preset number threshold, supplement the area to be supplemented based on a weighted kernel function, and determine that the supplemented sparse depth image is a semi-dense depth image.

[0106] In some embodiments, the coordinate projection module 703 is used to: determine the camera intrinsic parameter matrix; perform three-dimensional pose projection on the semi-dense depth image based on the camera intrinsic parameter matrix to determine the three-dimensional pose coordinates corresponding to the pixel points in the semi-dense depth image; and determine the three-dimensional pose image corresponding to the semi-dense depth image based on the three-dimensional pose coordinates.

[0107] In some embodiments, a dual-branch convolutional neural network includes: a semantic guidance branch and a depth refinement branch; the determination module 704 is used to: based on the semantic guidance branch, perform semantic guidance on the semi-dense depth image and the image to be processed to determine the semantic depth image corresponding to the image to be processed; based on the depth refinement branch, perform depth refinement on the semantic depth image and the three-dimensional pose image to determine the dense depth image corresponding to the image to be processed in the process of processing the three-dimensional pose image, the semi-dense depth image and the image to be processed based on the dual-branch convolutional neural network.

[0108] In some embodiments, the semantic guidance branch includes a dense jump connection structure. The determination module 704, in the process of performing semantic guidance on the semi-dense depth image and the image to be processed based on the semantic guidance branch to determine the semantic depth image corresponding to the image to be processed, is used to: perform multi-scale semantic fusion on the semi-dense depth image and the image to be processed based on the dense jump connection structure to determine the semantic depth image corresponding to the image to be processed.

[0109] It should be noted that the description of the device in the embodiment of the present application is similar to the description of the method embodiment above, and has similar beneficial effects as the method embodiment, so it will not be repeated. Figures 1 to 10 The present invention can be understood by referring to the description of any one of the accompanying drawings.

[0110] According to an embodiment of the present application, the present application also provides an electronic device and a non-transitory computer-readable storage medium.

[0111] Fig.11 A schematic block diagram of an example electronic device 800 that can be used to implement an embodiment of the present application is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workbenches, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present application described and / or required herein.

[0112] like Fig.11 As shown, the electronic device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. In the RAM 803, various programs and data required for the operation of the electronic device 800 can also be stored. The computing unit 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0113] Multiple components in the electronic device 800 are connected to the I / O interface 805, including: an input unit 806, such as a keyboard, a mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a disk, an optical disk, etc.; and a communication unit 809, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 809 allows the electronic device 800 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0114] The computing unit 801 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 801 performs the various methods and processes described above, such as image processing methods. For example, in some embodiments, the image processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 808. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 800 via ROM 802 and / or communication unit 809. When the computer program is loaded into RAM 803 and executed by the computing unit 801, one or more steps of the image processing method described above may be performed. Alternatively, in other embodiments, the computing unit 801 may be configured to perform the image processing method in any other appropriate manner (e.g., by means of firmware).

[0115] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0116] The program code for implementing the method of the present application can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, implements the functions / operations specified in the flow chart and / or block diagram. The program code can be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0117] In the context of the present application, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0118] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0119] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0120] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0121] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution disclosed in this application can be achieved, and this document is not limited here.

[0122] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0123] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An image processing method, characterized in that: The method comprises: Acquire an image to be processed and a sparse depth image corresponding to the image to be processed; Based on the sparse depth image, determining a semi-dense depth image corresponding to the sparse depth image; Based on the semi-dense depth image, determining a three-dimensional pose image corresponding to the semi-dense depth image; Based on a dual-branch convolutional neural network, the three-dimensional pose image, the semi-dense depth image and the image to be processed are processed to determine a dense depth image corresponding to the image to be processed; The dual-branch convolutional neural network includes: a semantic guidance branch and a depth refinement branch; based on the dual-branch convolutional neural network, the three-dimensional pose image, the semi-dense depth image and the image to be processed are processed to determine the dense depth image corresponding to the image to be processed, including: using the semi-dense depth map and the image to be processed as inputs of the semantic guidance branch, inputting the semi-dense depth map and the image to be processed into a semantic encoder, performing multi-scale semantic fusion on the semi-dense depth map and the image to be processed through the dense jump structure included in the semantic guidance branch, and determining the semantic image corresponding to the image to be processed. The depth map and semantic fusion features are used to output the semantic depth map through a semantic decoder as a semantic guidance branch; the semantic depth map and the three-dimensional pose image output by the semantic guidance branch are used as inputs of the depth refinement branch, the semantic depth map and the three-dimensional pose image output by the semantic guidance branch are input into a refinement encoder, and the depth information of the semantic depth map and the three-dimensional pose image are fused through a refinement encoder and a refinement decoder, and the semantic depth map and the three-dimensional pose image are deeply refined to determine a dense depth map corresponding to the image to be processed, and the dense depth map is output as the depth refinement branch through a refinement decoder.

2. The method according to claim 1, characterized in that: The determining, based on the sparse depth image, a semi-dense depth image corresponding to the sparse depth image includes: Determine the position of invalid value pixels in the sparse depth image; Determine a square area to be supplemented with the position of the invalid value pixel as the center and a side length of 5 pixels; In response to the number of effective value pixels in the area to be supplemented meeting a preset number threshold, the area to be supplemented is supplemented based on a weighted kernel function, and the supplemented sparse depth image is determined to be the semi-dense depth image.

3. The method according to claim 1, characterized in that The step of determining, based on the semi-dense depth image, a three-dimensional pose image corresponding to the semi-dense depth image comprises: Determine the camera intrinsic parameter matrix; Based on the camera intrinsic parameter matrix, perform three-dimensional pose projection on the semi-dense depth image to determine the three-dimensional pose coordinates corresponding to the pixel points in the semi-dense depth image; Based on the three-dimensional pose coordinates, the three-dimensional pose image corresponding to the semi-dense depth image is determined.

4. An image processing device, characterized in that: The image processing device comprises: An acquisition module, used to acquire an image to be processed and a sparse depth image corresponding to the image to be processed; An adaptive dense module, configured to determine a semi-dense depth image corresponding to the sparse depth image based on the sparse depth image; A coordinate projection module, configured to determine a three-dimensional pose image corresponding to the semi-dense depth image based on the semi-dense depth image; A determination module is used to process the three-dimensional pose image, the semi-dense depth image and the image to be processed based on a dual-branch convolutional neural network to determine the dense depth image corresponding to the image to be processed; the dual-branch convolutional neural network includes: a semantic guidance branch and a depth refinement branch; the three-dimensional pose image, the semi-dense depth image and the image to be processed based on the dual-branch convolutional neural network to determine the dense depth image corresponding to the image to be processed, including: taking the semi-dense depth map and the image to be processed as inputs of the semantic guidance branch, inputting the semi-dense depth map and the image to be processed into a semantic encoder, and processing the semi-dense depth map and the image to be processed through a dense skip connection structure included in the semantic guidance branch. The dense depth map and the image to be processed are subjected to multi-scale semantic fusion, the semantic depth map and semantic fusion features corresponding to the image to be processed are determined, and the semantic depth map is output as a semantic guidance branch through a semantic decoder; the semantic depth map and the three-dimensional pose image output by the semantic guidance branch are used as inputs of the depth refinement branch, and the semantic depth map and the three-dimensional pose image output by the semantic guidance branch are input into a refinement encoder, and depth information fusion is performed on the semantic depth map and the three-dimensional pose image through a refinement encoder and a refinement decoder, and depth refinement is performed on the semantic depth map and the three-dimensional pose image, and the dense depth map corresponding to the image to be processed is determined, and the dense depth map is output as the depth refinement branch through a refinement decoder.

5. The device according to claim 4, characterized in that The adaptive dense module is used to: Determine the position of invalid value pixels in the sparse depth image; Determine a square area to be supplemented with the position of the invalid value pixel as the center and a side length of 5 pixels; In response to the number of effective value pixels in the area to be supplemented meeting a preset number threshold, the area to be supplemented is supplemented based on a weighted kernel function, and the supplemented sparse depth image is determined to be the semi-dense depth image.

6. The device according to claim 4, characterized in that The coordinate projection module is used to: Determine the camera intrinsic parameter matrix; Based on the camera intrinsic parameter matrix, perform three-dimensional pose projection on the semi-dense depth image to determine the three-dimensional pose coordinates corresponding to the pixel points in the semi-dense depth image; Based on the three-dimensional pose coordinates, the three-dimensional pose image corresponding to the semi-dense depth image is determined.

7. An electronic device, characterized in that: include: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 3.

8. A non-transitory computer-readable storage medium storing computer instructions, characterized in that: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-3.

Images