Depth image completion method, device, equipment and storage medium

Abstract

The application relates to the field of image processing, and discloses a depth image completion method, device and equipment and a storage medium. The method comprises the following steps: obtaining a depth image and a color image of a to-be-completed image; inputting the to-be-completed image into a preset first neural network model to obtain a semantic segmentation result of the to-be-completed image; segmenting the depth image and the color image of the to-be-completed image according to the semantic segmentation result to obtain grouped images of different semantic categories; performing completion calculation on the depth information of the grouped images respectively through a preset second neural network model to obtain depth images of each group, and combining the depth images of each group to obtain a depth image of the to-be-completed image. Grouping convolution is performed according to semantic categories, which is beneficial to separating category interference, improving the reliability of depth completion calculation, and improving the perception accuracy.

Description

Technical Field

[0001] This application relates to the field of image processing, and in particular to a method, apparatus, device and storage medium for depth image completion. Background Technology

[0002] Depth cameras have been widely used in fields such as indoor robotics due to their high measurement accuracy and relatively low cost. However, some depth cameras currently in use, including those based on infrared structured light, laser Time-of-Flight (ToF), or binocular stereo vision, may fail to obtain depth values, resulting in incomplete depth images. In intelligent sensing systems, such as robot navigation and positioning systems and autonomous driving systems, this can easily lead to safety hazards such as missed detections and false alarms.

[0003] To fill in the missing depth information, the proposed solution includes texture consistency-based methods that utilize the neighborhood depth values ​​of invalid points and the texture similarity of the corresponding regions. However, if the texture corresponding to the invalid point region differs from the texture of other regions with depth values, the reliability of the reconstructed depth information is low, which is detrimental to improving the perception accuracy of intelligent sensing systems. Summary of the Invention

[0004] In view of this, embodiments of this application provide a depth image completion method, apparatus, device, and storage medium to solve the problem that when performing depth image completion in the prior art, if the texture corresponding to the invalid point region is different from the texture of other regions with depth values, the reliability of the reconstructed depth information is not high, which is not conducive to improving the perception accuracy of the intelligent perception system.

[0005] A first aspect of this application provides a depth image completion method, the method comprising:

[0006] Obtain the depth image and color image of the image to be completed;

[0007] The image to be completed is input into a preset first neural network model to obtain the semantic segmentation result of the image to be completed;

[0008] Based on the semantic segmentation results, the depth image and the color image of the image to be completed are segmented to obtain grouped images of different semantic categories;

[0009] The depth information of the grouped images is completed by a preset second neural network model to obtain the depth image of each group. The depth images of each group are then combined to obtain the depth image of the image to be completed.

[0010] In conjunction with the first aspect, in a first possible implementation of the first aspect, after inputting the image to be completed into a preset first neural network model to obtain the semantic segmentation result of the image to be completed, the method further includes:

[0011] The edges of the semantic segmentation result are optimized based on the input color image using the convolutional conditional random field method.

[0012] In conjunction with the first aspect, in a second possible implementation of the first aspect, the depth image and the color image of the image to be completed are segmented according to the semantic segmentation result to obtain grouped images of different semantic categories, including:

[0013] The semantic image is determined by identifying the mask composed of pixels of each semantic category based on the labels in the semantic segmentation results;

[0014] The depth image and color image of the image to be completed are segmented using the mask to obtain a semantically category-based segmented depth image and color image;

[0015] Based on the depth image, color image, and semantic image segmented according to the semantic category, grouped images corresponding to the same semantic category are generated.

[0016] In conjunction with the first aspect, in the third possible implementation of the first aspect, before performing depth information completion calculations on the grouped images using a preset second neural network model to obtain the depth images of each group, the method further includes:

[0017] Determine the reliability of depth information in grouped images of one of the semantic categories;

[0018] The loss of pixels in the grouped images whose reliability is lower than a predetermined reliability threshold is set to be less than a first loss value, and the loss of pixels whose reliability is higher than the predetermined reliability threshold is set to be greater than a second loss value, wherein the first loss value is less than the second loss value.

[0019] The second neural network model corresponding to the grouped images is trained based on the set loss value.

[0020] In conjunction with the third possible implementation of the first aspect, in the fourth possible implementation of the first aspect, the loss of pixels in the grouped images whose reliability is lower than a predetermined reliability threshold is set to 0.

[0021] In conjunction with the first aspect, in the fifth possible implementation of the first aspect, before inputting the image to be completed into a preset first neural network model, the method further includes:

[0022] Obtain the depth image from the image to be completed, and acquire the camera parameters of the depth image;

[0023] Based on the camera parameters and the depth image, determine the point cloud image of the image to be completed;

[0024] The step of inputting the image to be completed into a preset first neural network model is: inputting the color image and the point cloud image into the preset first neural network model.

[0025] In conjunction with the fifth possible implementation of the first aspect, in the sixth possible implementation of the first aspect, the camera parameters include focal length and optical center. Determining the point cloud image of the image to be completed based on the camera parameters and the depth image includes:

[0026] According to the formula Determine the point cloud image of the image to be completed, wherein, Let be the focal length in the x-direction. Let be the focal length in the y-direction. Let x and y represent the optical centers in the x and y directions, respectively, and let the coordinates of the pixels in the depth image be... , d The depth is denoted by a pixel, and the point cloud coordinates of a single pixel are given by... .

[0027] A second aspect of this application provides a depth image completion apparatus, the apparatus comprising:

[0028] The image acquisition unit is used to acquire the depth image and color image of the image to be completed;

[0029] A semantic segmentation unit is used to input the image to be completed into a preset first neural network model to obtain the semantic segmentation result of the image to be completed;

[0030] A grouping unit is used to segment the depth image and the color image of the image to be completed based on the semantic segmentation result, so as to obtain grouped images of different semantic categories;

[0031] The completion unit is used to perform completion calculations on the depth information of the grouped images respectively through a preset second neural network model to obtain the depth image of each group, and combine the depth images of each group to obtain the depth image of the image to be completed.

[0032] A third aspect of this application provides a depth image completion device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the method as described in any of the first aspects.

[0033] A fourth aspect of this application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the method as described in any of the first aspects.

[0034] The beneficial effects of this application embodiment compared with the prior art are as follows: This application performs semantic segmentation calculation on the depth image and color image of the image to be completed using a first neural network model to determine the semantic segmentation result in the image to be completed. Based on the semantic segmentation result, the color image and depth image in the image to be completed are segmented to obtain grouped images corresponding to different semantic categories. The grouped images are then completed using a second neural network model to obtain the depth image corresponding to each group. The depth image of the image to be completed is obtained by summing the results. This isolates the categories from each other and prevents interference between categories. As a result, the impact of depth mutation on depth completion calculation can be effectively mitigated, which is beneficial to improving the reliability of depth completion calculation and improving the perception accuracy of depth information. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0036] Figure 1 This is a schematic diagram illustrating the implementation process of a depth image completion method provided in an embodiment of this application;

[0037] Figure 2 This is a schematic diagram of the structure of a semantic segmentation network provided in an embodiment of this application;

[0038] Figure 3 This is a schematic diagram illustrating image segmentation based on semantic categories, provided in an embodiment of this application.

[0039] Figure 4 This is a schematic diagram of the structure of a second neural network model for deep completion provided in an embodiment of this application;

[0040] Figure 5 This is a schematic diagram of a depth image completion device provided in an embodiment of this application;

[0041] Figure 6 This is a schematic diagram of the depth image completion device provided in the embodiments of this application. Detailed Implementation

[0042] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0043] To illustrate the technical solution described in this application, specific embodiments are provided below.

[0044] In intelligent sensing systems, such as robotic systems or autonomous driving systems, depth images acquired by depth cameras may not be perfectly dense. Using incompletely dense depth images for navigation, positioning, or autonomous driving can easily lead to safety accidents during navigation or autonomous driving.

[0045] To improve the accuracy of depth images for robots, researchers have proposed various solutions. One approach is based on texture consistency, which uses the neighborhood depth values ​​of invalid points and the texture similarity of the corresponding regions for filling. However, in many cases, the texture corresponding to an invalid point region may be drastically different from the texture of other regions with depth values. Therefore, this method is unreliable for depth reconstruction of surfaces of some special materials or objects. Another approach is based on depth value interpolation, which interpolates the corresponding depth based on neighborhood depth values. However, this method is not suitable for regions with abrupt changes in depth or large areas without depth values.

[0046] Furthermore, to improve the accuracy of depth images, a joint learning method based on RGB color information for depth estimation and semantic segmentation was proposed. However, the semantic segmentation results of this method are easily affected by color information, leading to semantic classification errors. Moreover, this method heavily relies on the ground truth of the depth image, which may be difficult to obtain in practical applications, making it challenging to use this method for depth completion.

[0047] To address the aforementioned problems, this application proposes a depth image completion method, such as... Figure 1 The diagram shown is a schematic representation of the implementation flow of a depth image completion method provided in an embodiment of this application, which is described in detail below:

[0048] In S101, the depth image and color image of the image to be completed are obtained.

[0049] In this embodiment, the image to be completed is an image that includes lost depth information, i.e., a sparse depth image. When the image is acquired, the depth information in the image can be detected, and based on the detection results, it can be determined whether the acquired image is the image to be completed.

[0050] When inspecting acquired images, the depth information in the image can be detected based on the depth camera's acquisition principle and the corresponding depth loss characteristics. For example, when a depth camera acquires depth images using infrared structured light or laser time-of-flight, the acquired image is compared with the features of black light-absorbing areas and specular reflection areas to determine if any depth information is lost. When using an infrared structured light depth camera, for long-distance measurements, the accuracy of infrared structured light measurements at long distances is not high, and the pixels in the acquired image may be holes or invalid values. The image can be determined to be incomplete based on the distance.

[0051] The image to be completed includes a depth image and a color image. The color image can be a planar color image in RGB format. However, it is not limited to this; it can also be in other formats, such as a color image in YUV format.

[0052] The depth image can be an image in which the position of the pixel in the depth image is determined by the uv coordinate system, and the depth of the pixel is determined by the z value.

[0053] In one possible implementation, the depth image can be converted to obtain a point cloud image corresponding to the image to be converted.

[0054] When converting the depth image into a point cloud image, the conversion calculation can be performed based on the camera parameters and the depth image to determine the point cloud image corresponding to the depth image.

[0055] The conversion calculation formula can be expressed as:

[0056]

[0057] In the above formula, Let be the focal length in the x-direction. Let be the focal length in the y-direction. Let x and y represent the optical centers in the x and y directions, respectively, and let the coordinates of the pixels in the depth image be... , d The depth is denoted by a pixel, and the point cloud coordinates of a single pixel are given by... .

[0058] In S102, the image to be completed is input into a preset first neural network model to obtain the semantic segmentation result of the image to be completed.

[0059] In this embodiment of the application, the depth image and color image of the image to be completed can be input into a preset first neural network model, and the semantic segmentation result of the image to be completed can be obtained by calculation, that is, the semantic category included in the image to be completed.

[0060] For example, if the image to be completed includes semantic categories such as ground, cover, table, chair, and computer, the semantic category corresponding to each pixel can be obtained through recognition processing using the first neural network model. The image to be completed is then segmented into different regions according to semantic categories, and multiple regions corresponding to semantic categories can be used to form a semantic image.

[0061] Or, it can be like Figure 2 The semantic segmentation network shown converts the depth image into a point cloud image (xyz image), combines it with the color image (RGB image), and inputs it into the encoder of the preset first neural network model for encoding and the decoder for decoding to obtain the recognition results of each semantic category. By summing the recognition results of each semantic category, the semantic segmentation result of the image to be completed can be obtained.

[0062] When the point cloud image and the color image are input into the first neural network model, the three-channel point cloud image is cascaded with the three-channel color image to form a six-channel image which is then input into the first neural network model.

[0063] In real-world scenarios, each semantic category often has a certain correspondence with its spatial location. For example, assuming the plane where the ground is located is a spatial coordinate system... shaft and Given a plane defined by axes, the positions of semantic categories such as the ground, walls, tables, chairs, and computers in the spatial coordinate system in a real-world scenario exhibit certain patterns and characteristics. For example, walls are usually perpendicular to the ground; tables and chairs are usually connected to the ground and raised a certain distance above it; computers are usually located on tables, and so on. Based on the above layout relationships of semantic categories, the first neural network model is trained to obtain the trained first neural network model.

[0064] The first neural network is a convolutional neural network model that includes an encoding structure and a decoding structure, including semantic segmentation network models such as ESANet.

[0065] When the input data to the first neural network model includes point cloud images, the encoder structure for depth images in ESANet can be transformed into an encoder structure for point cloud images. For example, the RGB encoder structure of the input image can be changed into a joint RGBXYZ encoder structure.

[0066] When semantic segmentation is performed based on point cloud images and color images, edge modeling issues may arise. To address this problem, this application utilizes the convolutional conditional random field method to perform calculations on the input color image and optimizes the edges of the semantic segmentation result based on the calculation results.

[0067] Through computational processing using the first neural network model, semantic segmentation results at different scales can be obtained. For example, the original image resolution is... The resolution of the output semantic results includes , , , The loss function of the first neural network model can be set as the cross-entropy loss function.

[0068] In S103, the depth image and the color image of the image to be completed are segmented according to the semantic segmentation result to obtain grouped images of different semantic categories.

[0069] In convolution operations on 2D images, the kernel size is typically greater than or equal to 3, which can cause continuous depth changes in regions of abrupt depth changes, resulting in blurred depth image edges. To avoid edge blurring, this application utilizes convolutional conditional random fields to ensure that the edges of semantic segmentation are aligned with the edges of the image, thereby enabling further optimization of the depth image using these optimized semantic edges.

[0070] In this embodiment, upon obtaining the semantic segmentation result, the semantic category to which the pixels in the image to be completed belong can be determined. A mask corresponding to the same semantic category can be generated. Based on the positional information of this mask, the color image and depth image in the image to be completed can be segmented to obtain a depth image, a color image, and a semantic image segmented by semantic categories. The semantic image includes the regions corresponding to each semantic category.

[0071] Specifically, such as Figure 3 The diagram illustrates image segmentation based on semantic categories. This application first utilizes the label information obtained after semantic segmentation to segment the existing color and depth images. For example, by setting all ground pixel positions in the semantic labels to true and others to false, a mask corresponding to the ground region can be created. Using this mask, ground information can be extracted from the depth image and the RGB image. Similarly, depth and RGB information corresponding to all categories can be obtained. The sum of these masks forms the complete depth and color images. After separation, the semantic image, depth image, and color information corresponding to each category are grouped together. Assuming the number of semantic labels is L, the tensor dimension after concatenating these separated groups is 3L, i.e., the number of channels is 3L.

[0072] Regions of the same semantic category in the depth image, color image, and semantic image are merged into a group of images, i.e., grouped images are generated. A grouped image includes only pixels of the same semantic category. After edge optimization of semantic segmentation using the convolutional conditional random field method, the edge division within the same grouped image becomes more accurate.

[0073] In S104, the depth information of the grouped images is completed by a preset second neural network model to obtain the depth image of each group, and the depth images of each group are combined to obtain the depth image of the image to be completed.

[0074] The second neural network model can also employ an encoding-decoding structure. For example... Figure 4 The diagram shows a schematic of a second neural network model for depth completion provided in an embodiment of this application. To avoid interference between semantic categories and to ensure consistency between depth changes and semantic edges, this application performs convolution and deconvolution operations on each grouped image separately, i.e., to implement the encoding and decoding processes separately.

[0075] By using grouped convolution operations, the semantic categories can be kept separate, preventing interference between categories. The depth information optimized by the second neural network model can effectively adapt to the completion requirements of abrupt changes in depth. After decoding the structure, L independent depth images are obtained. These L depth images are then added together based on the previous mask to fuse them into a single overall depth image.

[0076] Furthermore, to achieve self-supervision, the input depth image can be randomly sampled during the training of the second neural network model. The sampled sparse depth image is used as the input depth image to train the second neural network model until a fully dense depth image can be output.

[0077] In this application, the loss function during the training process of the second neural network model can be set to L1 loss. Since the input image is an image to be completed and lacks ground truth values ​​of a completely dense depth image, this application only performs supervised learning on pixels with depth in the input depth image during training. That is, firstly, based on the input depth image, the mask corresponding to the depth value is solved. When solving for the loss value, the loss of pixels without ground truth values ​​is automatically set to be less than the first loss value, such as set to zero, based on the mask. The loss of pixels with ground truth values ​​is set to be greater than the second loss value, and the second loss value is greater than the first loss value.

[0078] For a sparse depth image input to the depth completion module, the actual completion involves adding ground truth depth values. These ground truth depth values ​​can constrain the model's inference and generalization capabilities. Therefore, a fully dense depth image can be obtained during the inference process.

[0079] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0080] Figure 5 This is a schematic diagram of the structure of a depth image completion device provided in an embodiment of this application, as shown below. Figure 5 As shown, the device includes:

[0081] Image acquisition unit 501 is used to acquire the depth image and color image of the image to be completed;

[0082] The semantic segmentation unit 502 is used to input the image to be completed into a preset first neural network model to obtain the semantic segmentation result of the image to be completed;

[0083] Grouping unit 503 is used to segment the depth image and the color image of the image to be completed according to the semantic segmentation result, so as to obtain grouped images of different semantic categories;

[0084] The completion unit 504 is used to perform completion calculations on the depth information of the grouped images respectively through a preset second neural network model to obtain the depth image of each group, and combine the depth images of each group to obtain the depth image of the image to be completed.

[0085] Figure 5 The depth image completion device shown is, with Figure 1 The depth image completion method shown corresponds to this.

[0086] Figure 6 This is a schematic diagram of a depth image completion device provided in an embodiment of this application. For example... Figure 6 As shown, the depth image completion device 6 of this embodiment includes: a processor 60, a memory 61, and a computer program 62 stored in the memory 61 and executable on the processor 60, such as a depth image completion program. When the processor 60 executes the computer program 62, it implements the steps in the various depth image completion method embodiments described above. Alternatively, when the processor 60 executes the computer program 62, it implements the functions of each module / unit in the various device embodiments described above.

[0087] For example, the computer program 62 may be divided into one or more modules / units, which are stored in the memory 61 and executed by the processor 60 to complete this application. The one or more modules / units may be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 62 in the depth image completion device 6.

[0088] The depth image completion device may include, but is not limited to, a processor 60 and a memory 61. Those skilled in the art will understand that... Figure 6 This is merely an example of the depth image completion device 6 and does not constitute a limitation on the depth image completion device 6. It may include more or fewer components than shown, or combine certain components, or different components. For example, the depth image completion device may also include input / output devices, network access devices, buses, etc.

[0089] The processor 60 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0090] The memory 61 can be an internal storage unit of the depth image completion device 6, such as a hard disk or RAM of the depth image completion device 6. The memory 61 can also be an external storage device of the depth image completion device 6, such as a plug-in hard disk, Smart Media Card (SMC), Secure Digital (SD) card, or Flash Card equipped on the depth image completion device 6. Furthermore, the memory 61 can include both internal and external storage units of the depth image completion device 6. The memory 61 is used to store the computer program and other programs and data required by the depth image completion device. The memory 61 can also be used to temporarily store data that has been output or will be output.

[0091] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0092] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0093] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0094] In the embodiments provided in this application, it should be understood that the disclosed devices / terminal equipment and methods can be implemented in other ways. For example, the device / terminal equipment embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the displayed or discussed mutual coupling or direct coupling or communication connection may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0095] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0096] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0097] If the integrated module / unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments can also be implemented by hardware related to computer program instructions. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include: any entity or device capable of carrying the computer program code, recording media, USB flash drives, portable hard drives, magnetic disks, optical disks, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signals, telecommunication signals, and software distribution media, etc. It should be noted that the content included in the computer-readable medium can be appropriately added or removed according to the requirements of legislation and patent practice in the jurisdiction. For example, in some jurisdictions, according to legislation and patent practice, computer-readable media do not include electrical carrier signals and telecommunication signals.

[0098] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A depth image completion method, characterized in that, The method includes: Obtain the depth image and color image of the image to be completed; The image to be completed is input into a preset first neural network model to obtain the semantic segmentation result of the image to be completed; Based on the semantic segmentation results, the depth image and the color image of the image to be completed are segmented to obtain grouped images of different semantic categories. The grouped images include a group of images in which the region of the same semantic category in the depth image, the region in the color image, and the region in the semantic image are merged. The depth information of the grouped images is completed by a preset second neural network model to obtain the depth image of each group. The depth images of each group are then combined to obtain the depth image of the image to be completed.

2. The method according to claim 1, characterized in that, After inputting the image to be completed into a preset first neural network model to obtain the semantic segmentation result of the image to be completed, the method further includes: The edges of the semantic segmentation result are optimized based on the input color image using the convolutional conditional random field method.

3. The method according to claim 1, characterized in that, Based on the semantic segmentation results, the depth image and color image of the image to be completed are segmented to obtain grouped images of different semantic categories, including: The semantic image is determined by identifying the mask composed of pixels of each semantic category based on the labels in the semantic segmentation results; The depth image and color image of the image to be completed are segmented using the mask to obtain a semantically category-based segmented depth image and color image; Based on the depth image, color image, and semantic image segmented according to the semantic category, grouped images corresponding to the same semantic category are generated.

4. The method according to claim 1, characterized in that, Before performing depth information completion calculations on the grouped images using a preset second neural network model to obtain the depth images of each group, the method further includes: Determine the reliability of depth information in grouped images of one of the semantic categories; The loss of pixels in the grouped images whose reliability is lower than a predetermined reliability threshold is set to be less than a first loss value, and the loss of pixels whose reliability is higher than the predetermined reliability threshold is set to be greater than a second loss value, wherein the first loss value is less than the second loss value. The second neural network model corresponding to the grouped images is trained based on the set loss value.

5. The method according to claim 4, characterized in that, The loss of pixels in the grouped images whose reliability is lower than a predetermined reliability threshold is set to 0.

6. The method according to claim 1, characterized in that, Before inputting the image to be completed into a preset first neural network model, the method further includes: Obtain the depth image from the image to be completed, and acquire the camera parameters of the depth image; Based on the camera parameters and the depth image, determine the point cloud image of the image to be completed; The step of inputting the image to be completed into a preset first neural network model is: inputting the color image and the point cloud image into the preset first neural network model.

7. The method according to claim 6, characterized in that, The camera parameters include focal length and optical center. Based on the camera parameters and the depth image, the point cloud image of the image to be completed is determined, including: According to the formula Determine the point cloud image of the image to be completed, wherein, Let be the focal length in the x-direction. Let be the focal length in the y-direction. Let x and y represent the optical centers in the x and y directions, respectively, and let the coordinates of the pixels in the depth image be... , d The depth is denoted by a pixel, and the point cloud coordinates of a single pixel are given by... .

8. A depth image completion device, characterized in that, The device includes: The image acquisition unit is used to acquire the depth image and color image of the image to be completed; A semantic segmentation unit is used to input the image to be completed into a preset first neural network model to obtain the semantic segmentation result of the image to be completed; A grouping unit is used to segment the depth image and the color image of the image to be completed according to the semantic segmentation result, so as to obtain grouped images of different semantic categories. The grouped images include a group of images in which the same semantic category is located in the depth image, in the color image, and in the semantic image. The completion unit is used to perform completion calculations on the depth information of the grouped images respectively through a preset second neural network model to obtain the depth image of each group, and combine the depth images of each group to obtain the depth image of the image to be completed.

9. A depth image completion device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method as described in any one of claims 1 to 7.

Images