Method, device and mobile robot for denoising and obstacle detection of depth images

Abstract

A method, device and mobile robot for denoising and obstacle detection of a depth image, the method comprising: obtaining multi-modal images collected for a same scene, the multi-modal images comprising a depth image and at least one other modality image; inputting the depth image and the at least one other modality image into a trained neural network respectively, predicting a noise region in the depth image based on fusion of features of different modality images by the neural network; removing the noise region in the depth image based on a prediction result of the neural network, obtaining and outputting an optimized depth image. The method and device for denoising of a depth image can recognize a noise region on the depth image by fusing the depth image and other image sources through the trained neural network, can denoise the depth image, and thus optimize the depth image.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and more specifically to a method, apparatus, and mobile robot for denoising and obstacle detection of depth images. Background Technology

[0002] Currently, depth images captured by depth cameras, especially indoor depth images, often exhibit bright areas under extreme lighting conditions, such as direct light, ground reflections, and highly reflective materials. These bright areas are often accompanied by erroneous depth values ​​and are therefore referred to as noise areas. Such depth images cannot accurately reflect the depth information of the captured scene. Performing other operations based on such depth maps, such as obstacle detection, can easily lead to incorrect detection results and false obstacle avoidance.

[0003] Therefore, a noise reduction scheme for depth images is needed to solve the above problems. Summary of the Invention

[0004] According to one aspect of this application, a method for denoising depth images is provided. The method includes: acquiring multimodal images collected for the same scene, the multimodal images including a depth image and at least one other modal image; inputting the depth image and the at least one other modal image into a trained neural network respectively, and having the neural network predict noise regions in the depth image based on the fusion of features of different modal images; removing the noise regions in the depth image based on the prediction results of the neural network, and obtaining and outputting an optimized depth image.

[0005] In one embodiment of this application, the neural network utilizes specific candidate anchor boxes when predicting the noise region, wherein the specific candidate anchor boxes are determined based on the source of the noise region in the depth image.

[0006] In one embodiment of this application, when the source of the noise region in the depth image includes an artificial light source, the specific candidate anchor box includes at least one of a circular candidate anchor box, an elliptical candidate anchor box, and a rectangular candidate anchor box.

[0007] In one embodiment of this application, the step of predicting noise regions in the depth image by fusing features of different modal images using the neural network includes: extracting features from the depth image and the at least one other modal image by the backbone network module of the neural network; fusing the features extracted by the backbone network module by the fusion network module of the neural network; and predicting noise regions in the depth image by the head network module of the neural network based on the features fused by the fusion network module.

[0008] In one embodiment of this application, the backbone network module includes a first backbone network module and a second backbone network module, wherein: the first backbone network module is used to encode the depth image; the second backbone network module is used to encode the other modal images; and the first backbone network module and the second backbone network module share weights.

[0009] In one embodiment of this application, the other modal images include at least one of infrared images, color images, and grayscale images.

[0010] In one embodiment of this application, the scene includes an indoor scene.

[0011] According to another aspect of this application, a noise reduction apparatus for a depth image is provided. The apparatus includes a memory and a processor. The memory stores a computer program that is executed by the processor. When the computer program is executed by the processor, it causes the processor to perform the noise reduction method for the depth image as described above.

[0012] According to another aspect of this application, an obstacle detection method is provided, the method comprising: acquiring a depth image of a scene to be detected, wherein the depth image is an optimized depth image obtained by the aforementioned depth image denoising method; performing obstacle detection on the scene based on the optimized depth image to obtain an obstacle detection result for the scene.

[0013] According to another aspect of this application, an obstacle detection device is provided, the device including a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the obstacle detection method as described above.

[0014] According to another aspect of this application, a mobile robot is provided, the mobile robot including an image acquisition device and an obstacle detection device, wherein: the image acquisition device is used to acquire images of a region to be moved by the mobile robot, the images including depth images and at least one other modal image; the obstacle detection device is used to perform obstacle detection based on the images acquired by the image acquisition device, so as to avoid obstacles during the movement of the mobile robot, wherein the obstacle detection device includes the obstacle detection device described above.

[0015] According to another aspect of this application, a storage medium is provided, wherein a computer program executed by a processor is stored on the storage medium, and the computer program, when executed by the processor, causes the processor to perform the depth image noise reduction method as described above or to perform the obstacle detection method as described above.

[0016] The depth image denoising method and apparatus according to embodiments of this application identify noisy regions in a depth image by fusing a trained neural network with a depth image and other image sources, thereby denoising and optimizing the depth image. The obstacle detection method and apparatus according to embodiments of this application identify noisy regions in a depth image by fusing a trained neural network with a depth image and other image sources, and remove the noisy regions from the depth image to obtain an optimized depth image for obstacle detection. This effectively reduces or even avoids false obstacle detections, thereby improving the accuracy of obstacle detection. The mobile robot according to embodiments of this application performs obstacle detection based on the above-described obstacle detection apparatus, effectively reducing or even avoiding false obstacle detections, improving the accuracy of obstacle detection, and thus improving obstacle avoidance accuracy. Attached Figure Description

[0017] The above and other objects, features, and advantages of this application will become more apparent from the more detailed description of the embodiments of this application in conjunction with the accompanying drawings. The drawings are provided to further illustrate the embodiments of this application and form part of the specification. They are used together with the embodiments of this application to explain this application and do not constitute a limitation thereof. In the drawings, the same reference numerals generally represent the same components or steps.

[0018] Figure 1 A schematic flowchart illustrating a method for denoising depth images according to an embodiment of this application is shown.

[0019] Figure 2 A schematic structural block diagram of a noise reduction apparatus for depth images according to an embodiment of this application is shown.

[0020] Figure 3 A schematic flowchart of an obstacle detection method according to an embodiment of this application is shown.

[0021] Figure 4 A schematic structural block diagram of an obstacle detection device according to an embodiment of this application is shown.

[0022] Figure 5 A schematic structural block diagram of a mobile robot according to an embodiment of this application is shown. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely a part of the embodiments of this application, and not all of the embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0024] First, refer to Figure 1 This describes a depth image noise reduction method 100 according to embodiments of this application. For example... Figure 1 As shown, the depth image denoising method 100 may include the following steps:

[0025] In step S110, multimodal images of the same scene are acquired, including depth images and at least one other modal image.

[0026] In step S120, the depth image and at least one other modal image are respectively input into the trained neural network, and the neural network predicts the noise region in the depth image based on the fusion of features of different modal images.

[0027] In step S130, noise regions in the depth image are removed based on the prediction results of the neural network, and an optimized depth image is obtained and output.

[0028] In the embodiments of this application, depth images and other image sources are fused to identify erroneous information (noisy regions) in the depth image. Erroneous information (i.e., erroneous depth values) in depth images, especially those of indoor scenes, is often caused by highly reflective, bright objects, including but not limited to onboard lights, reflections from floor tiles, and highly reflective pillars. These erroneous depth values ​​often differ significantly from surrounding depth values ​​(extremely far away, forming holes in the depth image; or extremely close, forming peaks in the depth image). In other modalities of imagery, such as infrared, color, or grayscale images, the intensity values ​​of these regions are extremely strong. Therefore, this information can be well modeled by deep neural networks, allowing for the detection of these regions using multimodal deep neural networks.

[0029] Based on this, in the embodiments of this application, different modal images of the same scene can be labeled and used as sample images to train the neural network. Specifically, for example, depth images and other modal images (such as at least one of infrared images, color images, or grayscale images) of scene A are collected, and noisy regions in the depth images are labeled, as well as regions with extremely strong intensity values ​​corresponding to the noisy regions in other modal images; similarly, depth images and other modal images of other scenes such as scenes B, C, D, etc. are collected and labeled respectively; finally, these labeled images are used as sample images and input into the neural network for training, so that the neural network can predict the noisy regions in the depth images after fusing the features of depth images and other modal images of the same scene. The parameters of the neural network are continuously optimized according to the difference between the output (prediction result) and the label (real result). When the difference meets the preset convergence condition, the trained neural network is obtained.

[0030] Based on a trained neural network, in practical applications, multimodal images (i.e., images of various modalities) are acquired for the same scene. The acquired images include not only depth images but also at least one other modal image, such as infrared images, color images, or grayscale images (this depends on which modal images(s) are used during training; in practical applications, the modality of the acquired images is consistent with the modality of the sample images used during training). Images of different modalities are input into the trained neural network, which extracts features from each modal image and performs feature fusion. Based on the fused features, noise regions in the depth image are predicted. Removing these noise regions yields the denoised depth image, i.e., the optimized depth image. Therefore, the depth image denoising method according to the embodiments of this application identifies noise regions in the depth image by fusing a trained neural network with the depth image and other image sources, thereby denoising and optimizing the depth image.

[0031] In embodiments of this application, the neural network used in step S120 may include a backbone network module, a fusion network module, and a head network module. Based on this, the step S120, in which the neural network predicts the noise region in the depth image by fusing features from different modal images, may include: the backbone network module extracting features from the depth image and at least one other modal image; the fusion network module fusing the features extracted by the backbone network module; and the head network module predicting the noise region in the depth image based on the features fused by the fusion network module.

[0032] In one example, the backbone network module of the aforementioned neural network may further include a first backbone network module and a second backbone network module, wherein: the first backbone network module is used to encode the depth image to obtain features extracted from the depth image; the second backbone network module is used to encode other modal images to obtain features extracted from other modal images; the first and second backbone network modules share weights, that is, the weights of the two backbone network modules are consistent. Using different backbone network modules with shared weights to encode the depth image and other modal images separately can improve encoding efficiency. In other examples, the same backbone network module can also be used to encode the depth image and other modal images separately to obtain their respective features.

[0033] In the embodiments of this application, the backbone network module encodes the two types of data and then inputs them into the fusion network module for fusion. During neural network training, the fusion network module restricts the two information sources to contain significantly different information (low cosine similarity) to ensure that they provide complementary information. Furthermore, due to the differentiability of the cosine function, the neural network can be trained end-to-end.

[0034] In the embodiments of this application, the aforementioned neural network (the head network module) utilizes specific candidate bounding boxes when predicting noisy regions in a depth image. These specific candidate bounding boxes are determined based on the source of the noise regions in the depth image. In this embodiment, instead of simply using traditional rectangular boxes as detection candidate bounding boxes, specific candidate bounding boxes are determined based on the source characteristics of the noise regions in the depth image, thus making it more advantageous to accurately detect noisy regions in the depth image.

[0035] In one example, the sources of noise regions in a depth image include artificial light sources (i.e., the scene of the depth image may be an indoor scene). Since the high-reflectivity regions (noise regions) generated by artificial light sources are mostly circular, elliptical, or rectangular in shape, based on this prior knowledge, the candidate anchor boxes of the neural network can be specified as at least one of circular, elliptical, or rectangular candidate anchor boxes, thereby significantly improving the detection accuracy and precision of noise regions in depth images.

[0036] The above description exemplarily illustrates a depth image denoising method 100 according to an embodiment of this application. Based on the above description, the depth image denoising method 100 according to an embodiment of this application identifies noise regions on the depth image by fusing the depth image with a trained neural network and other image sources, thereby denoising the depth image and optimizing the depth image.

[0037] The following is combined Figure 2 Describes a noise reduction apparatus 200 for depth images provided in another aspect of this application. Figure 2 A schematic structural block diagram of a depth image noise reduction apparatus 200 according to an embodiment of this application is shown. Figure 2 As shown, the depth image denoising apparatus 200 according to an embodiment of this application may include a memory 210 and a processor 220. The memory 210 stores a computer program executed by the processor 220. When the computer program is executed by the processor 220, it causes the processor 220 to perform the depth image denoising method 100 described above according to an embodiment of this application. Those skilled in the art can understand the specific operation of the depth image denoising apparatus 200 according to the embodiments of this application in conjunction with the foregoing description. For the sake of brevity, specific details will not be repeated here, only some main operations of the processor 220 will be described.

[0038] In one embodiment of this application, when the computer program is run by the processor 220, the processor 220 performs the following steps: acquiring multimodal images collected for the same scene, the multimodal images including a depth image and at least one other modal image; inputting the depth image and at least one other modal image into a trained neural network respectively, and having the neural network predict noise regions in the depth image based on the fusion of features of different modal images; removing noise regions in the depth image based on the prediction results of the neural network, and obtaining and outputting an optimized depth image.

[0039] In one embodiment of this application, the neural network utilizes specific candidate anchor boxes when predicting noisy regions, wherein the specific candidate anchor boxes are determined based on the source of the noisy regions in the depth image.

[0040] In one embodiment of this application, when the source of noise regions in a depth image includes an artificial light source, a specific candidate anchor box includes at least one of a circular candidate anchor box, an elliptical candidate anchor box, and a rectangular candidate anchor box.

[0041] In one embodiment of this application, when the computer program is run by the processor 220, the processor 220 executes the following: the prediction of noise regions in a depth image by fusing features of different modal images from a neural network, comprising: extracting features from the depth image and at least one other modal image by the backbone network module of the neural network; fusing the features extracted by the backbone network module by the fusion network module of the neural network; and predicting noise regions in the depth image by the head network module of the neural network based on the features fused by the fusion network module.

[0042] In one embodiment of this application, the backbone network module includes a first backbone network module and a second backbone network module, wherein: the first backbone network module is used to encode depth images; the second backbone network module is used to encode other modal images; and the first backbone network module and the second backbone network module share weights.

[0043] In one embodiment of this application, other modal images include at least one of infrared images, color images, and grayscale images.

[0044] In one embodiment of this application, the scene includes an indoor scene.

[0045] Based on the above description, the depth image denoising device 200 according to the embodiments of this application identifies noise regions on the depth image by fusing the depth image with a trained neural network and other image sources, thereby denoising the depth image and optimizing the depth image.

[0046] The following is combined Figure 3 Describes an obstacle detection method provided according to another aspect of this application. Figure 3 A schematic flowchart of an obstacle detection method 300 according to an embodiment of this application is shown. Figure 3 As shown, obstacle detection method 300 may include the following steps:

[0047] In step S310, multimodal images of the same scene are acquired, including depth images and at least one other modal image.

[0048] In step S320, the depth image and at least one other modal image are respectively input into the trained neural network, and the neural network predicts the noise region in the depth image based on the fusion of features of different modal images;

[0049] In step S330, noise regions in the depth image are removed based on the prediction results of the neural network to obtain an optimized depth image;

[0050] In step S340, obstacle detection is performed on the scene based on the optimized depth image to obtain the obstacle detection results of the scene.

[0051] In the embodiments of this application, depth images and other image sources are fused to identify erroneous information (noisy regions) in the depth image, and an optimized depth image is obtained by removing the noisy regions (the same as the depth image denoising method 100 described above). Obstacle detection is then performed based on the optimized depth image. Erroneous information (i.e., erroneous depth values) in depth images, especially those of indoor scenes, is often caused by highly reflective and bright objects, including but not limited to onboard lights, reflections from floor tiles, and highly reflective pillars. These erroneous depth values ​​often differ significantly from surrounding depth values ​​(extremely far, forming holes in the depth image; or extremely close, forming peaks in the depth image). In other modalities of images, such as infrared, color, or grayscale images, the intensity values ​​of these regions are extremely strong. Therefore, this information can be well modeled by deep neural networks, allowing multimodal deep neural networks to detect these regions. After removing the detected noisy regions, an optimized depth image is obtained. Obstacle detection based on such a depth image can effectively reduce or even avoid false obstacle detections, thereby improving the accuracy of obstacle detection.

[0052] In embodiments of this application, the neural network used in step S320 may include a backbone network module, a fusion network module, and a head network module. Based on this, the prediction of noise regions in the depth image by the neural network based on the fusion of features from different modal images in step S320 may include: the backbone network module of the neural network extracting features from the depth image and at least one other modal image; the fusion network module of the neural network fusing the features extracted by the backbone network module; and the head network module of the neural network predicting noise regions in the depth image based on the features fused by the fusion network module.

[0053] In one example, the backbone network module of the aforementioned neural network may further include a first backbone network module and a second backbone network module, wherein: the first backbone network module is used to encode the depth image to obtain features extracted from the depth image; the second backbone network module is used to encode other modal images to obtain features extracted from other modal images; the first and second backbone network modules share weights, that is, the weights of the two backbone network modules are consistent. Using different backbone network modules with shared weights to encode the depth image and other modal images separately can improve encoding efficiency. In other examples, the same backbone network module can also be used to encode the depth image and other modal images separately to obtain their respective features.

[0054] In the embodiments of this application, the backbone network module encodes the two types of data and then inputs them into the fusion network module for fusion. During neural network training, the fusion network module restricts the two information sources to contain significantly different information (low cosine similarity) to ensure that they provide complementary information. Furthermore, due to the differentiability of the cosine function, the neural network can be trained end-to-end.

[0055] In the embodiments of this application, the aforementioned neural network (the head network module) utilizes specific candidate anchor boxes when predicting noisy regions in a depth image. These specific candidate anchor boxes are determined based on the source of the noise regions in the depth image. In this embodiment, instead of simply using traditional rectangular boxes as detection candidate anchor boxes, specific candidate anchor boxes are determined based on the source characteristics of the noise regions in the depth image, thus making it more conducive to accurately detecting noisy regions in the depth image.

[0056] In one example, the sources of noise regions in a depth image include artificial light sources (i.e., the scene of the depth image may be an indoor scene). Since the high-reflectivity regions (noise regions) generated by artificial light sources are mostly circular, elliptical, or rectangular in shape, based on this prior knowledge, the candidate anchor boxes of the neural network can be specified as at least one of circular, elliptical, or rectangular candidate anchor boxes, thereby significantly improving the detection accuracy and precision of noise regions in depth images.

[0057] In one example, based on an optimized depth image, the pose and size of obstacles in the scene in three-dimensional space can be obtained by combining the intrinsic and extrinsic parameters of the camera that captured the depth image, thereby obtaining the obstacle detection result.

[0058] The obstacle detection method 300 according to an embodiment of this application has been described above by way of example. Based on the above description, the obstacle detection method 300 according to the embodiment of this application identifies noisy regions on the depth image by fusing a trained neural network with a depth image and other image sources, and removes the noisy regions on the depth image to obtain an optimized depth image for obstacle detection. This can effectively reduce or even avoid the occurrence of false obstacle detection, thereby improving the accuracy of obstacle detection.

[0059] The following is combined Figure 4 This application describes an obstacle detection device 400 provided in another aspect. Figure 4 A schematic structural block diagram of an obstacle detection device 400 according to an embodiment of this application is shown. Figure 4As shown, the obstacle detection device 400 according to an embodiment of this application may include a memory 410 and a processor 420. The memory 410 stores a computer program executed by the processor 420. When the computer program is executed by the processor 420, the processor 420 performs the obstacle detection method 300 described above according to an embodiment of this application. Those skilled in the art can understand the specific operation of the obstacle detection device 400 according to the embodiments of this application in conjunction with the foregoing description; for the sake of brevity, it will not be described again here.

[0060] The following is combined Figure 5 This application further describes the mobile robot 500 provided in another aspect. Figure 5 A schematic structural block diagram of a mobile robot 500 according to an embodiment of this application is shown. Figure 5 As shown, the mobile robot 500 according to an embodiment of this application may include an image acquisition device 510 and an obstacle detection device 520. The image acquisition device 510 is used to acquire images of the area to be moved by the mobile robot 500, including depth images and at least one other modal image. The obstacle detection device 520 is used to detect obstacles based on the images acquired by the image acquisition device 510, for obstacle avoidance during the movement of the mobile robot 500. The obstacle detection device 520 may be the aforementioned obstacle detection device 400. Those skilled in the art can understand the specific operation of the obstacle detection device 520 in conjunction with the foregoing description; for brevity, specific details are not repeated here.

[0061] In the embodiments of this application, the image acquisition device 510 of the mobile robot 500 acquires multimodal images of the same scene. The acquired images include not only depth images but also at least one other modal image, such as an infrared image, a color image, or a grayscale image. Here, the image acquisition device 510 of the mobile robot 500 may include a camera capable of providing multimodal images, or may include different cameras to provide images of different modalities. Images of different modalities are input into the obstacle detection device 520. The obstacle detection device 520 extracts features from the different modal images and performs feature fusion. Based on the fused features, it predicts noise regions in the depth image, removes the noise regions to obtain an optimized depth image, performs obstacle detection based on the optimized depth image, and avoids obstacles during movement based on the obstacle detection results.

[0062] As mentioned earlier, errors in depth images, especially those of indoor scenes, are often caused by highly reflective and bright objects, including but not limited to onboard lights, reflections from floor tiles, and highly reflective pillars. These erroneous depth values ​​often differ significantly from their surroundings (extremely far away, forming holes in the depth image; or extremely close, forming peaks). In other modalities of imagery, such as infrared, color, or grayscale images, the intensity values ​​of these regions are extremely strong. Therefore, this information can be well modeled by deep neural networks, allowing the detection of these regions using a multimodal deep neural network (including in the obstacle detection device 520). After removing the detected noise regions, an optimized depth image is obtained. Obstacle detection based on such a depth image can effectively reduce or even avoid false obstacle detections, improving the accuracy of obstacle detection and thus improving obstacle avoidance accuracy.

[0063] Furthermore, according to embodiments of this application, a storage medium is also provided, on which program instructions are stored. When executed by a computer or processor, these program instructions are used to perform corresponding steps of the depth image noise reduction method or obstacle detection method of the embodiments of this application. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0064] Based on the above description, the depth image denoising method and apparatus according to embodiments of this application identify noisy regions on the depth image by fusing a trained neural network with the depth image and other image sources, thereby denoising the depth image and optimizing it. The obstacle detection method and apparatus according to embodiments of this application identify noisy regions on the depth image by fusing a trained neural network with the depth image and other image sources, and remove the noisy regions from the depth image to obtain an optimized depth image for obstacle detection. This effectively reduces or even avoids false obstacle detections, thereby improving the accuracy of obstacle detection. The mobile robot according to embodiments of this application performs obstacle detection based on the above-described obstacle detection apparatus, effectively reducing or even avoiding false obstacle detections, improving the accuracy of obstacle detection, and thus improving obstacle avoidance accuracy.

[0065] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0066] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0067] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0068] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0069] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0070] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0071] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features included in other embodiments but not others, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0072] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0073] It should be noted that the above embodiments are illustrative of this application and not restrictive, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "a" or "an" preceding an element does not exclude the presence of a plurality of such elements. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0074] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A method for denoising depth images, characterized in that, The method includes: Acquire multimodal images of the same scene, wherein the multimodal images include depth images and at least one other modal image; The depth image and the at least one other modal image are respectively input into a trained neural network. The neural network predicts noise regions in the depth image based on the fusion of features from the different modal images. The noise regions represent erroneous information in the depth image. The neural network uses specific candidate anchor boxes when predicting the noise regions, and these specific candidate anchor boxes are determined based on the source of the noise regions in the depth image. The neural network includes a backbone network module, a fusion network module, and a head network module. The backbone network module is configured to extract features from the depth image and the at least one other modal image. The fusion network module is configured to fuse the features extracted by the backbone network module, and the fusion network module can limit the information sources corresponding to the depth image and the at least one other modal image to contain differing information. The head network module is configured to predict noise regions in the depth image based on the features fused by the fusion network module. The noise region in the depth image is removed based on the prediction result of the neural network, and an optimized depth image is obtained and output.

2. The method according to claim 1, characterized in that, When the source of noise regions in the depth image includes artificial light sources, the specific candidate anchor boxes include at least one of circular candidate anchor boxes, elliptical candidate anchor boxes, and rectangular candidate anchor boxes.

3. The method according to claim 1, characterized in that, The backbone network module includes a first backbone network module and a second backbone network module, wherein: The first backbone network module is used to encode the depth image; The second backbone network module is used to encode the other modal images; The first backbone network module and the second backbone network module share weights.

4. The method according to any one of claims 1-2, characterized in that, The other modal images include at least one of infrared images, color images, and grayscale images.

5. The method according to any one of claims 1-2, characterized in that, The scenario includes indoor scenarios.

6. A noise reduction device for depth images, characterized in that, The apparatus includes a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform a noise reduction method for a depth image as described in any one of claims 1-5.

7. An obstacle detection method, characterized in that, The method includes: Acquire a depth image of the scene to be obstacle detected, wherein the depth image is an optimized depth image obtained by the depth image denoising method according to any one of claims 1-5; Obstacle detection is performed on the scene based on the optimized depth image to obtain the obstacle detection results for the scene.

8. An obstacle detection device, characterized in that, The device includes a memory and a processor, the memory storing a computer program executed by the processor, the computer program, when executed by the processor, causing the processor to perform the obstacle detection method as described in claim 7.

9. A mobile robot, characterized in that, The mobile robot includes an image acquisition device and an obstacle detection device, wherein: The image acquisition device is used to acquire images of the area to be moved by the mobile robot, the images including depth images and at least one other modal image; The obstacle detection device is used to detect obstacles based on the images acquired by the image acquisition device, so as to avoid obstacles during the movement of the mobile robot, wherein the obstacle detection device includes the obstacle detection device according to claim 8.

10. A storage medium, characterized in that, The storage medium stores a computer program executed by a processor, which, when executed by the processor, causes the processor to perform a noise reduction method for a depth image as described in any one of claims 1-5 or an obstacle detection method as described in claim 7.

Images