A depth camera module with dirt detection function and robot

Abstract

A kind of depth camera module with dirt detection function, including laser emitter, for switchably emitting structured light or floodlight;Receiver, for receiving the reflection signal of the structured light or the floodlight;Controller, for controlling the laser emitter emits structured light and floodlight in a certain proportion, the receiver receives the first signal of adjacent structured light and the second signal of the floodlight, respectively generates first depth map and second depth map, obtains first area N1 with depth value less than N on the first depth map, obtains second area N2 with depth value less than N on the second depth map, according to the ratio of signal average intensity on the first area N1, the second area N2 judges whether there is dirt on lens or not.The present application does not need additional device, can carry out dirt detection at any time, and does not affect data acquisition, with very high efficiency and convenience.

Description

Technical Field

[0001] The present invention relates to the technical field of depth cameras, and in particular to a depth camera module and a robot with a dirt detection function. Background Art

[0002] Depth cameras are the eyes of many AI devices, such as robots, and are crucial for the efficient operation of various smart devices. Due to their unique characteristics, depth cameras are often damaged by environmental factors, such as water stains, dust, and scratches. When the depth camera lens is located on the outermost surface, it is susceptible to damage. When a depth camera has a protective lens on the outside, the protective lens is also susceptible to damage. Damage to both the depth camera lens and the protective lens will affect the data captured by the depth camera, making it crucial to promptly identify such contamination.

[0003] In the prior art, additional detection devices are often used to detect dirt on the depth camera lens or the protective lens.

[0004] For example, an invention discloses a dirt monitoring system for a transmitting module, a depth camera, an intelligent terminal, a dirt detection method and a computer-readable storage medium. The dirt monitoring system includes a light source, an optical element, a detection element and a processor. The light source is used to emit a light signal. The optical element is located on the projection path of the light source. The detection element generates a photocurrent after receiving the light signal reflected by the optical element. The processor is used to calculate the reflectivity of the optical element to the light signal based on the photocurrent. When the reflectivity of the optical element is greater than a first preset threshold, it is determined that the optical element is dirty.

[0005] However, these technologies are complex to operate and require additional equipment, making them difficult to be widely adopted in practical applications.

[0006] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of this application. Summary of the Invention

[0007] To this end, the present invention is aimed at a depth camera that can switch between projecting structured light and flood light, and uses the depth value to screen out the first area and the second area, and determines whether there is dirt on the lens based on the ratio of the average signal intensity of the first area and the second area. The present invention does not require additional equipment, can perform dirt detection at any time, and does not affect data acquisition, with very high efficiency and convenience.

[0008] In a first aspect, the present invention provides a depth camera module with a dirt detection function, characterized in that it includes a laser transmitter, a receiver and a controller;

[0009] The laser emitter is used to switchably emit structured light or flood light;

[0010] The receiver is configured to receive a reflection signal of the structured light or the flood light;

[0011] The controller is used to control the laser emitter to emit structured light and flood light in a certain proportion. The receiver receives the adjacent first signal of the structured light and the second signal of the flood light, and generates a first depth map and a second depth map respectively. A first area N1 with a depth value less than N is obtained on the first depth map, and a second area N2 with a depth value less than N is obtained on the second depth map. Whether there is dirt on the lens is determined based on the ratio of the average signal intensities in the first area N1 and the second area N2; wherein N is a preset value.

[0012] Optionally, the depth camera module with a dirt detection function is characterized in that N is positively correlated with an error value n of the depth camera module with a dirt detection function on the lens.

[0013] Optionally, the depth camera module with a dirt detection function is characterized in that when the laser emitter emits structured light or flood light, the error values ​​of the two at the lens are different, and the error value n is the larger value of the two errors.

[0014] Optionally, the depth camera module with a dirt detection function is characterized in that N=3n.

[0015] Optionally, the depth camera module with a dirt detection function is characterized in that if dirt is detected, specific features of the dirt are detected on the second depth map.

[0016] Optionally, the depth camera module with a dirt detection function is characterized in that dirt is detected using a water stain model, a scratch model, and a particle model in sequence.

[0017] Optionally, the depth camera module with a dirt detection function is characterized in that the average signal intensity of the first area N1 refers to the total intensity of the structured light beam projected on the first area N1 divided by the area illuminated by the structured light in the first area N1.

[0018] Optionally, the depth camera module with a dirt detection function is characterized in that the average signal strength of the first area N1 is M1, and the average signal strength of the second area N2 is M2. It is determined that there is dirt on the lens; wherein σ is the attenuation coefficient related to the lens.

[0019] In a second aspect, the present invention provides a robot, characterized in that it comprises a depth camera module with a dirt detection function according to any one of claims 1 to 8.

[0020] Optionally, the robot is characterized in that it further includes a distance sensor; the distance sensor is arranged adjacent to the depth camera module with a dirt detection function in the same direction, and when the distance sensor detects a close object, the depth camera module with a dirt detection function no longer determines that there is dirt.

[0021] Compared with the prior art, the present invention has the following beneficial effects:

[0022] The present invention utilizes the signal obtained by the depth camera module itself for detection to obtain dirt data information, does not require additional devices or equipment for detection, and is very convenient. At the same time, it can be performed synchronously with normal data collection to improve efficiency.

[0023] The present invention is aimed at a depth camera that can switch between projecting structured light and flood light. It uses the depth value to screen out the first area and the second area, and determines whether there is dirt on the lens based on the ratio of the average signal intensity of the first area and the second area. This greatly reduces the judgment area, is conducive to quickly obtaining detection results, and can achieve normalized real-time detection. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for the embodiments or the description of the prior art. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without inventive work. Other features, purposes and advantages of the present invention will become more apparent by reading the detailed description of the non-limiting embodiments with reference to the following drawings:

[0025] Figure 1 Schematic diagram of the structure of a depth camera module with a dirt detection function according to an embodiment of the present invention;

[0026] Figure 2 This is a schematic structural diagram of a laser transmitter according to an embodiment of the present invention;

[0027] Figure 3 This is a schematic structural diagram of another laser transmitter according to an embodiment of the present invention;

[0028] Figure 4 This is a schematic diagram of dirt in an embodiment of the present invention;

[0029] Figure 5Schematic diagram of the structure of a robot in an embodiment of the present invention. DETAILED DESCRIPTION

[0030] The present invention will be described in detail below with reference to specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those skilled in the art, several variations and improvements can be made without departing from the scope of the present invention. These all fall within the scope of protection of the present invention.

[0031] The terms "first," "second," "third," "fourth," and the like (if any) in the description and claims of the present invention and in the accompanying drawings are used to distinguish similar objects and are not necessarily used to describe a particular order or precedence. It should be understood that the terms used in this manner are interchangeable where appropriate, so that the embodiments of the invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or apparatus comprising a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or apparatus.

[0032] The embodiments of the present invention provide a depth camera module and system with a dirt detection function, aiming to solve the problems existing in the prior art.

[0033] The following describes in detail the technical solutions of the present invention and how the technical solutions of this application solve the above-mentioned technical problems using specific embodiments. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. The following embodiments of the present invention are described in conjunction with the accompanying drawings.

[0034] The present invention is aimed at a depth camera that can switch between projecting structured light and flood light. It uses the depth value to screen out the first area and the second area, and determines whether there is dirt on the lens based on the ratio of the average signal intensity of the first area and the second area. The present invention does not require additional equipment, can perform dirt detection at any time, and does not affect data acquisition. It has very high efficiency and convenience.

[0035] Figure 1 FIG is a structural diagram of a depth camera module with a dirt detection function according to an embodiment of the present invention. Figure 1 As shown, a depth camera module with a dirt detection function in an embodiment of the present invention includes a laser transmitter 1, a receiver 2 and a controller 3.

[0036] The laser emitter 1 is used for switchably emitting structured light or flood light.

[0037] Specifically, the laser emitter 1 can rapidly switch between structured light and flood light, and can project structured light and flood light in a specific ratio within a preset time period. For example, within 1 second, structured light and flood light can be projected 10 times and 5 times, respectively, with a ratio of 2:1. This allows for rapid switching between structured light and flood light. The laser emitter 1 can have a variety of structures, as long as it can achieve switching between structured light and flood light.

[0038] In some embodiments, Figure 2 As shown, the laser emitter 1 includes a structured light projector 101 and a modulator 102. The structured light projector 101 projects a structured light beam. The structured light beam is directed perpendicularly to the modulator 102. The modulator 102 is controlled by voltage and switches between a transparent state and a frosted state. When the modulator 102 is in the transparent state, the structured light beam passes through the modulator 102 and is emitted as a structured light beam. When the modulator 102 is in the frosted state, the structured light beam passes through the modulator 102 and is emitted as a flood light. In this embodiment, the intensity of the flood light is less than the intensity of the structured light. By controlling the voltage applied to the modulator 102, the laser emitter 1 can be quickly switched between structured light and flood light.

[0039] In some embodiments, Figure 3 As shown, the laser emitter 1 includes a structured light projector 101, a lens 103 and a diffraction optical element 104. The structured light projector 101 projects a structured light beam. The lens 103 is used to refract the structured light beam projected by the structured light projector 101 and project it toward the diffraction optical element 104. The diffraction optical element 104 diffracts the incident light beam to form structured light or floodlight illumination. By adjusting the relative distance between the lens 103 and the diffraction optical element 104, and changing the focal length of the lens 103 and the relative position of the diffraction optical element 104, the light beam emitted by the diffraction optical element 104 can be structured light or floodlight. During adjustment, the position of the lens 103 or the diffraction optical element 104 can be adjusted separately, or the positions of the lens 103 and the diffraction optical element 104 can be adjusted simultaneously.

[0040] The receiver 2 is configured to receive the reflection signal of the structured light or the floodlight.

[0041] Specifically, receiver 2 is a TOF receiver capable of receiving both floodlight and structured light signals. The signals received by receiver 2 include reflections from the target area. If the lens is contaminated, receiver 2 will also receive the reflections from the contaminants. However, due to the proximity of the lens to receiver 2, the depth data obtained will have significant errors, making it difficult to determine depth directly based on the data.

[0042] The controller 3 is used to control the laser emitter to emit structured light and flood light in a certain proportion, and the receiver receives the first signal of the adjacent structured light and the second signal of the flood light, and generates a first depth map and a second depth map respectively. A first area N1 with a depth value less than N is obtained on the first depth map, and a second area N2 with a depth value less than N is obtained on the second depth map. Whether there is dirt on the lens is determined based on the ratio of the average signal intensities in the first area N1 and the second area N2; wherein N is a preset value.

[0043] Specifically, the ratio of structured light to flood light projected by laser emitter 1 can be any ratio, such as 1:1, 1:2, 1:3, 2:1, 3:1, etc. When the ratio of structured light to flood light is 1:1, the images obtained by adjacent structured light and flood light are used to determine whether the lens is dirty. When the ratio of structured light to flood light is not 1:1, taking 2:1 as an example, each flood light laser is preceded and followed by a structured light laser. In this way, each flood light laser is calculated with the preceding and following structured light lasers to determine the dirtiness.

[0044] N is positively correlated with the lens error value n for depth camera modules with dirt detection capabilities. While laser emitter 1 can project both structured light and floodlight, the different light intensities and working principles result in different lens error values. Therefore, the error value n is the larger of the two, structured light and floodlight, at the lens. To effectively identify dirt while preventing objects from being misidentified as dirt during normal measurement, 5n > N > 2n. After testing with various depth camera types, N = 3n is suitable for most depth camera modules.

[0045] On the first depth map and the second depth map, the depth value N is used for screening. Objects farther away from the lens can be filtered out, and objects near the lens can be identified, thereby identifying dirt. Dirt exists in the same area of ​​the first area N1 and the second area N2. However, in addition to dirt, there may also be objects closer to the lens in the first area N1 and the second area N2. However, since the dirt is located on the lens, its depth value is different from that of other objects. The dirt can be identified by the ratio of the average signal intensity on the first area N1 and the second area N2. The average signal intensity of the first area N1 refers to the total intensity of the structured light beam projected on the first area N1 divided by the area illuminated by the structured light in the first area N1. When the ratio of the average signal intensity on the first area N1 and the second area N2 is greater than a, it is determined that dirt exists. a is a preset value and is related to the beam density of the structured light. The greater the beam density of the structured light, the greater a.

[0046] In some embodiments, if dirt is detected, specific features of the dirt are detected on the second depth map. Through further detection of the dirt, more features of the dirt can be obtained, such as the area of the dirt, the shape of the dirt, etc. Through the judgment of the area of the dirt, it can be judged whether cleaning is needed or the user is prompted to clean. Through the judgment of the shape of the dirt, the type of the dirt can be determined, whether it can be cleaned by the self-provided cleaning device, or the user is prompted to use what kind of cleaning means.

[0047] In some embodiments, the dirt is detected by using the water stain model, the scratch model, and the particle model in sequence. The particle model does not detect the areas detected by the water stain model and the scratch model. Figure 4 The main three forms of dirt existing on the robot are shown, which are water stain, scratch, and particle. The water stain is mainly caused by sewage and has certain area characteristics. The scratch is mainly caused by collision, etc. The particle is mainly caused by dust. In this embodiment, the particle can be a single particle or a connected area composed of multiple particles. Machine learning method is used to learn the water stain, the scratch, and the particle respectively to obtain the water stain model, the scratch model, and the particle model to improve the recognition effect. When detecting, the water stain model is used for detection first, and then the scratch model is used for detection. The water stain model and the scratch model detect all areas of the image. The particle model only detects the areas without recognized dirt to improve the efficiency.

[0048] In some embodiments, the average signal intensity of the first area N1 is M1, the average signal intensity of the second area N2 is M2, if then it is determined that there is dirt on the lens; wherein σ is an attenuation coefficient related to the lens. In this embodiment, the value of N is the same as that in the foregoing embodiments. The first area N1 and the second area N2 can be continuous areas or discontinuous multiple areas. The areas of the first area N1 and the second area N2 can be large or small. When the dirt is a particle, the areas of the first area N1 and the second area N2 are the smallest. σ is not only related to the parameters of the lens itself, but also related to the positions of the laser emitter 1 and the receiver 2 from the lens.

[0049] Figure 5 A structural schematic diagram of a robot in an embodiment of the present application. Figure 5The robot shown in FIG6 includes a depth camera 601, a robot body 602, and a display screen 603. Depth camera 601 is the depth camera module with dirt detection functionality described in any of the aforementioned embodiments. Depth camera 601 can capture the scene in front of the robot, thereby obtaining three-dimensional information. Typically, depth camera 601 also includes an RGB camera to obtain RGB images, which can be combined with the depth image to generate an RGBD image. Robot body 602 can have different functions depending on the robot type. For example, a food delivery robot may have a tray, a stand, drive wheels, etc.; a welcoming robot may have drive wheels, a humanoid shape, etc. Display screen 603 is used to display information and facilitate user interaction. Display screen 603 can be either unidirectional or bidirectional. By using a highly integrated, automatically switching depth camera, the robot can accommodate more sensors within the same size, thereby achieving better environmental perception capabilities. Of course, the robot can also be reduced in size to adapt to more demanding scenarios. The automatically switching depth camera can also use the monitoring range to set different trigger distances to automatically start the automatically switching depth camera at a close distance and shut down the automatically switching depth camera at a long distance, thereby saving energy consumption.

[0050] In some embodiments, a distance sensor 604 is further included; the distance sensor is arranged adjacent to the depth camera module with dirt detection function in the same direction, and when the distance sensor detects a close object, the depth camera module with dirt detection function no longer determines that it is dirty. The distance sensor 604 is used to detect close-range target objects, which can prevent the depth camera module with dirt detection function from judging close-range target objects as dirty, and can further improve the recognition accuracy of the depth camera module with dirt detection function. Only one distance sensor 604 needs to be set, and it must be set adjacent to the depth camera module with dirt detection function to improve stability and reliability. The distance between the distance sensor 604 and the edge of the depth camera module with dirt detection function does not exceed 1 cm.

[0051] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same and similar parts between the various embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be apparent to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

[0052] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various variations or modifications within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A depth camera module with a dirt detection function, characterized in that: Includes laser transmitter, receiver and controller; The laser emitter is used to switchably emit structured light or flood light; The receiver is configured to receive a reflection signal of the structured light or the flood light; The controller is used to control the laser emitter to emit structured light and flood light in a certain proportion. The receiver receives the adjacent first signal of the structured light and the second signal of the flood light, and generates a first depth map and a second depth map respectively. A first area N1 with a depth value less than N is obtained on the first depth map, and a second area N2 with a depth value less than N is obtained on the second depth map. Whether there is dirt on the lens is determined based on the ratio of the average signal intensities in the first area N1 and the second area N2; wherein N is a preset value.

2. The depth camera module with dirt detection function according to claim 1, characterized in that: The N is positively correlated with the error value n of the lens of the depth camera module with dirt detection function.

3. The depth camera module with dirt detection function according to claim 2, characterized in that: When the laser emitter emits structured light or flood light, the error values ​​of the two at the lens are different, and the error value n is the larger value of the two errors.

4. The depth camera module with dirt detection function according to claim 1, characterized in that: N=3n.

5. The depth camera module with dirt detection function according to claim 1, characterized in that: If dirt is detected, specific features of the dirt are detected in the second depth map.

6. The depth camera module with dirt detection function according to claim 5, characterized in that: The water stain model, scratch model and particle model are used in turn to detect dirt.

7. The depth camera module with dirt detection function according to claim 1, characterized in that: The average signal intensity of the first area N1 refers to the total intensity of the structured light beam projected on the first area N1 divided by the area illuminated by the structured light in the first area N1.

8. The depth camera module with dirt detection function according to claim 1, characterized in that: The average signal strength of the first area N1 is M1, and the average signal strength of the second area N2 is M2. It is determined that there is dirt on the lens; wherein σ is the attenuation coefficient related to the lens.

9. A robot, characterized in that: A depth camera module with a dirt detection function comprising any one of claims 1-8.

10. A robot according to claim 9, characterized in that: It also includes a distance sensor; the distance sensor is arranged adjacent to the depth camera module with a dirt detection function in the same direction, and when the distance sensor detects a close object, the depth camera module with a dirt detection function no longer determines whether there is dirt.

Images