Depth camera and sweeping robot

Abstract

The invention discloses a depth camera and a sweeping robot, and the camera comprises a projector which is used for projecting speckles and floodlight; the receiver is used for receiving reflection signals of the speckles and generating a speckle pattern; receiving a reflection signal of the floodlight, and generating an IR image; the RGB camera is used for obtaining an RGB image; and the processor is used for generating a depth map according to the speckle pattern or the IR image, determining a ground area according to the RGB image and / or the depth map, fusing the RGB image, the IR image and the speckle pattern corresponding to the ground area, and performing detection through a smudginess segmentation algorithm to obtain a smudginess range. According to the invention, the degree of liquid and solid smudginess can be effectively identified, and accurate segmentation and 3D modeling are carried out on the liquid and solid smudginess.

Description

Technical Field

[0001] The present invention relates to the technical field of depth cameras, and in particular to a depth camera and a sweeping robot. Background Art

[0002] With the rapid development of smart home technology, sweeping robots have become one of the important devices for household cleaning. However, existing sweeping robots still have certain limitations in identifying floor dirt, especially when faced with multiple types of floor dirt, the accuracy of identification and the intelligence of processing are insufficient. Common floor dirt includes two categories: liquid and solid, which have great differences in visual and physical properties. How to accurately identify these types of dirt and clean them effectively is an urgent problem to be solved.

[0003] Traditional sweeping robots usually rely on simple visual algorithms or basic sensor data to identify dirt. This approach often shows low recognition rates and unstable performance when facing complex ground environments. For example, transparent or translucent liquid dirt (such as water, oil, milk, etc.) and solid particles (such as gravel, paper scraps, etc.) are very different visually, and it is difficult to effectively identify and distinguish these types of dirt at the same time using simple image processing methods. In addition, traditional visual algorithms and sensors have difficulty calculating the precise 3D information of the dirty area, which is not conducive to the subsequent SLAM, obstacle avoidance, cleaning planning and other tasks of the sweeping robot.

[0004] In order to improve the intelligence and accuracy of ground dirt identification, a more accurate and robust system is needed that can utilize multiple types of image data and combine deep learning algorithms to perform multimodal data fusion and intelligent identification.

[0005] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0006] To this end, the present invention uses a deep learning algorithm based on the RGB images, IR images, speckle patterns and depth maps collected by the depth camera to intelligently identify and classify various types of dirt on the ground. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0007] In a first aspect, the present invention provides a depth camera, characterized in that it includes:

[0008] Projectors, used to project speckle and flood;

[0009] A receiver, configured to receive the reflection signal of the speckle to generate a speckle pattern; and receive the reflection signal of the flood light to generate an IR image;

[0010] RGB camera, used to obtain RGB images;

[0011] a processor, configured to generate a depth map according to the speckle map or the IR image, determine a ground area according to the RGB image and / or the depth map, fuse the RGB image, the IR image and the speckle map corresponding to the ground area, and detect through a dirt segmentation algorithm to obtain a dirt range.

[0012] Optionally, the depth camera is characterized in that the processor includes:

[0013] Step S1: generating a depth map according to the speckle pattern or the IR image;

[0014] Step S2: aligning the RGB image, the depth map, the IR image and the speckle map;

[0015] Step S3: determining an initial ground area according to the depth map, detecting the initial ground area on the RGB image, and determining a final ground area;

[0016] Step S4: The RGB image, the IR image and the speckle image corresponding to the final ground area are fused, and detected by a dirt segmentation algorithm to obtain a dirt range.

[0017] Optionally, the depth camera is characterized by further comprising:

[0018] Step S5: Fusing the dirt range with the depth map to obtain accurate 3D information of the dirt area.

[0019] Optionally, the depth camera is characterized by further comprising:

[0020] Step S6: performing dirt detection in the final ground area according to the RGB image, the IR image and the speckle pattern to obtain a detection frame and a classification result.

[0021] Optionally, the depth camera is characterized in that step S3 comprises:

[0022] Step S31: determining an initial ground area according to the depth value at the bottom of the depth map;

[0023] Step S32: obtaining an initial ground area on the RGB image, performing ground detection, and determining a final ground area.

[0024] Optionally, the depth camera is characterized in that step S3 comprises:

[0025] Step S33: determining an initial ground area according to the depth value at the bottom of the depth map;

[0026] Step S34: extracting the initial ground area of ​​the RGB image to obtain a lower RGB image, and extracting the initial ground area of ​​the depth map to obtain a lower depth map;

[0027] Step S35: performing depth reconstruction according to the lower RGB image and the lower depth map to obtain a lower point cloud;

[0028] Step S36: performing plane fitting according to the point cloud, determining the direction of the plane, and obtaining the ground area.

[0029] Optionally, the depth camera is characterized in that step S4 comprises:

[0030] Step S41: stitching the RGB image, the IR image and the speckle pattern into a first image;

[0031] Step S42: extracting features from the first image to obtain first features;

[0032] Step S43: Use the self-attention mechanism to process the first feature to obtain the dirt range.

[0033] Optionally, the depth camera is characterized in that step S4 comprises:

[0034] Step S44: extracting features from the RGB image to obtain a first feature map; extracting features from the IR image to obtain a second feature map; extracting features from the speckle map to obtain a third feature map; wherein the first feature map, the second feature map and the third feature map all contain position information;

[0035] Step S45: splicing the first feature map, the second feature map and the third feature map to obtain a second feature;

[0036] Step S46: Use the self-attention mechanism to process the second feature to obtain the dirt range.

[0037] Optionally, the depth camera is characterized in that step S4 comprises:

[0038] Step S47: extracting features from the RGB image to obtain a first feature map; extracting features from the IR image to obtain a second feature map; extracting features from the speckle pattern to obtain a third feature map;

[0039] Step S48: fusing the first feature map, the second feature map and the third feature map to obtain a third feature;

[0040] Step S49: Use the self-attention mechanism to process the third feature to obtain the dirt range.

[0041] In a second aspect, the present invention provides a sweeping robot, characterized in that it comprises a depth camera as described in any one of the above items.

[0042] Compared with the prior art, the present invention has the following beneficial effects:

[0043] Based on the RGB images, IR images, speckle patterns and depth maps collected by the depth camera, the present invention uses a deep learning algorithm to intelligently identify and classify various types of dirt on the ground. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0044] The present invention can accurately identify and classify various degrees of dirt on the ground based on multimodal image data and deep learning algorithms, and can calculate accurate 3D information of the dirty area. On the one hand, the present invention can more accurately and robustly identify and classify various degrees of dirt on the ground, making it easier for the sweeping robot to adopt a more effective cleaning strategy; on the other hand, it can calculate accurate 3D information of the dirty area for subsequent SLAM, obstacle avoidance, cleaning planning and other tasks.

[0045] The present invention uses multiple image sources (including RGB images, IR images, speckle images and depth images) for information fusion to improve the segmentation effect of liquid contamination targets. Different from the practice of only using RGB images or IR images in the prior art, the multimodal image fusion method of the present invention can capture the characteristics of liquid contamination more comprehensively, thereby overcoming the limitations of a single image source. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0047] Figure 1 is a schematic structural diagram of a depth camera according to an embodiment of the present invention;

[0048] Figure 2 A flowchart of the steps of a processor during processing in an embodiment of the present invention;

[0049] Figure 3 is a flowchart of steps of another processor during processing in an embodiment of the present invention;

[0050] Figure 4 is a flowchart of steps of another processor during processing in an embodiment of the present invention;

[0051] Figure 5 A flowchart of a step of determining a ground area in an embodiment of the present invention;

[0052] Figure 6 This is another flow chart of steps for determining a ground area in an embodiment of the present invention;

[0053] Figure 7 This is a flow chart of steps for obtaining a dirt range in an embodiment of the present invention;

[0054] Figure 8 This is another flowchart of steps for obtaining a dirt range in an embodiment of the present invention;

[0055] Fig. 9 This is another flowchart of the steps for obtaining the dirt range in an embodiment of the present invention. DETAILED DESCRIPTION

[0056] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0057] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0058] An embodiment of the present invention provides a depth camera, which aims to solve the problems existing in the prior art.

[0059] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0060] Based on the RGB images, IR images, speckle patterns and depth maps collected by the depth camera, the present invention uses a deep learning algorithm to intelligently identify and classify various types of dirt on the ground. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0061] In the present invention, multiple image sources (including RGB images, IR images, speckle images and depth images) are used for information fusion to improve the segmentation effect of liquid contamination targets. Different from the practice of only using RGB images or IR images in the prior art, the multimodal image fusion method of the present invention can capture the characteristics of liquid contamination more comprehensively, thereby overcoming the limitations of a single image source.

[0062] RGB images are often affected by changes in lighting, liquid transparency, and the similarity between liquid color and floor color. For example, when the liquid is transparent or the color of the liquid dirt is close to the color of the floor, the segmentation effect is often poor. Although IR images are highly robust to changes in lighting and can clearly present the outline of transparent liquid dirt, in the case of some special liquids or floor materials, the outline of the liquid dirt on the IR image may not be obvious or clear. A large number of tests have shown that when the outline of the liquid dirt in the IR image is not obvious, the liquid dirt area on the speckle image shows obvious features (laser scattering). Therefore, by introducing speckle patterns and depth map information, the deficiencies of RGB and IR images can be effectively supplemented, and the feature expression of liquid dirt targets can be significantly enhanced, making liquid dirt targets easier to identify, thereby improving the segmentation accuracy.

[0063] Figure 1 FIG. 1 is a schematic diagram of the structure of a depth camera according to an embodiment of the present invention. Figure 1 As shown, a depth camera in an embodiment of the present invention includes:

[0064] Projectors for projecting speckle and flood.

[0065] Specifically, the projector is responsible for projecting speckle and flood light onto the object being photographed. The speckle emitted by the projector is a specially modulated invisible infrared light. These speckle light points form a unique pattern on the object being photographed, and each speckle light point and the set of points within a certain range around it are unique at each position in the spatial distribution. This unique pattern helps the receiver capture and calculate the depth information of the object. In addition to speckle, the projector also emits flood infrared light. This light helps the receiver better identify the object being photographed in a low-light environment, thereby improving the recognition accuracy and robustness of the camera.

[0066] The structure of the projector can be one emitter or multiple emitters. When the projector is one emitter, it can switch between emitting speckle and floodlight to achieve the projection of speckle and floodlight. When the projector is multiple emitters, it includes a speckle projector and a floodlight projector. The speckle projector is used to project speckle, and the floodlight projector is used to emit floodlight.

[0067] The receiver is used to receive the reflection signal of the speckle to generate a speckle pattern; and receive the reflection signal of the flood light to generate an IR image.

[0068] Specifically, the receiver receives the reflected signals of the speckle and flood light emitted by the projector. The receiver captures the speckle pattern reflected back by the photographed object. Since the different depths of the object surface will cause the distortion of the speckle pattern, the receiver calculates the depth information of the object by analyzing this distortion. The receiver also receives the reflected signal of the flood light and generates an IR image. The IR image provides infrared information of the photographed object, which helps the processor to perform subsequent image processing and recognition. The receiver is an infrared receiver. The receiver can be a receiver that can receive both flood light and structured light, or it can be a combination of a TOF receiver that receives flood light and a structured light receiver.

[0069] RGB camera, used to obtain RGB images.

[0070] Specifically, RGB cameras are used to capture color images of objects. RGB cameras capture light in the scene through a lens module and convert it into digital signals. These digital signals are processed to form RGB images, providing rich information such as the color, texture, and shape of the object.

[0071] a processor, configured to generate a depth map according to the speckle map or the IR image, determine a ground area according to the RGB image and / or the depth map, fuse the RGB image, the IR image and the speckle map corresponding to the ground area, and detect through a dirt segmentation algorithm to obtain a dirt range.

[0072] Specifically, the processor is responsible for processing the image information captured by the receiver and the RGB camera, and generating a depth map, identifying the ground area, and detecting the scope of dirt. The processor calculates the depth information of each pixel through an algorithm based on the speckle pattern or IR image generated by the receiver to generate a depth map. The depth map provides three-dimensional structural information of objects in the scene. The processor also uses the RGB image and / or depth map to identify the ground area. This is usually achieved through image segmentation and recognition algorithms, which helps to fuse and process the image information of the ground area later. The processor fuses the RGB image, IR image and speckle map corresponding to the ground area through a dirt segmentation algorithm. The dirt segmentation algorithm can identify dirty areas in the image and calculate the scope and degree of dirt. This technology helps to monitor and maintain the cleanliness of the camera and ensure the quality of the captured images.

[0073] Figure 2 FIG. 1 is a flowchart of the steps of a processor in processing according to an embodiment of the present invention. Figure 2 As shown, the steps of a processor in processing in an embodiment of the present invention include:

[0074] Step S1: generating a depth map according to the speckle pattern or the IR image.

[0075] In this step, the depth map is generated to obtain the three-dimensional information of the objects in the scene, which is crucial for the subsequent determination of the ground area and dirt detection. When obtaining the depth map, it can be obtained by using the speckle map calculation or the IR map calculation. When using the speckle map calculation, the depth map is obtained by calculating the deformation of the spot. When using the IR map calculation, the depth map is obtained by calculating the time-of-flight algorithm. The accuracy of the depth map obtained by different methods is different, and you can choose according to the usage scenario to obtain higher accuracy. You can also fuse the depth map obtained by the speckle map with the depth map obtained by the IR map to generate a depth map with high accuracy in each range.

[0076] Step S2: aligning the RGB image, the depth map, the IR image and the speckle map.

[0077] In this step, all images are aligned to ensure that they are represented in the same coordinate system so that subsequent processing can accurately map information from one image to another. In this embodiment, the positions of the depth map, RGB image, IR image, and speckle pattern are fixed. The transformation matrix of each image obtained by the depth camera is calibrated and known, and no further calculation is required.

[0078] Step S3: determining an initial ground area according to the depth map, detecting the initial ground area on the RGB image, and determining a final ground area.

[0079] In this step, the ground area is the main target area for dirt detection, so it needs to be accurately identified first. Use the depth map: Based on the depth information in the depth map, you can preliminarily determine which areas may be the ground (usually areas with large depth values ​​and relatively flat). Use image segmentation, edge detection and other algorithms on the RGB image to further confirm and refine the ground area.

[0080] Step S4: The RGB image, the IR image and the speckle image corresponding to the final ground area are fused, and detected by a dirt segmentation algorithm to obtain a dirt range.

[0081] In this step, the dirty area is accurately detected by fusing multiple image information and using a dirt segmentation algorithm. The RGB image, IR image, and speckle map corresponding to the final ground area are fused to utilize the unique information of each image. Machine learning algorithms (such as convolutional neural networks) or traditional image processing algorithms (such as threshold segmentation, clustering, etc.) are applied to detect dirty areas. These algorithms are usually trained to recognize the characteristics of dirt, such as color, texture, temperature, etc.

[0082] Figure 3 FIG. 1 is a flowchart of another processing step of a processor in an embodiment of the present invention. Figure 3 As shown, compared with the above-mentioned embodiment, another processor in the embodiment of the present invention further includes:

[0083] Step S5: Fusing the dirt range with the depth map to obtain accurate 3D information of the dirt area.

[0084] In this step, the dirt range detected in the previous step is fused with the depth map to obtain the precise position and shape information of the dirty area in three-dimensional space. This is crucial for subsequent dirt cleaning, maintenance or monitoring applications. Based on the dirt range obtained in the previous step S4, the boundary and shape information of the dirty area are extracted.

[0085] Match the boundaries of the dirty area with the corresponding area in the depth map. This usually involves converting the pixel coordinates of the dirty area into coordinates in the depth map and finding the depth values ​​corresponding to these coordinates. Using the matched depth values ​​and the pixel coordinates of the dirty area, construct the point cloud data of the dirty area in three-dimensional space. Through point cloud data processing techniques (such as filtering, smoothing, etc.), further optimize the 3D model of the dirty area to improve its accuracy and completeness.

[0086] This embodiment makes the processing flow of the processor more perfect, and can provide users with more accurate and comprehensive 3D information of the dirty area.

[0087] Figure 4FIG. 1 is a flowchart of another processing step of a processor in an embodiment of the present invention. Figure 4 As shown, compared with the above-mentioned embodiment, another processor in the embodiment of the present invention further includes:

[0088] Step S6: performing dirt detection in the final ground area according to the RGB image, the IR image and the speckle pattern to obtain a detection frame and a classification result.

[0089] In this step, the goal is to use the rich information provided by RGB images, IR images, and speckle patterns to perform more detailed dirt detection in the final determined ground area. By marking the detection frame and classifying the dirt type, the user can more intuitively understand the specific location and nature of the dirt, and thus take more effective countermeasures.

[0090] The RGB image, IR image, and speckle pattern are fused in the final ground area. This can be achieved through techniques such as image overlay, feature extraction, and fusion algorithms to make full use of the unique information of each image. Apply a dirt detection algorithm on the fused image. These algorithms can be based on machine learning (such as deep learning models) or traditional image processing algorithms (such as edge detection, color segmentation, etc.). The goal of the algorithm is to identify the dirty area in the ground area and mark the detection box. The detection box should tightly surround the dirty area so that the user can accurately identify it. In addition to marking the detection box, the algorithm should also be able to classify the dirt. The basis for classification can be the color, texture, temperature characteristics of the dirt, etc. The classification results can be discrete (such as oil stains, water stains, dust, etc.) or continuous (such as a quantitative score of the degree of dirtiness). The detection box and classification results are output to the user in a visual way. This can be achieved by superimposing a label box, color coding, or text labels on the image.

[0091] This embodiment makes the processing flow of the processor more perfect and can provide users with more accurate and detailed dirt detection information.

[0092] Figure 5 FIG. 1 is a flow chart of steps for determining a ground area in an embodiment of the present invention. Figure 5 As shown, in an embodiment of the present invention, a step of determining a ground area includes:

[0093] Step S31: determining an initial ground area according to the depth value at the bottom of the depth map.

[0094] In this step, the approximate position of the ground area is preliminarily determined using the depth information in the depth map. Since the ground is usually located at the bottom of the scene and has a relatively flat and continuous depth value, the ground area can be identified by analyzing the depth value at the bottom of the depth map. Read the depth map and obtain the depth value of each pixel. Analyze the distribution of depth values ​​at the bottom of the depth map. Usually, the depth values ​​of the ground area are relatively concentrated in a range. The bottommost depth value in the depth map represents the closest distance to the depth camera. The bottommost depth value in the depth map represents the closest distance between the depth camera and the ground. Through the preset working mode, the approximate range of the initial ground area can be obtained. For example, when the bottommost depth value in the depth map is relatively small, it means that the depth camera is located at a relatively close distance from the ground, such as a sweeping robot, and the ground accounts for a large proportion in the image. When the bottommost depth value in the depth map is relatively large, it means that the depth camera is located at a relatively far distance from the ground, such as a service robot, and the ground accounts for a small proportion in the image. The initial ground area is obtained according to the preset model based on the depth value at the bottom of the depth map, which can quickly reduce the amount of data to be processed.

[0095] Step S32: obtaining an initial ground area on the RGB image, performing ground detection, and determining a final ground area.

[0096] In this step, this step aims to further detect and refine the initial ground area using features such as color, texture and shape in the RGB image to determine the final ground area. The initial ground area determined in step S31 is mapped to the RGB image. This can be achieved through techniques such as coordinate transformation or pixel mapping. A ground detection algorithm is applied to the RGB image. These algorithms can be based on machine learning (such as deep learning models) or traditional image processing algorithms (such as color segmentation, edge detection, etc.). The goal of the algorithm is to identify the ground area in the RGB image and refine it. Based on the output of the ground detection algorithm, the final ground area is determined. The final ground area should be more accurate and detailed, and can reflect the true shape and boundaries of the ground.

[0097] Figure 6 FIG. 1 is another flow chart of steps for determining a ground area in an embodiment of the present invention. Figure 6 As shown, another step of determining the ground area in the embodiment of the present invention includes:

[0098] Step S33: determining an initial ground area according to the depth value at the bottom of the depth map.

[0099] In this step, the approximate location of the ground area is preliminarily determined using the depth information in the depth map. Since the ground is usually located at the bottom of the scene and has relatively flat and continuous depth values, the ground area can be identified by analyzing the depth values ​​at the bottom of the depth map. Read the depth map and obtain the depth value of each pixel. Analyze the depth value distribution at the bottom of the depth map and set a depth threshold (usually a lower depth value range is selected because the ground is usually closer). Pixels in the depth map with depth values ​​below the set threshold are marked as ground areas to form an initial ground area.

[0100] Step S34: extracting the initial ground area of ​​the RGB image to obtain a lower RGB image, and extracting the initial ground area of ​​the depth map to obtain a lower depth map.

[0101] In this step, the purpose is to extract the portion corresponding to the initial ground area from the RGB image and the depth map for subsequent processing. According to the initial ground area determined in step S33, the corresponding pixel area is extracted from the RGB image to form a lower RGB image. Similarly, the depth value corresponding to the initial ground area is extracted from the depth map to form a lower depth map.

[0102] Step S35: performing depth reconstruction according to the lower RGB image and the lower depth map to obtain a lower point cloud.

[0103] In this step, the information in the lower RGB image and the lower depth map is used to perform 3D reconstruction to obtain the 3D point cloud data of the ground area. According to the depth value in the depth map, each pixel in the lower RGB image is converted into a point in 3D space to form point cloud data. These point cloud data represent the 3D shape and position of the ground area.

[0104] Step S36: performing plane fitting according to the point cloud, determining the direction of the plane, and obtaining the ground area.

[0105] In this step, the point cloud data is used to perform plane fitting to more accurately determine the position and shape of the ground area. The point cloud data obtained in step S35 is plane fitted to obtain the equation and parameters of the fitting plane. The direction of the fitting plane is determined. Usually, the plane corresponding to the ground area should be horizontal (i.e., the normal vector is close to the vertical direction). According to the equation and parameters of the fitting plane, the points belonging to the plane are extracted from the point cloud data to form the final ground area.

[0106] Figure 7 FIG. 1 is a flow chart of steps for obtaining a dirt range in an embodiment of the present invention. Figure 7 As shown, in an embodiment of the present invention, a step of obtaining a dirt range includes:

[0107] Step S41: stitching the RGB image, the IR image and the speckle pattern into a first image.

[0108] In this step, the RGB image, IR image and speckle pattern are stitched together to form a first image containing multiple information sources. The stitching process may involve alignment, scaling and merging of color channels of the image. The purpose of stitching is to integrate the information in different images together for subsequent feature extraction and dirt detection.

[0109] Step S42: extract features from the first image to obtain first features.

[0110] In this step, a feature extraction algorithm is applied to the stitched first image to obtain key information in the image. Feature extraction may involve color features, texture features, edge features, shape features, etc. The extracted features are called first features, and they contain various clues that may indicate the presence of dirt in the image.

[0111] Step S43: Use the self-attention mechanism to process the first feature to obtain the dirt range.

[0112] In this step, the self-attention mechanism is used for processing. The self-attention mechanism is a deep learning technique that can capture distance dependencies in data. In the dirt detection task, the self-attention mechanism can be used to identify the association between different regions in the image, especially the contrast between the dirty region and the surrounding environment. The self-attention mechanism calculates the similarity between different features and weights the features according to the similarity, thereby highlighting the dirty region and suppressing the background noise.

[0113] After the self-attention mechanism is processed, the dirty area in the image will be significantly highlighted. By setting an appropriate threshold, the highlighted area can be distinguished from the background, thereby obtaining a binary image or mask of the dirty range.

[0114] The dirty area can be represented in many forms, such as binary image (dirty area is white and the rest is black), grayscale image (dirty degree is represented by grayscale value) or polygonal area (dirty area is described by bounding box or contour line).

[0115] This embodiment can accurately detect and extract the dirt range on the ground by splicing multiple images, extracting features, and applying the self-attention mechanism, which provides an important basis for subsequent cleaning, maintenance, and management.

[0116] Figure 8 FIG. 1 is another flow chart of steps for obtaining the dirt range in an embodiment of the present invention. Figure 8 As shown, another step of obtaining the dirt range in the embodiment of the present invention includes:

[0117] Step S44: extracting features from the RGB image to obtain a first feature map; extracting features from the IR image to obtain a second feature map; and extracting features from the speckle pattern to obtain a third feature map.

[0118] In this step, a feature extraction algorithm is applied to the RGB image to obtain a first feature map containing position information. The feature extraction algorithm may involve deep learning models such as convolutional neural networks (CNNs), which can automatically learn features such as color, texture, and shape in images. The first feature map not only contains the feature information of the image, but also retains the position information of each feature in the image, which is crucial for subsequent dirt detection.

[0119] A similar feature extraction process is performed on the IR image to obtain a second feature map. The second feature map also contains position information, so that subsequent processing can accurately match the features to specific locations in the image.

[0120] Applying feature extraction algorithm to the speckle pattern, a third feature map is obtained. The features in the speckle pattern may be related to the slight deformation, roughness or laser speckle interference effect of the object surface. The third feature map also contains position information, which provides additional clues for subsequent dirt detection.

[0121] Step S45: concatenate the first feature map, the second feature map and the third feature map to obtain a second feature.

[0122] In this step, the first feature map, the second feature map, and the third feature map are spliced ​​to obtain a second feature containing multiple information sources. The splicing process may involve channel merging or spatial alignment of feature maps. The spliced ​​second feature not only contains the feature information in the RGB image, IR image, and speckle map, but also retains the positional relationship of these features in the image.

[0123] Step S46: Use the self-attention mechanism to process the second feature to obtain the dirt range.

[0124] In this step, the self-attention mechanism is applied to the concatenated second feature to capture the distance dependency between features. Since the second feature contains dirt and ground information, and the ground information is relatively consistent, while the dirt information has different characteristics, the self-attention mechanism can highlight the dirty area in the image and suppress the information of other areas, especially when the dirty area is significantly different from the surrounding environment in color, temperature or surface roughness.

[0125] After the self-attention mechanism is processed, the important areas in the image (i.e., dirty areas) will be significantly highlighted. By setting an appropriate threshold, the highlighted areas can be distinguished from the background, thereby obtaining a binary image or mask of the dirty range. The dirty range can be represented in many forms, such as a binary image (the dirty area is white and the rest is black), a grayscale image (the degree of dirtiness is represented by grayscale values), or a polygonal area (the dirty area is described by a bounding box or contour line).

[0126] After fusing the features, this embodiment uses a self-attention mechanism to process the feature map to further improve the segmentation performance and solve the problem of missed detection or incomplete recognition of liquid contamination targets; by respectively extracting the features of the RGB image, IR image and speckle map, and splicing these features into a second feature, and then applying the self-attention mechanism for processing, it is possible to accurately detect and extract the scope of contamination on the ground, and combines multiple image information sources and deep learning technology to improve the accuracy and robustness of contamination detection.

[0127] Fig. 9 FIG. 1 is another flow chart of steps for obtaining the dirt range in an embodiment of the present invention. Fig. 9 As shown, another step of obtaining the dirt range in the embodiment of the present invention includes:

[0128] Step S47: extracting features from the RGB image to obtain a first feature map; extracting features from the IR image to obtain a second feature map; extracting features from the speckle image to obtain a third feature map.

[0129] This step is the same as step S44 and will not be repeated here.

[0130] Step S48: Fusing the first feature map, the second feature map and the third feature map to obtain a third feature.

[0131] In this step, the first feature map, the second feature map and the third feature map are fused to obtain a third feature containing multiple information sources. The fusion process may involve channel merging, weighted summation, maximum pooling and other operations of the feature map. The fused third feature not only contains the feature information in the RGB image, IR image and speckle map, but also enhances the robustness and complementarity of the features through the fusion process.

[0132] Step S49: Use the self-attention mechanism to process the third feature to obtain the dirt range.

[0133] In this step, the self-attention mechanism is applied to the fused third feature. Since the second feature contains dirt and ground information, and the ground information is relatively consistent, while the dirt information has different characteristics, the self-attention mechanism can highlight the dirty area in the image and suppress the information of other areas, especially when the dirty area is significantly different from the surrounding environment in color, temperature or surface roughness.

[0134] After the self-attention mechanism is processed, the important areas in the image (i.e., dirty areas) will be significantly highlighted. By setting an appropriate threshold or applying a classifier, the highlighted areas can be distinguished from the background, thereby obtaining a binary image or mask of the dirty range.

[0135] The dirty area can be represented in many forms, such as binary image (dirty area is white and the rest is black), grayscale image (dirty degree is represented by grayscale value) or polygonal area (dirty area is described by bounding box or contour line).

[0136] After fusing the features, this embodiment uses a self-attention mechanism to process the feature map to further improve the segmentation performance and solve the problem of missed detection or incomplete recognition of liquid dirty targets; this embodiment can strengthen the focus on the key areas of dirty targets and improve the robustness and accuracy of the model in complex environments. This embodiment extracts the features of RGB images, IR images, and speckle images respectively, fuses these features into a third feature, and then applies a self-attention mechanism for processing, which can accurately detect and extract the dirt range on the ground. It combines multiple image information sources and deep learning technology to improve the accuracy and robustness of dirt detection.

[0137] In some embodiments, in step S48, fusion is performed according to the position information, and in step S49, higher weights are assigned to different parts of the third feature. A fusion strategy based on position information is adopted. This involves pixel-by-pixel fusion. Through the above fusion strategy, the first feature map, the second feature map, and the third feature map are fused into a third feature containing multiple information sources. This feature map not only contains rich feature information, but also enhances the spatial consistency and complementarity of the features through the fusion of position information.

[0138] During the self-attention process, different parts of the third feature (i.e. different pixels or regions) are given higher weights according to their importance or relevance to dirt detection. The weight assignment can be based on the similarity, significance, contextual information or pre-trained model weights of the feature vectors. For example, if a region shows significant dirt features in multiple feature maps, it is given a higher weight in the self-attention mechanism.

[0139] This embodiment combines multiple image information sources, deep learning technology and position information fusion strategy to improve the accuracy and robustness of dirt detection.

[0140] This specification also provides an embodiment of a sweeping robot. A sweeping robot is a smart home appliance that can autonomously complete the task of sweeping the ground. It is usually equipped with advanced sensors, navigation systems and cleaning mechanisms, and can autonomously navigate in complex environments, avoid obstacles, and effectively remove dust, hair, debris and other debris on the ground.

[0141] A depth camera is a technical device that can measure the distance of an object and generate a three-dimensional image. Introducing a depth camera in a sweeping robot can significantly improve the robot's environmental perception and intelligence level.

[0142] The depth camera can capture the three-dimensional information of the robot vacuum cleaner's surroundings in real time, including the location, shape, and size of obstacles. By analyzing this information, the robot vacuum cleaner can intelligently plan the cleaning path, avoid obstacles, and avoid collisions and falls.

[0143] The 3D data provided by the depth camera can help the robot vacuum cleaner achieve accurate positioning and determine its position in the home environment. Combined with advanced navigation algorithms, the robot vacuum cleaner can navigate autonomously and clean according to the predetermined cleaning route.

[0144] Combining RGB images and depth information, the depth camera can accurately detect dirty areas on the ground. By analyzing the color, texture, depth and other characteristics of the dirt, the sweeping robot can classify the dirt and adopt corresponding cleaning strategies.

[0145] The depth camera can capture three-dimensional information of the robot vacuum cleaner's surroundings in real time and build a three-dimensional map of the home environment.

[0146] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0147] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A depth camera, characterized in that: include: Projectors, used to project speckle and flood; A receiver, used for receiving a reflection signal of the speckle to generate a speckle pattern; receiving a reflection signal of the floodlight and generating an IR image; RGB camera, used to obtain RGB images; a processor, configured to generate a depth map according to the speckle map or the IR image, determine a ground area according to the RGB image and / or the depth map, fuse the RGB image, the IR image and the speckle map corresponding to the ground area, and detect through a dirt segmentation algorithm to obtain a dirt range.

2. A depth camera according to claim 1, characterized in that: The processor includes: Step S1: generating a depth map according to the speckle pattern or the IR image; Step S2: aligning the RGB image, the depth map, the IR image and the speckle map; Step S3: determining an initial ground area according to the depth map, detecting the initial ground area on the RGB image, and determining a final ground area; Step S4: The RGB image, the IR image and the speckle image corresponding to the final ground area are fused, and detected by a dirt segmentation algorithm to obtain a dirt range.

3. A depth camera according to claim 2, characterized in that: Also includes: Step S5: Fusing the dirt range with the depth map to obtain accurate 3D information of the dirt area.

4. A depth camera according to claim 2, characterized in that: Also includes: Step S6: performing dirt detection in the final ground area according to the RGB image, the IR image and the speckle pattern to obtain a detection frame and a classification result.

5. A depth camera according to claim 2, characterized in that: Step S3 includes: Step S31: determining an initial ground area according to the depth value at the bottom of the depth map; Step S32: obtaining an initial ground area on the RGB image, performing ground detection, and determining a final ground area.

6. A depth camera according to claim 2, characterized in that: Step S3 includes: Step S33: determining an initial ground area according to the depth value at the bottom of the depth map; Step S34: extracting the initial ground area of ​​the RGB image to obtain a lower RGB image, and extracting the initial ground area of ​​the depth map to obtain a lower depth map; Step S35: performing depth reconstruction according to the lower RGB image and the lower depth map to obtain a lower point cloud; Step S36: performing plane fitting according to the point cloud, determining the direction of the plane, and obtaining the ground area.

7. A depth camera according to claim 2, characterized in that: Step S4 includes: Step S41: stitching the RGB image, the IR image and the speckle pattern into a first image; Step S42: extracting features from the first image to obtain first features; Step S43: Use the self-attention mechanism to process the first feature to obtain the dirt range.

8. The depth camera according to claim 2, characterized in that: Step S4 includes: Step S44: extracting features from the RGB image to obtain a first feature map; extracting features from the IR image to obtain a second feature map; extracting features from the speckle map to obtain a third feature map; wherein the first feature map, the second feature map and the third feature map all contain position information; Step S45: splicing the first feature map, the second feature map and the third feature map to obtain a second feature; Step S46: Use the self-attention mechanism to process the second feature to obtain the dirt range.

9. The depth camera according to claim 2, characterized in that: Step S4 includes: Step S47: extracting features from the RGB image to obtain a first feature map; extracting features from the IR image to obtain a second feature map; extracting features from the speckle pattern to obtain a third feature map; Step S48: fusing the first feature map, the second feature map and the third feature map to obtain a third feature; Step S49: Use the self-attention mechanism to process the third feature to obtain the dirt range.

10. A sweeping robot, characterized in that: A depth camera comprising any one of claims 1-9.

Images