Multi-mode ground dirt segmentation method, system and device and storage medium

Abstract

The invention discloses a multi-modal ground dirt segmentation method, system and device and a storage medium. The method comprises the following steps: S1, acquiring a depth map, an IR image and a speckle map; s2, determining a ground area according to the IR image and the depth map; and S3, fusing the IR image and the speckle pattern, and detecting through a smudginess segmentation algorithm to obtain a smudginess range. According to the invention, the degree of liquid and solid smudginess can be effectively identified, and accurate segmentation and 3D modeling are carried out on the liquid and solid smudginess.

Description

Background Art

[0002] With the rapid development of smart home technology, sweeping robots have become one of the important devices for household cleaning. However, existing sweeping robots still have certain limitations in identifying floor dirt, especially when faced with multiple types of floor dirt, the accuracy of identification and the intelligence of processing are insufficient. Common floor dirt includes two categories: liquid and solid, which have great differences in visual and physical properties. How to accurately identify these types of dirt and clean them effectively is an urgent problem to be solved.

[0003] Traditional sweeping robots usually rely on simple visual algorithms or basic sensor data to identify dirt. This approach often shows low recognition rates and unstable performance when facing complex ground environments. For example, transparent or translucent liquid dirt (such as water, oil, milk, etc.) and solid particles (such as gravel, paper scraps, etc.) are very different visually, and it is difficult to effectively identify and distinguish these types of dirt at the same time using simple image processing methods. In addition, traditional visual algorithms and sensors have difficulty calculating the precise 3D information of the dirty area, which is not conducive to the subsequent SLAM, obstacle avoidance, cleaning planning and other tasks of the sweeping robot.

[0004] In order to improve the intelligence and accuracy of ground dirt identification, a more accurate and robust system is needed that can utilize multiple types of image data and combine deep learning algorithms to perform multimodal data fusion and intelligent identification.

[0005] The disclosure of the above background technology content is only used to assist in understanding the inventive concept and technical solution of the present invention. It does not necessarily belong to the prior art of this patent application. In the absence of clear evidence that the above content has been disclosed on the filing date of this patent application, the above background technology should not be used to evaluate the novelty and creativity of the present application. Summary of the invention

[0006] To this end, the present invention uses a deep learning algorithm based on the IR images, speckle patterns and depth maps collected by the depth camera to intelligently identify and classify various types of dirt on the ground. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0007] In a first aspect, the present invention provides a multimodal ground dirt segmentation method, characterized by comprising:

[0008] Step S1: Acquire a depth map, an IR image and a speckle map;

[0009] Step S2: determining a ground area according to the IR image and the depth map;

[0010] Step S3: Fusing the IR image and the speckle pattern, and detecting through a dirt segmentation algorithm to obtain a dirt range.

[0011] Optionally, the multimodal ground dirt segmentation method is characterized by further comprising:

[0012] Step S4: Fusing the dirt range with the depth map to obtain accurate 3D information of the dirt area.

[0013] Optionally, the multimodal ground dirt segmentation method is characterized in that step S2 comprises:

[0014] Step S21: extracting the lower half of the IR image to obtain a lower IR image, and extracting the lower half of the depth map to obtain a lower depth map; wherein the extraction ratios of the IR image and the depth map are the same;

[0015] Step S22: performing depth reconstruction according to the lower IR image and the lower depth map to obtain a lower point cloud;

[0016] Step S23: performing plane fitting according to the point cloud, determining the direction of the plane, and obtaining the ground area.

[0017] Optionally, the multimodal ground dirt segmentation method is characterized in that step S3 comprises:

[0018] Step S31: stitching the IR image and the speckle pattern into a first image;

[0019] Step S32: extracting features from the first image to obtain first features;

[0020] Step S33: Use the self-attention mechanism to process the first feature to obtain the dirt range.

[0021] Optionally, the multimodal ground dirt segmentation method is characterized in that step S3 comprises:

[0022] Step S34: extracting features from the IR image to obtain a first feature map; extracting features from the speckle map to obtain a second feature map; wherein both the first feature map and the second feature map contain position information;

[0023] Step S35: splicing the first feature map and the second feature map to obtain a second feature;

[0024] Step S36: Use the self-attention mechanism to process the second feature to obtain the dirt range.

[0025] Optionally, the multimodal ground dirt segmentation method is characterized in that step S3 comprises:

[0026] Step S37: extracting features from the IR image to obtain a first feature map; extracting features from the speckle map to obtain a second feature map; wherein both the first feature map and the second feature map contain position information;

[0027] Step S38: fusing the first feature map and the second feature map to obtain a third feature;

[0028] Step S39: Use the self-attention mechanism to process the third feature to obtain the dirt range.

[0029] Optionally, the multimodal ground dirt segmentation method is characterized in that, in step S38, fusion is performed according to the position information, and in step S39, higher weights are assigned to different parts of the third feature.

[0030] In a second aspect, the present invention provides a multimodal ground dirt segmentation system, which is used to implement any of the multimodal ground dirt segmentation methods described above, and is characterized by comprising:

[0031] An acquisition module, used for acquiring a depth map, an IR image and a speckle map;

[0032] A ground module, configured to determine a ground area according to the IR image and the depth map;

[0033] The segmentation module is used to fuse the IR image and the speckle pattern, and detect the dirt range through a dirt segmentation algorithm.

[0034] In a third aspect, the present invention provides a multi-modal ground dirt segmentation device, characterized in that it includes:

[0035] processor;

[0036] a memory storing executable instructions of the processor;

[0037] Wherein, the processor is configured to perform the steps of any of the aforementioned multimodal ground dirt segmentation methods by executing the executable instructions.

[0038] In a fourth aspect, the present invention provides a computer-readable storage medium for storing a program, characterized in that when the program is executed, the steps of any of the multimodal ground dirt segmentation methods described above are implemented.

[0039] Compared with the prior art, the present invention has the following beneficial effects:

[0040] Based on the IR images, speckle patterns and depth maps collected by the depth camera, the present invention uses a deep learning algorithm to intelligently identify and classify various types of dirt on the ground. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0041] The present invention can accurately identify and classify various degrees of dirt on the ground based on multimodal image data and deep learning algorithms, and can calculate accurate 3D information of the dirty area. On the one hand, the present invention can more accurately and robustly identify and classify various degrees of dirt on the ground, making it easier for the sweeping robot to adopt a more effective cleaning strategy; on the other hand, it can calculate accurate 3D information of the dirty area for subsequent SLAM, obstacle avoidance, cleaning planning and other tasks.

[0042] The present invention uses multiple image sources (including IR images, speckle images and depth images) for information fusion to improve the segmentation effect of liquid contamination targets. Different from the practice of only using RGB images or IR images in the prior art, the multimodal image fusion method of the present invention can capture the characteristics of liquid contamination more comprehensively, thereby overcoming the limitations of a single image source. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following will briefly introduce the drawings required for use in the embodiments or the prior art descriptions. Obviously, the drawings in the following descriptions are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without creative work. By reading the detailed description of the non-limiting embodiments with reference to the following drawings, other features, purposes and advantages of the present invention will become more obvious:

[0044] Figure 1 This is a flowchart of the steps of a multi-modal ground dirt segmentation method in an embodiment of the present invention;

[0045] Figure 2 This is a flowchart of another multi-modal ground dirt segmentation method in an embodiment of the present invention;

[0046] Figure 3 A flowchart of a step of determining a ground area in an embodiment of the present invention;

[0047] Figure 4 This is a flow chart of steps for obtaining a dirt range in an embodiment of the present invention;

[0048] Figure 5 This is another flowchart of steps for obtaining a dirt range in an embodiment of the present invention;

[0049] Figure 6This is another flowchart of steps for obtaining a dirt range in an embodiment of the present invention;

[0050] Figure 7 Schematic diagram of the structure of a multi-modal ground dirt segmentation system in an embodiment of the present invention;

[0051] Figure 8 is a schematic structural diagram of a multi-modal ground dirt segmentation device in an embodiment of the present invention; and

[0052] Fig. 9 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. DETAILED DESCRIPTION

[0053] The present invention is described in detail below in conjunction with specific embodiments. The following embodiments will help those skilled in the art to further understand the present invention, but are not intended to limit the present invention in any form. It should be noted that, for those of ordinary skill in the art, several variations and improvements may be made without departing from the concept of the present invention. These all belong to the protection scope of the present invention.

[0054] The terms "first", "second", "third", "fourth", etc. (if any) in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein, for example, can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0055] The embodiment of the present invention provides a multimodal ground dirt segmentation method, which aims to solve the problems existing in the prior art.

[0056] The following specific embodiments are used to describe in detail the technical solutions of the present invention and how the technical solutions of the present application solve the above-mentioned technical problems. The following specific embodiments can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present invention will be described below in conjunction with the accompanying drawings.

[0057] Based on the IR images, speckle patterns and depth maps collected by the depth camera, the present invention uses a deep learning algorithm to intelligently identify and classify various types of dirt on the ground. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0058] In the present invention, multiple image sources (including IR images, speckle images and depth images) are used for information fusion to improve the segmentation effect of liquid contamination targets. Different from the practice of only using RGB images or IR images in the prior art, the multimodal image fusion method of the present invention can capture the characteristics of liquid contamination more comprehensively, thereby overcoming the limitations of a single image source.

[0059] Although IR images are highly robust to changes in illumination and can clearly present the outline of transparent liquid stains, in the case of some special liquids or floor materials, the outline of liquid stains on IR images may not be obvious or clear. A large number of tests have shown that when the outline of liquid stains in IR images is not obvious, the liquid stain areas on the speckle images all show obvious features (laser scattering). Therefore, by introducing speckle images and depth map information, the deficiencies of IR images can be effectively supplemented, and the feature representation of liquid stain targets can be significantly enhanced, making liquid stain targets easier to identify, thereby improving segmentation accuracy.

[0060] Figure 1 FIG. 1 is a flowchart of a multi-modal ground dirt segmentation method according to an embodiment of the present invention. Figure 1 As shown, the steps of a multimodal ground dirt segmentation method in an embodiment of the present invention include:

[0061] Step S1: Acquire a depth map, an IR image, and a speckle map.

[0062] In this step, the depth map is the depth information of the ground obtained by the depth camera. The depth map can reflect the distance from the object to the camera, which helps to determine the three-dimensional structure of the ground area. The IR image is also obtained by the depth camera. Infrared images are sensitive to temperature differences and help detect thermal anomalies or dirty areas on the ground. The speckle image is obtained by laser speckle interferometry technology and can reflect the slight deformation or surface roughness of the ground. In this embodiment, multiple images are obtained by a depth camera, and the multiple images are aligned.

[0063] Step S2: Determine a ground area according to the IR image and the depth map.

[0064] In this step, the ground area usually has specific features in the IR image and depth map, such as temperature distribution, depth value range, etc. By analyzing these features, the scope of the ground area can be preliminarily determined. The ground area is segmented from the image using image segmentation algorithms, such as threshold segmentation, edge detection, etc. This step helps the focus and accuracy of the subsequent dirt detection algorithm.

[0065] Step S3: Fusing the IR image and the speckle pattern, and detecting through a dirt segmentation algorithm to obtain a dirt range.

[0066] In this step, the IR image and the speckle map are fused to make full use of the information in the two images. The fusion process may involve steps such as image registration and weight assignment to ensure that the fused image can accurately reflect the dirtiness of the ground. Advanced dirt segmentation algorithms such as deep learning-based object detection networks (such as YOLO, Faster R-CNN) and image segmentation algorithms (such as MaskR-CNN, U-Net) are used to detect dirty areas on the fused image. These algorithms can automatically identify and segment dirty areas in the image. According to the results of the dirt segmentation algorithm, the scope and location of the dirty area are determined. This step can provide important reference information for subsequent cleaning and maintenance work.

[0067] Figure 2 FIG. 1 is a flowchart of another multi-modal ground dirt segmentation method in an embodiment of the present invention. Figure 2 As shown, compared with the above-mentioned embodiment, another multimodal ground dirt segmentation method in the embodiment of the present invention further includes:

[0068] Step S4: Fusing the dirt range with the depth map to obtain accurate 3D information of the dirt area.

[0069] In this step, the dirty range is fused with the depth map to obtain accurate three-dimensional information of the dirty area. Since the dirty range map is derived based on two-dimensional image information such as IR images, and the depth map provides three-dimensional depth information of the ground, the two need to be registered. The purpose of registration is to ensure that the position of the dirty range on the two-dimensional image is consistent with the three-dimensional position of the corresponding point in the depth map. The registration process may involve steps such as image transformation and coordinate conversion to ensure that the dirty range and the depth map can accurately correspond.

[0070] After the registration is completed, the dirt range map is fused with the depth map. The fusion method may include associating the binary information (or grayscale information) of the dirt range map with the depth value in the depth map to generate a three-dimensional data set containing both the dirty area information and the depth information. The fused data set can be represented as a three-dimensional point cloud or a three-dimensional grid, where each point or grid unit contains the dirt state (dirty or not) and depth information (distance to the camera). The three-dimensional information of the dirty area is extracted from the fused data set. This may include geometric features such as the volume, surface area, shape, and other geometric features of the dirty area, as well as attribute information such as the degree of dirtiness.

[0071] This embodiment can obtain accurate three-dimensional information of the dirty area by fusing the dirt range with the depth map, providing strong data support for subsequent floor cleaning, maintenance and management.

[0072] Figure 3 FIG. 1 is a flow chart of steps for determining a ground area in an embodiment of the present invention. Figure 3 As shown, in an embodiment of the present invention, a step of determining a ground area includes:

[0073] Step S21: extracting the lower half of the IR image to obtain a lower IR image, and extracting the lower half of the depth map to obtain a lower depth map.

[0074] In this step, the extraction ratios of the IR image and the depth map are the same. Since the ground is usually located in the lower half of the image (assuming that the camera perspective is looking down or slightly tilted), extracting the lower half of the IR image and depth map can reduce the complexity and amount of calculation of subsequent processing. According to a preset extraction ratio (for example, 50%), an area of ​​corresponding proportion is cut upward from the bottom of the IR image and depth map to obtain a lower IR image and a lower depth map. Ensure that the extraction ratios of the two images are the same to maintain the spatial correspondence between them.

[0075] Step S22: performing depth reconstruction according to the lower IR image and the lower depth map to obtain a lower point cloud.

[0076] In this step, depth reconstruction converts the depth information in the depth map into point cloud data in three-dimensional space for subsequent spatial analysis and processing. Using the depth information provided by the depth map and the pixel position in the IR image, the three-dimensional spatial coordinates corresponding to each pixel can be calculated. Combining these coordinate points, we get the point cloud below. Each point in the point cloud contains its position information in three-dimensional space.

[0077] Step S23: performing plane fitting according to the point cloud, determining the direction of the plane, and obtaining the ground area.

[0078] In this step, a plane fitting is performed on the lower part of the point cloud. The purpose of plane fitting is to find an optimal plane so that most points in the point cloud are as close to this plane as possible. Plane fitting methods may include least squares method, random sampling consensus (RANSAC) and other algorithms. After the plane fitting is completed, it is necessary to determine the direction of the fitting plane. Usually, the plane direction of the ground is horizontal (that is, the normal vector is close to the vertical direction). By calculating the normal vector of the fitting plane and comparing it with the vertical direction, it can be determined whether the plane is the ground. If the direction of the fitting plane meets the characteristics of the ground (that is, horizontal or close to horizontal), the point cloud area covered by the plane is regarded as the ground area. The ground area may be represented in the form of a point cloud, or it may be described by geometric shapes such as bounding boxes and polygons.

[0079] This embodiment can accurately determine the ground area by extracting the lower half of the IR image and the depth map, performing depth reconstruction and point cloud generation, as well as plane fitting and ground area judgment, thereby providing an important basis for subsequent dirt segmentation and processing.

[0080] Figure 4 FIG. 1 is a flow chart of steps for obtaining a dirt range in an embodiment of the present invention. Figure 4 As shown, in an embodiment of the present invention, a step of obtaining a dirt range includes:

[0081] Step S31: stitching the IR image and the speckle pattern into a first image.

[0082] In this step, in order to make full use of the information contained in the IR image and the speckle pattern, they are spliced ​​together to form a new image (the first image), so that the new image contains both temperature distribution information (from the IR image) and surface roughness or texture information (from the speckle pattern), which are all useful for dirt detection.

[0083] When stitching, the IR image and the speckle pattern may be combined in some way (such as left-right stitching, top-bottom stitching, or superposition) to form a first image.

[0084] Step S32: extracting features from the first image to obtain first features.

[0085] In this step, useful features are extracted from the spliced ​​first image, and these features will be used for subsequent dirt range detection. Feature extraction is a key step in computer vision, which involves extracting information from an image that can represent the content or nature of the image. For dirt detection, the features that may need to be extracted include color, texture, edge, shape, etc. These features can be extracted by various image processing algorithms (such as filters, edge detectors, texture analyzers, etc.). The extracted feature set is called the first feature.

[0086] Step S33: Use the self-attention mechanism to process the first feature to obtain the dirt range.

[0087] In this step, the self-attention mechanism is a deep learning technique that helps the model pay more attention to important areas or features when processing images. In dirt detection, the use of the self-attention mechanism can help the model locate the dirty areas more accurately. First, you need to build a deep learning model that includes the self-attention mechanism. This model can be based on a convolutional neural network (CNN) or other types of neural networks. The self-attention mechanism is usually integrated into a layer of the model so that the weights can be dynamically adjusted when processing features.

[0088] The first extracted feature is input into the self-attention model. The model processes these features and outputs a feature set weighted by the self-attention mechanism. Finally, the model outputs an image or mask representing the dirty range. Each pixel in this mask indicates whether the corresponding position belongs to the dirty area. Through thresholding or post-processing steps (such as morphological operations), the mask can be converted into a more understandable representation of the dirty range (such as a bounding box, segmented area, etc.).

[0089] This embodiment can accurately detect and extract the dirt range on the ground by splicing multiple images, extracting features, and applying the self-attention mechanism, which provides an important basis for subsequent cleaning, maintenance, and management.

[0090] Figure 5 FIG. 1 is another flow chart of steps for obtaining the dirt range in an embodiment of the present invention. Figure 5 As shown, another step of obtaining the dirt range in the embodiment of the present invention includes:

[0091] Step S34: extracting features from the IR image to obtain a first feature map; extracting features from the speckle pattern to obtain a second feature map.

[0092] In this step, in order to more accurately utilize the information of the IR image and the speckle pattern, features are extracted from them respectively, and a feature map containing position information is generated. The IR image is processed using image processing or deep learning technology (such as convolutional neural network), and features related to dirt detection (such as temperature gradient, thermal anomaly, etc.) are extracted, and a first feature map is generated. Each pixel or area in this feature map corresponds to a position in the IR image and contains feature information of the position. Similarly, the speckle pattern is processed using image processing or deep learning technology, and features related to surface roughness, texture or shape changes are extracted, and a second feature map is generated. This feature map also contains position information, so that each pixel or area can be matched with the corresponding position in the speckle pattern.

[0093] Step S35: concatenate the first feature map and the second feature map to obtain a second feature.

[0094] In this step, the feature maps of the IR image and the speckle pattern are spliced ​​together to form a comprehensive feature representation so that the subsequent self-attention mechanism can simultaneously utilize the information of the two modalities. The first feature map and the second feature map are combined in a certain way (such as channel splicing) to form a new feature map (second feature). This new feature map contains both the temperature information in the IR image and the surface texture information in the speckle pattern, providing richer data support for dirt detection.

[0095] Step S36: Use the self-attention mechanism to process the second feature to obtain the dirt range.

[0096] In this step, the concatenated features are processed using a self-attention mechanism to dynamically adjust the weights and highlight the dirty areas. Build a deep learning model that includes a self-attention mechanism that can process the concatenated feature maps. Input the second feature into the self-attention model. The model will calculate the attention weights for each pixel or region based on the information in the feature map, which reflect their importance in dirt detection. Based on the calculated attention weights, the model will generate an image or mask representing the dirt range. Each pixel in this mask indicates whether the corresponding position belongs to a dirty area. Through threshold processing or post-processing steps, the mask can be converted into a more understandable representation of the dirt range (such as a bounding box, segmented area, etc.).

[0097] After fusion of features, this embodiment uses a self-attention mechanism to process the feature map to further improve segmentation performance and solve the problem of missed detection or incomplete recognition of liquid contamination targets; it combines the information of IR images and speckle patterns and uses a self-attention mechanism to achieve accurate detection of the contamination range. This method not only improves the accuracy of contamination detection, but also provides important reference information for subsequent cleaning and maintenance work.

[0098] Figure 6 FIG. 1 is another flow chart of steps for obtaining the dirt range in an embodiment of the present invention. Figure 6 As shown, another step of obtaining the dirt range in the embodiment of the present invention includes:

[0099] Step S37: extracting features from the IR image to obtain a first feature map; extracting features from the speckle map to obtain a second feature map; wherein both the first feature map and the second feature map contain position information.

[0100] This step is the same as step S34 and will not be repeated here.

[0101] Step S38: Fusing the first feature map and the second feature map to obtain a third feature.

[0102] In this step, the first feature map, the second feature map and the third feature map are fused to obtain a third feature containing multiple information sources. The fusion process may involve channel merging, weighted summation, maximum pooling and other operations of the feature map. The fused third feature not only contains the feature information in the IR image and the speckle map, but also enhances the robustness and complementarity of the features through the fusion process.

[0103] Step S39: Use the self-attention mechanism to process the third feature to obtain the dirt range.

[0104] In this step, the self-attention mechanism is applied to the fused third feature. Since the second feature contains dirt and ground information, and the ground information is relatively uniform, while the dirt information has different characteristics, the self-attention mechanism can highlight the dirty areas in the image and suppress the information of other areas, especially when the dirty areas are significantly different from the surrounding environment in color, temperature or surface roughness.

[0105] After the self-attention mechanism is processed, the important areas in the image (i.e., dirty areas) will be significantly highlighted. By setting an appropriate threshold or applying a classifier, the highlighted areas can be distinguished from the background, thereby obtaining a binary image or mask of the dirty range.

[0106] The dirty area can be represented in many forms, such as binary image (dirty area is white and the rest is black), grayscale image (dirty degree is represented by grayscale value) or polygonal area (dirty area is described by bounding box or contour line).

[0107] After fusing the features, this embodiment uses a self-attention mechanism to process the feature map to further improve the segmentation performance and solve the problem of missed detection or incomplete recognition of liquid dirty targets; this embodiment can strengthen the focus on the key areas of dirty targets and improve the robustness and accuracy of the model in complex environments. This embodiment extracts the features of the IR image and the speckle pattern separately, fuses these features into a third feature, and then applies the self-attention mechanism for processing, which can accurately detect and extract the dirt range on the ground. It combines multiple image information sources and deep learning technology to improve the accuracy and robustness of dirt detection.

[0108] In some embodiments, in step S38, fusion is performed according to the position information, and in step S39, higher weights are assigned to different parts of the third feature. A fusion strategy based on position information is adopted. This involves pixel-by-pixel fusion. Through the above fusion strategy, the first feature map, the second feature map, and the third feature map are fused into a third feature containing multiple information sources. This feature map not only contains rich feature information, but also enhances the spatial consistency and complementarity of the features through the fusion of position information.

[0109] During the self-attention process, different parts of the third feature (i.e. different pixels or regions) are given higher weights according to their importance or relevance to dirt detection. The weight assignment can be based on the similarity, significance, contextual information or pre-trained model weights of the feature vectors. For example, if a region shows significant dirt features in multiple feature maps, it is given a higher weight in the self-attention mechanism.

[0110] This embodiment combines multiple image information sources, deep learning technology and position information fusion strategy to improve the accuracy and robustness of dirt detection.

[0111] Figure 7 FIG. 1 is a schematic diagram of the structure of a multi-modal ground dirt segmentation system according to an embodiment of the present invention. Figure 7 As shown, a multimodal ground dirt segmentation system in an embodiment of the present invention includes:

[0112] An acquisition module, used for acquiring a depth map, an IR image and a speckle map;

[0113] A ground module, configured to determine a ground area according to the IR image and the depth map;

[0114] The segmentation module is used to fuse the IR image and the speckle pattern, and detect the dirt range through a dirt segmentation algorithm.

[0115] This embodiment uses a deep learning algorithm to intelligently identify and classify various types of dirt on the ground based on the IR images, speckle patterns and depth maps collected by the depth camera. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0116] The embodiment of the present invention further provides a multimodal ground dirt segmentation device, comprising a processor and a memory, wherein executable instructions of the processor are stored, wherein the processor is configured to execute the steps of a multimodal ground dirt segmentation method by executing the executable instructions.

[0117] As described above, this embodiment uses a deep learning algorithm to intelligently identify and classify various types of dirt on the ground based on the IR images, speckle patterns and depth maps collected by the depth camera. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0118] It will be appreciated by those skilled in the art that various aspects of the present invention may be implemented as systems, methods or program products. Therefore, various aspects of the present invention may be specifically implemented in the following forms, namely: complete hardware implementation, complete software implementation (including firmware, microcode, etc.), or a combination of hardware and software implementations, which may be collectively referred to herein as "circuits", "modules" or "platforms".

[0119] Figure 8 Schematic diagram of the structure of a multi-modal ground dirt segmentation device in an embodiment of the present invention. Figure 8 The electronic device 600 according to this embodiment of the present invention is described. Figure 8 The electronic device 600 shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present invention.

[0120] like Figure 8 As shown, the electronic device 600 is in the form of a general computing device. The components of the electronic device 600 may include, but are not limited to: at least one processing unit 610, at least one storage unit 620, a bus 630 connecting different platform components (including the storage unit 620 and the processing unit 610), a display unit 640, etc.

[0121] The storage unit stores program codes, which can be executed by the processing unit 610, so that the processing unit 610 executes the steps of various exemplary embodiments of the present invention described in the above-mentioned multimodal ground dirt segmentation method of this specification. For example, the processing unit 610 can execute the following steps: Figure 1 Follow the steps shown in .

[0122] The storage unit 620 may include a readable medium in the form of a volatile storage unit, such as a random access memory unit (RAM) 6201 and / or a cache memory unit 6202 , and may further include a read-only memory unit (ROM) 6203 .

[0123] The storage unit 620 may also include a program / utility 6204 having a set (at least one) of program modules 6205, such program modules 6205 including but not limited to: an operating system, one or more application programs, other program modules, and program data, each of which or some combination may include an implementation of a grid environment.

[0124] Bus 630 may represent one or more of several types of bus structures, including a memory unit bus or memory unit controller, a peripheral bus, an accelerated graphics port, a processing unit, or a local bus using any of a variety of bus architectures.

[0125] The electronic device 600 may also communicate with one or more external devices 700 (e.g., keyboards, pointing devices, Bluetooth devices, etc.), one or more devices that enable a user to interact with the electronic device 600, and / or any device that enables the electronic device 600 to communicate with one or more other computing devices (e.g., routers, modems, etc.). Such communication may be performed through an input / output (I / O) interface 650. Furthermore, the electronic device 600 may also communicate with one or more grids (e.g., a local area network (LAN), a wide area network (WAN), and / or a public grid, such as the Internet) through a grid adapter 660. The grid adapter 660 may communicate with other modules of the electronic device 600 through the bus 630. It should be understood that although Figure 8Not shown, other hardware and / or software modules may be used in conjunction with electronic device 600, including but not limited to: microcode, device drivers, redundant processing units, external disk drive arrays, RAID systems, tape drives, and data backup storage platforms.

[0126] In an embodiment of the present invention, a computer-readable storage medium is also provided for storing a program, and when the program is executed, the steps of a multimodal ground dirt segmentation method are implemented. In some possible implementations, various aspects of the present invention can also be implemented in the form of a program product, which includes a program code. When the program product is run on a terminal device, the program code is used to enable the terminal device to execute the steps of various exemplary embodiments of the present invention described in the above multimodal ground dirt segmentation method section of this specification.

[0127] As shown above, this embodiment uses a deep learning algorithm to intelligently identify and classify various types of dirt on the ground based on the IR images, speckle patterns and depth maps collected by the depth camera. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0128] Fig. 9 Schematic diagram of the structure of a computer-readable storage medium in an embodiment of the present invention. Fig. 9 As shown, a program product 800 for implementing the above method according to an embodiment of the present invention is described, which can adopt a portable compact disk read-only memory (CD-ROM) and include program code, and can be run on a terminal device, such as a personal computer. However, the program product of the present invention is not limited thereto. In this document, a readable storage medium can be any tangible medium containing or storing a program, which can be used by or in combination with an instruction execution system, an apparatus or a device.

[0129] The program product may use any combination of one or more readable media. The readable medium may be a readable signal medium or a readable storage medium. The readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of readable storage media (a non-exhaustive list) include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.

[0130] Computer readable storage media may include data signals propagated in baseband or as part of a carrier wave, wherein readable program codes are carried. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, device, or device. The program codes contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination thereof.

[0131] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of grid, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0132] This embodiment uses a deep learning algorithm to intelligently identify and classify various types of dirt on the ground based on the IR images, speckle patterns and depth maps collected by the depth camera. It can effectively identify the degree of liquid and solid dirt, and perform accurate segmentation and 3D modeling on it.

[0133] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments, and the same and similar parts between the embodiments can be referred to each other. The above description of the disclosed embodiments enables professionals and technicians in this field to implement or use the present invention. Various modifications to these embodiments will be obvious to professionals and technicians in this field, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present invention. Therefore, the present invention will not be limited to these embodiments shown in this article, but will comply with the widest range consistent with the principles and novel features disclosed herein.

[0134] The above describes the specific embodiments of the present invention. It should be understood that the present invention is not limited to the above specific embodiments, and those skilled in the art may make various modifications or variations within the scope of the claims, which do not affect the essence of the present invention.

Claims

1. A multimodal ground dirt segmentation method, characterized in that: include: Step S1: Acquire a depth map, an IR image and a speckle map; Step S2: determining a ground area according to the IR image and the depth map; Step S3: Fusing the IR image and the speckle pattern, and detecting through a dirt segmentation algorithm to obtain a dirt range.

2. A multimodal ground dirt segmentation method according to claim 1, characterized in that: Also includes: Step S4: Fusing the dirt range with the depth map to obtain accurate 3D information of the dirt area.

3. The multimodal ground dirt segmentation method according to claim 1, characterized in that: Step S2 includes: Step S21: extracting the lower half of the IR image to obtain a lower IR image, and extracting the lower half of the depth map to obtain a lower depth map; wherein the extraction ratios of the IR image and the depth map are the same; Step S22: performing depth reconstruction according to the lower IR image and the lower depth map to obtain a lower point cloud; Step S23: performing plane fitting according to the point cloud, determining the direction of the plane, and obtaining the ground area.

4. The multimodal ground dirt segmentation method according to claim 1, characterized in that: Step S3 includes: Step S31: stitching the IR image and the speckle pattern into a first image; Step S32: extracting features from the first image to obtain first features; Step S33: Use the self-attention mechanism to process the first feature to obtain the dirt range.

5. The multimodal ground dirt segmentation method according to claim 1, characterized in that: Step S3 includes: Step S34: extracting features from the IR image to obtain a first feature map; extracting features from the speckle map to obtain a second feature map; wherein both the first feature map and the second feature map contain position information; Step S35: splicing the first feature map and the second feature map to obtain a second feature; Step S36: Use the self-attention mechanism to process the second feature to obtain the dirt range.

6. The multimodal ground dirt segmentation method according to claim 1, characterized in that: Step S3 includes: Step S37: extracting features from the IR image to obtain a first feature map; extracting features from the speckle map to obtain a second feature map; wherein both the first feature map and the second feature map contain position information; Step S38: fusing the first feature map and the second feature map to obtain a third feature; Step S39: Use the self-attention mechanism to process the third feature to obtain the dirt range.

7. The multimodal ground dirt segmentation method according to claim 6, characterized in that: In step S38, fusion is performed based on the position information, and in step S39, higher weights are assigned to different parts of the third feature.

8. A multimodal ground dirt segmentation system, used to implement the multimodal ground dirt segmentation method according to any one of claims 1 to 7, characterized in that: include: An acquisition module, used for acquiring a depth map, an IR image and a speckle map; A ground module, configured to determine a ground area according to the IR image and the depth map; The segmentation module is used to fuse the IR image and the speckle pattern, and detect the dirt range through a dirt segmentation algorithm.

9. A multi-modal ground dirt segmentation device, characterized in that: include: processor; a memory storing executable instructions of the processor; Wherein, the processor is configured to perform the steps of the multimodal ground dirt segmentation method described in any one of claims 1 to 7 by executing the executable instructions.

10. A computer-readable storage medium for storing a program, characterized in that: When the program is executed, the steps of the multimodal ground dirt segmentation method described in any one of claims 1 to 7 are implemented.

Images