Cleaning equipment driving surface covering identification method and device and cleaning equipment
Through the integration of linear laser imaging technology and multi-modal image, the problem of high computing power demand for cleaning equipment when identifying the travel surface covering is solved, and efficient and low-latency covering recognition is achieved.
Patent Information
- Application Number
- CN202310561910.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-18
- Publication Date
- 2024-11-26
AI Technical Summary
When existing cleaning equipment recognizes coverings on the driving surface, the computing power demand is high, resulting in inefficiency.
Linear laser imaging technology is used to obtain infrared imaging images, cover recognition is performed through pre-trained cover recognition models, and multi-modal image fusion is performed with height mask images to reduce computing power requirements.
It realizes accurate identification of the covering position on the driving surface under low computing power, improves identification efficiency and accuracy, and reduces delays.
Smart Images

Figure CN119027798A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the technical field of cleaning equipment, and in particular relates to a method and device for identifying a covering on a running surface of a cleaning equipment, and the cleaning equipment. Background Art
[0002] With the development of science and technology, cleaning equipment such as sweeping robots or mopping robots are becoming more and more popular, bringing great convenience to people's lives. Sometimes there are coverings such as carpets on the ground in the working environment of the cleaning equipment, and the cleaning equipment needs to travel around these coverings. Therefore, it is very important for the cleaning equipment to identify the coverings on the driving surface during driving.
[0003] In the traditional method, the camera on the cleaning equipment generally collects images of the driving surface, and then performs semantic segmentation processing on the collected images to determine the location of the covered objects on the driving surface, so that the cleaning equipment can drive around the covered objects. However, this method of identifying covered objects through image collection and image segmentation results in high computing power requirements. Summary of the invention
[0004] Therefore, the technical problem to be solved by the present invention is how to reduce the computing power requirement for covering object recognition.
[0005] In order to solve the above technical problems, the present invention provides a method for identifying a covering on the running surface of a cleaning device, comprising:
[0006] Acquire a line laser imaging image collected by the cleaning device during driving; the line laser imaging image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning device projects the line laser;
[0007] For each of the line laser imaging images, performing cover recognition according to the line laser infrared imaging information in the line laser imaging image, and obtaining a cover recognition result of the line laser projection area corresponding to the line laser imaging image;
[0008] The position of the covering object on the driving surface on which the cleaning device is driving is determined according to the covering object recognition results respectively corresponding to the multiple line laser imaging images collected by the cleaning device.
[0009] Optionally, in the above-mentioned method for identifying a covering on the running surface of the cleaning equipment, the covering identification result is obtained by performing covering identification according to the line laser infrared imaging information in the line laser imaging image by using a pre-trained first covering identification model;
[0010] The training step of the first covering object recognition model comprises:
[0011] Extracting the image content of the target area from the first sampling image to obtain a sample line laser image, wherein the sample line laser image includes the imaging content of the line laser in the first sampling image;
[0012] Obtaining a labeling result corresponding to the sample line laser image;
[0013] The first covering object recognition model to be trained is iteratively trained according to the sample line laser image and the corresponding annotation result to obtain a trained first covering object recognition model.
[0014] Optionally, in the above-mentioned method for identifying a covering on the travel surface of a cleaning device, extracting the image content of the target area from the first sampling image to obtain a sample line laser image includes:
[0015] The image contents of multiple target areas are extracted from the central axis of the imaging contents of the line laser in the first sampling image to obtain multiple sample line laser images.
[0016] Optionally, in the above-mentioned method for identifying a covering on the travel surface of a cleaning device, extracting image contents of multiple target areas on the central axis of the imaging contents of the line laser in the first sampling image to obtain multiple sample line laser images includes:
[0017] In the first sampling image, along the central axis of the imaging content of the line laser, the image content in the target area that meets the preset size is extracted in sequence with a preset step length to obtain multiple sample line laser images.
[0018] Optionally, in the above-mentioned method for identifying a covering on the travel surface of a cleaning device, the step of obtaining a labeling result corresponding to the sample line laser image includes:
[0019] Inputting the sample line laser image into an initial cover recognition model, and outputting an initial annotation result corresponding to the sample line laser image;
[0020] Obtaining a labeling result corresponding to the sample line laser image determined according to the initial labeling result;
[0021] The initial covering object recognition model is obtained by pre-training the model based on the sample image extracted from the second sampling image and the corresponding annotation results.
[0022] Optionally, in the above-mentioned method for identifying a covering on the driving surface of the cleaning equipment, before performing covering identification on each of the line laser imaging images according to the line laser infrared imaging information in the line laser imaging image to obtain a covering identification result of the line laser projection area corresponding to the line laser imaging image, the method further includes:
[0023] Acquire a height mask image corresponding to each of the line laser imaging images; the height mask image contains height information of each position on the line laser in the line laser imaging image from the actual distance to the driving surface;
[0024] Fusing each of the line laser imaging images with the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image;
[0025] For each of the line laser imaging images, the covering object identification is performed according to the line laser infrared imaging information in the line laser imaging image, and the covering object identification result of the line laser projection area corresponding to the line laser imaging image is obtained, which includes:
[0026] For each of the line laser imaging images, covering object recognition is performed according to the multimodal image corresponding to the line laser imaging image to obtain a covering object recognition result of the line laser projection area corresponding to the line laser imaging image.
[0027] Optionally, in the above-mentioned method for identifying a covering on the driving surface of a cleaning device, the step of fusing each of the line laser imaging images with the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image comprises:
[0028] Averaging the pixel values of corresponding pixel points in each of the line laser imaging images and the corresponding height mask image;
[0029] According to the average values corresponding to the respective pixel points obtained by the averaging process, a fused multimodal image corresponding to the line laser imaging image is generated.
[0030] Optionally, in the above-mentioned method for identifying a covering on the driving surface of a cleaning device, the step of fusing each of the line laser imaging images with the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image comprises:
[0031] At least one identical line laser imaging image and at least one corresponding height mask image are merged into a multi-channel image to obtain a fused multimodal image corresponding to the line laser imaging image.
[0032] Optionally, in the above-mentioned method for identifying covers on the driving surface of cleaning equipment, the cover identification result is obtained by performing cover identification on the multimodal image corresponding to the line laser imaging image using a pre-trained second cover identification model; the second cover identification model is obtained by model training based on a sample image extracted from a multimodal sampling image and the corresponding annotation results; the multimodal sampling image is obtained by fusing the first sampling image and the corresponding height mask sampling image.
[0033] Optionally, in the above-mentioned method for identifying a covering on the driving surface of the cleaning device, determining the position of the covering on the driving surface on which the cleaning device is driving according to the covering identification results corresponding to the multiple line laser imaging images collected by the cleaning device includes:
[0034] Determine the positions of the respective line laser projection areas where the cleaning device is traveling and has covered objects on the driving surface on which the cleaning device is traveling, according to the results of identifying covered objects in the line laser projection areas respectively corresponding to the plurality of line laser imaging images collected by the cleaning device;
[0035] The position of the area occupied by the covering is determined based on the position of each linear laser projection area where the covering exists.
[0036] Optionally, in the above-mentioned method for identifying an object covered on the driving surface of a cleaning device, the step of acquiring a line laser imaging image collected by the cleaning device during driving comprises:
[0037] Projects line lasers around while driving;
[0038] The line laser projection area is subjected to infrared imaging to obtain a line laser imaging diagram.
[0039] Optionally, in the above-mentioned method for identifying objects covered on the driving surface of the cleaning device, the cleaning device is provided with at least two line laser units; and projecting the line laser to the surroundings during driving includes:
[0040] The line laser units are used to alternately project the line laser obliquely forward in the driving direction of the cleaning device.
[0041] The present invention also provides a cleaning equipment running surface covering identification device, comprising:
[0042] An image acquisition module is used to acquire a line laser image captured by the cleaning device during driving; the line laser image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning device projects the line laser;
[0043] A covering object recognition module, for performing covering object recognition according to the line laser infrared imaging information in each line laser imaging image, and obtaining a covering object recognition result of the line laser projection area corresponding to the line laser imaging image;
[0044] The covering object position determination module is used to determine the position of the covering object on the driving surface on which the cleaning device is driving according to the covering object recognition results respectively corresponding to the multiple line laser imaging images collected by the cleaning device.
[0045] The present invention also provides a cleaning device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method for identifying a travel surface cover of the cleaning device in each embodiment of the present application is implemented.
[0046] The present invention also provides a computer-readable storage medium having a computer program stored thereon, and when the computer program is executed by a processor, the method for identifying the travel surface cover of the cleaning equipment in each embodiment of the present application is implemented.
[0047] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements the method for identifying the travel surface cover of the cleaning equipment in each embodiment of the present application.
[0048] The technical solution provided by the present invention obtains line laser imaging images collected by the cleaning equipment during driving, and for each line laser imaging image, performs covering recognition according to the line laser infrared imaging information in the line laser imaging image to obtain the covering recognition result of the line laser projection area corresponding to the line laser imaging image, so that the covering recognition result can be obtained by simply identifying the line laser infrared imaging information in the line laser imaging image. Compared with the method of performing semantic segmentation using color images collected by a camera, the computing power requirement is greatly reduced. Finally, the position of the covering on the driving surface on which the cleaning equipment is driving is determined according to the covering recognition results corresponding to the multiple line laser imaging images collected by the cleaning equipment, and the position of the covering on the driving surface can be determined with lower computing power. BRIEF DESCRIPTION OF THE DRAWINGS
[0049] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0050] Figure 1 A diagram showing an application environment of a method for identifying an object covering the travel surface of a cleaning device in one embodiment;
[0051] Figure 2 A schematic diagram of a flow chart of a method for identifying an object covered on the travel surface of a cleaning device in one embodiment;
[0052] Figure 3 A schematic diagram of a linear laser imaging diagram in one embodiment;
[0053] Figure 4 A schematic diagram of determining a position of an overlay in a map in one embodiment;
[0054] Figure 5 is a schematic diagram of extracting image content of a target area from a first sampling image in one embodiment;
[0055] Figure 6 A schematic diagram of a process of labeling a sample line laser image by using a model in one embodiment;
[0056] Figure 7 is a schematic diagram of data fusion in one embodiment;
[0057] Figure 8 It is a structural block diagram of a device for identifying a covering on the running surface of a cleaning device in one embodiment;
[0058] Fig. 9 It is a structural block diagram of a device for identifying an object covered on the travel surface of a cleaning device in another embodiment;
[0059] Fig.10 Schematic diagram of the internal structure of a cleaning device in one embodiment. DETAILED DESCRIPTION
[0060] The technical solution of the present invention will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are part of the embodiments of the present invention, rather than all of the embodiments. The present invention will be described in detail below with reference to the accompanying drawings and in conjunction with the embodiments. It should be noted that the embodiments of the present invention and the features in the embodiments can be combined with each other without conflict.
[0061] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.
[0062] In the present invention, unless otherwise specified, the directional words used, such as "up, down, top, bottom", usually refer to the directions shown in the drawings, or to the components themselves in the vertical, perpendicular or gravity directions; similarly, for ease of understanding and description, "inside and outside" refer to the inside and outside relative to the outline of each component itself, but the above-mentioned directional words are not used to limit the present invention.
[0063] The method for identifying a covering on the travel surface of a cleaning device provided in the embodiment of the present application can be applied to Figure 1In the application environment shown. Among them, the cleaning device 102 travels on the driving surface 104, and there is a covering 106 on the driving surface 104. The cleaning device 102 can execute the cleaning device driving surface covering identification method in each embodiment of the present application to determine the position of the covering 106 on the driving surface 104. After determining the position of the covering 106, the cleaning device 102 can bypass the area occupied by the covering 106 during driving. Among them, the cleaning device 102 can be but not limited to a self-moving cleaning device such as a sweeping robot, a mopping robot, or a sweeping and mopping robot. The driving surface 104 can be the ground or platform on which the cleaning device travels. The covering 106 can be but not limited to a carpet or a floor mat.
[0064] Example 1
[0065] This embodiment provides a method for identifying a covering on the travel surface of a cleaning device. Figure 2 As shown, this method is applied to Figure 1 Taking the cleaning device 102 in the figure as an example, the following steps are included:
[0066] Step 202, obtaining a line laser imaging image collected by the cleaning device during driving; the line laser imaging image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning device projects the line laser.
[0067] The line laser projection area is an area including the projected line laser.
[0068] The cleaning equipment can project a line laser while driving, and perform infrared imaging of the line laser projection area to obtain a line laser imaging map.
[0069] The cleaning equipment can project a line laser at preset time intervals during driving, and perform infrared imaging of the line laser projection area, thereby collecting multiple line laser imaging images during driving.
[0070] The cleaning device is provided with a line laser unit and an infrared imaging unit. The cleaning device can project a line laser through the line laser unit, and perform infrared imaging of the line laser projection area through the infrared imaging unit to obtain a line laser imaging image.
[0071] Step 204 , for each line laser imaging image, performing covering object recognition according to the line laser infrared imaging information in the line laser imaging image, and obtaining a covering object recognition result of the line laser projection area corresponding to the line laser imaging image.
[0072] Cover recognition may be a process of recognizing whether a cover exists in an image. The cover recognition result may be whether a cover exists or not.
[0073] For each line laser imaging image, the cleaning device can identify the covering according to the line laser infrared imaging information in the image content of the preset area in the line laser imaging image, and obtain the covering identification result of the line laser projection area corresponding to the line laser imaging image. The preset area can be the central area of the imaging content of the line laser in the line laser imaging image. The position and size of the preset area can be set in advance according to the orientation of the line laser unit.
[0074] The image content of the preset area refers to the content within the preset area in the image. For example, if the size of the preset area is 500 pixels * 500 pixels, then the image content within the area with a size of 500 pixels * 500 pixels at the position of the preset area in the image is the image content of the preset area. The imaging content of the line laser refers to the content displayed in the image when the line laser is imaged into the image. For example: Figure 3 302 in (1) is the imaging content of the line laser.
[0075] For each line laser imaging image, the cleaning equipment can extract the image content of a preset area from the line laser imaging image to obtain a target line laser image, and then perform covering identification based on the line laser infrared imaging information in the target line laser image to obtain a covering identification result for the line laser projection area corresponding to the line laser imaging image.
[0076] The cleaning device can input the target line laser image into the pre-trained first covering recognition model, and the first covering recognition model can perform covering recognition according to the line laser infrared imaging information in the target line laser image to obtain the covering recognition result of the line laser projection area corresponding to the target line laser image. The first covering recognition model is obtained by pre-training the model according to the sample line laser image extracted from the first sampling image and the corresponding annotation result.
[0077] The first covering recognition model may be a neural network model. The first covering recognition model may be a convolutional neural network model.
[0078] like Figure 3 (1) is the line laser imaging image collected when the line laser is projected onto a non-carpet surface. Figure 3 (2) is a line laser image captured when the line laser is projected onto the carpet. It can be seen that there is a difference between the two line laser images. The covering can be easily and accurately identified based on the line laser imaging information in the line laser imaging image.
[0079] Step 206 , determining the position of the covering on the driving surface on which the cleaning device is driving according to the covering identification results corresponding to the multiple line laser imaging images collected by the cleaning device.
[0080] During the driving process, the cleaning device collects a line laser image each time, and can perform step 204 according to the line laser image to determine the cover recognition result of the line laser projection area corresponding to the line laser image, and output the position of the cover when the position of the complete cover is obtained according to multiple cover recognition results. After outputting the position of the cover, the cleaning device can bypass the cover according to the position of the cover during driving, and can continue to collect line laser images to determine the positions of other covers.
[0081] The position of the covering may be the position of the area occupied by the covering. The position of the area occupied by the covering may be the position of the boundary of the covering. Figure 4 As shown, the cleaning device can determine the location of the boundaries of each covering in the figure.
[0082] The cleaning equipment can determine the positions of multiple line laser imaging areas where covers exist on the driving surface on which the cleaning equipment is driving based on the results of identifying covers in the line laser imaging areas corresponding to multiple line laser imaging images collected by the cleaning equipment, and determine the position of the covers based on the positions of the line laser imaging areas where covers exist.
[0083] The cleaning device can map the position of the line laser imaging area corresponding to the covering object recognition result to the map of the cleaning environment based on the position information of the cleaning device when collecting the line laser imaging image, so as to determine the position of the covering object in the map according to the map.
[0084] The cleaning device can determine the position information of the cleaning device when the line laser imaging image is collected according to the time information corresponding to each line laser imaging image. The time information can be a timestamp.
[0085] After the position of the covering is determined, the cleaning device can bypass the covering according to the position of the covering during driving.
[0086] The cleaning device can directly bypass the covering according to the position of the boundary of the covering. Alternatively, the cleaning device can expand the determined boundary position outward by a preset distance to obtain the position of the expanded boundary, and then bypass the covering according to the position of the expanded boundary. For example: Figure 4 The dotted box outside the 402 in the figure is the expanded boundary.
[0087] The method for identifying a covering on the running surface of a cleaning device in each embodiment of the present application can be applied to an ARM (Advanced RISC Machines) processor in a cleaning device. The method for identifying a covering on the running surface of a cleaning device in each embodiment of the present application can be applied to a low-computing-power platform of an ARM processor in a cleaning device.
[0088] The above-mentioned method for identifying coverings on the driving surface of the cleaning equipment obtains the line laser imaging images collected by the cleaning equipment during driving, and for each line laser imaging image, performs covering identification according to the line laser infrared imaging information in the line laser imaging image, and obtains the covering identification result of the line laser projection area corresponding to the line laser imaging image, thereby obtaining the covering identification result by simply identifying the line laser infrared imaging information in the line laser imaging image, which greatly reduces the computing power requirement compared to the method of using color images collected by the camera for semantic segmentation, and finally determines the position of the covering on the driving surface on which the cleaning equipment is driving according to the covering identification results corresponding to the multiple line laser imaging images collected by the cleaning equipment, and can determine the position of the covering on the driving surface with lower computing power. Moreover, due to the lower computing power requirement, the recognition efficiency is improved, the delay is reduced, and the covering identification can be performed in real time.
[0089] The covering recognition result is obtained by performing covering recognition according to the line laser infrared imaging information in the line laser imaging image by using the pre-trained first covering recognition model. The training steps of the first covering recognition model include: extracting the image content of the target area from the first sampling image to obtain a sample line laser image; the sample line laser image includes the imaging content of the line laser in the first sampling image; obtaining the annotation result corresponding to the sample line laser image; iteratively training the first covering recognition model to be trained according to the sample line laser image and the corresponding annotation result to obtain the trained first covering recognition model.
[0090] The first sampling image is a line laser imaging image used in the model training stage. The target region (ROI, ie, region of interest) is a preset region for extracting image content from the first sampling image.
[0091] For each line laser imaging image, the cleaning device can use a pre-trained first covering object recognition model to perform covering object recognition according to the line laser infrared imaging information in the line laser imaging image to obtain a covering object recognition result.
[0092] For each line laser imaging image, the cleaning equipment can extract the image content of a preset area from the line laser imaging image to obtain a target line laser image, and then input the target line laser image into a pre-trained first covering object recognition model. The first covering object recognition model performs covering object recognition based on the line laser infrared imaging information in the target line laser image to obtain a covering recognition result.
[0093] During the model training stage, the cleaning equipment can be pre-driven to collect multiple line laser imaging images as the first sampling images.
[0094] The first sampling image collected by the cleaning device can be transmitted to a computer device, and the computer device performs the model training step. The computer device can be a server or a terminal.
[0095] The target area may be a central area of the imaging content of the line laser in the first sampling image. In other embodiments, the target area may be a plurality of areas on the central axis of the imaging content of the line laser in the first sampling image.
[0096] The marking result is the result of marking whether there is a covering in the sample line laser image. The marking result can be whether there is a covering in the sample line laser image or not.
[0097] The computer device can obtain the labeling result corresponding to the sample line laser image in response to the labeling operation on each sample line laser image.
[0098] In other embodiments, the computer device can perform cover recognition on the sample line laser image through the initial cover recognition model to obtain the labeling result corresponding to the sample line laser image. The initial cover recognition model is obtained by pre-training the model based on the sample image extracted from the second sampling image and the corresponding labeling result.
[0099] The computer device can input the sample line laser image and the corresponding annotation result into the first covering recognition model to be trained for covering recognition, output the predicted covering recognition result, and iteratively adjust the model parameters of the first covering recognition model to be trained according to the difference between the predicted covering recognition result and the annotation result until the iteration stop condition is met, thereby obtaining the trained first covering recognition model.
[0100] The first covering object recognition model to be trained may be an initial covering object recognition model or an original model, wherein the original model is an untrained model.
[0101] In the above embodiment, the image content of the target area is extracted from the first sampling image to obtain a sample line laser image, and then the labeling result corresponding to the sample line laser image is obtained. According to the sample line laser image and the corresponding labeling result, the first cover recognition model to be trained is iteratively trained to obtain a first cover recognition model that can accurately recognize the cover. Since the line laser imaging image is used for cover recognition, the computing power requirement is relatively low, so it can be realized by using a lightweight model, which reduces resource consumption.
[0102] Extracting the image content of the target area from the first sampling image to obtain the sample line laser image includes: extracting the image content of multiple target areas from the central axis of the imaging content of the line laser in the first sampling image to obtain multiple sample line laser images.
[0103] The computer device can extract image contents of multiple target areas that meet preset sizes from the central axis of the imaging contents of the line laser in the first sampling image to obtain multiple sample line laser images, wherein the preset size is a size pre-set for the target area.
[0104] The plurality of target areas include a central area of the imaging content of the linear laser in the first sampling pattern.
[0105] After obtaining multiple sample line laser images, the multiple sample line laser images can be screened to obtain a final sample line laser image, and then model training is performed based on the final sample line laser image and the corresponding annotation results. Screening is a process of selecting sample line laser images containing imaging content of line lasers from multiple sample line laser images and removing sample line laser images that do not contain imaging content of line lasers.
[0106] In the above embodiment, the image content of multiple target areas is extracted from the central axis of the imaging content of the line laser in the first sampling image to obtain multiple sample line laser images, thereby improving the data utilization rate of the first sampling image, realizing data expansion and enhancement of the training samples, and further improving the robustness of the trained model based on the expanded and enhanced training samples.
[0107] Extracting image contents of multiple target areas from the central axis of the imaging content of the line laser in the first sampling image to obtain multiple sample line laser images includes: in the first sampling image, along the central axis of the imaging content of the line laser, extracting image contents in target areas that meet a preset size in sequence with a preset step length to obtain multiple sample line laser images.
[0108] like Figure 5 As shown, 502 is the imaging content of the line laser, and 504 is the central area of the imaging content of the line laser (that is, one of the target areas). The image contents of multiple target areas (that is, the dotted boxes in the figure) with the same size as 504 can be extracted in sequence along the central axis of 502 with a preset step size.
[0109] In the above embodiment, in the first sampling image, along the central axis of the imaging content of the line laser, the image content in the target area that meets the preset size is extracted in sequence with a preset step length to obtain multiple sample line laser images. This can improve the data utilization rate of the first sampling image, realize data expansion and enhancement of the training samples, and further improve the robustness of the trained model based on the expanded and enhanced training samples.
[0110] Obtaining the annotation result corresponding to the sample line laser image includes: inputting the sample line laser image into the initial cover recognition model, outputting the initial annotation result corresponding to the sample line laser image; and obtaining the annotation result corresponding to the sample line laser image determined according to the initial annotation result. The initial cover recognition model is obtained by pre-training the model according to the sample image extracted from the second sampling image and the corresponding annotation result.
[0111] The computer device may extract the image content of the target area from the second sampling image in advance to obtain a sample image, and obtain a labeling result of the sample image in response to a labeling operation on the sample image.
[0112] The computer device may perform model training in advance based on the sample image extracted from the second sampling image and the corresponding annotation results to obtain an initial covering object recognition model.
[0113] The number of the second sampling maps may be smaller than the number of the first sampling maps.
[0114] The second sampling map can be a part of the first sampling map, or a sampling map other than the first sampling map. For example, the cleaning equipment collects 100,000 sampling maps, 5,000 of which are selected as the second sampling map, and the 100,000 sampling maps are used as the first sampling map, or the sampling maps other than the 5,000 second sampling maps in the 100,000 sampling maps are used as the first sampling map.
[0115] The server may input the sample line laser image into the initial cover recognition model, output the initial annotation result corresponding to the sample line laser image, and then obtain the annotation result corresponding to the sample line laser image in response to the adjustment operation on the initial annotation result.
[0116] like Figure 6 As shown, the process of annotating the sample line laser image through the initial covering recognition model includes the following steps: inputting the sample line laser image into the initial covering recognition model, and outputting whether there is a covering through the initial covering recognition model. If so, annotating the sample line laser image as having a covering; if not, annotating the sample line laser image as not having a covering, thereby obtaining the initial annotation result of the sample line laser image.
[0117] In the above embodiment, the sample line laser image is annotated by using the initial cover recognition model to obtain the annotation result of the sample line laser image, which greatly improves the efficiency of annotating the sample line laser image.
[0118] Before, for each line laser imaging image, performing covering recognition according to the line laser infrared imaging information in the line laser imaging image, and obtaining the covering recognition result of the line laser projection area corresponding to the line laser imaging image, the method also includes: obtaining a height mask image corresponding to each line laser imaging image; the height mask image includes the height information of the actual distance from the driving surface to each position on the line laser in the line laser imaging image; fusing each line laser imaging image with the corresponding height mask image, and obtaining a fused multimodal image corresponding to the line laser imaging image; for each line laser imaging image, performing covering recognition according to the line laser infrared imaging information in the line laser imaging image, and obtaining the covering recognition result of the line laser projection area corresponding to the line laser imaging image includes: for each line laser imaging image, performing covering recognition according to the multimodal image corresponding to the line laser imaging image, and obtaining the covering recognition result of the line laser projection area corresponding to the line laser imaging image.
[0119] Among them, the multimodal image is an image obtained by fusing the line laser imaging image and the corresponding height mask image.
[0120] The pixel value of each pixel point in the height mask image represents the height value of the corresponding pixel point in the corresponding line laser imaging image at the actual mid-range driving surface.
[0121] For each line laser imaging image, the cleaning device can first normalize the line laser imaging image and the corresponding height mask image, and then fuse the normalized line laser imaging image and the height mask image to obtain a fused multimodal image corresponding to the line laser imaging image.
[0122] The normalization process may adopt any one of the normalization processes such as maximum and minimum value normalization, L1 normalization, L2 norm square sum root normalization and L2 norm square normalization.
[0123] like Figure 7 As shown, the fusion process includes the following steps: normalizing the line laser imaging image and the height mask image respectively, and then fusing the normalized line laser imaging image and the height mask image to obtain a fused multimodal image corresponding to the line laser imaging image.
[0124] For each line laser imaging image, the cleaning device can use a pre-trained second covering recognition model to perform covering recognition based on the multimodal image corresponding to the line laser imaging image to obtain a covering recognition result for the line laser projection area corresponding to the line laser imaging image.
[0125] The cleaning device can extract the image content of a preset area from the multimodal image to obtain a target fusion map, and then input the target fusion map into a pre-trained second covering recognition model to obtain a covering recognition result of the line laser projection area corresponding to the multimodal image.
[0126] The second covering recognition model may be a neural network model. The second covering recognition model may be a convolutional neural network model.
[0127] In the above embodiment, each line laser imaging image is fused with the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image. Compared with the line laser image, the multimodal image has increased height information and is more informative. Then, covering identification is performed based on the multimodal image with richer information to obtain a covering identification result of the line laser projection area. The covering identification is performed using richer information, thereby improving the accuracy of covering identification.
[0128] Fusing each line laser imaging image with the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image includes: averaging the pixel values of corresponding pixel points in each line laser imaging image and the corresponding height mask image; generating a fused multimodal image corresponding to the line laser imaging image according to the average values corresponding to each pixel point obtained by the averaging process.
[0129] The averaging process may be any of arithmetic mean and geometric mean.
[0130] The pixel value of each pixel in the multimodal image is the average value of each pixel obtained by averaging.
[0131] In the above embodiment, the pixel values of the corresponding pixel points in each line laser imaging image and the corresponding height mask image are averaged to generate a fused multimodal image, so that the information in the multimodal image is fused with the line laser imaging information in the line laser imaging image and the height information in the height mask image, and the information is richer. Using richer information for covering object recognition improves the accuracy of covering object recognition.
[0132] Fusion of each line laser imaging image and the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image includes: merging at least one identical line laser imaging image and at least one corresponding height mask image into a multi-channel image to obtain a fused multimodal image corresponding to the line laser imaging image.
[0133] The cleaning device can copy at least one of each line laser imaging image and the corresponding height mask image, and then merge the copied at least one line laser imaging image and the corresponding at least one height mask image into a multi-channel image to obtain a fused multimodal image corresponding to the line laser imaging image.
[0134] The fused multimodal image may be a two-channel image, a three-channel image, a four-channel image, a five-channel image, a six-channel image, etc., without limitation.
[0135] For example: the cleaning equipment can copy the line laser imaging image to obtain two line laser imaging images (i.e., two identical line laser imaging images), and then merge the two line laser imaging images and a corresponding height mask image into a three-channel image to obtain a fused multimodal image corresponding to the line laser imaging image.
[0136] For another example: the cleaning equipment can copy the line laser imaging image and the height mask image to obtain the two line laser imaging images and the two height mask images (i.e., two identical line laser imaging images and two identical height mask images), and then merge the two line laser imaging images at the position and the corresponding two height mask images into a four-channel image to obtain a fused multimodal image corresponding to the line laser imaging image.
[0137] In the above embodiment, at least one identical line laser imaging image and at least one corresponding height mask image are merged into a multi-channel image to obtain a fused multimodal image corresponding to the line laser imaging image, so that the multimodal image has both line laser imaging information and height information, and the information is richer. The richer information is used to identify the covering, thereby improving the accuracy of covering identification.
[0138] The covering recognition result is obtained by performing covering recognition on the multimodal image corresponding to the line laser imaging image through a pre-trained second covering recognition model; the second covering recognition model is obtained by model training based on the sample image extracted from the multimodal sampling image and the corresponding annotation results; the multimodal sampling image is obtained by fusing the first sampling image and the corresponding height mask sampling image.
[0139] The height mask sampling map contains the height information of each position on the line laser in the first sampling map from the actual distance to the driving surface.
[0140] The training step of the second covering recognition model includes: fusing the first sampling map and the corresponding height mask sampling map to obtain a multimodal sampling map, then extracting the image content of the target area from the multimodal sampling map to obtain a sample map, then obtaining the annotation result corresponding to the sample map, and iteratively training the second covering recognition model to be trained according to the sample map and the corresponding annotation result to obtain a trained second covering recognition model. The sample map contains the imaging content of the line laser in the multimodal sampling map.
[0141] The step of training the second cover recognition model may be performed by a computer device, which may be a server or a terminal.
[0142] The computer device can extract the image content of multiple target areas from the central axis of the imaging content of the linear laser in the multi-modal sampling image to obtain multiple sample images.
[0143] The computer device can extract the image content in the target area with a preset size in a preset step length in a multi-modal sampling image along the central axis of the imaging content of the line laser to obtain multiple sample images.
[0144] The computer device may input the sample image into the initial second covering object recognition model, output an initial annotation result corresponding to the sample image, and then obtain an annotation result corresponding to the sample image determined according to the initial annotation result.
[0145] The computer device can fuse the second sampling image and the corresponding height mask image in advance to obtain a second multimodal sampling image, and then extract the second sample image from the second multimodal sampling image, perform model training based on the second sample image and the corresponding annotation results, and obtain an initial second covering object recognition model.
[0146] In the above embodiment, model training is performed based on sample images extracted from multimodal sampling images and corresponding annotation results to obtain a second covering recognition model, so that covering recognition can be performed based on the fused rich information through the second covering recognition model, thereby improving the accuracy of covering recognition.
[0147] Determining the position of a covering on the traveling surface on which the cleaning equipment is traveling based on the covering identification results corresponding to the multiple line laser imaging images collected by the cleaning equipment includes: determining the positions of each line laser projection area where a covering exists on the traveling surface on which the cleaning equipment is traveling based on the covering identification results of the line laser projection areas corresponding to the multiple line laser imaging images collected by the cleaning equipment; determining the position of the area occupied by the covering based on the position of each line laser projection area where a covering exists.
[0148] The cleaning device can determine the position of the boundary of the area occupied by the covering according to the position of the linear laser projection area where each covering exists.
[0149] After determining the position of the area occupied by the covering, the cleaning device can bypass the covering according to the position of the area occupied by the covering during driving.
[0150] In the above embodiment, based on the cover recognition results of the line laser projection areas corresponding to the multiple line laser imaging images collected by the cleaning equipment, the positions of the line laser projection areas where the covers exist on the driving surface on which the cleaning equipment is driving are determined, and based on the positions of the line laser projection areas where the covers exist, the position of the area occupied by the covers is determined. The position of the area occupied by the covers can be accurately determined, so that the cleaning equipment can accurately bypass the covers.
[0151] Acquiring the line laser imaging map collected by the cleaning device during driving includes: projecting the line laser to the surroundings during driving; performing infrared imaging on the line laser projection area to obtain the line laser imaging map.
[0152] During driving, the cleaning device can project a line laser to the front area in the driving direction of the cleaning device at preset time intervals, perform infrared imaging on the line laser projection area in the front area where the line laser is projected, and obtain the corresponding line laser imaging map at each position.
[0153] In the above embodiment, the cleaning equipment projects a line laser to the surrounding area while driving, performs infrared imaging on the line laser projection area, and obtains a line laser imaging map, so that the covering object recognition result can be obtained by simply identifying the line laser infrared imaging information in the line laser imaging map. Compared with the method of using color images captured by the camera for semantic segmentation, the computing power is greatly reduced.
[0154] At least two line laser units are arranged on the cleaning device; projecting the line laser to the surroundings during the driving process includes: alternately projecting the line laser toward the oblique front of the driving direction of the cleaning device through each line laser unit.
[0155] Among them, the line laser unit is a hardware unit on the cleaning equipment used to project line laser.
[0156] Two line laser units are arranged on the cleaning device. The left line laser unit projects a line laser toward the right front of the traveling direction of the cleaning device, and the right line laser unit projects a line laser toward the left front of the traveling direction of the cleaning device. The two line laser units project line lasers alternately.
[0157] In the above embodiment, the cleaning device alternately projects a line laser toward the oblique front of the traveling direction of the cleaning device through each line laser unit. On the one hand, the multiple line laser units can expand the area of the line laser projection area, thereby improving the efficiency of covering object recognition. On the other hand, the alternating projection of multiple line laser units can increase the frequency of collecting line laser imaging images, thereby further improving the efficiency of covering object recognition and improving the accuracy of covering object recognition.
[0158] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.
[0159] Example 2
[0160] Based on the same inventive concept, the embodiment of the present application also provides a cleaning equipment running surface cover identification device for implementing the above-mentioned cleaning equipment running surface cover identification method. The implementation scheme for solving the problem provided by the device is similar to the implementation scheme recorded in the above-mentioned method, so the specific limitations in one or more cleaning equipment running surface cover identification device embodiments provided below can refer to the limitations of the cleaning equipment running surface cover identification method above, and will not be repeated here.
[0161] This embodiment provides a cleaning equipment running surface covering identification device 800, such as Figure 8 As shown, including:
[0162] The image acquisition module 802 is used to acquire the line laser imaging image collected by the cleaning device during driving; the line laser imaging image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning device projects the line laser.
[0163] The covering object recognition module 804 is used to perform covering object recognition for each line laser imaging image according to the line laser infrared imaging information in the line laser imaging image, and obtain a covering object recognition result of the line laser projection area corresponding to the line laser imaging image.
[0164] The covering object position determination module 806 is used to determine the position of the covering object on the driving surface on which the cleaning device is driving according to the covering object recognition results corresponding to the multiple line laser imaging images collected by the cleaning device.
[0165] The covering object recognition result is obtained by performing covering object recognition based on the line laser infrared imaging information in the line laser imaging image using the pre-trained first covering object recognition model. Fig. 9 As shown, the cleaning equipment running surface covering identification device 800 also includes:
[0166] The model training module 808 is used to extract the image content of the target area from the first sampling image to obtain a sample line laser image; the sample line laser image includes the imaging content of the line laser in the first sampling image; obtain the annotation result corresponding to the sample line laser image; according to the sample line laser image and the corresponding annotation result, iteratively train the first covering recognition model to be trained to obtain the trained first covering recognition model.
[0167] The model training module 808 is also used to extract the image content of multiple target areas from the central axis of the imaging content of the line laser in the first sampling image to obtain multiple sample line laser images.
[0168] The model training module 808 is also used to extract the image content in the target area that meets the preset size in the first sampling image along the central axis of the imaging content of the line laser in a preset step size to obtain multiple sample line laser images.
[0169] The model training module 808 is also used to input the sample line laser image into the initial covering object recognition model, and output the initial annotation result corresponding to the sample line laser image; obtain the annotation result corresponding to the sample line laser image determined according to the initial annotation result; wherein, the initial covering object recognition model is obtained by pre-model training based on the sample image extracted from the second sampling image and the corresponding annotation result.
[0170] like Fig. 9 As shown, the cleaning equipment running surface covering identification device 800 also includes:
[0171] The image fusion module 810 is used to obtain a height mask map corresponding to each line laser imaging image; the height mask map contains the height information of each position on the line laser in the line laser imaging image from the actual distance to the driving surface; each line laser imaging image and the corresponding height mask map are fused to obtain a fused multimodal image corresponding to the line laser imaging image.
[0172] The covering object recognition module 804 is further used to perform covering object recognition for each line laser imaging image according to the multimodal image corresponding to the line laser imaging image, and obtain a covering object recognition result of the line laser projection area corresponding to the line laser imaging image.
[0173] The image fusion module 810 is also used to average the pixel values of the corresponding pixel points in each line laser imaging image and the corresponding height mask image; based on the average values corresponding to each pixel point obtained by the averaging process, a fused multimodal image corresponding to the line laser imaging image is generated.
[0174] The image fusion module 810 is further used to merge at least one identical line laser imaging image and at least one corresponding height mask image into a multi-channel image to obtain a fused multi-modal image corresponding to the line laser imaging image.
[0175] The covering recognition result is obtained by performing covering recognition on the multimodal image corresponding to the line laser imaging image through a pre-trained second covering recognition model; the second covering recognition model is obtained by model training based on the sample image extracted from the multimodal sampling image and the corresponding annotation results; the multimodal sampling image is obtained by fusing the first sampling image and the corresponding height mask sampling image.
[0176] The covering object position determination module 806 is also used to determine the positions of each line laser projection area where covering objects exist on the driving surface on which the cleaning equipment is traveling, based on the covering object recognition results of the line laser projection areas corresponding to the multiple line laser imaging images collected by the cleaning equipment; and determine the position of the area occupied by the covering object based on the positions of each line laser projection area where covering objects exist.
[0177] The image acquisition module 802 is also used to project a line laser to the surroundings during driving; perform infrared imaging on the line laser projection area to obtain a line laser imaging image.
[0178] The cleaning device is provided with at least two line laser units. The image acquisition module 802 is also used to project line lasers alternately toward the front obliquely of the driving direction of the cleaning device through each line laser unit.
[0179] The above-mentioned cleaning equipment driving surface covering recognition device obtains the line laser imaging image collected by the cleaning equipment during the driving process, and for each line laser imaging image, performs covering recognition according to the line laser infrared imaging information in the line laser imaging image, and obtains the covering recognition result of the line laser projection area corresponding to the line laser imaging image, so that the covering recognition result can be obtained by simply identifying the line laser infrared imaging information in the line laser imaging image. Compared with the method of using color images collected by the camera for semantic segmentation, the computing power is greatly reduced. Finally, according to the covering recognition results corresponding to the multiple line laser imaging images collected by the cleaning equipment, the position of the covering on the driving surface on which the cleaning equipment is driving is determined, and the position of the covering on the driving surface can be determined with lower computing power.
[0180] Each module in the above cleaning equipment running surface cover identification device can be implemented in whole or in part by software, hardware and their combination. Each module can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the corresponding operations of each module.
[0181] Example 3
[0182] This embodiment provides a cleaning device, and its internal structure diagram can be shown as follows: Fig.10As shown. The cleaning device includes a processor, a memory and a network interface connected through a system bus. Among them, the processor of the cleaning device is used to provide computing and control capabilities. The memory of the cleaning device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the cleaning device is used to communicate with an external terminal through a network connection. When the computer program is executed by the processor, a method for identifying a travel surface cover of a cleaning device is implemented.
[0183] Those skilled in the art will understand that Fig.10 The structure shown in the figure is only a block diagram of a part of the structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components.
[0184] In one embodiment, a cleaning device is provided, including a memory and a processor. The memory stores a computer program, and the processor implements the steps in the above method embodiments when executing the computer program.
[0185] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps in the above-mentioned method embodiments are implemented.
[0186] In one embodiment, a computer program product is provided, including a computer program, which implements the steps in the above method embodiments when executed by a processor.
[0187] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetoresistive random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, etc., but are not limited to this.
[0188] Obviously, the embodiments described above are only some embodiments of the present invention, rather than all embodiments. Based on the embodiments of the present invention, ordinary technicians in this field can make other different forms of changes or modifications without creative work, which should fall within the scope of protection of the present invention.
[0189] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.
Claims
1. A method for identifying a covering on the running surface of a cleaning device, characterized in that: The method comprises: Acquire a line laser imaging image collected by the cleaning device during driving; the line laser imaging image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning device projects the line laser; For each of the line laser imaging images, performing cover recognition according to the line laser infrared imaging information in the line laser imaging image, and obtaining a cover recognition result of the line laser projection area corresponding to the line laser imaging image; The position of the covering object on the driving surface on which the cleaning device is driving is determined according to the covering object recognition results respectively corresponding to the multiple line laser imaging images collected by the cleaning device.
2. The method according to claim 1, characterized in that: The covering object recognition result is obtained by performing covering object recognition according to the line laser infrared imaging information in the line laser imaging image by using a pre-trained first covering object recognition model; The training step of the first covering object recognition model comprises: Extracting the image content of the target area from the first sampling image to obtain a sample line laser image; the sample line laser image includes the imaging content of the line laser in the first sampling image; Obtaining a labeling result corresponding to the sample line laser image; The first covering object recognition model to be trained is iteratively trained according to the sample line laser image and the corresponding annotation result to obtain a trained first covering object recognition model.
3. The method according to claim 2, characterized in that Extracting the image content of the target area from the first sampling image to obtain the sample line laser image includes: The image contents of multiple target areas are extracted from the central axis of the imaging contents of the line laser in the first sampling image to obtain multiple sample line laser images.
4. The method according to claim 3, characterized in that The step of extracting image contents of multiple target areas from the central axis of the imaging contents of the line laser in the first sampling image to obtain multiple sample line laser images comprises: In the first sampling image, along the central axis of the imaging content of the line laser, the image content in the target area that meets the preset size is extracted in sequence with a preset step length to obtain multiple sample line laser images.
5. The method according to claim 2, characterized in that: The obtaining of the labeling result corresponding to the sample line laser image includes: Inputting the sample line laser image into an initial cover recognition model, and outputting an initial annotation result corresponding to the sample line laser image; Obtaining a labeling result corresponding to the sample line laser image determined according to the initial labeling result; The initial covering object recognition model is obtained by pre-training the model based on the sample image extracted from the second sampling image and the corresponding annotation results.
6. The method according to claim 1, characterized in that Before performing covering identification for each of the line laser imaging images according to the line laser infrared imaging information in the line laser imaging image to obtain a covering identification result of the line laser projection area corresponding to the line laser imaging image, the method further includes: Acquire a height mask image corresponding to each of the line laser imaging images; the height mask image contains height information of each position on the line laser in the line laser imaging image from the actual distance to the driving surface; Fusing each of the line laser imaging images with the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image; For each of the line laser imaging images, the covering object identification is performed according to the line laser infrared imaging information in the line laser imaging image, and the covering object identification result of the line laser projection area corresponding to the line laser imaging image is obtained, which includes: For each of the line laser imaging images, covering object recognition is performed according to the multimodal image corresponding to the line laser imaging image to obtain a covering object recognition result of the line laser projection area corresponding to the line laser imaging image.
7. The method according to claim 6, characterized in that The step of fusing each of the line laser imaging images with the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image comprises: Averaging the pixel values of corresponding pixel points in each of the line laser imaging images and the corresponding height mask image; According to the average values corresponding to the respective pixel points obtained by the averaging process, a fused multimodal image corresponding to the line laser imaging image is generated.
8. The method according to claim 6, characterized in that The step of fusing each of the line laser imaging images with the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser imaging image comprises: At least one identical line laser imaging image and at least one corresponding height mask image are merged into a multi-channel image to obtain a fused multimodal image corresponding to the line laser imaging image.
9. The method according to claim 6, characterized in that The covering recognition result is obtained by performing covering recognition on the basis of the multimodal image corresponding to the line laser imaging image using a pre-trained second covering recognition model; the second covering recognition model is obtained by performing model training based on sample images extracted from a multimodal sampling image and corresponding annotation results; the multimodal sampling image is obtained by fusing the first sampling image and the corresponding height mask sampling image.
10. The method according to any one of claims 1 to 9, characterized in that The determining the position of the covering on the driving surface on which the cleaning device is driving according to the covering identification results respectively corresponding to the multiple line laser imaging images collected by the cleaning device comprises: Determine the positions of the respective line laser projection areas where the cleaning device is traveling and has covered objects on the driving surface on which the cleaning device is traveling, according to the results of identifying covered objects in the line laser projection areas respectively corresponding to the plurality of line laser imaging images collected by the cleaning device; The position of the area occupied by the covering is determined based on the position of each linear laser projection area where the covering exists.
11. The method according to any one of claims 1 to 9, characterized in that The method of obtaining a line laser imaging image collected by the cleaning device during driving includes: Projects line lasers around while driving; The line laser projection area is subjected to infrared imaging to obtain a line laser imaging diagram.
12. The method according to claim 11, characterized in that The cleaning device is provided with at least two line laser units; The projecting of a line laser to the surroundings during driving includes: The line laser units are used to alternately project the line laser obliquely forward in the driving direction of the cleaning device.
13. A device for identifying a covering on the running surface of a cleaning device, characterized in that: The device comprises: An image acquisition module is used to acquire a line laser image captured by the cleaning device during driving; the line laser image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning device projects the line laser; A covering object recognition module, for performing covering object recognition according to the line laser infrared imaging information in each line laser imaging image, and obtaining a covering object recognition result of the line laser projection area corresponding to the line laser imaging image; The covering object position determination module is used to determine the position of the covering object on the driving surface on which the cleaning device is driving according to the covering object recognition results respectively corresponding to the multiple line laser imaging images collected by the cleaning device.
14. A cleaning device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 12 are implemented.
15. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.
16. A computer program product comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 12 are implemented.
