Training method, device and equipment of cleaning equipment driving surface covering identification model

Abstract

The invention discloses a cleaning equipment driving surface covering identification method and device and cleaning equipment. The method comprises the steps that a line laser imaging picture collected by the cleaning equipment in the driving process is acquired; the line laser imaging picture is an image obtained by performing infrared imaging on a line laser projection area; the line laser projection area is an area where the cleaning equipment projects line laser; aiming at each line laser imaging picture, according to line laser infrared imaging information in the line laser imaging picture, carrying out covering object identification to obtain a covering object identification result of a line laser projection area corresponding to the line laser imaging picture; and determining the position of a covering on a driving surface on which the cleaning equipment runs according to the covering identification results corresponding to the plurality of line laser imaging pictures collected by the cleaning equipment. The computing power demand of covering identification can be reduced.

Description

Technical Field

[0001] This application is a divisional application filed with the Chinese Patent Office on May 18, 2023, with application number 2023105619106, entitled "Method, Apparatus and Cleaning Equipment for Identifying Covers on Driving Surface of Cleaning Equipment", the entire contents of which are incorporated herein by reference. Background Technology

[0002] With the development of science and technology, cleaning equipment such as robotic vacuum cleaners and robotic mops are becoming increasingly popular, bringing great convenience to people's lives. In the working environment of these cleaning devices, there are sometimes carpets or other coverings on the floor, which the devices need to navigate around. Therefore, it is crucial for the cleaning equipment to identify coverings on the surface during its operation.

[0003] Traditional methods typically involve using a camera on the cleaning equipment to capture images of the driving surface, then performing semantic segmentation on the images to determine the location of any coverings on the surface, allowing the cleaning equipment to navigate around the coverings. However, this method of identifying coverings through image acquisition and segmentation results in high computational demands. Summary of the Invention

[0004] Therefore, the technical problem to be solved by the present invention is how to reduce the computing power requirements for cover recognition.

[0005] To solve the above-mentioned technical problems, the present invention provides a method for identifying the surface covering of cleaning equipment, comprising:

[0006] Acquire a line laser image captured by the cleaning equipment during operation; the line laser image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning equipment projects the line laser.

[0007] For each line laser imaging image, the covering is identified based on the line laser infrared imaging information in the line laser imaging image, and the covering identification result of the line laser projection area corresponding to the line laser imaging image is obtained.

[0008] Based on the identification results of the coverings corresponding to the multiple line laser images collected by the cleaning equipment, the position of the coverings on the driving surface on which the cleaning equipment is traveling is determined.

[0009] Optionally, in the above-mentioned method for identifying the covering on the driving surface of the cleaning equipment, the covering identification result is obtained by a pre-trained first covering identification model based on the line laser infrared imaging information in the line laser imaging image.

[0010] The training steps for the first covering recognition model include:

[0011] The image content of the target area is extracted from the first sampling image to obtain a sample line laser image, which contains the imaging content of the line laser in the first sampling image;

[0012] Obtain the annotation results corresponding to the laser image of the sample line;

[0013] Based on the sample line laser image and the corresponding annotation results, the first covering recognition model to be trained is iteratively trained to obtain the trained first covering recognition model.

[0014] Optionally, in the above-described method for identifying the surface covering of cleaning equipment, the step of extracting the image content of the target area from the first sampling image to obtain a sample line laser image includes:

[0015] From the central axis of the line laser imaging content in the first sampling image, the image content of multiple target areas is extracted to obtain multiple sample line laser images.

[0016] Optionally, in the above-described method for identifying the surface covering of cleaning equipment, the step of extracting image content of multiple target areas from the central axis of the linear laser imaging content in the first sampling image to obtain multiple sample linear laser images includes:

[0017] In the first sampling image, along the central axis of the line laser imaging content, the image content of the target area that meets the preset size is extracted sequentially with a preset step size to obtain multiple sample line laser images.

[0018] Optionally, in the above-described method for identifying the surface covering of cleaning equipment, obtaining the annotation result corresponding to the sample line laser image includes:

[0019] The sample line laser image is input into the initial coverage recognition model, and the initial annotation result corresponding to the sample line laser image is output.

[0020] Obtain the annotation result corresponding to the laser pattern of the sample line determined based on the initial annotation result;

[0021] The initial overlay recognition model is obtained by training the model in advance based on the sample images extracted from the second sampling image and the corresponding annotation results.

[0022] Optionally, in the above-described method for identifying the covering on the driving surface of cleaning equipment, before performing covering identification based on the line laser infrared imaging information in each of the line laser imaging images to obtain the covering identification result for the line laser projection area corresponding to the line laser imaging image, the method further includes:

[0023] Obtain a height mask image corresponding to each of the line laser imaging images; the height mask image contains the actual height information of each position on the line laser in the line laser imaging image from the driving surface;

[0024] Each line laser image and its corresponding height mask image are fused to obtain a fused multimodal image corresponding to the line laser image.

[0025] The step of identifying the covering in each line laser imaging image based on the line laser infrared imaging information in the line laser imaging image, and obtaining the covering identification result of the line laser projection area corresponding to the line laser imaging image, includes:

[0026] For each of the line laser imaging images, the covering is identified based on the multimodal image corresponding to the line laser imaging image, and the covering identification result of the line laser projection area corresponding to the line laser imaging image is obtained.

[0027] Optionally, in the above-described method for identifying the surface covering of cleaning equipment, fusing each of the line laser imaging images and the corresponding height mask image to obtain the fused multimodal image corresponding to the line laser imaging image includes:

[0028] The pixel values ​​of corresponding pixels in each of the line laser imaging images and the corresponding height mask images are averaged.

[0029] Based on the average value of each pixel obtained through averaging, a fused multimodal image corresponding to the line laser imaging map is generated.

[0030] Optionally, in the above-described method for identifying the surface covering of cleaning equipment, fusing each of the line laser imaging images and the corresponding height mask image to obtain the fused multimodal image corresponding to the line laser imaging image includes:

[0031] At least one identical line laser imaging image and at least one corresponding height mask image are merged into a multi-channel image to obtain a fused multimodal image corresponding to the line laser imaging image.

[0032] Optionally, in the above-described method for identifying the surface covering of the cleaning equipment, the result of the surface covering identification is obtained by a pre-trained second surface covering identification model based on the multimodal image corresponding to the line laser imaging map; the second surface covering identification model is obtained by training the model based on the sample map extracted from the multimodal sampling map and the corresponding annotation result; the multimodal sampling map is obtained by fusing the first sampling map and the corresponding height mask sampling map.

[0033] Optionally, in the above-described method for identifying the covering on the driving surface of the cleaning equipment, determining the position of the covering on the driving surface on which the cleaning equipment is traveling, based on the covering identification results corresponding to the multiple line laser imaging images collected by the cleaning equipment, includes:

[0034] Based on the covering identification results of the line laser projection areas corresponding to the multiple line laser imaging images collected by the cleaning equipment, the positions of each line laser projection area with coverings on the driving surface on which the cleaning equipment is traveling are determined;

[0035] The location of the area occupied by the covering is determined based on the location of the line laser projection area where each covering exists.

[0036] Optionally, in the above-described method for identifying the surface covering of the cleaning equipment during operation, acquiring the line laser image collected by the cleaning equipment during operation includes:

[0037] It projects laser beams into the surrounding area while in motion;

[0038] Infrared imaging is performed on the area to which the line laser is projected to obtain a line laser image.

[0039] Optionally, in the above-described method for identifying the surface covering of a cleaning device, the cleaning device is equipped with at least two line laser units; the projection of line lasers onto the surroundings during travel includes:

[0040] Each of the aforementioned line laser units alternately projects line lasers diagonally forward in the direction of travel of the cleaning equipment.

[0041] The present invention also provides a cleaning equipment driving surface covering identification device, comprising:

[0042] The image acquisition module is used to acquire a line laser image captured by the cleaning equipment during operation; the line laser image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning equipment projects the line laser.

[0043] The covering recognition module is used to identify the coverings for each line laser imaging image based on the line laser infrared imaging information in the line laser imaging image, and to obtain the covering recognition result of the line laser projection area corresponding to the line laser imaging image.

[0044] The covering location determination module is used to determine the location of the covering on the driving surface on which the cleaning equipment is traveling, based on the covering identification results corresponding to the multiple line laser imaging images collected by the cleaning equipment.

[0045] The present invention also provides a cleaning device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the cleaning device driving surface covering identification method in various embodiments of the present application.

[0046] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the cleaning equipment driving surface covering identification method in various embodiments of the present application.

[0047] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the cleaning equipment driving surface covering identification method in various embodiments of the present application.

[0048] The technical solution provided by this invention acquires line laser imaging images collected by cleaning equipment during operation. For each line laser imaging image, the covering is identified based on the line laser infrared imaging information in the image, resulting in the covering identification result for the corresponding line laser projection area. Thus, by simply identifying the line laser infrared imaging information in the line laser imaging image, the covering identification result can be obtained. Compared with the method of semantic segmentation using color images acquired by a camera, this greatly reduces the computational power requirement. Finally, based on the covering identification results corresponding to the multiple line laser imaging images acquired by the cleaning equipment, the position of the covering on the driving surface on which the cleaning equipment is traveling is determined, enabling the determination of the position of the covering on the driving surface with relatively low computational power. Attached Figure Description

[0049] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0050] Figure 1 This is an application environment diagram of the cleaning equipment driving surface covering identification method in one embodiment;

[0051] Figure 2 This is a flowchart illustrating a method for identifying the surface covering of a cleaning device in one embodiment;

[0052] Figure 3 This is a schematic diagram of a linear laser imaging pattern in one embodiment;

[0053] Figure 4 This is a schematic diagram illustrating the determination of the location of an overlay on a map in one embodiment;

[0054] Figure 5 This is a schematic diagram illustrating the extraction of image content of a target region from a first sampled image in one embodiment;

[0055] Figure 6 This is a schematic diagram illustrating the process of annotating sample line laser maps using a model in one embodiment;

[0056] Figure 7 This is a schematic diagram illustrating data fusion in one embodiment;

[0057] Figure 8 This is a structural block diagram of a cleaning equipment driving surface covering identification device in one embodiment;

[0058] Figure 9 This is a structural block diagram of a cleaning equipment driving surface covering identification device in another embodiment;

[0059] Figure 10 This is a schematic diagram of the internal structure of a cleaning device in one embodiment. Detailed Implementation

[0060] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. The present invention will be described in detail below with reference to the accompanying drawings and embodiments. It should be noted that, unless otherwise specified, the embodiments and features in the embodiments of the present invention can be combined with each other.

[0061] It should be noted that the terms "first," "second," etc., in the specification, claims, and drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0062] In this invention, unless otherwise stated, directional terms such as "upper," "lower," "top," and "bottom" are generally used in relation to the direction shown in the accompanying drawings, or in relation to the vertical, perpendicular, or gravitational direction of the component itself; similarly, for ease of understanding and description, "inner" and "outer" refer to the inner and outer contours of each component itself, but the above directional terms are not intended to limit this invention.

[0063] The cleaning equipment driving surface covering identification method provided in this application embodiment can be applied to, for example... Figure 1In the application environment shown, the cleaning device 102 travels on a driving surface 104, where a covering 106 is present. The cleaning device 102 can execute the cleaning device driving surface covering identification method in various embodiments of this application to determine the position of the covering 106 on the driving surface 104. After determining the position of the covering 106, the cleaning device 102 can bypass the area occupied by the covering 106 during its travel. The cleaning device 102 can be, but is not limited to, a self-moving cleaning device such as a sweeping robot, a mopping robot, or a combined sweeping and mopping robot. The driving surface 104 can be the ground or platform on which the cleaning device travels. The covering 106 can be, but is not limited to, a carpet or floor mat.

[0064] Example 1

[0065] This embodiment provides a method for identifying the surface covering of cleaning equipment during operation, such as... Figure 2 As shown, this method is applied to Figure 1 Taking cleaning equipment 102 as an example, the steps include the following:

[0066] Step 202: Obtain the line laser image captured by the cleaning equipment during operation; the line laser image is an image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning equipment projects the line laser.

[0067] The line laser projection area is the region containing the projected line laser.

[0068] The cleaning equipment can project a line laser while in motion and perform infrared imaging on the area where the line laser is projected to obtain a line laser image.

[0069] The cleaning equipment can project line lasers at preset time intervals during operation and perform infrared imaging on the line laser projection area, thereby collecting multiple line laser images during operation.

[0070] The cleaning equipment is equipped with a line laser unit and an infrared imaging unit. The cleaning equipment can project a line laser through the line laser unit and perform infrared imaging on the line laser projection area through the infrared imaging unit to obtain a line laser image.

[0071] Step 204: For each line laser imaging image, identify the covering based on the line laser infrared imaging information in the line laser imaging image to obtain the covering identification result of the line laser projection area corresponding to the line laser imaging image.

[0072] Coverage recognition can be the process of identifying whether an image contains an covering. The result of coverage recognition can be either that an covering exists or that no covering exists.

[0073] For each line laser image, the cleaning equipment can identify the covering based on the line laser infrared imaging information in a preset area of ​​the line laser image, thus obtaining the covering identification result for the corresponding line laser projection area. The preset area can be the central region of the line laser imaging content in the line laser image. The position and size of the preset area can be pre-set according to the orientation of the line laser unit.

[0074] The image content of the preset area refers to the content located within the preset area in the image. For example, if the size of the preset area is 500 pixels * 500 pixels, then the image content within the 500-pixel * 500-pixel area at the preset area's location in the image is the image content of the preset area. The imaging content of the line laser refers to the content displayed in the image after the line laser is imaged onto it. For example: Figure 3 In (1), 302 is the imaging content of the line laser.

[0075] For each line laser image, the cleaning equipment can extract the image content of a preset area from the line laser image to obtain the target line laser image. Then, based on the line laser infrared imaging information in the target line laser image, the covering is identified to obtain the covering identification result of the line laser projection area corresponding to the line laser image.

[0076] The cleaning equipment can input the target line laser image into a pre-trained first covering recognition model. The first covering recognition model identifies the covering based on the line laser infrared imaging information in the target line laser image, obtaining the covering recognition result for the line laser projection area corresponding to the target line laser image. The first covering recognition model is pre-trained using sample line laser images extracted from a first sampling image and the corresponding annotation results.

[0077] The first covering recognition model can be a neural network model. The first covering recognition model can be a convolutional neural network model.

[0078] like Figure 3 Image (1) is a line laser image captured when the line laser is projected onto a non-carpet surface. Figure 3 (2) is a line laser image collected when the line laser is projected onto the carpet. It can be seen that there is a difference between the two line laser images. The covering can be easily and accurately identified based on the line laser imaging information in the line laser image.

[0079] Step 206: Based on the cover identification results corresponding to the multiple line laser imaging images collected by the cleaning equipment, determine the position of the cover on the driving surface on which the cleaning equipment is traveling.

[0080] During operation, the cleaning equipment acquires a line laser image each time. Based on this image, step 204 is executed to determine the coverage identification result within the corresponding line laser projection area. After obtaining the complete location of the coverage based on multiple coverage identification results, the location of the coverage is output. After outputting the coverage location, the cleaning equipment can bypass the coverage during operation and continue acquiring line laser images to determine the locations of other coverages.

[0081] The location of a covering can be the location of the area it occupies. The location of the area occupied by the covering can also be the location of its boundaries. For example... Figure 4 As shown, the cleaning equipment can determine the location of the boundaries of each covering in the diagram.

[0082] The cleaning equipment can determine the location of multiple linear laser imaging areas with coverings on the driving surface on which the cleaning equipment is traveling based on the covering identification results of the linear laser imaging areas corresponding to the multiple linear laser imaging images collected by the cleaning equipment. Based on the location of each linear laser imaging area with coverings, the location of the coverings can be determined.

[0083] The cleaning equipment can map the location of the line laser imaging area corresponding to the cover identification result onto a map of the cleaning environment based on the location information of the cleaning equipment when the line laser imaging image is collected, so as to determine the location of the cover on the map.

[0084] The cleaning equipment can determine its location when acquiring the line laser imaging image based on the time information corresponding to each line laser image. The time information can be a timestamp.

[0085] After determining the location of the covering, the cleaning equipment can avoid the covering during its operation.

[0086] The cleaning equipment can directly bypass the covering based on its boundary position. Alternatively, it can extend the determined boundary position outward by a preset distance to obtain the extended boundary position, and then bypass the covering based on this extended boundary position. For example: Figure 4 The dashed box outside 402 in the image represents the expanded boundary.

[0087] The cleaning equipment driving surface covering identification method in the various embodiments of this application can be applied to ARM (Advanced RISC Machines) processors in cleaning equipment. The cleaning equipment driving surface covering identification method in the various embodiments of this application can be applied to low-computing-power platforms of ARM processors in cleaning equipment.

[0088] The aforementioned method for identifying coverings on the driving surface of cleaning equipment involves acquiring linear laser images collected during the equipment's operation. For each linear laser image, coverings are identified based on the linear laser infrared imaging information within the image, yielding the covering identification result for the corresponding linear laser projection area. This method achieves covering identification simply by recognizing the linear laser infrared imaging information within the linear laser image, significantly reducing computational requirements compared to semantic segmentation of color images acquired by a camera. Finally, based on the covering identification results corresponding to multiple linear laser images collected by the cleaning equipment, the position of the coverings on the driving surface of the equipment is determined. This method enables the determination of covering positions on the driving surface with relatively low computational power. Furthermore, due to the lower computational power requirement, it improves recognition efficiency, reduces latency, and allows for real-time covering identification.

[0089] The cover recognition result is obtained by identifying the cover based on the line laser infrared imaging information in the line laser imaging map using a pre-trained first cover recognition model. The training steps of the first cover recognition model include: extracting the image content of the target area from the first sample map to obtain a sample line laser map; the sample line laser map contains the imaging content of the line laser in the first sample map; obtaining the annotation results corresponding to the sample line laser map; and iteratively training the first cover recognition model to be trained based on the sample line laser map and the corresponding annotation results to obtain the trained first cover recognition model.

[0090] The first sampled image is a line laser imaging image used during the model training phase. The target region (ROI, i.e., region of interest) is a predefined region from which image content is extracted from the first sampled image.

[0091] For each line laser image, the cleaning equipment can identify the covering based on the line laser infrared imaging information in the line laser image using a pre-trained first covering recognition model, and obtain the covering recognition result.

[0092] For each line laser image, the cleaning equipment can extract the image content of a preset area from the line laser image to obtain the target line laser image. Then, the target line laser image is input into a pre-trained first cover recognition model. The first cover recognition model performs cover recognition based on the line laser infrared imaging information in the target line laser image to obtain the cover recognition result.

[0093] During the model training phase, the cleaning equipment can pre-travel and collect multiple line laser imaging images as the first sampling images.

[0094] The initial sampled images collected by the cleaning equipment can be transmitted to a computer device, which can then perform model training steps. The computer device can be a server or a terminal.

[0095] The target region may be the central region of the line laser imaging content in the first sampling image. In other embodiments, the target region may be multiple regions along the central axis of the line laser imaging content in the first sampling image.

[0096] The annotation result indicates whether there is any covering in the laser image of the sample line. The annotation result can be either the presence or absence of covering in the laser image of the sample line.

[0097] The computer device can respond to the annotation operation for each sample line laser pattern and obtain the annotation results corresponding to the sample line laser pattern.

[0098] In other embodiments, the computer device can use an initial covering recognition model to identify the covering in the sample line laser image and obtain the corresponding annotation results. The initial covering recognition model is obtained by pre-training the model based on the sample image extracted from the second sampling image and the corresponding annotation results.

[0099] The computer device can input the sample line laser image and the corresponding annotation results into the first cover recognition model to be trained for cover recognition, output the predicted cover recognition result, and iteratively adjust the model parameters of the first cover recognition model to be trained according to the difference between the predicted cover recognition result and the annotation result until the iteration stopping condition is met, thus obtaining the first cover recognition model that has been trained.

[0100] The first object recognition model to be trained can be either an initial object recognition model or a raw model. The raw model is the untrained model.

[0101] In the above embodiments, the image content of the target region is extracted from the first sampling image to obtain a sample line laser image. Then, the annotation results corresponding to the sample line laser image are obtained. Based on the sample line laser image and the corresponding annotation results, the first covering recognition model to be trained is iteratively trained to obtain a first covering recognition model that can accurately identify coverings. Since covering recognition is performed using line laser imaging images, the computational requirements are low, so a lightweight model can be used, reducing resource consumption.

[0102] Extracting image content of target regions from the first sampling image to obtain sample line laser images includes: extracting image content of multiple target regions from the central axis of the line laser imaging content in the first sampling image to obtain multiple sample line laser images.

[0103] The computer device can extract image content of multiple target areas conforming to a preset size from the central axis of the line laser imaging content in the first sample image, thus obtaining multiple sample line laser images. The preset size is a size pre-set for the target area.

[0104] Multiple target areas include the central region of the imaging content of the linear laser in the first sampling image.

[0105] After obtaining multiple sample line laser images, these images can be filtered to obtain the final sample line laser images. Then, model training is performed based on the final sample line laser images and their corresponding annotations. The filtering process involves selecting sample line laser images that contain line laser imaging content from the multiple sample line laser images and removing those that do not.

[0106] In the above embodiments, image content of multiple target regions is extracted from the central axis of the imaging content of the line laser in the first sampling image to obtain multiple sample line laser images. This can improve the data utilization rate of the first sampling image, realize the data expansion and enhancement of the training samples, and further improve the robustness of the trained model based on the expanded and enhanced training samples.

[0107] Extracting image content from multiple target regions along the central axis of the line laser imaging content in the first sampling image to obtain multiple sample line laser images includes: in the first sampling image, extracting image content from target regions of a preset size sequentially along the central axis of the line laser imaging content with a preset step size to obtain multiple sample line laser images.

[0108] like Figure 5 As shown, 502 is the imaging content of the line laser, and 504 is the central region of the imaging content of the line laser (that is, one of the target regions). The image content of multiple target regions (that is, the dashed boxes in the figure) with the same size as 504 can be extracted sequentially along the central axis of 502 with a preset step size.

[0109] In the above embodiments, in the first sampling image, along the central axis of the imaging content of the line laser, the image content of the target area that conforms to the preset size is extracted sequentially with a preset step size to obtain multiple sample line laser images. This can improve the data utilization rate of the first sampling image, realize the data expansion and enhancement of the training samples, and thus improve the robustness of the trained model based on the expanded and enhanced training samples.

[0110] Obtaining the annotation results corresponding to the sample line laser image includes: inputting the sample line laser image into the initial covering recognition model, outputting the initial annotation results corresponding to the sample line laser image; and obtaining the annotation results corresponding to the sample line laser image determined based on the initial annotation results. The initial covering recognition model is pre-trained using sample images extracted from the second sampling image and their corresponding annotation results.

[0111] The computer device can extract the image content of the target region from the second sampled image in advance to obtain a sample image, and in response to the annotation operation on the sample image, obtain the annotation result of the sample image.

[0112] The computer equipment can train the model in advance based on the sample image extracted from the second sampling image and the corresponding annotation results to obtain an initial overlay recognition model.

[0113] The number of second sampled images can be less than the number of first sampled images.

[0114] The second sampled image can be a portion of the first sampled image, or it can be a sampled image other than the first sampled image. For example, a cleaning device collects 100,000 sampled images. 5,000 images are selected from these 100,000 images as the second sampled image, and the remaining 100,000 images are used as the first sampled image. Alternatively, the remaining images from the 100,000 sampled images, excluding the 5,000 second sampled images, can be used as the first sampled image.

[0115] The server can input the sample line laser image into the initial coverage recognition model, output the initial annotation result corresponding to the sample line laser image, and then, in response to the adjustment operation of the initial annotation result, obtain the annotation result corresponding to the sample line laser image.

[0116] like Figure 6 As shown, the process of annotating the sample line laser image using the initial covering recognition model includes the following steps: inputting the sample line laser image into the initial covering recognition model, and outputting whether there is a covering through the initial covering recognition model. If yes, the sample line laser image is marked as having a covering; otherwise, the sample line laser image is marked as not having a covering, thus obtaining the initial annotation result of the sample line laser image.

[0117] In the above embodiments, the sample line laser map is annotated using the initial covering recognition model to obtain the annotation result of the sample line laser map, which greatly improves the efficiency of annotating the sample line laser map.

[0118] Before identifying the covering in each line laser image based on the line laser infrared imaging information in the line laser image to obtain the covering identification result of the line laser projection area corresponding to the line laser image, the method further includes: obtaining a height mask image corresponding to each line laser image; the height mask image contains the actual height information of each position on the line laser in the line laser image from the driving surface; fusing each line laser image and the corresponding height mask image to obtain a fused multimodal image corresponding to the line laser image; for each line laser image, identifying the covering based on the line laser infrared imaging information in the line laser image to obtain the covering identification result of the line laser projection area corresponding to the line laser image includes: for each line laser image, identifying the covering based on the multimodal image corresponding to the line laser image to obtain the covering identification result of the line laser projection area corresponding to the line laser image.

[0119] Among them, the multimodal image is the image obtained by fusing the line laser image and the corresponding height mask image.

[0120] The pixel value of each pixel in the height mask image represents the actual height of the corresponding pixel in the line laser imaging image from the driving surface.

[0121] For each line laser image, the cleaning equipment can first normalize the line laser image and the corresponding height mask image, and then fuse the normalized line laser image and the height mask image to obtain the fused multimodal image corresponding to the line laser image.

[0122] Normalization can be performed using any of the following methods: maximum-minimum normalization, L1 normalization, L2 norm square root normalization, and L2 norm square normalization.

[0123] like Figure 7 As shown, the fusion process includes the following steps: normalizing the line laser image and the height mask image respectively, and then fusing the normalized line laser image and the height mask image to obtain the fused multimodal image corresponding to the line laser image.

[0124] For each line laser image, the cleaning equipment can use a pre-trained second cover recognition model to identify the cover based on the multimodal image corresponding to the line laser image, and obtain the cover recognition result for the line laser projection area corresponding to the line laser image.

[0125] The cleaning equipment can extract the image content of a preset area from a multimodal image to obtain a target fusion map. Then, the target fusion map is input into a pre-trained second cover recognition model to obtain the cover recognition result of the line laser projection area corresponding to the multimodal image.

[0126] The second covering recognition model can be a neural network model. Specifically, the second covering recognition model can be a convolutional neural network model.

[0127] In the above embodiments, by fusing each line laser image with the corresponding height mask image, a fused multimodal image corresponding to the line laser image is obtained. Compared with the line laser image, the multimodal image has added height information, making the information richer. Then, based on the information-rich multimodal image, the covering is identified to obtain the covering identification result of the line laser projection area. Using richer information for covering identification improves the accuracy of covering identification.

[0128] The process of fusing each line laser image with its corresponding height mask to obtain a fused multimodal image corresponding to the line laser image includes: averaging the pixel values ​​of corresponding pixels in each line laser image and its corresponding height mask; and generating the fused multimodal image corresponding to the line laser image based on the average value of each pixel obtained from the averaging process.

[0129] The averaging process can be any of the following: arithmetic mean, geometric mean, etc.

[0130] The pixel value of each pixel in a multimodal image is the average value of each pixel obtained through averaging.

[0131] In the above embodiments, the pixel values ​​of corresponding pixels in each line laser imaging image and the corresponding height mask image are averaged to generate a fused multimodal image. This allows the multimodal image to integrate the line laser imaging information from the line laser imaging image and the height information from the height mask image, resulting in richer information. Using this richer information for object recognition improves the accuracy of object recognition.

[0132] The process of fusing each line laser image and its corresponding height mask to obtain a fused multimodal image corresponding to the line laser image includes: merging at least one identical line laser image and its corresponding height mask into a multi-channel image to obtain a fused multimodal image corresponding to the line laser image.

[0133] The cleaning equipment can copy at least one of each line laser image and corresponding height mask image, and then merge the copied at least one line laser image and corresponding at least one height mask image into a multi-channel image to obtain a fused multimodal image corresponding to the line laser image.

[0134] The fused multimodal image can be a two-channel image, a three-channel image, a four-channel image, a five-channel image, or a six-channel image, etc., without limitation.

[0135] For example, a cleaning device can copy a line laser image to obtain two line laser images (i.e., two identical line laser images), and then merge these two line laser images with a corresponding height mask image into a three-channel image to obtain the fused multimodal image corresponding to the line laser image.

[0136] For example, a cleaning device can copy a line laser image and a height mask to obtain two line laser images and two height masks (i.e., two identical line laser images and two identical height masks). Then, the two line laser images and the corresponding two height masks at that location are merged into a four-channel image to obtain the fused multimodal image corresponding to the line laser image.

[0137] In the above embodiments, at least one identical line laser imaging image and at least one corresponding height mask image are merged into a multi-channel image to obtain a fused multimodal image corresponding to the line laser imaging image. This makes the multimodal image contain both line laser imaging information and height information, resulting in richer information. Using richer information for covering recognition improves the accuracy of covering recognition.

[0138] The cover recognition result is obtained by identifying the cover based on the multimodal image corresponding to the line laser imaging map through a pre-trained second cover recognition model. The second cover recognition model is trained based on the sample map extracted from the multimodal sampling map and the corresponding annotation results. The multimodal sampling map is obtained by fusing the first sampling map and the corresponding height mask sampling map.

[0139] The height mask sampling image contains information about the actual height of each position on the line laser in the first sampling image from the driving surface.

[0140] The training steps of the second covering recognition model include: fusing the first sampling image and the corresponding height mask sampling image to obtain a multimodal sampling image; extracting the image content of the target region from the multimodal sampling image to obtain a sample image; obtaining the annotation results corresponding to the sample image; and iteratively training the second covering recognition model to be trained based on the sample image and the corresponding annotation results to obtain the trained second covering recognition model. The sample image includes the imaging content of the line laser in the multimodal sampling image.

[0141] The training steps for the second object recognition model can be performed by a computer device. The computer device can be a server or a terminal.

[0142] Computer equipment can extract image content of multiple target areas from the central axis of the imaging content of the laser in a multimodal sampling image, and obtain multiple sample images.

[0143] The computer device can extract image content from target areas of a preset size sequentially along the central axis of the line laser imaging content in a multimodal sampling image, with a preset step size, to obtain multiple sample images.

[0144] The computer device can input the sample image into the initial second overlay recognition model, output the initial annotation result corresponding to the sample image, and then obtain the annotation result corresponding to the sample image determined based on the initial annotation result.

[0145] The computer device can first fuse the second sampling image and the corresponding height mask image to obtain the second multimodal sampling image, then extract the second sample image from the second multimodal sampling image, and train the model based on the second sample image and the corresponding annotation results to obtain the initial second cover recognition model.

[0146] In the above embodiments, a second covering recognition model is obtained by training the model based on the sample map extracted from the multimodal sampling map and the corresponding annotation results. This allows the covering to be recognized based on the rich information after fusion, thereby improving the accuracy of covering recognition.

[0147] Based on the cover identification results corresponding to the multiple line laser images collected by the cleaning equipment, the position of the cover on the driving surface on which the cleaning equipment travels is determined, including: based on the cover identification results of the line laser projection areas corresponding to the multiple line laser images collected by the cleaning equipment, the position of each line laser projection area on the driving surface on which the cleaning equipment travels is determined; based on the position of each line laser projection area on which the cover is located, the position of the area occupied by the cover is determined.

[0148] The cleaning equipment can determine the boundary of the area occupied by the covering by the laser projection area of ​​each covering.

[0149] After determining the location of the area occupied by the covering, the cleaning equipment can avoid the covering during its operation.

[0150] In the above embodiments, based on the covering identification results of the line laser projection areas corresponding to the multiple line laser imaging images collected by the cleaning equipment, the positions of each line laser projection area with coverings on the driving surface on which the cleaning equipment travels are determined. Based on the positions of each line laser projection area with coverings, the position of the area occupied by the covering is determined. This can accurately determine the position of the area occupied by the covering, thereby enabling the cleaning equipment to accurately bypass the covering.

[0151] Acquiring a line laser image captured by the cleaning equipment during operation includes: projecting a line laser into the surrounding area during operation; and performing infrared imaging on the area where the line laser is projected to obtain the line laser image.

[0152] The cleaning equipment can project line lasers into the area in front of the cleaning equipment at preset time intervals during operation, and perform infrared imaging on the line laser projection area in the area in front of the equipment to obtain the corresponding line laser image at each location.

[0153] In the above embodiments, the cleaning equipment projects line lasers into the surrounding area during operation and performs infrared imaging on the line laser projection area to obtain a line laser image. Thus, by simply identifying the line laser infrared imaging information in the line laser image, the covering identification result can be obtained. Compared with the method of semantic segmentation using color images captured by a camera, the computing power is greatly reduced.

[0154] The cleaning equipment is equipped with at least two line laser units; the projection of line lasers into the surrounding area during travel includes: projecting line lasers alternately in the direction of travel of the cleaning equipment through each line laser unit.

[0155] The line laser unit is a hardware unit on the cleaning equipment used to project line lasers.

[0156] The cleaning equipment is equipped with two line laser units. The left line laser unit projects a line laser towards the right front of the direction in which the cleaning equipment travels, while the right line laser unit projects a line laser towards the left front of the direction in which the cleaning equipment travels. The two line laser units project line lasers alternately.

[0157] In the above embodiments, the cleaning equipment alternately projects line lasers in the oblique front direction of the cleaning equipment's travel direction through each line laser unit. On the one hand, the area of ​​the line laser projection area can be expanded by multiple line laser units, thereby improving the efficiency of covering identification. On the other hand, the alternating projection of multiple line laser units can increase the frequency of acquiring line laser images, thereby further improving the efficiency of covering identification and the accuracy of covering identification.

[0158] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the above embodiments may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0159] Example 2

[0160] Based on the same inventive concept, this application also provides a cleaning equipment driving surface covering identification device for implementing the above-mentioned cleaning equipment driving surface covering identification method. The solution provided by this device is similar to the solution described in the above method. Therefore, the specific limitations of one or more cleaning equipment driving surface covering identification device embodiments provided below can be found in the limitations of the cleaning equipment driving surface covering identification method above, and will not be repeated here.

[0161] This embodiment provides a cleaning equipment driving surface covering identification device 800, such as... Figure 8 As shown, it includes:

[0162] The image acquisition module 802 is used to acquire the line laser image captured by the cleaning equipment during operation; the line laser image is the image obtained by infrared imaging of the line laser projection area; the line laser projection area is the area where the cleaning equipment projects the line laser.

[0163] The covering recognition module 804 is used to identify the covering for each line laser imaging image based on the line laser infrared imaging information in the line laser imaging image, and obtain the covering recognition result of the line laser projection area corresponding to the line laser imaging image.

[0164] The cover location determination module 806 is used to determine the location of the cover on the driving surface on which the cleaning equipment is traveling, based on the cover identification results corresponding to the multiple line laser imaging images collected by the cleaning equipment.

[0165] The covering identification result is obtained by identifying the covering based on the line laser infrared imaging information in the line laser imaging map using a pre-trained first covering identification model. For example... Figure 9 As shown, the cleaning equipment driving surface covering recognition device 800 also includes:

[0166] The model training module 808 is used to extract the image content of the target region from the first sampling image to obtain a sample line laser image; the sample line laser image contains the imaging content of the line laser in the first sampling image; obtain the annotation results corresponding to the sample line laser image; and iteratively train the first covering recognition model to be trained based on the sample line laser image and the corresponding annotation results to obtain the trained first covering recognition model.

[0167] The model training module 808 is also used to extract image content of multiple target regions from the central axis of the imaging content of the line laser in the first sampling image, and obtain multiple sample line laser images.

[0168] The model training module 808 is also used to extract image content in target areas that meet preset sizes sequentially along the central axis of the line laser imaging content in the first sampling image, with a preset step size, to obtain multiple sample line laser images.

[0169] The model training module 808 is also used to input the sample line laser image into the initial covering recognition model and output the initial annotation result corresponding to the sample line laser image; and to obtain the annotation result corresponding to the sample line laser image determined according to the initial annotation result; wherein, the initial covering recognition model is obtained by pre-training the model based on the sample image extracted from the second sampling image and the corresponding annotation result.

[0170] like Figure 9 As shown, the cleaning equipment driving surface covering recognition device 800 also includes:

[0171] The image fusion module 810 is used to acquire the height mask map corresponding to each line laser imaging image; the height mask map contains the actual height information of each position on the line laser in the line laser imaging image from the driving surface; and the image fusion module 810 fuses each line laser imaging image and the corresponding height mask map to obtain the fused multimodal image corresponding to the line laser imaging image.

[0172] The covering recognition module 804 is also used to perform covering recognition based on the multimodal image corresponding to each line laser imaging image, and obtain the covering recognition result of the line laser projection area corresponding to the line laser imaging image.

[0173] The image fusion module 810 is also used to average the pixel values ​​of corresponding pixels in each line laser imaging image and the corresponding height mask image; and to generate a fused multimodal image corresponding to the line laser imaging image based on the average value of each pixel obtained by the averaging process.

[0174] The image fusion module 810 is also used to merge at least one identical line laser imaging image and at least one corresponding height mask image into a multi-channel image to obtain a fused multimodal image corresponding to the line laser imaging image.

[0175] The cover recognition result is obtained by identifying the cover based on the multimodal image corresponding to the line laser imaging map through a pre-trained second cover recognition model. The second cover recognition model is trained based on the sample map extracted from the multimodal sampling map and the corresponding annotation results. The multimodal sampling map is obtained by fusing the first sampling map and the corresponding height mask sampling map.

[0176] The covering location determination module 806 is also used to determine the location of each line laser projection area with a covering on the driving surface on which the cleaning equipment travels, based on the covering identification results of the line laser projection areas corresponding to the multiple line laser imaging images collected by the cleaning equipment; and to determine the location of the area occupied by the covering based on the location of each line laser projection area with a covering.

[0177] The image acquisition module 802 is also used to project line lasers into the surrounding area during driving; and to perform infrared imaging on the line laser projection area to obtain a line laser image.

[0178] The cleaning equipment is equipped with at least two line laser units. The image acquisition module 802 is also used to alternately project line lasers in the oblique front direction of the cleaning equipment's travel direction through each line laser unit.

[0179] The aforementioned cleaning equipment driving surface covering identification device acquires line laser imaging images collected during the cleaning equipment's operation. For each line laser imaging image, it identifies the covering based on the line laser infrared imaging information within the image, obtaining the covering identification result for the corresponding line laser projection area. Thus, by simply identifying the line laser infrared imaging information in the line laser imaging image, the covering identification result can be obtained. Compared to using color images acquired by a camera for semantic segmentation, this significantly reduces computational power. Finally, based on the covering identification results corresponding to the multiple line laser imaging images acquired by the cleaning equipment, the position of the covering on the driving surface on which the cleaning equipment is driving is determined, enabling the determination of the covering position on the driving surface with relatively low computational power.

[0180] Each module in the aforementioned cleaning equipment driving surface covering recognition device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in a computer device in hardware form, or stored in the memory of a computer device in software form, so that the processor can call and execute the corresponding operations of each module.

[0181] Example 3

[0182] This embodiment provides a cleaning device, the internal structure of which can be shown in the following diagram. Figure 10As shown, the cleaning device includes a processor, memory, and network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The network interface is used to communicate with external terminals via a network connection. When the computer program is executed by the processor, it implements a method for identifying the surface covering of the cleaning device.

[0183] Those skilled in the art will understand that Figure 10 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0184] In one embodiment, a cleaning device is provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.

[0185] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon that, when executed by a processor, implements the steps in the above method embodiments.

[0186] In one embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above method embodiments.

[0187] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM). The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0188] Obviously, the embodiments described above are merely some, not all, embodiments of the present invention. Based on the embodiments of the present invention, those skilled in the art can make other variations or modifications without creative effort, and all such variations or modifications should fall within the scope of protection of the present invention.

[0189] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of this patent application. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this application should be determined by the appended claims.

Claims

1. A training method for a model of surface covering recognition for cleaning equipment, characterized in that, The method includes: Image content of the target region is extracted from the first sampling image to obtain a sample line laser image; the target region is multiple regions on the central axis of the line laser imaging content in the first sampling image; the sample line laser image contains the line laser imaging content in the first sampling image. In response to the annotation operation for each sample line laser pattern, the annotation result corresponding to the sample line laser pattern is obtained; the annotation result is the result of annotating whether there is an overlay in the sample line laser pattern; Based on the sample line laser image and the corresponding annotation results, the first covering recognition model to be trained is iteratively trained to obtain the trained first covering recognition model.

2. The method according to claim 1, characterized in that, The step of extracting the image content of the target region from the first sampling image to obtain the sample line laser image includes: Image content of multiple target regions is extracted from the central axis of the line laser imaging content in the first sampling image to obtain multiple sample line laser images; wherein, the multiple target regions include the central region of the line laser imaging content in the first sampling image.

3. The method according to claim 2, characterized in that, The process involves extracting image content from multiple target regions along the central axis of the line laser imaging content in the first sample image to obtain multiple sample line laser images, including: In the first sampling image, along the central axis of the line laser imaging content, the image content of the target area that conforms to the preset size is extracted sequentially with a preset step size to obtain multiple sample line laser images; The sample line laser images are obtained by filtering the multiple sample line laser images.

4. The method according to claim 3, characterized in that, The step of filtering the multiple sample line laser images to obtain the sample line laser images includes: The sample line laser image is obtained by selecting the sample line laser image containing the imaging content of line laser from the multiple sample line laser images and removing the sample line laser images that do not contain the imaging content of line laser.

5. The method according to claim 1, characterized in that, The step of obtaining the annotation results corresponding to the sample line laser pattern in response to the annotation operation for each sample line laser pattern includes: The sample line laser image is input into the initial coverage recognition model, and the initial annotation result corresponding to the sample line laser image is output. Obtain the annotation result corresponding to the laser pattern of the sample line determined based on the initial annotation result; The initial overlay recognition model is obtained by training the model in advance based on the sample images extracted from the second sampling image and the corresponding annotation results.

6. The method according to claim 5, characterized in that, The training steps for the initial covering recognition model include: Image content of the target region is extracted from the second sampled image in advance to obtain a sample image, and the annotation result of the sample image is obtained in response to the annotation operation on the sample image; wherein, the number of the second sampled images is less than the number of the first sampled images; The model is trained based on the sample images extracted from the second sampling image and the corresponding annotation results to obtain the initial cover recognition model.

7. A training device for recognizing the surface covering of cleaning equipment, characterized in that, The device includes: An acquisition module is used to extract the image content of a target region from a first sampling image to obtain a sample line laser image; the target region is multiple regions on the central axis of the line laser imaging content in the first sampling image; the sample line laser image contains the line laser imaging content in the first sampling image; The annotation module is used to respond to the annotation operation for each sample line laser pattern and obtain the annotation result corresponding to the sample line laser pattern; the annotation result is the result of annotating whether there is an overlay in the sample line laser pattern; The model training module is used to iteratively train the first covering recognition model to be trained based on the sample line laser image and the corresponding annotation results, so as to obtain the trained first covering recognition model.

8. A cleaning device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 6.

Images