Ground stain detection method, system and storage medium based on multispectral imaging

Abstract

The present invention relates to a ground stain detection method, system, and storage medium based on multispectral imaging. The method comprises: collecting a multispectral image sequence of the ground using a multispectral imaging device; fusing the multispectral image sequence using an information entropy weighted fusion method to obtain a fused image; detecting stains in the fused image based on a preset target detection model to obtain stain coordinate information; classifying the stains in the multispectral image sequence based on a preset target classification model and the stain coordinate information to obtain stain classification information; and combining the stain coordinate information and the stain classification information to output as a ground stain detection result. The present invention combines the information entropy weighted fusion method with multispectral images to highlight information-rich bands and enhance the significance of stain areas. It also combines multispectral images with target detection and classification to ensure efficient recognition and classification capabilities and have good environmental adaptability.

Description

Technical Field

[0001] The present invention is applicable to the field of machine vision technology applications, and in particular relates to a ground stain detection method, system and storage medium based on multispectral imaging. Background Art

[0002] With the rapid adoption of smart home and service robots, sweeping robots have become essential automated cleaning devices in both home and commercial environments. Currently, most products are equipped with visual and distance sensors such as RGB cameras, infrared range finders, or lidar to enable environmental mapping, obstacle avoidance, and path planning. However, these systems, which rely on visible light or laser reflections, often fail to accurately identify and locate floor stains in low light, strong reflections, shadows, or diverse floor textures. This can lead to missed detections, false detections, or positioning errors, compromising cleaning performance and user experience.

[0003] To improve stain detection accuracy, some research and commercial solutions have introduced deep learning object detection models (such as YOLOv5) to perform end-to-end detection directly on RGB images or pseudo-color multispectral fusion images. These single-stage networks have some advantages in locating coarse target areas, but because they only utilize spatial and color features, they lack the ability to exploit fine-grained spectral reflectance information. This makes it difficult to distinguish between stain types with similar composition or varying transparency, often resulting in inaccurate classification. Summary of the Invention

[0004] The present invention aims to solve the technical problems of insufficient stain detection accuracy and inaccurate category recognition during the operation of existing sweeping robot systems.

[0005] To solve the above technical problems, in a first aspect, the present invention provides a method for detecting floor stains based on multispectral imaging, comprising the following steps:

[0006] Collecting multispectral image sequences of the ground through a multispectral imaging device;

[0007] Using an information entropy weighted fusion method to fuse the multispectral image sequence to obtain a fused image;

[0008] Detecting stains in the fused image based on a preset target detection model to obtain stain coordinate information;

[0009] Classifying the stains in the multispectral image sequence based on a preset target classification model and the stain coordinate information to obtain stain classification information;

[0010] The stain coordinate information and the stain classification information are combined and output as a ground stain detection result.

[0011] Furthermore, the step of fusing the multispectral image sequence using an information entropy weighted fusion method to obtain a fused image includes the following sub-steps:

[0012] The multispectral image sequence is defined as ,in, I i Indicates the i Band images, n is the total number of band channels;

[0013] For each of the band images in the multispectral image sequence I i , calculate the corresponding information entropy H i , the following conditions are met:

[0014] ;

[0015] in, Indicates the i The first of the band images j The probability of gray levels, L is the grayscale level;

[0016] Each of the band images I i The information entropy is normalized to obtain the corresponding weight value , the following conditions are met:

[0017] ;

[0018] Based on each of the band images I i The corresponding weight value , perform weighted fusion on all the band images to obtain the fused image , the following conditions are met:

[0019] .

[0020] Furthermore, before the step of detecting stains in the fused image based on a preset target detection model to obtain stain coordinate information, the method further includes the following steps:

[0021] Normalize the fused image so that the pixel values ​​of the fused image are mapped to the interval [0, 1] to obtain a normalized image I norm , the following conditions are met:

[0022] ;

[0023] in, I min and I max are the minimum and maximum pixel values ​​in the fused image respectively;

[0024] The preset target detection model performs stain detection based on the normalized image.

[0025] Furthermore, the step of classifying the stains in the multispectral image sequence based on the preset target classification model and the stain coordinate information to obtain the stain classification information further includes:

[0026] According to the stain coordinate information, corresponding regions are cropped from different band images of the multispectral image sequence, and the cropped regional images are integrated as a multi-band image set. ,in, I i,j Indicates the i The stained area is j Band images, n is the total number of bands;

[0027] The multi-band image set is normalized, and the preset target classification model performs stain classification based on the normalized multi-band image set.

[0028] Furthermore, the step of collecting a multispectral image sequence of the ground using a multispectral imaging device further includes:

[0029] The average light intensity of the current environment is obtained through the multispectral imaging device. If the average light intensity is less than a preset light threshold, the environment is supplemented with light through the multispectral imaging device.

[0030] Furthermore, the preset target detection model is implemented based on the YOLOv5 target detection model.

[0031] Furthermore, the preset target classification model is implemented based on the MobileViT classification model.

[0032] In a second aspect, the present invention further provides a ground stain detection system based on multispectral imaging, comprising:

[0033] An acquisition module, configured to acquire a multispectral image sequence of the ground through a multispectral imaging device;

[0034] A multispectral image fusion module is used to fuse the multispectral image sequence using an information entropy weighted fusion method to obtain a fused image;

[0035] A stain detection module, configured to detect stains in the fused image based on a preset target detection model to obtain stain coordinate information;

[0036] a stain classification module, configured to classify stains in the multispectral image sequence based on a preset target classification model and the stain coordinate information to obtain stain classification information;

[0037] An output module is used to combine the stain coordinate information and the stain classification information and output them as a ground stain detection result.

[0038] In a third aspect, the present invention also provides a computer device comprising: a memory, a processor, and a multispectral imaging-based ground stain detection program stored in the memory and executable on the processor, wherein when the processor executes the multispectral imaging-based ground stain detection program, the processor implements the steps of the multispectral imaging-based ground stain detection method as described in any one of the above embodiments.

[0039] In a fourth aspect, the present invention also provides a storage medium, on which a ground stain detection program based on multispectral imaging is stored. When the ground stain detection program based on multispectral imaging is executed by a processor, the steps of the ground stain detection method based on multispectral imaging as described in any one of the above embodiments are implemented.

[0040] The beneficial effect achieved by the present invention lies in proposing a ground stain detection method based on multispectral imaging. This method can improve the perception ability of the sweeping robot for the ground environment based on the rich spectral information of the multispectral image, and combine the information entropy weighted fusion method to highlight the information-rich bands and enhance the significance of the stain area; this method also combines multispectral images with target detection and target classification to ensure efficient recognition and classification capabilities, and has good environmental adaptability. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] Figure 1 1 is a schematic diagram of the steps of a method for detecting floor stains based on multispectral imaging provided by an embodiment of the present invention;

[0042] Figure 2 Schematic diagram of the structure of a ground stain detection system based on multispectral imaging provided by an embodiment of the present invention;

[0043] Figure 3 It is a structural diagram of a computer device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solutions and advantages of the present invention more clearly understood, the present invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present invention and are not intended to limit the present invention.

[0045] Please refer to Figure 1 , Figure 1 1 is a schematic flow chart of the steps of a method for detecting floor stains based on multispectral imaging provided by an embodiment of the present invention. The method for detecting floor stains based on multispectral imaging includes the following steps:

[0046] S101. Collect a multispectral image sequence of the ground using a multispectral imaging device.

[0047] The multispectral imaging device in the embodiment of the present invention can be installed in the image capture device of a sweeping robot. Compared to ordinary RGB images, multispectral images have a wider multi-band coverage range and perform better in tasks requiring precise material identification, robustness in complex environments, or professional field detection. The embodiment of the present invention uses a multispectral imaging device to collect multispectral image sequences of the ground for the detection of ground stains. Furthermore, to improve imaging quality, the following steps are also included before step S101:

[0048] The average light intensity of the current environment is obtained through the multispectral imaging device. If the average light intensity is less than a preset light threshold, the environment is supplemented with light through the multispectral imaging device.

[0049] In this way, the illumination adaptability of the ground stain detection method based on multispectral imaging proposed in the embodiment of the present invention can be improved.

[0050] S102 , using an information entropy weighted fusion method to fuse the multispectral image sequence to obtain a fused image.

[0051] A characteristic of multispectral image sequences is that they produce images in different wavelength bands simultaneously. To facilitate subsequent target detection, and considering the differences in information content and sensitivity to stain characteristics between the various wavelength bands, the present invention employs an information entropy weighted fusion method to fuse the multi-band images into a single-channel fused image. Specifically, step S102 includes the following sub-steps:

[0052] The multispectral image sequence is defined as ,in, I i Indicates the i Band images, n is the total number of band channels;

[0053] For each of the band images in the multispectral image sequenceI i , calculate the corresponding information entropy H i , the following conditions are met:

[0054] ;

[0055] in, Indicates the i The first of the band images j The probability of gray levels, L is the grayscale level;

[0056] Each of the band images I i The information entropy is normalized to obtain the corresponding weight value , the following conditions are met:

[0057] ;

[0058] Based on each of the band images I i The corresponding weight value , perform weighted fusion on all the band images to obtain the fused image , the following conditions are met:

[0059] .

[0060] Through the above method, the bands with rich information content can be highlighted in the fused image, thereby enhancing the saliency of the stain area in the image.

[0061] S103: Detect stains in the fused image based on a preset target detection model to obtain stain coordinate information.

[0062] The preset target detection model is implemented based on the YOLOv5 target detection model. The YOLOv5 model can quickly extract image features through its unique network structure and output the coordinate information of the stain's bounding box. Before step S103, the following steps are also included:

[0063] Normalize the fused image so that the pixel values ​​of the fused image are mapped to the interval [0, 1] to obtain a normalized image I norm , the following conditions are met:

[0064] ;

[0065] in, I min and I maxare the minimum and maximum pixel values ​​in the fused image respectively;

[0066] The preset target detection model performs stain detection based on the normalized image. Normalization processing helps to improve the stability and convergence speed of the model.

[0067] S104 : Classify the stains in the multispectral image sequence based on a preset target classification model and the stain coordinate information to obtain stain classification information.

[0068] The preset target classification model is implemented based on the MobileViT classification model. The MobileViT classification model combines the local feature extraction capabilities of convolutional neural networks (CNNs) with the global modeling capabilities of visual transformers (ViTs). It is lightweight and efficient, making it suitable for mobile devices and embedded systems. In this embodiment of the present invention, the MobileViT classification model determines the specific type of stain, such as oil, water, or tea, based on the category corresponding to the highest probability.

[0069] Step S104 further includes:

[0070] According to the stain coordinate information, corresponding regions are cropped from different band images of the multispectral image sequence, and the cropped regional images are integrated as a multi-band image set. ,in, I i,j Indicates the i The stained area is j Band images, n is the total number of bands;

[0071] The multi-band image set is normalized, and the preset target classification model performs stain classification based on the normalized multi-band image set.

[0072] It is understood that the image normalization processes performed for different models in steps S103 and S104 are different steps. For the preset target classification model, the object it processes is a multi-band image. After normalization, in order to facilitate model input and target classification, the multi-band images can also be stacked to form a multi-channel tensor image.

[0073] S105: Combine the stain coordinate information and the stain classification information and output them as a ground stain detection result.

[0074] During the implementation process, the ground stain detection results are not only used to judge the work results of the sweeping robot, but also to formulate corresponding cleaning strategies based on the detection results to guide the cleaning behavior of the sweeping robot.

[0075] The beneficial effect achieved by the present invention lies in proposing a ground stain detection method based on multispectral imaging. This method can improve the perception ability of the sweeping robot for the ground environment based on the rich spectral information of the multispectral image, and combine the information entropy weighted fusion method to highlight the information-rich bands and enhance the significance of the stain area; this method also combines multispectral images with target detection and target classification to ensure efficient recognition and classification capabilities, and has good environmental adaptability.

[0076] The embodiment of the present invention also provides a ground stain detection system 200 based on multispectral imaging, please refer to Figure 2 , Figure 2 : is a schematic structural diagram of a floor stain detection system based on multispectral imaging provided by an embodiment of the present invention, which includes:

[0077] An acquisition module 201 is configured to acquire a multispectral image sequence of the ground using a multispectral imaging device;

[0078] The multispectral image fusion module 202 is configured to fuse the multispectral image sequence using an information entropy weighted fusion method to obtain a fused image.

[0079] A stain detection module 203 is configured to detect stains in the fused image based on a preset target detection model to obtain stain coordinate information;

[0080] A stain classification module 204 is configured to classify stains in the multispectral image sequence based on a preset target classification model and the stain coordinate information to obtain stain classification information;

[0081] The output module 205 is configured to combine the stain coordinate information and the stain classification information and output the result as a ground stain detection result.

[0082] The multispectral imaging-based ground stain detection system 200 can implement the steps in the multispectral imaging-based ground stain detection method in the above embodiment, and can achieve the same technical effects. Please refer to the description in the above embodiment and will not repeat them here.

[0083] The embodiment of the present invention also provides a computer device, please refer to Figure 3 , Figure 3 3 is a structural diagram of a computer device provided in an embodiment of the present invention. The computer device 300 includes: a memory 302, a processor 301, and a ground stain detection program based on multispectral imaging stored in the memory 302 and executable on the processor 301.

[0084] The processor 301 calls the multispectral imaging-based ground stain detection program stored in the memory 302 to execute the steps of the multispectral imaging-based ground stain detection method provided in the embodiment of the present invention. Figure 1 , specifically including the following steps:

[0085] S101. Collect a multispectral image sequence of the ground using a multispectral imaging device.

[0086] Before step S101, the following steps are also included:

[0087] The average light intensity of the current environment is obtained through the multispectral imaging device. If the average light intensity is less than a preset light threshold, the environment is supplemented with light through the multispectral imaging device.

[0088] S102 , using an information entropy weighted fusion method to fuse the multispectral image sequence to obtain a fused image.

[0089] Step S102 includes the following sub-steps:

[0090] The multispectral image sequence is defined as ,in, I i Indicates the i Band images, n is the total number of band channels;

[0091] For each of the band images in the multispectral image sequence I i , calculate the corresponding information entropy H i , the following conditions are met:

[0092] ;

[0093] in, Indicates the i The first of the band images j The probability of gray levels, L is the grayscale level;

[0094] Each of the band images I i The information entropy is normalized to obtain the corresponding weight value , the following conditions are met:

[0095] ;

[0096] Based on each of the band images I i The corresponding weight value , perform weighted fusion on all the band images to obtain the fused image , the following conditions are met:

[0097] .

[0098] S103: Detect stains in the fused image based on a preset target detection model to obtain stain coordinate information.

[0099] The preset target detection model is implemented based on the YOLOv5 target detection model. Before step S103, the following steps are also included:

[0100] Normalize the fused image so that the pixel values ​​of the fused image are mapped to the interval [0, 1] to obtain a normalized image I norm , the following conditions are met:

[0101] ;

[0102] in, I min and I max are the minimum and maximum pixel values ​​in the fused image respectively;

[0103] The preset target detection model performs stain detection based on the normalized image.

[0104] S104 : Classify the stains in the multispectral image sequence based on a preset target classification model and the stain coordinate information to obtain stain classification information.

[0105] The preset target classification model is implemented based on the MobileViT classification model. Step S104 also includes:

[0106] According to the stain coordinate information, corresponding regions are cropped from different band images of the multispectral image sequence, and the cropped regional images are integrated as a multi-band image set. ,in, I i,j Indicates the i The stained area is j Band images, n is the total number of bands;

[0107] The multi-band image set is normalized, and the preset target classification model performs stain classification based on the normalized multi-band image set.

[0108] S105: Combine the stain coordinate information and the stain classification information and output them as a ground stain detection result.

[0109] The computer device 300 provided in the embodiment of the present invention can implement the steps in the ground stain detection method based on multispectral imaging in the above embodiment and can achieve the same technical effects. Please refer to the description in the above embodiment and will not be repeated here.

[0110] An embodiment of the present invention also provides a storage medium, on which a ground stain detection program based on multispectral imaging is stored. When the ground stain detection program based on multispectral imaging is executed by a processor, the various processes and steps in the ground stain detection method based on multispectral imaging provided by an embodiment of the present invention are implemented, and the same technical effect can be achieved. To avoid repetition, it will not be repeated here.

[0111] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by using a multispectral imaging-based floor stain detection program to instruct related hardware (such as a mobile phone, computer, server, air conditioner, or network equipment). The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The storage medium can be a magnetic disk, an optical disk, a read-only memory (ROM), or a random access memory (RAM).

[0112] It should be noted that, in this document, the terms "comprises," "includes," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a process, method, article, or apparatus comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such process, method, article, or apparatus. In the absence of further limitations, an element defined by the phrase "comprising a ..." does not exclude the presence of other identical elements in the process, method, article, or apparatus comprising the element.

[0113] The embodiments of the present invention are described above in conjunction with the accompanying drawings. What is disclosed is only a preferred embodiment of the present invention. However, the present invention is not limited to the above-mentioned specific implementation methods. The above-mentioned specific implementation methods are merely illustrative and not restrictive. Under the guidance of the present invention, ordinary technicians in this field can also make many equivalent changes in form without departing from the scope of protection of the purpose of the present invention and the claims, which are all within the protection of the present invention.

Claims

1. A method for detecting ground stains based on multispectral imaging, characterized in that: The following steps are involved: Collecting multispectral image sequences of the ground through a multispectral imaging device; Using an information entropy weighted fusion method to fuse the multispectral image sequence to obtain a fused image; Detecting stains in the fused image based on a preset target detection model to obtain stain coordinate information, wherein the preset target detection model is implemented based on a YOLOv5 target detection model; Based on a preset target classification model and the stain coordinate information, the stains in the multispectral image sequence are classified to obtain stain classification information, wherein the preset target classification model is implemented based on the MobileViT classification model; Combining the stain coordinate information and the stain classification information and outputting them as a ground stain detection result; The step of detecting stains in the fused image based on a preset target detection model to obtain stain coordinate information further includes the following steps: Normalize the fused image so that the pixel values ​​of the fused image are mapped to the interval [0, 1] to obtain a normalized image I norm , the following conditions are met: ； in, is the fused image, I min and I max are the minimum and maximum pixel values ​​in the fused image respectively; The preset target detection model performs stain detection based on the normalized image; The step of classifying the stains in the multispectral image sequence based on a preset target classification model and the stain coordinate information to obtain stain classification information further includes: According to the stain coordinate information, corresponding regions are cropped from different band images of the multispectral image sequence, and the cropped regional images are integrated as a multi-band image set. ,in, I x,y Indicates the x The stained area is y Band images, n is the total number of bands; The multi-band image set is normalized, and the preset target classification model performs stain classification based on the normalized multi-band image set.

2. The method for detecting ground stains based on multispectral imaging according to claim 1, characterized in that: The step of fusing the multispectral image sequence using an information entropy weighted fusion method to obtain a fused image comprises the following sub-steps: The multispectral image sequence is defined as ,in, I i Indicates the i Band images, n is the total number of band channels; For each of the band images in the multispectral image sequence I i , calculate the corresponding information entropy H i , the following conditions are met: ； in, Indicates the i The first of the band images j The probability of gray levels, L is the grayscale level; Each of the band images I i The information entropy is normalized to obtain the corresponding weight value , the following conditions are met: ； Based on each of the band images I i The corresponding weight value , perform weighted fusion on all the band images to obtain the fused image , the following conditions are met: 。 3. The method for detecting ground stains based on multispectral imaging according to claim 1, characterized in that: The step of collecting a multispectral image sequence of the ground by the multispectral imaging device also includes: The average light intensity of the current environment is obtained through the multispectral imaging device. If the average light intensity is less than a preset light threshold, the environment is supplemented with light through the multispectral imaging device.

4. A ground stain detection system based on multispectral imaging, characterized in that: include: An acquisition module, configured to acquire a multispectral image sequence of the ground through a multispectral imaging device; A multispectral image fusion module is used to fuse the multispectral image sequence using an information entropy weighted fusion method to obtain a fused image; A stain detection module is used to detect stains in the fused image based on a preset target detection model to obtain stain coordinate information, wherein the preset target detection model is implemented based on the YOLOv5 target detection model; a stain classification module, configured to classify stains in the multispectral image sequence based on a preset target classification model and the stain coordinate information to obtain stain classification information, wherein the preset target classification model is implemented based on the MobileViT classification model; an output module, configured to combine the stain coordinate information and the stain classification information and output the combined information as a ground stain detection result; The stain detection module is further configured to normalize the fused image so that the pixel values ​​of the fused image are mapped to the interval [0, 1] to obtain a normalized image. I norm , the following conditions are met: ； in, is the fused image, I min and I max are the minimum and maximum pixel values ​​in the fused image respectively; The preset target detection model performs stain detection based on the normalized image The stain classification module is further configured to: cut out corresponding regions from different band images of the multispectral image sequence according to the stain coordinate information, and integrate the cut-out region images as a multi-band image set. ,in, I x,y Indicates the x The stained area is y Band images, n is the total number of bands; The multi-band image set is normalized, and the preset target classification model performs stain classification based on the normalized multi-band image set.

5. A computer device, characterized in that: include: A memory, a processor, and a multispectral imaging-based floor stain detection program stored in the memory and executable on the processor, wherein the processor implements the steps of the multispectral imaging-based floor stain detection method as described in any one of claims 1 to 3 when executing the multispectral imaging-based floor stain detection program.

6. A computer storage medium, characterized in that The computer storage medium stores a multispectral imaging-based ground stain detection program, which, when executed by the processor, implements the steps of the multispectral imaging-based ground stain detection method as described in any one of claims 1 to 3.

Images