Stain detection method and device based on spectral abnormal area perception

Abstract

The present application provides a stain detection method and device based on spectral abnormal region perception, which relates to the field of computer vision technology. The method comprises: inputting a second multispectral image of the area of ​​interest into an image encoder to obtain a feature map of the area of ​​interest, utilizing an attention mechanism to obtain a channel attention feature map and a spatial attention feature map corresponding to the feature map, and fusing them to obtain a feature map to be spliced; fusing the principal component feature map with the feature map to be spliced ​​in the channel dimension to obtain a joint feature map; inputting the joint feature map into a mask decoder to obtain a stain mask, and determining the stain area within the target area based on the stain mask. The purpose of the present application is to provide a stain detection method and device based on spectral abnormal region perception, which accurately identifies stains on the ground through multispectral images, enabling a sweeping robot to focus on cleaning the stained area, thereby improving the cleaning efficiency of the sweeping robot.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a stain detection method and device based on spectral abnormal region perception. Background Art

[0002] With the rapid development of artificial intelligence and the Internet of Things (IoT), smart home cleaning devices have gradually achieved basic environmental perception and path planning capabilities. However, existing technologies still have significant limitations. For example, while smart sweeping robots have basic spatial navigation capabilities, their cleaning logic remains at the mechanized stage of "indiscriminate coverage." These devices often perform repetitive cleaning of the entire home according to preset routines. They are unable to proactively identify key areas of contamination and struggle to differentiate treatment for different stain types, resulting in low cleaning efficiency.

[0003] Based on this, there is an urgent need for a method that can identify dirty areas, reduce the repetitive work of the sweeping robot, and improve cleaning efficiency. Summary of the Invention

[0004] The purpose of this application is to provide a stain detection method and device based on spectral abnormal area perception, which accurately identifies stains on the ground through multispectral images, so that the sweeping robot can focus on cleaning the stained areas and improve the cleaning efficiency of the sweeping robot.

[0005] This application provides a stain detection method based on spectral abnormal region perception, including:

[0006] A first multispectral image of a target area is obtained, and a second multispectral image is obtained after masking a non-interest area in the first multispectral image; the first multispectral image includes: spectral images corresponding to multiple different bands, and one spectral image corresponds to one band; the second multispectral image is input into a first image encoder to obtain a feature map of the interest area, and a channel attention feature map and a spatial attention feature map corresponding to the feature map of the interest area are obtained by using an attention mechanism; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; the channel attention feature map and the spatial attention feature map are fused to obtain a feature map to be spliced, and the principal component feature map of the second multispectral image and the feature map to be spliced ​​are fused by splicing in the channel dimension to obtain a joint feature map; the joint feature map is input into a first mask decoder to obtain a stain mask, and the stain area in the target area is determined based on the stain mask.

[0007] Optionally, the masking of the non-focus area in the first multispectral image to obtain a second multispectral image includes: inputting the multispectral image into a pre-trained second image encoder to obtain a multispectral feature map, and inputting the multispectral feature map into a text encoder to obtain a semantic feature vector; inputting the semantic feature vector as a query and the multispectral feature map as a key and value into a cross-attention network to obtain a fusion feature corresponding to the multispectral image; inputting the fusion feature into a second mask decoder to obtain the non-focus area image mask, and multiplying the non-focus area image mask with the spectral image in the first multispectral image to obtain a second multispectral image of the target area; wherein the non-focus area image mask is used to mask the non-focus area in the first multispectral image.

[0008] Optionally, inputting the second multispectral image into the first image encoder to obtain a feature map of the area of ​​interest includes: inputting the second multispectral image into the first image encoder, using a shallow layer in the first image encoder to capture local texture features, and using a deep layer in the first image encoder to expand the receptive field, and finally obtaining the feature map of the area of ​​interest; wherein the shallow layer includes: a convolution kernel of a preset size; and the deep layer includes: a dilated convolution with a preset dilation rate.

[0009] Optionally, the use of the attention mechanism to obtain the channel attention feature map corresponding to the feature map of the focus area includes: calculating the global statistics of each channel based on the eigenvalue of each channel at any pixel position in the feature map of the focus area; calculating the channel attention weight vector used to characterize the importance of each channel based on the global statistics of each channel; and obtaining the channel attention feature map after performing an XOR operation on the channel attention weight vector and the feature map of the focus area.

[0010] Optionally, the use of the attention mechanism to obtain the spatial attention feature map corresponding to the feature map of the focus area includes: calculating a first feature map after average pooling of the feature map of the focus area along the spatial dimension and a second feature map after maximum pooling along the spatial dimension; performing a splicing operation on the first feature map and the second feature map in the spatial dimension, and generating a spatial attention weight; performing an XOR operation on the spatial attention weight vector and the feature map of the focus area to obtain the spatial attention feature map.

[0011] Optionally, the joint feature map is input into a first mask decoder to obtain a stain mask, and the stain area within the target area is determined based on the stain mask, including: upsampling the joint feature map and inputting it into the first mask decoder to obtain a preliminary stain mask; processing the preliminary stain mask using morphological post-processing, and fine-tuning the preliminary stain mask to obtain a final stain mask; the morphological post-processing includes: performing an erosion-dilation sequence operation using an elliptical kernel with a size adaptive to the stain area; multiplying the final stain mask with the second multispectral image to obtain a stain multispectral image, and determining the stain area within the target area based on the stain multispectral image.

[0012] The present application also provides a stain detection device based on spectral abnormal region perception, comprising:

[0013] An acquisition module is configured to acquire a first multispectral image of a target area; an image processing module is configured to mask a non-interest area in the first multispectral image to obtain a second multispectral image; the first multispectral image includes: spectral images corresponding to multiple different bands, with one spectral image corresponding to each band; a feature extraction module is configured to input the second multispectral image into a first image encoder to obtain a feature map of the interest area, and to use an attention mechanism to obtain a channel attention feature map and a spatial attention feature map corresponding to the feature map of the interest area; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; the feature extraction module is further configured to fuse the channel attention feature map and the spatial attention feature map to obtain a feature map to be spliced, and to splice and fuse the principal component feature map of the second multispectral image with the feature map to be spliced ​​in the channel dimension to obtain a joint feature map; the image processing module is further configured to input the joint feature map into a first mask decoder to obtain a stain mask; and a stain detection module is configured to determine a stain area within the target area based on the stain mask.

[0014] Optionally, the image processing module is specifically used to input the multispectral image into a pre-trained second image encoder to obtain a multispectral feature map, and input the multispectral feature map into a text encoder to obtain a semantic feature vector; the image processing module is also specifically used to input the semantic feature vector as a query and the multispectral feature map as a key and value into a cross-attention network to obtain a fusion feature corresponding to the multispectral image; the image processing module is also specifically used to input the fusion feature into a second mask decoder to obtain the non-focus area image mask, and multiply the non-focus area image mask with the spectral image in the first multispectral image to obtain a second multispectral image of the target area; wherein, the non-focus area image mask is used to mask the non-focus area in the first multispectral image.

[0015] Optionally, the feature extraction module is specifically used to input the second multispectral image into the first image encoder, use the shallow layer in the first image encoder to capture local texture features, and use the deep layer in the first image encoder to expand the receptive field, and finally obtain the feature map of the area of ​​interest; wherein the shallow layer includes: a convolution kernel of a preset size; the deep layer includes: a dilated convolution with a preset dilation rate.

[0016] Optionally, the feature extraction module is specifically used to calculate the global statistics of each channel based on the eigenvalue of each channel at any pixel position in the feature map of the focus area; the feature extraction module is also specifically used to calculate the channel attention weight vector used to characterize the importance of each channel based on the global statistics of each channel; the feature extraction module is also specifically used to obtain the channel attention feature map after performing an XOR operation on the channel attention weight vector and the feature map of the focus area.

[0017] Optionally, the feature extraction module is specifically used to calculate a first feature map after average pooling of the feature map of the focus area along the spatial dimension and a second feature map after maximum pooling along the spatial dimension; the feature extraction module is also specifically used to generate a spatial attention weight after performing a splicing operation on the first feature map and the second feature map in the spatial dimension; the feature extraction module is also specifically used to obtain the spatial attention feature map after performing an XOR operation on the spatial attention weight vector and the feature map of the focus area.

[0018] Optionally, the image processing module is specifically used to upsample the joint feature map and input it into the first mask decoder to obtain a preliminary stain mask; the image processing module is specifically used to process the preliminary stain mask using morphological post-processing, and perform fine-tuning on the preliminary stain mask to obtain a final stain mask; the morphological post-processing includes: performing an erosion-dilation sequence operation using an elliptical kernel with a size adaptive to the stain area; the image processing module is specifically used to multiply the final stain mask with the second multispectral image to obtain a stain multispectral image; the stain detection module is specifically used to determine the stain area within the target area based on the stain multispectral image.

[0019] The present application also provides a computer program product, including a computer program / instruction, which, when executed by a processor, implements the steps of any of the above-mentioned stain detection methods based on spectral abnormal area perception.

[0020] The present application also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of any of the above-described stain detection methods based on spectral abnormality region perception are implemented.

[0021] The present application also provides a computer-readable storage medium having a computer program stored thereon. When the computer program is executed by a processor, the steps of any of the above-mentioned stain detection methods based on spectral abnormal area perception are implemented.

[0022] The stain detection method and device based on spectral abnormal area perception provided by the present application obtain a first multispectral image of the target area, and after masking the non-interest area in the first multispectral image, obtain a second multispectral image; the first multispectral image includes: spectral images corresponding to multiple different bands, and one band corresponds to one spectral image; the second multispectral image is input into the first image encoder to obtain a feature map of the interest area, and the channel attention feature map and the spatial attention feature map corresponding to the feature map of the interest area are obtained by using the attention mechanism; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; after fusing the channel attention feature map and the spatial attention feature map, a feature map to be spliced ​​is obtained, and the principal component feature map of the second multispectral image and the feature map to be spliced ​​are spliced ​​and fused in the channel dimension to obtain a joint feature map; the joint feature map is input into the first mask decoder to obtain a stain mask, and the stain area in the target area is determined based on the stain mask. In this way, the stains on the ground are accurately identified by the multispectral image, so that the sweeping robot can focus on cleaning the stain area, thereby improving the cleaning efficiency of the sweeping robot. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the present application or the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] Figure 1 Schematic diagram of a hardware environment for an interactive method of a smart device according to an embodiment of the present application;

[0025] Figure 2 This is a flow chart of a stain detection method based on spectral abnormal region perception provided by the present application;

[0026] Figure 3 This is a schematic structural diagram of a stain detection device based on spectral abnormal region perception provided by the present application;

[0027] Figure 4 It is a structural diagram of the electronic device provided in this application. DETAILED DESCRIPTION

[0028] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments in the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts should fall within the scope of protection of this application.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequential order. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in a sequence other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0030] According to one aspect of an embodiment of the present application, a stain detection method based on spectral abnormality region perception is provided. The stain detection method based on spectral abnormality region perception is widely used in smart home (Smart Home), smart home, smart home device ecology, smart residence (Intelligence House) ecology and other whole-house intelligent digital control application scenarios. Optionally, in this embodiment, the stain detection method based on spectral abnormality region perception can be applied to Figure 2 In the hardware environment shown in FIG. 1 , which is composed of a terminal device 102 and a server 104. Figure 2 As shown, the server 104 is connected to the terminal device 102 via a network, and can be used to provide services (such as application services, etc.) for the terminal or a client installed on the terminal. A database can be set up on the server or independently of the server to provide data storage services for the server 104. Cloud computing and / or edge computing services can be configured on the server or independently of the server to provide data computing services for the server 104.

[0031] The aforementioned network may include, but is not limited to, at least one of the following: a wired network and a wireless network. The aforementioned wired network may include, but is not limited to, at least one of the following: a wide area network, a metropolitan area network, and a local area network. The aforementioned wireless network may include, but is not limited to, at least one of the following: Wi-Fi (Wireless Fidelity) and Bluetooth. The terminal device 102 may include, but is not limited to, a PC, a mobile phone, a tablet computer, a smart air conditioner, a smart range hood, a smart refrigerator, a smart oven, a smart stove, a smart washing machine, a smart water heater, a smart washing machine, a smart dishwasher, a smart projector, a smart TV, a smart clothes drying rack, smart curtains, a smart audio / video system, a smart socket, a smart speaker, a smart fresh air system, smart kitchen and bathroom equipment, smart bathroom equipment, a smart robot vacuum, a smart window cleaning robot, a smart robot mop, a smart air purifier, a smart steamer, a smart microwave oven, a smart kitchen appliance, a smart purifier, a smart water dispenser, a smart door lock, and the like.

[0032] Related art stain detection methods have several existing technical approaches and limitations: Image segmentation-feature extraction architectures improve the efficiency of multi-target parallel processing through regionalized detection strategies, but fail to effectively distinguish stain features from complex backgrounds; RGB channel selective detection methods detect stains only in the blue and green channels of an image, but still suffer from insufficient spectral resolution and cannot capture the near-infrared and short-wave infrared characteristics of specific stains. Furthermore, these methods have difficulty detecting stains with reflective properties similar to those of the floor substrate, and are prone to missed or false detections against cluttered backgrounds, leading to redundant cleaning path planning. These shortcomings of existing technical solutions can result in a poor user experience.

[0033] In summary, regionalized stain detection alone cannot resolve background interference, and stain detection solely in the blue and green channels cannot improve the detection of difficult stains. In order to enable the robot vacuum to accurately identify floor stains, reduce false detections and missed detections, and thus achieve a precise cleaning route guided by stain detection.

[0034] Based on this, an embodiment of the present application provides a stain detection method that integrates multispectral images. This method eliminates cluttered background interference by finding a ground mask, enhances the spatial-spectral information of the acquired image by integrating multispectral features with image enhancement algorithms, and constructs a stain detection system with strong anti-interference ability.

[0035] The stain detection method based on spectral abnormal region perception provided by the embodiment of the present application is described in detail below with reference to specific embodiments and their application scenarios in conjunction with the accompanying drawings.

[0036] like Figure 2 As shown, an embodiment of the present application provides a stain detection method based on spectral abnormal region perception, which may include the following steps 201 to 204:

[0037] Step 201: Acquire a first multispectral image of a target area, and mask a non-interest area in the first multispectral image to obtain a second multispectral image.

[0038] The first multispectral image includes: spectral images corresponding to multiple different bands, and one band corresponds to one spectral image.

[0039] Exemplarily, the target area is any area on the cleaning path of a sweeping robot or a window cleaning robot. The sweeping robot or the window cleaning robot can collect a multispectral image of the target area through a sensor.

[0040] It should be noted that the multispectral image collected by the sensor includes spectral images corresponding to different bands. In the embodiment of the present application, spectral images of 10 bands (i.e., 10 spectral images of the target area) are used as an example for description.

[0041] For example, after obtaining the first multispectral image, in order to eliminate some interference and narrow the processing range, a mask can be used to mask out non-interest areas. For example, in a robot vacuum cleaner, non-ground areas in the image can be masked out.

[0042] Specifically, the step of masking the non-interest area in the first multispectral image to obtain the second multispectral image in step 201 may further include the following steps 201a1 to 201a3:

[0043] Step 201a1: input the multispectral image into a pre-trained second image encoder to obtain a multispectral feature map, and input the multispectral feature map into a text encoder to obtain a semantic feature vector.

[0044] Step 201a2: Input the semantic feature vector as a query and the multispectral feature map as a key and a value into a cross-attention network to obtain a fusion feature corresponding to the multispectral image.

[0045] Step 201a3: input the fused features into a second mask decoder to obtain the non-interest region image mask, and multiply the non-interest region image mask with the spectral image in the first multispectral image to obtain a second multispectral image of the target area.

[0046] The non-interest region image mask is used to mask the non-interest region in the first multispectral image.

[0047] For example, in this embodiment, a cross-attention mechanism of semantic and image features is used to generate a binary mask of the ground area. Through semantic guidance and interaction with multispectral features, non-ground interference targets are removed and the processing scope is limited to the key ground area.

[0048] For example, in order to obtain a multispectral image of the region of interest, the first spectral image may be pre-trained with an image encoder using contrastive learning to obtain a multispectral feature map. , where H is the width of the multispectral feature map, W is the height of the multispectral feature map, is the number of channels of the extracted multispectral feature map, as shown in the following formula 1:

[0049] (Formula 1)

[0050] in, is a multispectral image feature extractor, that is, the second image encoder mentioned above.

[0051] For example, after this, it is also necessary to obtain the Generate static semantic feature vectors using pre-trained CLIP text encoder Input to the cross attention module to get , as shown in the following formula 2:

[0052] (Formula 2)

[0053] in, For the cross attention module, text features As the query Query, image features As the key Key and value Value, focus on the ground-related area through the attention weight.

[0054] For example, the features obtained by the cross attention module Input the mask decoder (i.e. the second decoder above) to get the binary mask (i.e. the above-mentioned non-interest area image mask), as shown below:

[0055]

[0056]

[0057]

[0058] in, is the deconvolution upsampling network, , is the mask decoder, is the weight obtained.

[0059] Finally, the binary mask Multiply them with the first multispectral image respectively to obtain the required ground multispectral image , that is, the second multispectral image mentioned above.

[0060] It should be noted that in the training phase, the loss function is designed to be Dice Loss (i.e. ) and Focal Loss (i.e. ), the formula is as follows:

[0061]

[0062]

[0063]

[0064] in, Pixels i The probability of predicting it as a stain, is the true label (1 for stains, 0 for background), =0.25, =2, suppressing the loss weight of simple samples (background), , Dice Loss dominates the global structure, and Focal Loss refines local details.

[0065] Step 202: Input the second multispectral image into the first image encoder to obtain a feature map of the focus area, and use the attention mechanism to obtain a channel attention feature map and a spatial attention feature map corresponding to the feature map of the focus area.

[0066] The number of channels of the first image encoder is the same as the number of bands of the second multispectral image.

[0067] For example, after obtaining a multispectral image of the region of interest, principal component analysis (PCA) is used to perform orthogonal decomposition of the multi-channel multispectral image to extract low-dimensional principal component features. A dual-branch attention mechanism, channel-wise and spatial-wise, is then introduced to extract key features based on spectral sensitivity and spatial distribution, respectively. The channel-wise branch then adaptively enhances the discriminative stain bands, while the spatial branch focuses on localized abnormal textures. The principal component features and dual-branch attention features jointly capture global spectral anomaly characteristics, and the two are fused to produce a joint feature map.

[0068] Specifically, the step of inputting the second multispectral image into the first image encoder to obtain the feature map of the region of interest in the above step 202 may further include the following step 202a:

[0069] Step 202a: input the second multispectral image into the first image encoder, use the shallow layer of the first image encoder to capture local texture features, and use the deep layer of the first image encoder to expand the receptive field, and finally obtain a feature map of the region of interest.

[0070] Among them, the shallow layer includes: a convolution kernel of a preset size; the deep layer includes: a dilated convolution with a preset dilation rate.

[0071] For example, the second multispectral image obtained Input into the first image encoder to obtain the multispectral ground feature map , as shown in the following formula 3:

[0072] (Formula 3)

[0073] in, To expand the UNet input channel count to ten, the pre-trained weights are initialized by replicating the RGB channels. The shallow layers (3×3 convolutions) are designed to capture local textures (such as oil spots and water stain edges). Deeper layers (atrous convolutions) use a dilation ratio of r=2 to expand the receptive field to 15×15, enabling detection of larger stains.

[0074] Specifically, in the above step 202, the step of using the attention mechanism to obtain the channel attention feature map corresponding to the feature map of the focus area may further include the following steps 202b1 to 202b3:

[0075] Step 202b1: Based on the feature value of each channel at any pixel position in the feature map of the target area, calculate and obtain the global statistics of each channel.

[0076] Step 202b2: Based on the global statistics of each channel, a channel attention weight vector is calculated to characterize the importance of each channel.

[0077] Step 202b3: Perform an XOR operation on the channel attention weight vector and the feature map of the attention area to obtain the channel attention feature map.

[0078] For example, the obtained multispectral ground feature map Input into the dual attention module, and obtain the channel attention feature map using the dual attention mechanism and spatial attention feature maps .

[0079] For example, the channel attention formula is as follows:

[0080]

[0081] in, For the cth channel at position The eigenvalues ​​of is the global statistic of the c-th channel, is the weight matrix of the fully connected layer, is the ReLU activation function, is the Sigmoid function, s is the channel attention weight vector, indicating the importance of each channel.

[0082] Specifically, in the above step 202, the step of using the attention mechanism to obtain the spatial attention feature map corresponding to the feature map of the focus area may further include the following steps 202c1 to 202c3:

[0083] Step 202c1: Calculate a first feature map after average pooling of the feature map of the focus area along the spatial dimension and a second feature map after maximum pooling along the spatial dimension.

[0084] Step 202c2: After performing a splicing operation on the first feature map and the second feature map in the spatial dimension, a spatial attention weight is generated.

[0085] Step 202c3: Perform an XOR operation on the spatial attention weight vector and the feature map of the attention area to obtain the spatial attention feature map.

[0086] For example, the spatial attention formula is as follows:

[0087]

[0088] in, is the feature map after average pooling along the spatial dimension, It is the feature map after the maximum pooling along the spatial dimension. Features obtained by the dual attention mechanism .

[0089] Step 203: After fusing the channel attention feature map and the spatial attention feature map, a feature map to be spliced ​​is obtained, and the principal component feature map of the second multispectral image and the feature map to be spliced ​​are fused by splicing in the channel dimension to obtain a joint feature map.

[0090] For example, the second multispectral image obtained Perform principal component analysis (PCA) to reduce dimensionality and extract the principal components of the global spectrum , as shown in the following formula:

[0091]

[0092] in, , is the eigenvector matrix corresponding to the first three largest eigenvalues.

[0093] For example, the principal component characteristic diagram obtained by principal component analysis is Features obtained from the dual attention mechanism module Form a joint feature map by splicing and fusing the channel dimensions .

[0094]

[0095] in, .

[0096] Step 204: Input the joint feature map into a first mask decoder to obtain a stain mask, and determine the stain area within the target area based on the stain mask.

[0097] For example, after the joint feature map is input into the decoding network, a preliminary high-resolution stain mask is generated to distinguish the real stain area from the ground area. The obtained preliminary mask is subjected to morphological post-processing. Isolated noise points are eliminated by first corroding and then dilating. Internal holes are filled by first dilating and then corroding. The smoothness and continuity of the mask boundary are optimized, and finally the stain positioning result is output.

[0098] Specifically, the above step 204 may further include the following steps 204a1 to 204a3:

[0099] Step 204a1: Upsample the joint feature map and input it into the first mask decoder to obtain a preliminary stain mask.

[0100] Step 204a2: Process the preliminary stain mask using morphological post-processing, perform fine adjustments on the preliminary stain mask, and obtain a final stain mask.

[0101] The morphological post-processing includes: performing an erosion-dilation sequence operation using an elliptical kernel with a size adaptive to the stain area.

[0102] Step 204a3: Multiply the final stain mask by the second multispectral image to obtain a stain multispectral image, and determine the stain area within the target area based on the stain multispectral image.

[0103] For example, the joint feature map Input mask decoder to get high-resolution preliminary stain mask , as shown below:

[0104]

[0105] in, is the deconvolution upsampling network, , is the mask decoder, is the weight obtained.

[0106] It should be noted that in the training phase, the loss function is also designed as Dice Loss and Focal Loss. The specific formula is as follows:

[0107]

[0108] For example, after initially obtaining the stain area mask, The mask is fine-tuned through the morphological post-processing optimization module to obtain the high-resolution stain final mask. The specific process is as follows: Use an N×N elliptical kernel (N is adaptively adjusted according to the stain area) to perform an erosion-dilation sequence: eliminate isolated noise points, fill holes (such as hair occlusion areas), and preserve the smoothness of the stain edge. and The final stain multispectral image is obtained by multiplication, that is, the stain area is found.

[0109] In one possible implementation, during the training phase, data is collected using a CM020 series hyperspectral imaging device. This camera has a wide spectral response range of 350-950nm, and its imaging sensor features a 1600×1200 pixel array. To overcome interference from specular reflections from liquid contaminants, a polarization filter is placed in front of the optical lens. By adjusting the angle of the two polarizers, a depolarized light path is formed, suppressing specular reflections from the liquid surface and smooth surfaces due to ambient light.

[0110] For example, the multispectral camera captures color images, grayscale images, and QS-format files containing multispectral information. Spectral inversion software is used to generate a multispectral image in BMP format. During inversion, the band range and the desired number of bands can be selected. Here, we select the 410-950 band range, a total of ten bands, for inversion, ultimately resulting in the acquired multispectral image.

[0111] Afterwards, the obtained multispectral image is preprocessed, including: geometric correction and spatial transformation to denoise the multispectral image; scaling the multispectral image to 640×640 pixels, The multispectral image is obtained through preprocessing. The multispectral image is used to train the above encoders and decoders.

[0112] The stain detection method based on spectral anomaly region perception provided by the embodiment of the present application has the following advantages over the stain detection solutions in the related art: ① By introducing multispectral data of ten bands (410-950nm), the specific reflection characteristics of stains under different spectra are captured, solving the problem of false detection of similar textures by traditional RGB images and improving the model's cross-material generalization ability. ② A combined polarizer is used to suppress mirror reflection noise, effectively eliminating high-light interference on smooth floors and liquid contaminant surfaces. ③ A cross-attention method based on semantic features and multispectral images is constructed to generate accurate ground masks, excluding non-ground interference areas such as furniture and shadows, which helps to find high-precision stain areas in the future. ④ A principal component analysis combined with a dual attention mechanism is designed to focus on spectral anomaly areas and improve feature decoding efficiency.

[0113] The stain detection method based on spectral abnormal area perception provided by the embodiment of the present application obtains a first multispectral image of the target area, and after masking the non-interest area in the first multispectral image, obtains a second multispectral image; the first multispectral image includes: spectral images corresponding to multiple different bands, and one spectral image corresponds to one band; the second multispectral image is input into the first image encoder to obtain a feature map of the interest area, and the channel attention feature map and the spatial attention feature map corresponding to the feature map of the interest area are obtained by using the attention mechanism; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; after fusing the channel attention feature map and the spatial attention feature map, a feature map to be spliced ​​is obtained, and the principal component feature map of the second multispectral image is spliced ​​and fused with the feature map to be spliced ​​by the channel dimension to obtain a joint feature map; the joint feature map is input into the first mask decoder to obtain a stain mask, and the stain area in the target area is determined based on the stain mask. In this way, the stains on the ground are accurately identified by the multispectral image, so that the sweeping robot can focus on cleaning the stain area, thereby improving the cleaning efficiency of the sweeping robot.

[0114] It should be noted that the stain detection method based on spectral abnormality region perception provided in the embodiments of the present application can be executed by a stain detection device based on spectral abnormality region perception, or by a control module within the stain detection device based on spectral abnormality region perception that is configured to execute the stain detection method based on spectral abnormality region perception. In the embodiments of the present application, the stain detection device based on spectral abnormality region perception provided in the embodiments of the present application is described by taking the stain detection device based on spectral abnormality region perception executing the stain detection method based on spectral abnormality region perception as an example.

[0115] It should be noted that in the embodiments of this application, the stain detection methods based on spectral abnormal region perception shown in the above-mentioned method figures are each illustrated by way of example in conjunction with one of the figures in the embodiments of this application. In specific implementations, the stain detection methods based on spectral abnormal region perception shown in the above-mentioned method figures can also be implemented in conjunction with any other combinable figures shown in the above-mentioned embodiments, and will not be further described here.

[0116] The stain detection device based on spectral abnormal region perception provided by the present application is described below. The stain detection method based on spectral abnormal region perception described below and the stain detection method based on spectral abnormal region perception described above can be referenced to each other.

[0117] Figure 3 This is a schematic diagram of the structure of a stain detection device based on spectral abnormality region perception provided in an embodiment of the present application, as shown in FIG. Figure 3 As shown, specifically including:

[0118] An acquisition module 301 is configured to acquire a first multispectral image of a target area. An image processing module 302 is configured to mask a non-interest area in the first multispectral image to obtain a second multispectral image. The first multispectral image includes spectral images corresponding to multiple different bands, with one spectral image corresponding to each band. A feature extraction module 303 is configured to input the second multispectral image into a first image encoder to obtain a feature map of the interest area, and to use an attention mechanism to obtain a channel attention feature map and a spatial attention feature map corresponding to the feature map of the interest area. The number of channels of the first image encoder is the same as the number of bands of the second multispectral image. The feature extraction module 303 is further configured to fuse the channel attention feature map and the spatial attention feature map to obtain a feature map to be spliced, and to splice and fuse the principal component feature map of the second multispectral image with the feature map to be spliced ​​in the channel dimension to obtain a joint feature map. The image processing module 302 is further configured to input the joint feature map into a first mask decoder to obtain a stain mask. A stain detection module 304 is configured to determine a stain area within the target area based on the stain mask.

[0119] Optionally, the image processing module 302 is specifically used to input the multispectral image into a pre-trained second image encoder to obtain a multispectral feature map, and input the multispectral feature map into a text encoder to obtain a semantic feature vector; the image processing module 302 is also specifically used to input the semantic feature vector as a query and the multispectral feature map as a key and value into a cross-attention network to obtain a fusion feature corresponding to the multispectral image; the image processing module 302 is also specifically used to input the fusion feature into a second mask decoder to obtain the non-focus area image mask, and multiply the non-focus area image mask with the spectral image in the first multispectral image to obtain a second multispectral image of the target area; wherein, the non-focus area image mask is used to mask the non-focus area in the first multispectral image.

[0120] Optionally, the feature extraction module 303 is specifically used to input the second multispectral image into the first image encoder, use the shallow layer in the first image encoder to capture local texture features, and use the deep layer in the first image encoder to expand the receptive field, and finally obtain the feature map of the area of ​​interest; wherein the shallow layer includes: a convolution kernel of a preset size; the deep layer includes: a dilated convolution with a preset dilation rate.

[0121] Optionally, the feature extraction module 303 is specifically used to calculate the global statistics of each channel based on the characteristic value of each channel at any pixel position in the feature map of the focus area; the feature extraction module 303 is also specifically used to calculate the channel attention weight vector used to characterize the importance of each channel based on the global statistics of each channel; the feature extraction module 303 is also specifically used to obtain the channel attention feature map after performing an XOR operation on the channel attention weight vector and the feature map of the focus area.

[0122] Optionally, the feature extraction module 303 is specifically used to calculate a first feature map after average pooling of the feature map of the focus area along the spatial dimension and a second feature map after maximum pooling along the spatial dimension; the feature extraction module 303 is also specifically used to generate a spatial attention weight after performing a splicing operation on the first feature map and the second feature map in the spatial dimension; the feature extraction module 303 is also specifically used to obtain the spatial attention feature map after performing an XOR operation on the spatial attention weight vector and the feature map of the focus area.

[0123] Optionally, the image processing module 302 is specifically used to upsample the joint feature map and input it into the first mask decoder to obtain a preliminary stain mask; the image processing module 302 is specifically used to process the preliminary stain mask using morphological post-processing, and fine-tune the preliminary stain mask to obtain a final stain mask; the morphological post-processing includes: performing an erosion-dilation sequence operation using an elliptical kernel with a size adaptive to the stain area; the image processing module 302 is specifically used to multiply the final stain mask with the second multispectral image to obtain a stain multispectral image; the stain detection module 304 is specifically used to determine the stain area within the target area based on the stain multispectral image.

[0124] The stain detection device based on spectral abnormal area perception provided by the present application obtains a first multispectral image of a target area, and after masking the non-interest area in the first multispectral image, obtains a second multispectral image; the first multispectral image includes: spectral images corresponding to multiple different bands, and one band corresponds to one spectral image; the second multispectral image is input into the first image encoder to obtain a feature map of the interest area, and the channel attention feature map and the spatial attention feature map corresponding to the feature map of the interest area are obtained by using the attention mechanism; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; after fusing the channel attention feature map and the spatial attention feature map, a feature map to be spliced ​​is obtained, and the principal component feature map of the second multispectral image is spliced ​​and fused with the feature map to be spliced ​​by the channel dimension to obtain a joint feature map; the joint feature map is input into the first mask decoder to obtain a stain mask, and the stain area in the target area is determined based on the stain mask. In this way, the stains on the ground are accurately identified by the multispectral image, so that the sweeping robot can focus on cleaning the stain area, thereby improving the cleaning efficiency of the sweeping robot.

[0125] Figure 4 An example of a physical structure diagram of an electronic device is shown in FIG. Figure 4As shown, the electronic device may include: a processor (processor) 410 , a communication interface (Communications Interface) 420 , a memory (memory) 430 and a communication bus 440 , wherein the processor 410 , the communication interface 420 and the memory 430 communicate with each other via the communication bus 440 . The processor 410 can call the logic instructions in the memory 430 to execute a stain detection method based on spectral abnormal area perception, which includes: obtaining a first multispectral image of the target area, and masking the non-interest area in the first multispectral image to obtain a second multispectral image; the first multispectral image includes: spectral images corresponding to multiple different bands, and one band corresponds to one spectral image; inputting the second multispectral image into the first image encoder to obtain a feature map of the interest area, and using the attention mechanism to obtain a channel attention feature map and a spatial attention feature map corresponding to the feature map of the interest area; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; after fusing the channel attention feature map and the spatial attention feature map, a feature map to be spliced ​​is obtained, and the principal component feature map of the second multispectral image and the feature map to be spliced ​​are spliced ​​and fused in the channel dimension to obtain a joint feature map; the joint feature map is input into the first mask decoder to obtain a stain mask, and the stain area in the target area is determined based on the stain mask. In this way, the stains on the ground can be accurately identified through multispectral images, so that the sweeping robot can focus on cleaning the stained areas, thereby improving the cleaning efficiency of the sweeping robot.

[0126] In addition, the logical instructions in the above-mentioned memory 430 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when sold or used as an independent product. Based on this understanding, the technical solution of the present application, or the part that contributes to the existing technology, or the part of the technical solution, can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, and other media that can store program code.

[0127] On the other hand, the present application also provides a computer program product, which includes a computer program stored on a computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the stain detection method based on spectral abnormality area perception provided by the above methods, the method including: obtaining a first multispectral image of a target area, and masking a non-interest area in the first multispectral image to obtain a second multispectral image; the first multispectral image includes: spectral images corresponding to multiple different bands, and one band corresponds to one spectral image; the second multispectral image is input into a second multispectral image. In an image encoder, a feature map of the area of ​​interest is obtained, and the attention mechanism is used to obtain a channel attention feature map and a spatial attention feature map corresponding to the feature map of the area of ​​interest; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; after fusing the channel attention feature map and the spatial attention feature map, a feature map to be spliced ​​is obtained, and the principal component feature map of the second multispectral image is fused with the feature map to be spliced ​​in the channel dimension to obtain a joint feature map; the joint feature map is input into a first mask decoder to obtain a stain mask, and the stain area within the target area is determined based on the stain mask. In this way, stains on the ground are accurately identified through multispectral images, allowing the sweeping robot to focus on cleaning the stained area, thereby improving the cleaning efficiency of the sweeping robot.

[0128] On the other hand, the present application also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the above-mentioned stain detection method based on spectral abnormal area perception, the method comprising: obtaining a first multispectral image of a target area, and masking a non-interest area in the first multispectral image to obtain a second multispectral image; the first multispectral image comprises: spectral images corresponding to multiple different bands, and one spectral image corresponds to one band; the second multispectral image is input into a first image encoder to obtain a feature map of the interest area, and a channel attention feature map and a spatial attention feature map corresponding to the feature map of the interest area are obtained by using an attention mechanism; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; the channel attention feature map and the spatial attention feature map are fused to obtain a feature map to be spliced, and the principal component feature map of the second multispectral image is fused with the feature map to be spliced ​​in the channel dimension to obtain a joint feature map; the joint feature map is input into a first mask decoder to obtain a stain mask, and the stain area in the target area is determined based on the stain mask. In this way, the stains on the ground can be accurately identified through multispectral images, so that the sweeping robot can focus on cleaning the stained areas, thereby improving the cleaning efficiency of the sweeping robot.

[0129] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, i.e., they may be located in one location or distributed across multiple network units. Some or all of the modules may be selected based on actual needs to achieve the objectives of the present embodiment. Persons of ordinary skill in the art will be able to understand and implement the present invention without inventive effort.

[0130] Through the above description of the embodiments, those skilled in the art will clearly understand that each embodiment can be implemented using software plus a necessary general-purpose hardware platform, or of course, hardware. Based on this understanding, the essence of the above technical solution, or the portion that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a magnetic disk, or an optical disk, and includes a number of instructions for causing a computer device (such as a personal computer, server, or network device) to execute the methods described in each embodiment or certain portions of the embodiments.

[0131] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A stain detection method based on spectral abnormal region perception, characterized in that: include: Acquire a first multispectral image of a target area, and obtain a second multispectral image after masking a non-interest area in the first multispectral image; The first multispectral image includes: spectral images corresponding to a plurality of different wavebands, and one waveband corresponds to one spectral image; Inputting the second multispectral image into a first image encoder to obtain a feature map of a region of interest, and using an attention mechanism to obtain a channel attention feature map and a spatial attention feature map corresponding to the feature map of the region of interest; the number of channels of the first image encoder is the same as the number of bands of the second multispectral image; After fusing the channel attention feature map and the spatial attention feature map, a feature map to be spliced ​​is obtained, and the principal component feature map of the second multispectral image and the feature map to be spliced ​​are fused through splicing in the channel dimension to obtain a joint feature map; Inputting the joint feature map into a first mask decoder to obtain a stain mask, and determining a stain area within the target area based on the stain mask; Obtaining a second multispectral image after masking the non-interest area in the first multispectral image includes: Inputting the multispectral image into a pre-trained second image encoder to obtain a multispectral feature map, and inputting the multispectral feature map into a text encoder to obtain a semantic feature vector; Inputting the semantic feature vector as a query and the multispectral feature map as a key and a value into a cross-attention network to obtain a fusion feature corresponding to the multispectral image; Inputting the fused features into a second mask decoder to obtain a non-interest region image mask, and multiplying the non-interest region image mask with the spectral image in the first multispectral image respectively to obtain a second multispectral image of the target area; The non-interest region image mask is used to mask the non-interest region in the first multispectral image; Inputting the joint feature map into a first mask decoder to obtain a stain mask, and determining the stain area within the target area based on the stain mask, includes: Upsampling the joint feature map and inputting it into the first mask decoder to obtain a preliminary stain mask; The preliminary stain mask is processed by morphological post-processing, and the preliminary stain mask is fine-tuned to obtain a final stain mask; the morphological post-processing includes: performing an erosion-dilation sequence operation using an elliptical kernel with a size adaptive to the stain area; After multiplying the final stain mask and the second multispectral image, a stain multispectral image is obtained, and the stain area in the target area is determined based on the stain multispectral image.

2. The stain detection method based on spectral abnormal region perception according to claim 1 is characterized in that: Inputting the second multispectral image into the first image encoder to obtain a feature map of the region of interest includes: Inputting the second multispectral image into the first image encoder, using a shallow layer in the first image encoder to capture local texture features, and using a deep layer in the first image encoder to expand the receptive field, ultimately obtaining a feature map of the region of interest; Among them, the shallow layer includes: a convolution kernel of a preset size; the deep layer includes: a dilated convolution with a preset dilation rate.

3. The stain detection method based on spectral abnormal region perception according to claim 1 or 2, characterized in that: The method of using the attention mechanism to obtain a channel attention feature map corresponding to the feature map of the attention area includes: Calculating a global statistic for each channel based on the eigenvalue of each channel at any pixel position in the feature map of the region of interest; Based on the global statistics of each channel, the channel attention weight vector used to characterize the importance of each channel is calculated; After performing an XOR operation on the channel attention weight vector and the feature map of the attention area, the channel attention feature map is obtained.

4. The stain detection method based on spectral abnormal region perception according to claim 1 or 2, characterized in that: The method of using the attention mechanism to obtain a spatial attention feature map corresponding to the feature map of the focus area includes: Calculating a first feature map after average pooling of the feature map of the focus area along the spatial dimension and a second feature map after maximum pooling along the spatial dimension; After performing a splicing operation on the first feature map and the second feature map in a spatial dimension, generating a spatial attention weight; After performing an XOR operation on the spatial attention weight vector and the feature map of the attention area, the spatial attention feature map is obtained.

5. A stain detection device based on spectral abnormal region perception, characterized in that: The device comprises: an acquisition module, configured to acquire a first multispectral image of a target area; An image processing module is configured to mask a non-interest area in the first multispectral image to obtain a second multispectral image; the first multispectral image includes: spectral images corresponding to a plurality of different bands, with one spectral image corresponding to each band; a feature extraction module, configured to input the second multispectral image into a first image encoder to obtain a feature map of a region of interest, and to use an attention mechanism to obtain a channel attention feature map and a spatial attention feature map corresponding to the feature map of the region of interest; the number of channels of the first image encoder being the same as the number of bands of the second multispectral image; The feature extraction module is further configured to fuse the channel attention feature map and the spatial attention feature map to obtain a feature map to be spliced, and to splice and fuse the principal component feature map of the second multispectral image with the feature map to be spliced ​​in the channel dimension to obtain a joint feature map; The image processing module is further configured to input the joint feature map into a first mask decoder to obtain a stain mask; a stain detection module, configured to determine a stain area within the target area based on the stain mask; The image processing module is specifically configured to input the multispectral image into a pre-trained second image encoder to obtain a multispectral feature map, and input the multispectral feature map into a text encoder to obtain a semantic feature vector; The image processing module is further configured to input the semantic feature vector as a query and the multispectral feature map as a key and a value into a cross attention network to obtain a fusion feature corresponding to the multispectral image; The image processing module is further configured to input the fused features into a second mask decoder to obtain a non-interest region image mask, and multiply the non-interest region image mask by the spectral images in the first multispectral image to obtain a second multispectral image of the target area; The non-interest region image mask is used to mask the non-interest region in the first multispectral image; The image processing module is specifically used to upsample the joint feature map and input it into the first mask decoder to obtain a preliminary stain mask; the image processing module is specifically used to process the preliminary stain mask using morphological post-processing, and fine-tune the preliminary stain mask to obtain a final stain mask; the morphological post-processing includes: performing an erosion-dilation sequence operation using an elliptical kernel with an adaptive size of the stain area; the image processing module is specifically used to multiply the final stain mask with the second multispectral image to obtain a stain multispectral image; the stain detection module is specifically used to determine the stain area within the target area based on the stain multispectral image.

6. An electronic device, characterized in that: The method comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the stain detection method based on spectral abnormal area perception as claimed in any one of claims 1 to 4 are implemented.

7. A computer-readable storage medium, characterized in that A computer program is stored thereon, and when the computer program is executed by a processor, the steps of the stain detection method based on spectral abnormal area perception as claimed in any one of claims 1 to 4 are implemented.

Images