Dirt identifying and cleaning method, device and equipment

Abstract

The invention is suitable for the field of intelligent jet cleaning, and provides a dirt recognition cleaning method, device and equipment, and the dirt recognition cleaning method comprises the steps: obtaining an original image corresponding to target dirt and a sound reflection signal corresponding to the target dirt; converting the sound reflection signal corresponding to the target dirt into an image representation; obtaining a fused image corresponding to the target dirt according to the original image corresponding to the target dirt, the image representation and a preset multi-channel image fusion algorithm; inputting a fusion image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and size data of the target dirt; adjusting the pose of the dirt cleaning equipment according to the segmentation mask data and a preset coordinate conversion algorithm; according to the size data of the target dirt, dirt cleaning parameters are adjusted, and dirt cleaning equipment is controlled to execute cleaning operation. According to the method, the dirt recognition and cleaning precision and efficiency are remarkably improved, and various cleaning requirements and operation environments can be better met.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent jet cleaning, and particularly relates to a dirt identification and cleaning method, device and equipment. BACKGROUND

[0002] Nuclear industry, precision manufacturing, shipbuilding and the like have different degrees of requirements for surface cleanliness, and automatic identification and accurate cleaning of surface dirt become key problems in industrial automation. Traditional dirt identification methods mostly rely on gray scale or texture analysis of visual images, are easily disturbed by surface material and illumination conditions, and have limited identification accuracy. Moreover, traditional jet cleaning methods mostly use fixed pressure and path, lack adaptive response capability to dirt distribution, and are prone to cause energy waste or incomplete cleaning. In recent years, deep learning methods have been widely introduced into image segmentation and defect identification, but still face challenges in dirt detection. First, a neural network is not accurate enough in identifying dirt edges, especially in the case of weak contrast and irregular distribution. Second, the single nature of image information makes the model lack generalization capability and is greatly affected by environmental illumination and the like. SUMMARY

[0003] Embodiments of the application provide a dirt identification and cleaning method, device and equipment, which can solve the above problems.

[0004] In a first aspect, embodiments of the application provide a dirt identification and cleaning method, comprising:

[0005] obtaining an original image corresponding to target dirt and an acoustic reflection signal corresponding to the target dirt;

[0006] converting the acoustic reflection signal corresponding to the target dirt into an image representation;

[0007] obtaining a fusion image corresponding to the target dirt according to the original image corresponding to the target dirt, the image representation and a preset multi-channel image fusion algorithm;

[0008] inputting the fusion image corresponding to the target dirt into an improved deep neural network to obtain segmentation mask data and size data of the target dirt; wherein the improved deep neural network is integrated with a double-attention fusion module and a boundary perception enhancement module, the double-attention fusion module includes a channel attention submodule and a spatial attention submodule, the channel attention submodule is used to capture the relative importance of different feature channels to dirt identification, the spatial attention submodule is used to strengthen dirt boundary features, and the boundary perception enhancement module is used to highlight edge change significant areas;

[0009] adjusting the pose of a dirt cleaning device according to the segmentation mask data and a preset coordinate conversion algorithm;

[0010] Adjust a dirt cleaning parameter according to the size data of the target dirt, and control the dirt cleaning device to perform a cleaning operation.

[0011] In a second aspect, the embodiments of the present application provide a dirt identification and cleaning device, comprising:

[0012] An acquisition unit is configured to acquire an original image corresponding to target dirt and an acoustic reflection signal corresponding to the target dirt;

[0013] A first processing unit is configured to convert the acoustic reflection signal corresponding to the target dirt into an image representation;

[0014] A second processing unit is configured to obtain a fusion image corresponding to the target dirt according to the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm;

[0015] A third processing unit is configured to input the fusion image corresponding to the target dirt into an improved deep neural network to obtain segmentation mask data and size data of the target dirt, wherein the improved deep neural network is integrated with a dual-attention fusion module and a boundary perception enhancement module, the dual-attention fusion module includes a channel attention submodule and a spatial attention submodule, the channel attention submodule is configured to capture the relative importance of different feature channels to dirt identification, the spatial attention submodule is configured to strengthen dirt boundary features, and the boundary perception enhancement module is configured to highlight edge change significant regions;

[0016] A fourth processing unit is configured to adjust a pose of a dirt cleaning device according to the segmentation mask data and a preset coordinate conversion algorithm;

[0017] A fifth processing unit is configured to adjust a dirt cleaning parameter according to the size data of the target dirt, and control the dirt cleaning device to perform a cleaning operation.

[0018] In a third aspect, the embodiments of the present application provide a dirt identification and cleaning device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of the first aspect when executing the computer program.

[0019] In a fourth aspect, the embodiments of the present application provide a computer readable storage medium, which stores a computer program, and the computer program is executable on a processor to implement the method of the first aspect.

[0020] In this embodiment, the original image and acoustic reflection signal corresponding to the target dirt are acquired; the acoustic reflection signal is converted into an image representation; a fused image is obtained based on the original image, image representation, and a preset multi-channel image fusion algorithm; the fused image is input into an improved deep neural network to obtain segmentation mask data and target dirt size data; the pose of the dirt cleaning device is adjusted based on the segmentation mask data and a preset coordinate transformation algorithm; the dirt cleaning parameters are adjusted based on the target dirt size data, and the dirt cleaning device is controlled to perform cleaning operations. By fusing images and using an improved deep learning network, the accuracy of dirt edge recognition is significantly improved, especially in complex environments and low-contrast conditions. The cleaning device adjusts its pose and cleaning parameters according to the actual distribution and characteristics of the dirt, achieving precise cleaning and reducing energy consumption and resource waste. The introduction of deep learning and acoustic signal processing promotes the intelligentization of the cleaning process, adapting to different cleaning needs and environmental changes. This method is applicable to surface cleaning of different materials and shapes, and has broad application prospects. With the above implementation plan, the accuracy and efficiency of dirt identification and cleaning will be significantly improved, better meeting various cleaning needs and operating environments. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic flowchart of a dirt identification and cleaning method provided in the first embodiment of this application;

[0023] Figure 2 This is a schematic flowchart of step S103 in a dirt identification and cleaning method provided in the first embodiment of this application;

[0024] Figure 3 This is a schematic flowchart of steps S107-S108 in a dirt identification and cleaning method provided in the first embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the dirt identification and cleaning device provided in the second embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the dirt identification and cleaning device provided in the third embodiment of this application. Detailed Implementation

[0027] In the following description, for purposes of explanation and not limitation, specific details are set forth such as particular architectures, techniques, etc. in order to provide a thorough understanding of the embodiments of the present application. However, it will be apparent to those skilled in the art that the present application can be practiced in other embodiments that depart from these specific details. In other instances, detailed descriptions of well-known methods, devices, circuits, and

[0028] It is to be understood that the terminology "includes", "has", "holds", "contains" or "comprising", "including", "having" and the like, when used in the present specification and in the accompanying claims, are used in the sense of "including but not limited to", "including but not limited to", "including but not limited to" and "including but not limited to" and "including but not limited to", respectively, and are not used in the sense of "consist only of", "consist only of", "consist only of", "consist only of" and "consist only of", respectively, unless otherwise described.

[0029] It should also be understood that the term "and / or" as used herein, when used in the context of listing items, means that one or more of the listed items can be present, and that the listing of items should be construed to be inclusive of one or more of the items being present. That is, the term "and / or" as used herein, when used in the context of listing items, means that one or more of the listed items can be present, and that the listing of items should be construed to be inclusive of one or more of the items being present.

[0030] As used in the present specification and in the accompanying claims, the term "if" can be interpreted as meaning "when", or "once", or "in response to a determination", or "in response to a detection" depending on the context. Similarly, the phrase "if it is determined" or "if [a described condition or event] is detected" can be interpreted to mean "once it is determined" or "in response to the determination" or "once [the described condition or event] is detected" or "in response to the detection [of the described condition or event]", depending on the context.

[0031] In addition, the terms "first", "second", "third", etc. as used in the description of the specification and the appended claims are not used to denote or imply relative importance but are used to distinguish one element from another.

[0032] Reference throughout this specification to "one embodiment", "some embodiments", or "an embodiment" means that a particular feature, structure, or characteristic described in connection with the embodiment is included in at least one embodiment of the application. Thus, the appearances of the phrases "in one embodiment", "in some embodiments", "in other embodiments", "in additional embodiments", and so on, in various places throughout this specification are not necessarily all referring to the same embodiment, unless otherwise specifically stated. The terms "comprising", "including", "containing", and variations thereof, as well as the terms "consisting of" and "consisting essentially of" when used in the specification and in the following claims, mean "including, but not limited to", "including, but not limited to", "including, but not limited to", and "including, but not limited to", respectively, unless otherwise specifically stated.

[0033] See Figure 1 , Figure 1is a schematic flowchart of a dirt identification cleaning method provided by the first embodiment of the present application. The execution subject of the dirt identification cleaning method in this embodiment is a device with dirt identification cleaning function, such as a desktop computer, a server, and the like. As shown in the dirt identification cleaning method can include: Figure 1

[0034] S101: Obtain the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt.

[0035] Use a high-definition camera to take pictures of the target cleaning surface, and ensure that multiple images are collected under different lighting conditions to improve the accuracy of subsequent analysis.

[0036] Obtain the original image corresponding to the target dirt by taking pictures.

[0037] Use an ultrasonic sensor to emit acoustic waves and receive reflected signals, record the intensity and time information of the acoustic waves reflected by the target dirt, and obtain the acoustic reflection signal corresponding to the target dirt. Different frequencies of acoustic waves can be selected to adapt to surfaces of different materials.

[0038] S102: Convert the acoustic reflection signal corresponding to the target dirt into an image representation.

[0039] The device can filter, amplify, and feature extract the obtained acoustic reflection signal corresponding to the target dirt, and perform operations such as noise removal.

[0040] The device can generate a grayscale image or a heat map using the reflection intensity and time information of the acoustic waves, so that the acoustic reflection signal is converted into an image representation, which can reflect the distribution of dirt.

[0041] Pseudo-color processing can also be used to enhance the visualization effect.

[0042] In one implementation, S102 can include: performing GAF conversion and MTF conversion on the acoustic reflection signal respectively to obtain a first image representation and a second image representation; wherein the acoustic reflection signal is the acoustic reflection signal corresponding to the target dirt collected by the ultrasonic sensor, the GAF conversion refers to Gram angle field conversion, and the MTF conversion refers to Markov transition field conversion.

[0043] Use an ultrasonic sensor to emit ultrasonic signals to the target dirt. These signals will be reflected when they encounter dirt and surfaces.

[0044] Collect the reflected acoustic signals and convert them into electrical signals. These signals contain a wealth of information about the shape, thickness, and material properties of the dirt.

[0045] ​GAF (Gramian Angular Field) transformation is a technique for converting time series data, such as acoustic reflection signals, into images by encoding the angle and amplitude information of the signal into a two-dimensional image. First, the phase and amplitude of the acoustic reflection signal are calculated, and then a two-dimensional matrix is generated using this data, where each point in the matrix represents the state of the signal at a specific time point. The acoustic reflection signal is normalized and converted to polar coordinates. The angle (θ) and amplitude (r) of each time slice can be calculated and mapped to the coordinate system of the image to generate the first image representation.

[0046] MTF (Markov Transition Field) transformation is used to capture the state transition information of the signal by constructing a Markov chain to analyze the dynamic characteristics of the data. This method models the state changes of the acoustic reflection signal as a state transition matrix and converts this matrix into an image representation. The acoustic reflection signal is divided into multiple state segments, and the transition probabilities between each state are calculated to form a transition matrix. By mapping the transition matrix to the image space, a second image representation is generated, showing the dynamic characteristics of the dirt.

[0047] Specifically, GAF transformation and MTF transformation are used to encode one-dimensional time series acoustic reflection signals into two-dimensional images, respectively, which are used to describe the global trend and local state transition characteristics of the acoustic signal. After standardization, the acoustic reflection signal is converted to polar coordinates:

[0048]

[0049] Considering the time correlation of each data point in the acoustic signal time series at different time intervals, the one-dimensional data is converted to a GAF image:

[0050]

[0051]

[0052] By dividing the time series into Q quantile bins, each data point is assigned to the corresponding quantile bin . Based on the first-order Markov chain, an MTF is constructed to convert the one-dimensional data into an MTF image:

[0053]

[0054] where represents the transition probability from quantile bin to .

[0055] Specifically, the GAF and MTF transformations are performed on the acoustic reflection signal S(t) to obtain two-dimensional image forms With wherein:

[0056]

[0057]

[0058] wherein, , respectively represent the row and column indices in the GAF matrix, corresponding to the two time steps and relationship; , is the angular representation of the signal at time steps and is the value of the signal at time point is the amplitude of the entire signal , represent the signal values of the signal at time steps and is the transition probability from state to state In the MTF method, this value represents the probability of the signal transitioning to state given the past state

[0059] In this embodiment, through GAF and MTF conversion, the information of the acoustic reflection signal can be visualized, facilitating subsequent image processing and analysis, and improving the recognition rate of the dirt. The MTF conversion provides dynamic transition information of the dirt, which can analyze the change trend of the dirt, so as to identify different types of dirt. Through the combination of multiple ultrasonic sensors, the dirt information can be obtained from different angles, making the recognition result more comprehensive and accurate. This method can adapt to different types of dirt and their distribution, and is suitable for various cleaning scenes such as industrial equipment, building exterior walls, and vehicles. The automatic dirt recognition process reduces manual intervention and improves the simplicity and efficiency of operation.

[0060] S103: Obtain a fusion image corresponding to the target dirt according to the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm.

[0061] ​​​​​The device can employ multi-channel image fusion algorithms (such as weighted average, wavelet transform, etc.) to fuse the original image and image representation corresponding to the target dirt, obtaining a fused image corresponding to the target dirt. Different fusion weights can be set according to actual needs to highlight the features of a particular image.

[0062] In one embodiment, step S102 includes performing GAF conversion and MTF conversion on the acoustic reflection signal to obtain a first image representation and a second image representation; wherein the acoustic reflection signal is the acoustic reflection signal corresponding to the target dirt collected by an ultrasonic sensor, the GAF conversion refers to Gram angle field conversion, and the MTF conversion refers to Markov transfer field conversion; step S103 may include steps S1031~S1032, such as... Figure 2 As shown, S1031~S1032 are as follows:

[0063] S1031: Perform image enhancement operation on the original image corresponding to the target dirt to obtain an enhanced original image; wherein, the image enhancement operation includes bilateral filtering denoising and / or finite contrast adaptive histogram equalization.

[0064] For the original image of the target dirt, image enhancement is performed to improve image quality and recognition capability.

[0065] Image enhancement methods may include bilateral filtering for denoising and finite contrast adaptive histogram equalization (CLAHE).

[0066] Bilateral filtering is a filtering technique that considers both spatial distance and color similarity, effectively removing image noise while preserving edge features. By setting appropriate spatial and color parameters, image clarity can be improved.

[0067] Finite contrast adaptive histogram equalization can enhance image contrast in local areas, making it particularly suitable for processing images with uneven lighting. The original image is divided into multiple small blocks, each of which undergoes histogram equalization, and then the blocks are stitched together to form the overall image.

[0068] Specifically, for high-frequency noise that may exist in the original image, a bilateral filter is used for smoothing while preserving image edge information:

[0069]

[0070]

[0071] in, For a spatial Gaussian kernel, Gaussian weights for the differences in grayscale values. For pixels The local neighborhood, and is the corresponding pixel gray value. Then, in order to enhance the local contrast of the image, histogram equalization is performed in each region, and a clipping threshold T is introduced to the pixel histogram to limit the contrast amplification and prevent noise enhancement:

[0072]

[0073]

[0074] wherein, is the number of pixels of the image gray level is the clipped histogram value, is the cumulative distribution function, is the image size.

[0075] S1032: Splice the enhanced original image, the first image representation and the second image representation to obtain the fusion image corresponding to the target dirt.

[0076] The enhanced original image is spliced with the first image representation and the second image representation obtained in S102 to form a fusion image.

[0077] In this embodiment, the enhanced original image is spliced with the GAF and MTF images to form a comprehensive image, providing multi-dimensional information support.

[0078] This method not only improves the accuracy of dirt identification and the scientificity of cleaning decision, but also can adapt to various cleaning scenes, improve operation efficiency and reduce the risk of false cleaning.

[0079] S104: Input the fusion image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and size data of the target dirt; wherein the improved deep neural network is integrated with a double attention fusion module and a boundary perception enhancement module, the double attention fusion module includes a channel attention submodule and a spatial attention submodule, the channel attention submodule is used to capture the relative importance of different feature channels to dirt identification, the spatial attention submodule is used to strengthen the boundary features of dirt, and the boundary perception enhancement module is used to highlight the edge change significant area.

[0080] The improved deep neural network is pre-stored in the device, and the improved deep neural network is integrated with a double attention fusion module and a boundary perception enhancement module.

[0081] ​The double attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt identification. The channel attention submodule can use a fully connected layer to calculate the importance of each feature channel and dynamically adjust the weight of the feature channel.

[0082] The spatial attention submodule is used to enhance the dirt boundary features. The spatial attention submodule can generate a spatial attention map through a convolution layer and a pooling layer to emphasize important areas.

[0083] The boundary perception enhancement module is used to highlight areas with significant edge changes. The boundary perception enhancement module is designed with a specific edge detection layer that can use methods such as the Sobel operator to highlight edge changes and enhance the identification of dirt boundaries.

[0084] The device inputs the fusion image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and size data of the target dirt.

[0085] In one implementation, S104 can specifically include: inputting the fusion image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and size data of the target dirt .

[0086] wherein,

[0087]

[0088] the fusion image corresponding to the target dirt , represents the height of the fusion image corresponding to the target dirt, represents the width of the fusion image corresponding to the target dirt, represents the number of channels of the fusion image corresponding to the target dirt.

[0089] In one implementation, the channel attention mechanism is used to capture the relative importance of different feature channels in the identification task, and to distinguish the importance of the original image, the GAF image, and the MTF image for dirt identification, so as to highlight key channels and suppress redundant features. The execution steps of the channel attention submodule in the improved deep neural network include:

[0090] obtaining a feature image corresponding to the target dirt ;

[0091] performing global average pooling and global maximum pooling on the feature image corresponding to the target dirt to obtain a global average pooling result​ and global max pooling results ;

[0092] Input the global average pooling result and the global max pooling result To the fully connected layer and The activation function yields the attention weights for the first channel. Second channel attention weight ;

[0093] Based on the first channel attention weight Second channel attention weight as well as Activation function to obtain target channel attention weights ;

[0094] Based on the target channel attention weight Feature image corresponding to the target dirt The output of the channel attention submodule is obtained. ;

[0095] in,

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] Here, through

[0103]

[0104]

[0105] Perform global average pooling and global max pooling.

[0106] Through the fully connected layer

[0107]

[0108]

[0109] The activation function yields the channel attention weights.

[0110] The channel attention weight calculation formula is:

[0111]

[0112] The channel attention weight calculation formula is applied as:

[0113]

[0114] The feature image corresponding to the target dirt , represents the width of the feature image corresponding to the target dirt, represents the height of the feature image corresponding to the target dirt, represents the number of channels of the feature image corresponding to the target dirt, may be 3, represents the first fully connected layer, represents the second fully connected layer, represents an activation function, represents channel-by-channel multiplication.

[0115] In an embodiment, the spatial attention mechanism is used to strengthen the response of key spatial regions in the feature map, i.e., the dirt boundary features. The execution steps of the spatial attention sub-module in the improved deep neural network include:

[0116] obtaining the output result of the channel attention sub-module ;

[0117] performing global average pooling, global maximum pooling, and convolution operations on the output feature map of the channel attention sub-module to obtain a global average pooling result and a global maximum pooling result ;

[0118] obtaining a spatial attention weight according to the global average pooling result , the global maximum pooling result , and an activation function ;

[0119] obtaining the output result of the spatial attention sub-module according to the spatial attention weight and the output result of the channel attention sub-module ;

[0120] wherein,

[0121]

[0122]

[0123]

[0124]

[0125] denotes global average pooling, denotes global max pooling, denotes activation function, denotes a 1x1 convolution layer, denotes channel-wise multiplication.

[0126] Specifically, according to the feature map output by the channel attention module , first, global average pooling and global max pooling are used for channel dimension compression:

[0127]

[0128]

[0129] In the formula, is a 1x1 convolution layer.

[0130] The spatial attention weight calculation formula is:

[0131]

[0132] In the formula, is the spatial attention weight.

[0133] The application of the spatial attention weight calculation formula is:

[0134]

[0135] In the formula, denotes element-wise multiplication.

[0136] In one embodiment, the execution steps of the boundary perception enhancement module in the improved deep neural network include:

[0137] The output results of the channel attention submodule and the output results of the spatial attention submodule are obtained;

[0138] According to the output results of the channel attention submodule and the edge detection operator , gradient information is extracted;

[0139] convolve the gradient information to obtain a feature mapping result;

[0140] According to the feature mapping result and an activation function, an edge response map is obtained ;

[0141] fuse the edge response map and the output result of the spatial attention sub-module to obtain the output result of the boundary perception enhancement module ;

[0142] wherein,

[0143]

[0144]

[0145] represents a 1x1 convolution layer, represents an activation function, represents a weight coefficient.

[0146] Specifically, the improved deep neural network integrates a boundary perception enhancement module to assist the main branch in extracting the fine boundary of the dirt area, highlighting the area with significant edge changes in the image, thereby enhancing the model's perception ability of the dirt edge contour. The branch first applies a Sobel operator to the intermediate feature map to extract gradient information, then compresses and maps features through a 1x1 convolution, and then outputs an edge response map through a Sigmoid activation function . The obtained edge map E is fused with the spatial attention mechanism as guide information to strengthen the activation response of the edge area in the main branch feature map. The fusion method is weighted multiplication:

[0147]

[0148] is a weight coefficient, is the fused attention map.

[0149] S105: Adjust the pose of the dirt cleaning device according to the segmentation mask data and a preset coordinate conversion algorithm.

[0150] According to the segmentation mask data, the target position is converted into the coordinate system of the cleaning device through geometric transformation (such as affine transformation).

[0151] The pose of the cleaning device is adjusted through the controller to accurately align the dirt position.

[0152] S106: Adjust the dirt cleaning parameters according to the size data of the target dirt, and control the dirt cleaning device to perform cleaning operation.

[0153] According to the size data of the target dirt, set the parameters of the cleaning device such as pressure, flow, cleaning time, etc. to ensure the cleaning effect.

[0154] Control the cleaning device to start, and use dynamic adjustment of cleaning path and pressure to ensure efficient cleaning and reduce energy consumption.

[0155] In one embodiment, as shown in S106, S107~S108 can also be included after S106, S107~S108 are as follows: Figure 3

[0156] S107: After the completion of the cleaning operation, re-identify the target dirt to obtain the target dirt residual area.

[0157] After completing the preliminary cleaning operation, the cleaning device should perform a preliminary assessment of the target dirt.

[0158] The cleaning device can use sensors such as cameras, laser scanners or other imaging technologies to collect surface images and data.

[0159] Use image processing algorithms (such as machine learning, deep learning models) to analyze the cleaned surface and identify the residual dirt area.

[0160] Use image segmentation techniques (such as U-Net, FCN, etc.) to accurately identify the boundary of the dirt.

[0161] Calculate the area of the identified dirt area to obtain the "target dirt residual area".

[0162] It can be converted to actual area (such as square centimeters) by pixel counting and corrected using a scale.

[0163] S108: If the target dirt residual area is greater than the preset threshold, adjust the pose of the dirt cleaning device and the dirt cleaning parameters, and control the dirt cleaning device to perform cleaning operation again. Repeat the above operation until the target dirt residual area is not greater than the preset threshold.

[0164] Compare the calculated target dirt residual area with the preset cleaning threshold. This threshold can be set according to the cleaning requirements, dirt type and cleaning environment.

[0165] If the residual area is greater than the preset threshold, the cleaning device needs to adjust its position and angle according to the distribution of the dirt to ensure more comprehensive coverage of the cleaning area. For example, by repositioning the cleaning device through a mechanical arm or a mobile chassis to ensure that the cleaning head can contact the uncleaned dirt area.​

[0166] According to the properties of the dirt (such as adhesion strength, type, etc.), adjust the cleaning parameters, such as water pressure, cleaning agent concentration, cleaning speed, etc.

[0167] Repeat steps S107 and S108 until the target dirt residual area is less than or equal to the preset threshold. A maximum number of cycles can be set to avoid wasting resources in ineffective cleaning.

[0168] In this embodiment, intelligent recognition and feedback adjustment reduce the time and resource waste of repeated cleaning. Automated dirt recognition and parameter adjustment reduce labor costs and improve work efficiency. It can accurately identify and clean different types of dirt to ensure that the surface cleaning meets the expected standards. Through the accumulation of historical data, the cleaning system can continuously optimize the cleaning parameters to improve the overall cleaning effect. This method can be flexibly adjusted according to different environments and dirt characteristics, has wide adaptability, and is suitable for various cleaning scenarios.

[0169] In this embodiment, the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt are obtained; the acoustic reflection signal corresponding to the target dirt is converted into an image representation; a fusion image corresponding to the target dirt is obtained according to the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm; the fusion image corresponding to the target dirt is input into an improved deep neural network to obtain segmentation mask data and size data of the target dirt; the pose of the dirt cleaning device is adjusted according to the segmentation mask data and a preset coordinate conversion algorithm; the dirt cleaning parameters are adjusted according to the size data of the target dirt, and the dirt cleaning device is controlled to perform a cleaning operation. Through the fusion image and the improved deep learning network, the recognition accuracy of the dirt edge is significantly improved, especially in complex environments and low contrast conditions. The cleaning device adjusts the pose and cleaning parameters according to the actual distribution and characteristics of the dirt to achieve precise cleaning and reduce energy consumption and resource waste. The introduction of deep learning and acoustic signal processing promotes the intelligentization of the cleaning process and adapts to different cleaning needs and environmental changes. This method can be applied to the cleaning of surfaces of different materials and shapes, and has wide application prospects. Through the above implementation schemes, the accuracy and efficiency of dirt recognition and cleaning will be significantly improved, and various cleaning needs and operating environments can be better met.

[0170] In addition, by fusing the original image collected by the CCD camera and the acoustic reflection signal obtained by the ultrasonic sensor, the one-dimensional acoustic signal is converted into image information by using the GAF and MTF methods, a multi-channel fusion image input network is constructed, the shortcomings of single image information are effectively made up, and the representation ability of the dirt area is enhanced. A deep segmentation network combining channel attention, spatial attention and enhanced boundary perception mechanism is designed, which can highlight the dirt contour boundary features, significantly improve the clarity and accuracy of the segmentation edge, and is suitable for weak contrast or irregular dirt identification scenes. According to the dirt position and size information output by the segmentation result, the system can dynamically adjust the nozzle pose and jet parameters, realize fine control of the cleaning path, pressure and time, and avoid energy waste and cleaning blind area of the traditional fixed cleaning mode.

[0171] It should be understood that the size of the serial number of each step in the above embodiment does not mean the order of execution, and the execution order of each process should be determined according to its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0172] Please refer to Figure 4 , Figure 4 is a schematic diagram of the dirt identification and cleaning device provided by the second embodiment of the present application. Each unit included is used to execute Figures 1 to 3 each step in the corresponding embodiment. For details, please refer to Figures 1 to 3 the related description in the corresponding embodiment. For the sake of convenience, only the part related to the present embodiment is shown. Please refer to Figure 4 , the dirt identification and cleaning device 4 comprises:

[0173] The acquisition unit 410 is configured to acquire an original image corresponding to a target dirt and an acoustic reflection signal corresponding to the target dirt.

[0174] The first processing unit 420 is configured to convert the acoustic reflection signal corresponding to the target dirt into an image representation.

[0175] The second processing unit 430 is configured to obtain a fusion image corresponding to the target dirt according to the original image corresponding to the target dirt, the image representation and a preset multi-channel image fusion algorithm.

[0176] The third processing unit 440 is configured to input the fusion image corresponding to the target dirt into an improved deep neural network to obtain segmentation mask data and size data of the target dirt; the improved deep neural network is integrated with a double-attention fusion module and a boundary perception enhancement module; the double-attention fusion module includes a channel attention submodule and a spatial attention submodule; the channel attention submodule is configured to capture the relative importance of different feature channels to dirt identification; the spatial attention submodule is configured to strengthen dirt boundary features; and the boundary perception enhancement module is configured to highlight edge change significant regions.

[0177] The fourth processing unit 450 is configured to adjust the pose of the dirt cleaning device according to the segmentation mask data and a preset coordinate conversion algorithm.

[0178] The fifth processing unit 460 is configured to adjust dirt cleaning parameters according to the size data of the target dirt and control the dirt cleaning device to perform cleaning operations.

[0179] Further, the dirt identification and cleaning device 4 further comprises:

[0180] The sixth processing unit is configured to re-identify the target dirt after the completion of the cleaning operation to obtain a target dirt residual area.

[0181] The seventh processing unit is configured to re-adjust the pose of the dirt cleaning device and the dirt cleaning parameters if the target dirt residual area is greater than a preset threshold, control the dirt cleaning device to perform cleaning operations again, and repeat the above operations until the target dirt residual area is not greater than the preset threshold.

[0182] Further, the first processing unit is specifically configured to:

[0183] The sound reflection signal is subjected to GAF conversion and MTF conversion to obtain a first image representation and a second image representation; the sound reflection signal is a sound reflection signal corresponding to the target dirt collected by an ultrasonic sensor; the GAF conversion refers to Gram angle field conversion; and the MTF conversion refers to Markov transition field conversion.

[0184] Further, the second processing unit is specifically configured to:

[0185] The original image corresponding to the target dirt is subjected to an image enhancement operation to obtain an enhanced original image; the image enhancement operation includes bilateral filter denoising and / or limited contrast adaptive histogram equalization.

[0186] The enhanced original image, the first image representation and the second image representation are spliced to obtain a fusion image corresponding to the target dirt.

[0187] Further, the third processing unit is specifically configured to:

[0188] input the fusion image corresponding to the target dirt to the improved deep neural network to obtain segmentation mask data and size data of the target dirt ;

[0189] wherein,

[0190]

[0191] the fusion image corresponding to the target dirt , represents the height of the fusion image corresponding to the target dirt, represents the width of the fusion image corresponding to the target dirt, represents the number of channels of the fusion image corresponding to the target dirt.

[0192] Further, the third processing unit is specifically configured to:

[0193] obtain a feature image corresponding to the target dirt ;

[0194] perform global average pooling and global maximum pooling on the feature image corresponding to the target dirt to obtain a global average pooling result and a global maximum pooling result ;

[0195] input the global average pooling result and the global maximum pooling result to a fully connected layer and an activation function to obtain first channel attention weights and second channel attention weights ;

[0196] obtain target channel attention weights according to the first channel attention weights , the second channel attention weights , and an activation function ;

[0197] obtain an output result of the channel attention sub-module according to the target channel attention weights and the feature image corresponding to the target dirt ;

[0198] wherein,

[0199]

[0200]

[0201]

[0202]

[0203]

[0204]

[0205] a feature image corresponding to the target dirt , a width of the feature image corresponding to the target dirt, a height of the feature image corresponding to the target dirt, a number of channels of the feature image corresponding to the target dirt, a first fully connected layer, a second fully connected layer, an activation function, channel-wise multiplication.

[0206] Further, the third processing unit is specifically further configured to:

[0207] obtain an output result of the channel attention sub-module ;

[0208] perform global average pooling, global maximum pooling and convolution operation on the output feature map of the channel attention sub-module to obtain a global average pooling result and a global maximum pooling result ;

[0209] obtain a spatial attention weight according to the global average pooling result , the global maximum pooling result and an activation function ;

[0210] obtain an output result of the spatial attention sub-module according to the spatial attention weight and the output result of the channel attention sub-module ;

[0211] wherein, ​​​

[0212]

[0213]

[0214]

[0215]

[0216] represents global average pooling, represents global maximum pooling, represents activation function, represents a 1x1 convolution layer, represents channel-wise multiplication.

[0217] Further, the third processing unit is specifically further configured to:

[0218] obtain an output result of the channel attention submodule and an output result of the spatial attention submodule ;

[0219] extract gradient information according to the output result of the channel attention submodule and an edge detection operator ;

[0220] perform a convolution operation on the gradient information to obtain a feature mapping result;

[0221] obtain an edge response map according to the feature mapping result and activation function ;

[0222] fuse the edge response map and the output result of the spatial attention submodule to obtain an output result of the boundary perception enhancement module ;

[0223] wherein,

[0224]

[0225]

[0226] represents a 1x1 convolution layer, represents activation function, represents a weight coefficient.

[0227] Figure 5This is a schematic diagram of the dirt identification and cleaning device provided in the third embodiment of this application. Figure 5 As shown, the dirt identification and cleaning device 5 of this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50, such as a dirt identification and cleaning program. When the processor 50 executes the computer program 52, it implements the steps in the various dirt identification and cleaning method embodiments described above, for example... Figure 1 Steps 101 to 106 are shown. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of units 410 to 460 are shown.

[0228] For example, the computer program 52 can be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 52 in the dirt identification and cleaning device 5. For example, the computer program 52 can be divided into an acquisition unit, a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit, with the specific functions of each unit as follows:

[0229] The acquisition unit is used to acquire the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt;

[0230] The first processing unit is used to convert the acoustic reflection signal corresponding to the target dirt into an image representation;

[0231] The second processing unit is used to obtain a fused image corresponding to the target dirt based on the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm;

[0232] The third processing unit is used to input the fused image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and the size data of the target dirt; wherein, the improved deep neural network integrates a dual attention fusion module and a boundary awareness enhancement module. The dual attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt recognition, the spatial attention submodule is used to enhance the boundary features of dirt, and the boundary awareness enhancement module is used to highlight areas with significant edge changes.

[0233] a fourth processing unit, configured to adjust a pose of the dirt cleaning device according to the segmentation mask data and a preset coordinate conversion algorithm;

[0234] a fifth processing unit, configured to adjust a dirt cleaning parameter according to the size data of the target dirt, and control the dirt cleaning device to perform a cleaning operation.

[0235] The dirt identification and cleaning device can include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art can understand that, Figure 5 The dirt identification and cleaning device 5 is only an example and does not constitute a limitation on the dirt identification and cleaning device 5, and can include more or fewer components than shown, or combine certain components, or different components, for example, the dirt identification and cleaning device can also include an input and output device, a network access device, a bus, etc.

[0236] The processor 50 can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0237] The memory 51 can be an internal storage unit of the dirt identification and cleaning device 5, for example, a hard disk or a memory of the dirt identification and cleaning device 5. The memory 51 can also be an external storage device of the dirt identification and cleaning device 5, for example, a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, etc. Further, the dirt identification and cleaning device 5 can include both the internal storage unit and the external storage device of the dirt identification and cleaning device 5. The memory 51 is used to store the computer program and other programs and data required by the dirt identification and cleaning device. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0238] It should be noted that the information interaction, execution process and the like between the above apparatuses / units are based on the same concept as the method embodiments of the present application, and the specific functions and brought technical effects can be referred to the method embodiments part, which will not be repeated here.

[0239] The embodiments of the present application further provide a network device, comprising at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor implements the steps in any of the above method embodiments when executing the computer program.

[0240] The embodiments of the present application further provide a computer readable storage medium, which stores a computer program, wherein the computer program is executable by a processor to implement the steps in any of the above method embodiments.

[0241] The embodiments of the present application provide a computer program product, which, when executed on a mobile terminal, enables the mobile terminal to implement the steps in any of the above method embodiments.

[0242] The integrated unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the present application can implement all or part of the processes in the above embodiments by a computer program to instruct related hardware to complete, and the computer program can be stored in a computer readable storage medium. The computer program, when executed by a processor, can implement the steps in any of the above method embodiments. The computer program includes computer program code, which can be in the form of source code, object code, executable file or some intermediate form. The computer readable medium at least includes any entity or device capable of carrying the computer program code to the photographing device / terminal equipment, recording medium, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal and software distribution medium. For example, U disk, mobile hard disk, magnetic disk or optical disk, etc.

[0243] In the above embodiments, the description of each embodiment has its own focus, and the parts not described or recorded in detail in a certain embodiment can be referred to the related description of other embodiments.

[0244] Those skilled in the art can understand that the units and algorithm steps of each example described in combination with the embodiments disclosed herein can be realized in electronic hardware or a combination of computer software and electronic hardware. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0245] In the embodiments provided in the present application, it should be understood that the disclosed apparatus / network device and method can be implemented by other ways. For example, the apparatus / network device embodiments described above are only schematic, for example, the division of the modules or units is only a logical function division, and there can be another division in actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between different parts can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical, mechanical or other forms.

[0246] The units described as separate components can or can not be physically separate, and the components shown as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments.

[0247] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that the technical solutions recorded in the foregoing embodiments can be modified, or some technical features can be replaced by equivalents; and these modifications or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A soil discriminating cleaning method characterized by, The method comprises the steps of: obtaining an original image corresponding to a target dirt and an acoustic reflection signal corresponding to the target dirt; converting the acoustic reflection signal corresponding to the target dirt into an image representation; obtaining a fusion image corresponding to the target dirt according to the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm; inputting the fusion image corresponding to the target dirt into an improved deep neural network to obtain segmentation mask data and size data of the target dirt; wherein the improved deep neural network is integrated with a double-attention fusion module and a boundary perception enhancement module, the double-attention fusion module comprises a channel attention submodule and a spatial attention submodule, the channel attention submodule is used to capture the relative importance of different feature channels to dirt identification, and the spatial attention submodule is used to strengthen the boundary features of the dirt, and the boundary perception enhancement module is used to highlight the edge change significant area; adjusting the pose of a dirt cleaning device according to the segmentation mask data and a preset coordinate conversion algorithm; adjusting dirt cleaning parameters according to the size data of the target dirt and controlling the dirt cleaning device to perform a cleaning operation.

2. The method of claim 1, wherein the cleaning is performed by a cleaning robot. The method further comprises the steps of: re-identifying the target dirt after the cleaning operation is completed to obtain a target dirt residual area; if the target dirt residual area is greater than a preset threshold, re-adjusting the pose of the dirt cleaning device and the dirt cleaning parameters, controlling the dirt cleaning device to perform a cleaning operation again, and repeating the above operations until the target dirt residual area is not greater than the preset threshold.

3. The soil discriminating cleaning method according to claim 1 or 2, characterized by, The step of converting the acoustic reflection signal corresponding to the target dirt into an image representation comprises the steps of: performing GAF conversion and MTF conversion on the acoustic reflection signal respectively to obtain a first image representation and a second image representation; wherein the acoustic reflection signal is the acoustic reflection signal corresponding to the target dirt collected by an ultrasonic sensor, the GAF conversion refers to Gram angle field conversion, and the MTF conversion refers to Markov transition field conversion.

4. The method of claim 3, wherein the cleaning is performed by a cleaning robot. The step of obtaining a fusion image corresponding to the target dirt according to the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm comprises the steps of: performing image enhancement operations on the original image corresponding to the target dirt to obtain an enhanced original image; wherein the image enhancement operations include bilateral filter denoising and / or limited contrast adaptive histogram equalization; splicing the enhanced original image, the first image representation, and the second image representation to obtain the fusion image corresponding to the target dirt.

5. The method according to claim 1 or 2, wherein The step of inputting the fusion image corresponding to the target dirt into an improved deep neural network to obtain segmentation mask data and size data of the target dirt comprises the steps of: corresponding to the target dirt input to the improved deep neural network , obtaining segmentation mask data and size data of the target dirt ; wherein ； the fusion image corresponding to the target dirt , a height of the fusion image corresponding to the target dirt, a width of the fusion image corresponding to the target dirt, a channel number of the fusion image corresponding to the target dirt.

6. The scale identification cleaning method according to claim 5, wherein the execution steps of the channel attention submodule in the improved deep neural network comprise: acquiring a feature image corresponding to the target dirt ; A feature image corresponding to the target dirt Global average pooling and global maximum pooling are performed to obtain a global average pooling result and a global maximum pooling result ; input the global average pooling result and the global maximum pooling result to a full connection layer and activation function to obtain first channel attention weight and second channel attention weight ; According to the first channel attention weight , the second channel attention weight , and an activation function, to obtain a target channel attention weight ; According to the target channel attention weight And the feature image corresponding to the target dirt , Get the output result of the channel attention submodule ; wherein ； ； ； ； ； ； the feature image corresponding to the target dirt , denotes a width of the feature image corresponding to the target dirt, denotes a height of the feature image corresponding to the target dirt, denotes a number of channels of the feature image corresponding to the target dirt, denotes a first fully connected layer, denotes a second fully connected layer, denotes an activation function, denotes a channel-wise multiplication.

7. The scale detection cleaning method according to claim 5, wherein the execution steps of the spatial attention submodule in the improved deep neural network comprise: obtaining an output result of the channel attention submodule ; output feature maps of the channel attention sub-module performing global average pooling, global maximum pooling and convolution operation to obtain global average pooling result and global maximum pooling result ; According to the global average pooling result , a global maximum pooling result , and an activation function, spatial attention weights are obtained ; According to the spatial attention weight and an output result of the channel attention submodule , obtaining an output result of the spatial attention submodule ; wherein ； ； ； ； denotes global average pooling, denotes global max pooling, denotes activation function, denotes a 1 x 1 convolutional layer, denotes channel-wise multiplication.

8. The method of claim 5, wherein the cleaning is performed by a cleaning robot. the execution steps of the boundary perception enhancement module in the improved deep neural network comprise: obtaining an output result of the channel attention submodule and an output result of the spatial attention submodule ; According to the output result of the channel attention sub-module and an edge detection operator , gradient information is extracted; performing a convolution operation on the gradient information to obtain a feature mapping result; According to the feature mapping result and an activation function, to obtain an edge response map ; in response to the edge map and an output result of the spatial attention sub-module fusion, obtaining an output result of the boundary perception enhancement module ; wherein, ； ； represents a convolutional layer of 1x1, represents an activation function, represents a weight coefficient.

9. A soil discriminating cleaning apparatus comprising: Processor, memory, and computer program stored in the memory and executable on the processor, characterized in that the processor implements the steps of the method according to any one of claims 1 to 8 when executing the computer program.

10. A computer-readable storage medium storing a computer program, the computer program comprising instructions that, when executed by a computer, cause the computer to perform the method of any one of claims 1 to 9. The computer program is executed by the processor to implement the method according to any one of claims 1 to 8.

Images