A method, apparatus and device for soil identification cleaning

Abstract

This application pertains to the field of intelligent jet cleaning, providing a method, apparatus, and equipment for dirt identification and cleaning. The dirt identification and cleaning method includes: acquiring the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt; converting the acoustic reflection signal corresponding to the target dirt into an image representation; obtaining a fused image corresponding to the target dirt based on the original image, image representation, and a preset multi-channel image fusion algorithm; inputting the fused image corresponding to the target dirt into an improved deep neural network to obtain segmentation mask data and target dirt size data; adjusting the pose of the dirt cleaning equipment based on the segmentation mask data and a preset coordinate transformation algorithm; adjusting the dirt cleaning parameters based on the target dirt size data, and controlling the dirt cleaning equipment to perform the cleaning operation. The above method significantly improves the accuracy and efficiency of dirt identification and cleaning, better meeting various cleaning needs and operating environments.

Description

Technical Field

[0001] This application belongs to the field of intelligent jet cleaning technology, and in particular relates to a dirt identification and cleaning method, device and equipment. Background Technology

[0002] Industries such as nuclear power, precision manufacturing, and shipbuilding all have varying requirements for surface cleanliness, making the automatic identification and precise cleaning of surface contaminants a key issue in industrial automation. Traditional contaminant identification methods largely rely on grayscale or texture analysis of visual images, which are easily affected by surface materials and lighting conditions, resulting in limited identification accuracy. Furthermore, traditional jet cleaning methods often employ fixed pressure and paths, lacking adaptive response capabilities to contaminant distribution, easily leading to energy waste or incomplete cleaning. In recent years, deep learning methods have been widely introduced into image segmentation and defect recognition, but they still face challenges in contaminant detection. Firstly, neural networks are not accurate enough in identifying contaminant edges, especially in cases of low contrast and irregular distribution. Secondly, the limited generalization ability of images results in insufficient model generalization, and they are significantly affected by factors such as ambient lighting. Summary of the Invention

[0003] This application provides a method, apparatus, and device for identifying and cleaning dirt, which can solve the above-mentioned problems.

[0004] In a first aspect, embodiments of this application provide a dirt identification and cleaning method, including:

[0005] Acquire the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt;

[0006] The acoustic reflection signal corresponding to the target dirt is converted into an image representation;

[0007] Based on the original image corresponding to the target dirt, the image representation, and the preset multi-channel image fusion algorithm, a fused image corresponding to the target dirt is obtained;

[0008] The fused image corresponding to the target dirt is input into the improved deep neural network to obtain segmentation mask data and the size data of the target dirt. The improved deep neural network integrates a dual attention fusion module and a boundary awareness enhancement module. The dual attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt recognition, the spatial attention submodule is used to enhance the boundary features of the dirt, and the boundary awareness enhancement module is used to highlight areas with significant edge changes.

[0009] The pose of the dirt cleaning device is adjusted according to the segmentation mask data and the preset coordinate transformation algorithm;

[0010] Adjust the dirt cleaning parameters according to the size data of the target dirt, and control the dirt cleaning equipment to perform the cleaning operation.

[0011] Secondly, embodiments of this application provide a dirt identification and cleaning device, comprising:

[0012] The acquisition unit is used to acquire the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt;

[0013] The first processing unit is used to convert the acoustic reflection signal corresponding to the target dirt into an image representation;

[0014] The second processing unit is used to obtain a fused image corresponding to the target dirt based on the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm;

[0015] The third processing unit is used to input the fused image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and the size data of the target dirt; wherein, the improved deep neural network integrates a dual attention fusion module and a boundary awareness enhancement module. The dual attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt recognition, the spatial attention submodule is used to enhance the boundary features of dirt, and the boundary awareness enhancement module is used to highlight areas with significant edge changes.

[0016] The fourth processing unit is used to adjust the pose of the dirt cleaning device according to the segmentation mask data and the preset coordinate transformation algorithm.

[0017] The fifth processing unit is used to adjust the dirt cleaning parameters according to the size data of the target dirt, and to control the dirt cleaning equipment to perform the cleaning operation.

[0018] Thirdly, embodiments of this application provide a dirt identification and cleaning device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0019] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0020] In this embodiment, the original image and acoustic reflection signal corresponding to the target dirt are acquired; the acoustic reflection signal is converted into an image representation; a fused image is obtained based on the original image, image representation, and a preset multi-channel image fusion algorithm; the fused image is input into an improved deep neural network to obtain segmentation mask data and target dirt size data; the pose of the dirt cleaning device is adjusted based on the segmentation mask data and a preset coordinate transformation algorithm; the dirt cleaning parameters are adjusted based on the target dirt size data, and the dirt cleaning device is controlled to perform cleaning operations. By fusing images and using an improved deep learning network, the accuracy of dirt edge recognition is significantly improved, especially in complex environments and low-contrast conditions. The cleaning device adjusts its pose and cleaning parameters according to the actual distribution and characteristics of the dirt, achieving precise cleaning and reducing energy consumption and resource waste. The introduction of deep learning and acoustic signal processing promotes the intelligentization of the cleaning process, adapting to different cleaning needs and environmental changes. This method is applicable to surface cleaning of different materials and shapes, and has broad application prospects. With the above implementation plan, the accuracy and efficiency of dirt identification and cleaning will be significantly improved, better meeting various cleaning needs and operating environments. Attached Figure Description

[0021] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0022] Figure 1 This is a schematic flowchart of a dirt identification and cleaning method provided in the first embodiment of this application;

[0023] Figure 2 This is a schematic flowchart of step S103 in a dirt identification and cleaning method provided in the first embodiment of this application;

[0024] Figure 3 This is a schematic flowchart of steps S107-S108 in a dirt identification and cleaning method provided in the first embodiment of this application;

[0025] Figure 4 This is a schematic diagram of the dirt identification and cleaning device provided in the second embodiment of this application;

[0026] Figure 5 This is a schematic diagram of the dirt identification and cleaning device provided in the third embodiment of this application. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] It should be understood that, when used in this application specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or a collection thereof.

[0029] It should also be understood that the term “and / or” as used in this application specification and the appended claims means any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0030] As used in this application specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrases "if determined" or "if detected [the described condition or event]" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once detected [the described condition or event]," or "in response to detection [the described condition or event]."

[0031] Furthermore, in the description of this application and the appended claims, the terms "first," "second," "third," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0032] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0033] Please see Figure 1 , Figure 1This is a schematic flowchart illustrating a dirt identification and cleaning method according to the first embodiment of this application. In this embodiment, the subject executing the dirt identification and cleaning method is a device with dirt identification and cleaning functions, such as a desktop computer, server, etc. Figure 1 The dirt identification and cleaning method shown may include:

[0034] S101: Acquire the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt.

[0035] High-definition cameras were used to photograph the target cleaning surface, ensuring that multiple images were acquired under different lighting conditions to improve the accuracy of subsequent analysis.

[0036] The original image corresponding to the target dirt is obtained by taking a picture.

[0037] An ultrasonic sensor is used to emit sound waves and receive reflected signals. The intensity and timing of the sound wave reflection at the target dirt are recorded to obtain the corresponding acoustic reflection signal. Different frequencies of sound waves can be selected to suit different surface materials.

[0038] S102: Convert the acoustic reflection signal corresponding to the target dirt into an image representation.

[0039] The equipment can filter, amplify, and extract features from the acoustic reflection signals corresponding to the target dirt, and remove noise.

[0040] The device can generate grayscale images or thermal images using the intensity and time information of sound wave reflection. In this way, the sound reflection signal is converted into an image representation, which can reflect the distribution of dirt.

[0041] Pseudo-color processing can also be used to enhance the visualization effect.

[0042] In one embodiment, S102 may include: performing GAF conversion and MTF conversion on the acoustic reflection signal to obtain a first image representation and a second image representation; wherein, the acoustic reflection signal is the acoustic reflection signal corresponding to the target dirt collected by the ultrasonic sensor, the GAF conversion refers to Gram angle field conversion, and the MTF conversion refers to Markov transfer field conversion.

[0043] An ultrasonic sensor is used to emit ultrasonic signals toward the target dirt. These signals are reflected when they encounter the dirt and the surface.

[0044] The reflected sound waves are collected and converted into electrical signals. These signals contain a wealth of information about the shape, thickness, and material properties of the dirt.

[0045] Gramian Angular Field (GAF) transformation is a technique for converting time-series data (such as acoustic reflection signals) into an image by encoding the angle and amplitude information of the signal into a two-dimensional image. First, the phase and amplitude of the acoustic reflection signal are calculated. Then, this data is used to generate a two-dimensional matrix where each point represents the state of the signal at a specific time point. The acoustic reflection signal is then normalized, converting it to polar coordinates. The angle (θ) and amplitude (r) for each time segment can be calculated and mapped to the coordinate system of the image, generating the first image representation.

[0046] MTF (Markov Transition Field) is used to capture the state transition information of a signal, analyzing the dynamic characteristics of the data by constructing a Markov chain. This method models the state changes of the acoustic reflection signal as a state transition matrix and converts this matrix into an image representation. The acoustic reflection signal is divided into multiple state segments, and the transition probabilities between each state are calculated to form a transition matrix. By mapping the transition matrix to image space, a second image representation is generated, showcasing the dynamic characteristics of the dirt.

[0047] Specifically, GAF conversion and MTF conversion are used to encode one-dimensional time-series acoustic reflection signals into two-dimensional images. The two-dimensional images are used to describe the global trend and local state transition characteristics of the acoustic signal. The acoustic reflection signals are then normalized and subjected to polar coordinate transformation.

[0048]

[0049] Considering the temporal correlation of each data point in the acoustic signal time series within different time intervals, the one-dimensional data is converted into a GAF image:

[0050]

[0051]

[0052] By dividing the time series into Q quantile bins, each data point Assigned to the corresponding partition box Based on a first-order Markov chain, an MTF is constructed to convert one-dimensional data into an MTF image.

[0053]

[0054] in Indicates the partition box arrive The transition probability.

[0055] Specifically, the acoustic reflection signal S(t) is converted using GAF and MTF methods to obtain a two-dimensional image. and ,in:

[0056]

[0057]

[0058] in, , These represent the row and column indices in the GAF matrix, corresponding to two time steps in the signal. and Relationship; , It is the signal at time step and The angle above; It is the signal at a certain point in time. The value, It is the entire signal The amplitude; , Indicates the signal at time step and The signal value on; From state to state The transition probability, in the MTF method, represents the probability given a past state. Under these conditions, the signal transitions to state. The probability of.

[0059] In this embodiment, GAF and MTF conversion visualizes the acoustic reflection signal information, facilitating subsequent image processing and analysis and improving the dirt recognition rate. MTF conversion provides dynamic dirt transfer information, enabling analysis of dirt change trends and thus identifying different types of dirt. The combination of multiple ultrasonic sensors allows for the acquisition of dirt information from different angles, resulting in more comprehensive and accurate recognition results. This method is adaptable to different types of dirt and their distribution, suitable for various cleaning scenarios such as industrial equipment, building exteriors, and vehicles. The automated dirt recognition process reduces manual intervention, improving ease of operation and efficiency.

[0060] S103: Based on the original image corresponding to the target dirt, the image representation, and the preset multi-channel image fusion algorithm, obtain the fused image corresponding to the target dirt.

[0061] The device can employ multi-channel image fusion algorithms (such as weighted average, wavelet transform, etc.) to fuse the original image and image representation corresponding to the target dirt, obtaining a fused image corresponding to the target dirt. Different fusion weights can be set according to actual needs to highlight the features of a particular image.

[0062] In one embodiment, step S102 includes performing GAF conversion and MTF conversion on the acoustic reflection signal to obtain a first image representation and a second image representation; wherein the acoustic reflection signal is the acoustic reflection signal corresponding to the target dirt collected by an ultrasonic sensor, the GAF conversion refers to Gram angle field conversion, and the MTF conversion refers to Markov transfer field conversion; step S103 may include steps S1031~S1032, such as... Figure 2 As shown, S1031~S1032 are as follows:

[0063] S1031: Perform image enhancement operation on the original image corresponding to the target dirt to obtain the enhanced original image; wherein, the image enhancement operation includes bilateral filtering denoising and / or finite contrast adaptive histogram equalization.

[0064] For the original image of the target dirt, image enhancement is performed to improve image quality and recognition capability.

[0065] Image enhancement methods may include bilateral filtering for denoising and finite contrast adaptive histogram equalization (CLAHE).

[0066] Bilateral filtering is a filtering technique that considers both spatial distance and color similarity, effectively removing image noise while preserving edge features. By setting appropriate spatial and color parameters, image clarity can be improved.

[0067] Finite contrast adaptive histogram equalization can enhance image contrast in local areas, making it particularly suitable for processing images with uneven lighting. The original image is divided into multiple small blocks, each of which undergoes histogram equalization, and then the blocks are stitched together to form the overall image.

[0068] Specifically, for high-frequency noise that may exist in the original image, a bilateral filter is used for smoothing while preserving image edge information:

[0069]

[0070]

[0071] in, For a spatial Gaussian kernel, Gaussian weights for the differences in grayscale values. For pixels The local neighborhood, and This corresponds to the grayscale value of the pixel. Subsequently, to enhance local contrast in the image, histogram equalization is performed within each region, and a cropping threshold T is introduced into the pixel histogram to limit contrast amplification and prevent noise enhancement.

[0072]

[0073]

[0074] in, Image gray levels The number of pixels, These are the histogram values ​​after cropping. Let be the cumulative distribution function. This refers to the image size.

[0075] S1032: By stitching together the enhanced original image, the first image representation, and the second image representation, a fused image corresponding to the target dirt is obtained.

[0076] The enhanced original image is then stitched together with the first and second image representations obtained in S102 to form a fused image.

[0077] In this embodiment, the enhanced original image is stitched together with the GAF and MTF images to form a composite image, providing multi-dimensional information support.

[0078] This method not only improves the accuracy of dirt identification and the scientific nature of cleaning decisions, but also adapts to various cleaning scenarios, increases operational efficiency, and reduces the risk of accidental cleaning.

[0079] S104: Input the fused image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and the size data of the target dirt; wherein, the improved deep neural network integrates a dual attention fusion module and a boundary awareness enhancement module. The dual attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt recognition. The spatial attention submodule is used to enhance the boundary features of the dirt. The boundary awareness enhancement module is used to highlight areas with significant edge changes.

[0080] The device pre-stores an improved deep neural network, which integrates a dual attention fusion module and a boundary awareness enhancement module.

[0081] The dual-attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt recognition. The channel attention submodule can use fully connected layers to calculate the importance of each feature channel and dynamically adjust the weights of the feature channels.

[0082] The spatial attention submodule is used to enhance the features of dirt boundaries. It can generate a spatial attention map through convolutional layers and pooling layers to emphasize important regions.

[0083] The boundary awareness enhancement module is used to highlight areas with significant edge changes. This module employs a specific edge detection layer that can use methods such as the Sobel operator to emphasize edge variations and enhance the identification of dirt boundaries.

[0084] The device inputs the fused image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and target dirt size data.

[0085] In one embodiment, S104 may specifically include: fusing the image corresponding to the target dirt. Input to the improved deep neural network In the process, the segmentation mask data is obtained. and the size data of the target dirt ;

[0086] in,

[0087]

[0088] The fused image corresponding to the target dirt , This indicates the height of the fused image corresponding to the target dirt. This represents the width of the fused image corresponding to the target dirt. This indicates the number of channels in the fused image corresponding to the target dirt.

[0089] In one implementation, a channel attention mechanism is used to capture the relative importance of different feature channels in the recognition task, distinguishing the importance of the original image, GAF image, and MTF image for dirt recognition, thereby highlighting key channels and suppressing redundant features. The execution steps of the channel attention submodule in the improved deep neural network include:

[0090] Obtain the feature image corresponding to the target dirt. ;

[0091] Feature image corresponding to the target dirt Perform global average pooling and global max pooling to obtain the global average pooling result. and global max pooling results ;

[0092] Input the global average pooling result and the global max pooling result To the fully connected layer and The activation function yields the attention weights for the first channel. Second channel attention weight ;

[0093] Based on the first channel attention weight Second channel attention weight as well as Activation function to obtain target channel attention weights ;

[0094] Based on the target channel attention weight Feature image corresponding to the target dirt The output of the channel attention submodule is obtained. ;

[0095] in,

[0096]

[0097]

[0098]

[0099]

[0100]

[0101]

[0102] Here, through

[0103]

[0104]

[0105] Perform global average pooling and global max pooling.

[0106] Through the fully connected layer

[0107]

[0108]

[0109] The activation function yields the channel attention weights.

[0110] The formula for calculating channel attention weights is:

[0111]

[0112] The formula for calculating the channel attention weight is as follows:

[0113]

[0114] The feature image corresponding to the target dirt , This represents the width of the feature image corresponding to the target dirt. This indicates the height of the feature image corresponding to the target dirt. This represents the number of channels in the feature image corresponding to the target dirt. It can be 3. This indicates the first fully connected layer. This indicates the second fully connected layer. express Activation function This indicates multiplication by channel.

[0115] In one implementation, a spatial attention mechanism is used to enhance the response of key spatial regions in the feature map, namely, dirt boundary features. The execution steps of the spatial attention submodule in the improved deep neural network include:

[0116] Obtain the output of the channel attention submodule. ;

[0117] Output feature map of the channel attention submodule Perform global average pooling, global max pooling, and convolution operations to obtain the global average pooling result. and global max pooling results ;

[0118] Based on the global average pooling result Global max pooling results as well as Activation function to obtain spatial attention weights ;

[0119] Based on the spatial attention weight and the output of the channel attention submodule The output of the spatial attention submodule is obtained. ;

[0120] in,

[0121]

[0122]

[0123]

[0124]

[0125] Indicates global average pooling. Indicates global max pooling. express Activation function This represents a 1×1 convolutional layer. This indicates multiplication by channel.

[0126] Specifically, based on the feature map output by the channel attention module First, global average pooling is used. and global max pooling Perform channel dimension compression:

[0127]

[0128]

[0129] In the formula, It is a 1×1 convolutional layer.

[0130] The formula for calculating spatial attention weights is:

[0131]

[0132] In the formula, It is spatial attention weight.

[0133] The formula for calculating spatial attention weights is as follows:

[0134]

[0135] In the formula, This indicates element-wise multiplication.

[0136] In one embodiment, the execution steps of the boundary awareness enhancement module in the improved deep neural network include:

[0137] Obtain the output of the channel attention submodule. and the output of the spatial attention submodule ;

[0138] Based on the output of the channel attention submodule and edge detection operators Extract gradient information;

[0139] The gradient information is convolved to obtain the feature mapping result;

[0140] Based on the feature mapping results and Activation function to obtain edge response map ;

[0141] For the edge response map and the output of the spatial attention submodule The results are fused to obtain the output of the boundary perception enhancement module. ;

[0142] in,

[0143]

[0144]

[0145] This represents a 1×1 convolutional layer. express Activation function This represents the weighting coefficient.

[0146] Specifically, the improved deep neural network integrates a boundary-aware enhancement module to assist the main branch in extracting fine boundaries of dirt regions, highlighting areas with significant edge changes in the image, thereby enhancing the model's ability to perceive dirt edge contours. This branch first processes the intermediate feature map... The Sobel operator is applied to extract gradient information, which is then compressed and mapped using a 1×1 convolution. Finally, the edge response map is output after passing through the Sigmoid activation function. The resulting edge map E is fused with the spatial attention mechanism to serve as guiding information to strengthen the activation responses near the edge regions in the main branch feature map. The fusion method is weighted multiplication.

[0147]

[0148] These are the weighting coefficients. This is the merged attention map.

[0149] S105: Adjust the pose of the dirt cleaning device according to the segmentation mask data and the preset coordinate transformation algorithm.

[0150] Based on the segmentation mask data, the target position is converted into the coordinate system of the cleaning equipment through geometric transformations (such as affine transformations).

[0151] The position of the cleaning equipment is adjusted by the controller to ensure it is accurately aligned with the dirt.

[0152] S106: Adjust the dirt cleaning parameters according to the size data of the target dirt, and control the dirt cleaning equipment to perform the cleaning operation.

[0153] Based on the size data of the target dirt, set parameters such as pressure, flow rate, and cleaning time for the cleaning equipment to ensure cleaning effectiveness.

[0154] The cleaning equipment is controlled to start up by dynamically adjusting the cleaning path and pressure to ensure efficient cleaning and reduce energy consumption.

[0155] In one implementation, such as Figure 3 As shown, S107~S108 can be included after S106, and the details of S107~S108 are as follows:

[0156] S107: After the cleaning operation is completed, the target dirt is re-identified to obtain the residual area of ​​the target dirt.

[0157] After completing the initial cleaning operation, the cleaning equipment should perform a preliminary assessment of the target dirt.

[0158] Cleaning equipment can use sensors such as cameras, laser scanners, or other imaging technologies to collect surface images and data.

[0159] Image processing algorithms (such as machine learning and deep learning models) are used to analyze the cleaned surface and identify areas of residual dirt.

[0160] Combine image segmentation techniques (such as U-Net, FCN, etc.) to accurately identify the boundaries of dirt.

[0161] The area of ​​the identified dirt-covered areas is calculated to obtain the "target dirt residue area".

[0162] The pixel count can be converted to the actual area (e.g., square centimeters), and then corrected using a scale.

[0163] S108: If the target dirt residue area is greater than the preset threshold, readjust the position of the dirt cleaning device and the dirt cleaning parameters, control the dirt cleaning device to perform the cleaning operation again, and repeat the above operation until the target dirt residue area is not greater than the preset threshold.

[0164] The calculated target dirt residue area is compared with a preset cleaning threshold. This threshold can be set according to cleaning requirements, dirt type, and cleaning environment.

[0165] If the residual area exceeds a preset threshold, the cleaning equipment needs to adjust its position and angle according to the distribution of dirt to ensure more comprehensive coverage of the cleaning area. For example, the cleaning equipment can be repositioned using a robotic arm or mobile chassis to ensure that the cleaning head can reach the uncleaned dirt areas.

[0166] Adjust cleaning parameters such as water pressure, cleaning agent concentration, and cleaning speed according to the nature of the dirt (such as adhesion strength and type).

[0167] Repeat steps S107 and S108 until the target dirt residue area is less than or equal to the preset threshold. A maximum number of cycles can be set to avoid wasting resources on ineffective cleaning.

[0168] In this embodiment, intelligent identification and feedback adjustment reduce the time and resource waste associated with repeated cleaning. Automated dirt identification and parameter adjustment reduce labor costs and improve work efficiency. It can accurately identify and clean different types of dirt, ensuring that surface cleanliness meets expected standards. Through the accumulation of historical data, the cleaning system can continuously optimize cleaning parameters, improving overall cleaning results. This method can be flexibly adjusted according to different environments and dirt characteristics, making it widely adaptable and suitable for various cleaning scenarios.

[0169] In this embodiment, the original image and acoustic reflection signal corresponding to the target dirt are acquired; the acoustic reflection signal corresponding to the target dirt is converted into an image representation; based on the original image, image representation, and a preset multi-channel image fusion algorithm, a fused image corresponding to the target dirt is obtained; the fused image corresponding to the target dirt is input into an improved deep neural network to obtain segmentation mask data and target dirt size data; the pose of the dirt cleaning device is adjusted according to the segmentation mask data and a preset coordinate transformation algorithm; the dirt cleaning parameters are adjusted according to the target dirt size data, and the dirt cleaning device is controlled to perform the cleaning operation. By fusing images and using an improved deep learning network, the accuracy of dirt edge recognition is significantly improved, especially in complex environments and low contrast conditions. The cleaning device adjusts its pose and cleaning parameters according to the actual distribution and characteristics of the dirt, achieving precise cleaning and reducing energy consumption and resource waste. The introduction of deep learning and acoustic signal processing promotes the intelligentization of the cleaning process, adapting to different cleaning needs and environmental changes. This method is applicable to surface cleaning of different materials and shapes and has broad application prospects. With the above implementation plan, the accuracy and efficiency of dirt identification and cleaning will be significantly improved, better meeting various cleaning needs and operating environments.

[0170] Furthermore, by fusing the original images acquired by the CCD camera with the acoustic reflection signals obtained by the ultrasonic sensor, the one-dimensional acoustic signals are converted into image information using GAF and MTF methods. A multi-channel fusion image input network is constructed to effectively compensate for the deficiencies of single image information and enhance the representation ability of dirt areas. A deep segmentation network combining channel attention, spatial attention, and enhanced boundary perception mechanisms is designed, which can highlight the contour boundary features of dirt, significantly improving the clarity and accuracy of segmentation edges, and is suitable for low-contrast or irregular dirt recognition scenarios. Based on the dirt location and size information output from the segmentation results, the system can dynamically adjust the nozzle pose and jet parameters, achieving precise control over the cleaning path, pressure, and time, avoiding the energy waste and blind spots of traditional fixed cleaning methods.

[0171] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0172] Please see Figure 4 , Figure 4 This is a schematic diagram of the dirt identification and cleaning device provided in the second embodiment of this application. The included units are used for performing... Figures 1-3 The steps in the corresponding embodiments. Please refer to the details. Figures 1-3 The relevant descriptions in the corresponding embodiments are shown below. For ease of explanation, only the parts relevant to this embodiment are shown. See also... Figure 4 The dirt identification and cleaning device 4 includes:

[0173] The acquisition unit 410 is used to acquire the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt;

[0174] The first processing unit 420 is used to convert the acoustic reflection signal corresponding to the target dirt into an image representation;

[0175] The second processing unit 430 is used to obtain a fused image corresponding to the target dirt based on the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm;

[0176] The third processing unit 440 is used to input the fused image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and the size data of the target dirt; wherein, the improved deep neural network integrates a dual attention fusion module and a boundary awareness enhancement module. The dual attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt recognition. The spatial attention submodule is used to enhance the boundary features of the dirt. The boundary awareness enhancement module is used to highlight areas with significant edge changes.

[0177] The fourth processing unit 450 is used to adjust the pose of the dirt cleaning device according to the segmentation mask data and the preset coordinate transformation algorithm.

[0178] The fifth processing unit 460 is used to adjust the dirt cleaning parameters according to the size data of the target dirt, and control the dirt cleaning equipment to perform cleaning operations.

[0179] Furthermore, the dirt identification and cleaning device 4 also includes:

[0180] The sixth processing unit is used to re-identify the target dirt after the cleaning operation is completed, and to obtain the residual area of ​​the target dirt;

[0181] The seventh processing unit is used to readjust the position and cleaning parameters of the dirt cleaning device if the target dirt residue area is greater than a preset threshold, and control the dirt cleaning device to perform the cleaning operation again, repeating the above operation until the target dirt residue area is not greater than the preset threshold.

[0182] Further, the first processing unit is specifically used for:

[0183] The acoustic reflection signals are subjected to GAF conversion and MTF conversion respectively to obtain a first image representation and a second image representation; wherein, the acoustic reflection signal is the acoustic reflection signal corresponding to the target dirt collected by the ultrasonic sensor, the GAF conversion refers to Gram angle field conversion, and the MTF conversion refers to Markov transfer field conversion.

[0184] Furthermore, the second processing unit is specifically used for:

[0185] An image enhancement operation is performed on the original image corresponding to the target dirt to obtain an enhanced original image; wherein, the image enhancement operation includes bilateral filtering denoising and / or finite contrast adaptive histogram equalization;

[0186] By stitching together the enhanced original image, the first image representation, and the second image representation, a fused image corresponding to the target dirt is obtained.

[0187] Furthermore, the third processing unit is specifically used for:

[0188] The fused image corresponding to the target dirt Input to the improved deep neural network In the process, the segmentation mask data is obtained. and the size data of the target dirt ;

[0189] in,

[0190]

[0191] The fused image corresponding to the target dirt , This indicates the height of the fused image corresponding to the target dirt. This represents the width of the fused image corresponding to the target dirt. This indicates the number of channels in the fused image corresponding to the target dirt.

[0192] Furthermore, the third processing unit is specifically used for:

[0193] Obtain the feature image corresponding to the target dirt. ;

[0194] Feature image corresponding to the target dirt Perform global average pooling and global max pooling to obtain the global average pooling result. and global max pooling results ;

[0195] Input the global average pooling result and the global max pooling result To the fully connected layer and The activation function yields the attention weights for the first channel. Second channel attention weight ;

[0196] Based on the first channel attention weight Second channel attention weight as well as Activation function to obtain target channel attention weights ;

[0197] Based on the target channel attention weight Feature image corresponding to the target dirt The output of the channel attention submodule is obtained. ;

[0198] in,

[0199]

[0200]

[0201]

[0202]

[0203]

[0204]

[0205] The feature image corresponding to the target dirt , This represents the width of the feature image corresponding to the target dirt. This indicates the height of the feature image corresponding to the target dirt. This represents the number of channels in the feature image corresponding to the target dirt. This indicates the first fully connected layer. This indicates the second fully connected layer. express Activation function This indicates multiplication by channel.

[0206] Furthermore, the third processing unit is specifically used for:

[0207] Obtain the output of the channel attention submodule. ;

[0208] Output feature map of the channel attention submodule Perform global average pooling, global max pooling, and convolution operations to obtain the global average pooling result. and global max pooling results ;

[0209] Based on the global average pooling result Global max pooling results as well as Activation function to obtain spatial attention weights ;

[0210] Based on the spatial attention weight and the output of the channel attention submodule The output of the spatial attention submodule is obtained. ;

[0211] in,

[0212]

[0213]

[0214]

[0215]

[0216] Indicates global average pooling. Indicates global max pooling. express Activation function This represents a 1×1 convolutional layer. This indicates multiplication by channel.

[0217] Furthermore, the third processing unit is specifically used for:

[0218] Obtain the output of the channel attention submodule. and the output of the spatial attention submodule ;

[0219] Based on the output of the channel attention submodule and edge detection operators Extract gradient information;

[0220] The gradient information is convolved to obtain the feature mapping result;

[0221] Based on the feature mapping results and Activation function to obtain edge response map ;

[0222] For the edge response map and the output of the spatial attention submodule The results are fused to obtain the output of the boundary perception enhancement module. ;

[0223] in,

[0224]

[0225]

[0226] This represents a 1×1 convolutional layer. express Activation function This represents the weighting coefficient.

[0227] Figure 5This is a schematic diagram of the dirt identification and cleaning device provided in the third embodiment of this application. Figure 5 As shown, the dirt identification and cleaning device 5 of this embodiment includes: a processor 50, a memory 51, and a computer program 52 stored in the memory 51 and executable on the processor 50, such as a dirt identification and cleaning program. When the processor 50 executes the computer program 52, it implements the steps in the various dirt identification and cleaning method embodiments described above, for example... Figure 1 Steps 101 to 106 are shown. Alternatively, when the processor 50 executes the computer program 52, it implements the functions of each module / unit in the above-described device embodiments, for example... Figure 4 The functions of units 410 to 460 are shown.

[0228] For example, the computer program 52 can be divided into one or more modules / units, which are stored in the memory 51 and executed by the processor 50 to complete this application. The one or more modules / units can be a series of computer program instruction segments capable of performing specific functions, which describe the execution process of the computer program 52 in the dirt identification and cleaning device 5. For example, the computer program 52 can be divided into an acquisition unit, a first processing unit, a second processing unit, a third processing unit, a fourth processing unit, and a fifth processing unit, with the specific functions of each unit as follows:

[0229] The acquisition unit is used to acquire the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt;

[0230] The first processing unit is used to convert the acoustic reflection signal corresponding to the target dirt into an image representation;

[0231] The second processing unit is used to obtain a fused image corresponding to the target dirt based on the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm;

[0232] The third processing unit is used to input the fused image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and the size data of the target dirt; wherein, the improved deep neural network integrates a dual attention fusion module and a boundary awareness enhancement module. The dual attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt recognition, the spatial attention submodule is used to enhance the boundary features of dirt, and the boundary awareness enhancement module is used to highlight areas with significant edge changes.

[0233] The fourth processing unit is used to adjust the pose of the dirt cleaning device according to the segmentation mask data and the preset coordinate transformation algorithm.

[0234] The fifth processing unit is used to adjust the dirt cleaning parameters according to the size data of the target dirt, and to control the dirt cleaning equipment to perform the cleaning operation.

[0235] The dirt identification and cleaning device may include, but is not limited to, a processor 50 and a memory 51. Those skilled in the art will understand that... Figure 5 This is merely an example of the dirt identification and cleaning device 5 and does not constitute a limitation on the dirt identification and cleaning device 5. It may include more or fewer components than shown, or combine certain components, or different components. For example, the dirt identification and cleaning device may also include input / output devices, network access devices, buses, etc.

[0236] The processor 50 may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0237] The memory 51 can be an internal storage unit of the dirt identification and cleaning device 5, such as a hard drive or memory. The memory 51 can also be an external storage device, such as a plug-in hard drive, smart media card (SMC), secure digital card (SD), flash card, etc., equipped on the dirt identification and cleaning device 5. Furthermore, the dirt identification and cleaning device 5 can include both internal and external storage units. The memory 51 is used to store the computer program and other programs and data required by the dirt identification and cleaning device. The memory 51 can also be used to temporarily store data that has been output or will be output.

[0238] It should be noted that the information interaction and execution process between the above-mentioned devices / units are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, and they will not be repeated here.

[0239] This application also provides a network device, which includes: at least one processor, a memory, and a computer program stored in the memory and executable on the at least one processor, wherein the processor executes the computer program to implement the steps in any of the above method embodiments.

[0240] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps described in the various method embodiments above.

[0241] This application provides a computer program product that, when run on a mobile terminal, enables the mobile terminal to implement the steps described in the above-described method embodiments.

[0242] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying the computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks.

[0243] In the above embodiments, the descriptions of each embodiment have different focuses. For parts that are not described in detail or recorded in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0244] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0245] In the embodiments provided in this application, it should be understood that the disclosed apparatus / network devices and methods can be implemented in other ways. For example, the apparatus / network device embodiments described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0246] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0247] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application, and should all be included within the protection scope of this application.

Claims

1. A method for identifying and cleaning dirt, characterized in that, Including the following steps: Acquire the original image corresponding to the target dirt and the acoustic reflection signal corresponding to the target dirt; The acoustic reflection signal corresponding to the target dirt is converted into an image representation; The step of converting the acoustic reflection signal corresponding to the target dirt into an image representation includes the following steps: The acoustic reflection signals are subjected to GAF conversion and MTF conversion respectively to obtain a first image representation and a second image representation; wherein, the acoustic reflection signal is the acoustic reflection signal corresponding to the target dirt collected by the ultrasonic sensor, the GAF conversion refers to Gram angle field conversion, and the MTF conversion refers to Markov transfer field conversion; Based on the original image corresponding to the target dirt, the image representation, and the preset multi-channel image fusion algorithm, a fused image corresponding to the target dirt is obtained; The step of obtaining the fused image corresponding to the target dirt based on the original image corresponding to the target dirt, the image representation, and a preset multi-channel image fusion algorithm includes the following steps: An image enhancement operation is performed on the original image corresponding to the target dirt to obtain an enhanced original image; wherein, the image enhancement operation includes bilateral filtering denoising and / or finite contrast adaptive histogram equalization; By stitching together the enhanced original image, the first image representation, and the second image representation, a fused image corresponding to the target dirt is obtained; The fused image corresponding to the target dirt is input into the improved deep neural network to obtain segmentation mask data and the size data of the target dirt. The improved deep neural network integrates a dual attention fusion module and a boundary awareness enhancement module. The dual attention fusion module includes a channel attention submodule and a spatial attention submodule. The channel attention submodule is used to capture the relative importance of different feature channels for dirt recognition, the spatial attention submodule is used to enhance the boundary features of the dirt, and the boundary awareness enhancement module is used to highlight areas with significant edge changes. The execution steps of the boundary awareness enhancement module in the improved deep neural network include: Obtain the output of the channel attention submodule. and the output of the spatial attention submodule ; Based on the output of the channel attention submodule and edge detection operators Extract gradient information; The gradient information is convolved to obtain the feature mapping result; Based on the feature mapping results and Activation function to obtain edge response map ; For the edge response map and the output of the spatial attention submodule The results are fused to obtain the output of the boundary perception enhancement module. ; in, ； ； This represents a 1×1 convolutional layer. express Activation function Indicates the weighting coefficient; The pose of the dirt cleaning device is adjusted according to the segmentation mask data and the preset coordinate transformation algorithm. Adjust the dirt cleaning parameters according to the size data of the target dirt, and control the dirt cleaning equipment to perform the cleaning operation.

2. The dirt identification and cleaning method according to claim 1, characterized in that, The method further includes the following steps: After the cleaning operation is completed, the target dirt is re-identified to obtain the residual area of ​​the target dirt; If the target dirt residue area is greater than a preset threshold, the position and dirt cleaning parameters of the dirt cleaning device are readjusted, and the dirt cleaning device is controlled to perform the cleaning operation again. The above operation is repeated until the target dirt residue area is no greater than the preset threshold.

3. The dirt identification and cleaning method according to claim 1 or 2, characterized in that, The step of inputting the fused image corresponding to the target dirt into the improved deep neural network to obtain segmentation mask data and the size data of the target dirt includes the following steps: The fused image corresponding to the target dirt Input to the improved deep neural network In the process, the segmentation mask data is obtained. and the size data of the target dirt ; in, ； The fused image corresponding to the target dirt , This indicates the height of the fused image corresponding to the target dirt. This represents the width of the fused image corresponding to the target dirt. This indicates the number of channels in the fused image corresponding to the target dirt.

4. The dirt identification and cleaning method according to claim 3, characterized in that, The execution steps of the channel attention submodule in the improved deep neural network include: Obtain the feature image corresponding to the target dirt. ; Feature image corresponding to the target dirt Perform global average pooling and global max pooling to obtain the global average pooling result. and global max pooling results ; Input the global average pooling result and the global max pooling result To the fully connected layer and The activation function yields the attention weights for the first channel. Second channel attention weight ; Based on the first channel attention weight Second channel attention weight as well as Activation function to obtain target channel attention weights ; Based on the target channel attention weight Feature image corresponding to the target dirt The output of the channel attention submodule is obtained. ; in, ； ； ； ； ； ； The feature image corresponding to the target dirt , This indicates the height of the feature image corresponding to the target dirt. This represents the width of the feature image corresponding to the target dirt. This represents the number of channels in the fused image corresponding to the target contaminant. This means taking the data from all channels at the h-th row and w-th column position in the feature image F corresponding to the target dirt. This indicates that global average pooling is performed on the feature image corresponding to the target dirt. This indicates that global max pooling is performed on the feature image corresponding to the target dirt. This indicates the first fully connected layer. This indicates the second fully connected layer. express The activation function represents channel-wise multiplication.

5. The dirt identification and cleaning method according to claim 3, characterized in that, The execution steps of the spatial attention submodule in the improved deep neural network include: Obtain the output of the channel attention submodule. ; Output feature map of the channel attention submodule Perform global average pooling, global max pooling, and convolution operations to obtain the global average pooling result. and global max pooling results ; Based on the global average pooling result Global max pooling results as well as Activation function to obtain spatial attention weights ; Based on the spatial attention weight and the output of the channel attention submodule The output of the spatial attention submodule is obtained. ; in, ； ； ； ； Indicates global average pooling. Indicates global max pooling. express Activation function This represents a 1×1 convolutional layer. This indicates multiplication by channel.

6. A dirt identification and cleaning device, comprising: A processor, a memory, and a computer program stored in the memory and executable on the processor, characterized in that the processor, when executing the computer program, implements the steps of the method as claimed in any one of claims 1 to 5.

7. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 5.

Images