A window cleaning machine multi-modal sensor data fusion processing method and system

Abstract

The application relates to the technical field of data processing, and provides a window cleaning machine multi-modal sensor data fusion processing method and system, which comprises the following steps: performing feature extraction on visual information of a glass surface, comparing the extracted features with preset features, determining a to-be-investigated area according to a comparison result, in response to the determination of the to-be-investigated area, suspending a current operation, and starting a fine investigation process, moving a cleaning mechanism to the to-be-investigated area, making the cleaning mechanism contact the to-be-investigated area with a controlled acting force, and acquiring spectral reflection information and contact mechanics response information of the to-be-investigated area, combining the spectral reflection information and the contact mechanics response information, inferring physical properties of the to-be-investigated area, and identifying a stain type of the to-be-investigated area. The application can more accurately infer the physical properties and the type of the stain, and provides a more reliable basis for subsequent cleaning operations.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and more specifically, to a method and system for fusion processing of multimodal sensor data from a window cleaning machine. Background Technology

[0002] Automated window cleaning machines play an indispensable role in the maintenance of modern high-rise buildings. These devices typically rely on multiple sensors to perceive the surrounding environment and ensure smooth cleaning operations. However, in actual operation, window cleaning machines often face various challenges. For example, under complex lighting conditions, reflections on the glass surface or rain and fog can cause a single sensor to fail to accurately acquire environmental information, leading to incorrect environmental assessments. Furthermore, with long-term operation, sensor connection cables may suffer subtle, imperceptible damage, affecting the stability and time synchronization of data transmission. Even worse, when new industrial pollutants appear in the surrounding environment, these pollutants can form stubborn stains with complex optical properties and strong adhesion on the glass surface, making it difficult for the existing sensing and cleaning strategies of the window cleaning machine to cope, ultimately affecting its ability to accurately assess and optimize its own operational quality.

[0003] During long-term operation of window cleaning machines, localized fatigue in the data cables of key vision sensors leads to intermittent weak signal distortion and transmission delay jitter. These anomalies fail to be effectively identified by existing failure mechanisms, resulting in persistent minor deviations in the timestamp alignment of multimodal sensor data. When the working environment is suddenly exposed to new industrial pollutants that form a stubborn stain layer with complex optical properties on the glass surface, the data fusion weight allocation method, affected by the aforementioned sensor data defects, systematically misjudges key parameters such as stain type and adhesion without recognizing the failure of the vision data. This misjudgment causes the cleaning organization to fail to adapt to the characteristics of the stain during operation, potentially causing microscopic scratches or forming new stubborn residues on the glass surface that are difficult to detect with the naked eye. Summary of the Invention

[0004] This application provides a method and system for multimodal sensor data fusion processing of a window cleaning machine to solve at least one of the above-mentioned technical problems.

[0005] To achieve the above objectives, this application adopts the following technical solution: In a first aspect, this application discloses a method for multimodal sensor data fusion processing of a window cleaning machine, including: Obtain visual information about the glass surface; Visual information is used to extract features, and the extracted features are compared with preset features. The area to be explored is determined based on the comparison results. In response to the identification of the area to be explored, the current operation is suspended and a detailed exploration process is initiated, which includes: The cleaning unit is moved to the area to be explored, and the cleaning unit contacts the area with a controlled force, and the spectral reflectance information and contact mechanical response information of the area to be explored are obtained; By combining spectral reflectance information and contact mechanical response information, the physical properties of the area to be explored can be inferred, and the type of stain in the area to be explored can be identified.

[0006] Secondly, this application also discloses a multimodal sensor data fusion processing system for a window cleaning machine, comprising: The visual information acquisition module is used to acquire visual information about the glass surface; The feature extraction and comparison module is used to extract features from visual information, compare the extracted features with preset features, and determine the area to be explored based on the comparison results. The operation control module is used to pause the current operation and start the fine exploration process in response to the determination of the area to be explored; The cleaning mechanism control module is used to respond to the fine detection process to move the cleaning mechanism to the area to be detected, so that the cleaning mechanism contacts the area to be detected with a controlled force, and acquires the spectral reflectance information and contact mechanical response information of the area to be detected; The information processing and identification module is used to combine spectral reflectance information and contact mechanical response information to infer the physical properties of the area to be explored and to identify the type of stain in the area to be explored.

[0007] Compared with the prior art, this application has at least the following beneficial effects: This application enables accurate identification and physical property inference of stains on glass surfaces, providing window cleaning machines with more reliable operational basis, thereby enhancing their intelligent diagnosis, evaluation and adaptive optimization capabilities for their own operational quality, eliminating the need for manual re-inspection, and significantly improving cleaning efficiency and quality. Attached Figure Description

[0008] Figure 1 A flowchart illustrating a multimodal sensor data fusion processing method for a window cleaning machine provided in this application; Figure 2 This is a schematic diagram of the structure of a multimodal sensor data fusion processing system for a window cleaning machine provided in this application. Detailed Implementation

[0009] The technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments.

[0010] like Figure 1 As shown in the embodiment of this application, a method for multimodal sensor data fusion processing of a window cleaning machine is proposed, including: Obtain visual information about the glass surface; Visual information is used to extract features, and the extracted features are compared with preset features. The area to be explored is determined based on the comparison results. In response to the identification of the area to be explored, the current operation is suspended and a detailed exploration process is initiated, which includes: The cleaning unit is moved to the area to be explored, and the cleaning unit contacts the area with a controlled force, and the spectral reflectance information and contact mechanical response information of the area to be explored are obtained; By combining spectral reflectance information and contact mechanical response information, the physical properties of the area to be explored can be inferred, and the type of stain in the area to be explored can be identified.

[0011] Among them, visual information of glass surface refers to image or video data about glass surface obtained through visual sensors (such as cameras), which includes macroscopic visual features such as the shape, color, and distribution of stains.

[0012] Feature extraction refers to extracting representative values ​​or patterns from raw visual information to distinguish different stains or areas, such as texture features, color histograms, and edge information.

[0013] Preset features refer to pre-stored feature templates corresponding to known stain types or cleaned areas, which are used to compare with the extracted features.

[0014] The area to be investigated refers to the glass surface area that is suspected of having stains or requires further detailed inspection based on preliminary visual information analysis.

[0015] The cleaning mechanism refers to the components on a window cleaning machine used to perform cleaning operations. It typically includes brushes, scrapers, nozzles, etc., and integrates force sensors and spectral sensors.

[0016] Controlled force refers to the precise control of the magnitude and direction of the force applied by the cleaning mechanism when it comes into contact with the glass surface, in order to avoid damage to the glass surface and ensure the accuracy of sensor data acquisition.

[0017] Spectral reflectance information refers to the light intensity data at different wavelengths collected by a spectral sensor after light shines on the area to be investigated and is reflected. This data reflects the optical properties of the stain.

[0018] Contact mechanical response information refers to the data such as force, pressure, and friction collected by the cleaning agency through force sensors when it comes into contact with the area to be inspected. These data reflect the physical characteristics of the stain, such as adhesion and hardness.

[0019] Feature analysis refers to the processing and analysis of spectral reflectance information and contact mechanical response information to extract key features related to stain type and physical properties.

[0020] Physical properties refer to the physical characteristics of stains, such as their adhesion, hardness, stickiness, and particle size.

[0021] Stain type refers to the specific type of stain, such as dust, oil stains, limescale, bird droppings, etc.

[0022] The following is a detailed description of the multimodal sensor data fusion processing method for window cleaning machines proposed in this application: First, window cleaning machines can be equipped with high-resolution visible light cameras to continuously capture images of the glass surface during operation. These images can be transmitted to the control unit of the window cleaning machine for preliminary analysis. Window cleaning machines can also integrate infrared cameras to acquire thermal images of the glass surface at night or in low-light conditions to help identify areas of abnormal temperature, which may correspond to the presence of stains.

[0023] Secondly, visual information is input into the image processing module. This module can use edge detection algorithms (such as the Canny operator) to identify boundaries in the image, or use color segmentation techniques to distinguish regions of different colors. Extracted features, such as the area of ​​a specific color region and the roughness of the texture, are compared with a pre-defined stain feature library. This library stores typical visual features of various known stains. When the comparison results show that the features of a certain region are highly similar to a stain type in the stain feature library, that region is marked as a region to be explored. For example, if a large dark patch appears in the image, and its texture features are similar to pre-defined oil stain features, then that patch region will be identified as a region to be explored.

[0024] Secondly, if the window cleaning robot is performing a large-area cleaning operation, the cleaning action will stop immediately after identifying the area to be inspected. The system will then initiate a specialized fine-tuning process. This process aims to perform a deeper and more detailed analysis of the marked area to obtain more accurate stain information.

[0025] Next, a servo motor precisely controls the downward pressure of the cleaning mechanism to ensure it makes full contact with the stain without damaging the glass. Simultaneously, a miniature spectral sensor integrated into the cleaning mechanism emits a beam of light of a specific wavelength and receives the spectral information reflected from the stain surface. At the same time, a force sensor on the cleaning mechanism records the pressure, friction, and other mechanical response information generated during the contact process in real time.

[0026] Finally, the shape of the spectral curve, reflectance at specific wavelengths, and the position and intensity of absorption peaks are analyzed to identify the chemical composition or optical properties of the stain. Simultaneously, the instantaneous changes, average values, and peak values ​​of the force signal are analyzed to assess the stain's adhesion, hardness, or stickiness. Subsequently, this characteristically analyzed spectral and mechanical information is fused. For example, if spectral analysis indicates the stain contains organic components, while mechanical analysis shows strong adhesion and a certain degree of stickiness, the system can comprehensively determine that the stain is oil. Through this combination of multimodal information, the system can more accurately infer the physical properties of the stain and identify the specific stain type.

[0027] This application acquires visual information about the glass surface and uses feature extraction and comparison with preset features to quickly filter out potential stain areas, i.e., areas to be investigated. This preliminary identification process can efficiently locate key areas of interest, avoiding indiscriminate fine-tuning of the entire glass surface, thereby improving operational efficiency.

[0028] Once the area to be explored is identified, the system immediately pauses the current routine cleaning operation and initiates a more refined and in-depth exploration process. In this refined exploration process, the cleaning mechanism is precisely moved to the area to be explored. At this point, the cleaning mechanism is no longer merely a tool for performing cleaning tasks; it is endowed with more important sensing capabilities. By contacting the area to be explored with controlled force, the cleaning mechanism can simultaneously acquire spectral reflectance information and contact mechanical response information of that area. Spectral reflectance information reveals the optical properties and chemical composition of the stain, while contact mechanical response information reflects the physical properties of the stain, such as adhesion and hardness.

[0029] Subsequently, this multimodal sensor data is received and subjected to in-depth feature analysis. Spectral reflectance information is analyzed to extract optical features such as reflectance ratios and absorption peak positions at specific wavelengths; contact mechanical response information is analyzed to calculate mechanical features such as the mean, variance, and high-frequency components of the force signal during the scratching process. By comprehensively analyzing and fusing these two distinct yet complementary pieces of information, the system can overcome the limitations of single sensors in identifying stains in complex environments. For example, some stains may not be visually apparent, but their unique mechanical responses or spectral characteristics can be accurately captured.

[0030] Ultimately, by combining spectral reflectance and contact mechanical response information, the system can infer the physical properties of the area to be inspected and identify the specific type of stain. This process can accurately distinguish not only common stains such as dust, oil, and limescale, but also complex and stubborn stains formed by emerging industrial pollutants. In this way, this application can provide a precise basis for subsequent cleaning strategy adjustments, ensuring that the window cleaning machine can adopt the most suitable cleaning method according to the characteristics of different stains, thereby avoiding damage to the glass surface and significantly improving the cleaning effect.

[0031] In some embodiments, the steps of combining spectral reflectance information and contact mechanical response information to infer the physical properties of the area to be explored and to identify the type of stain in the area to be explored include: Feature analysis is performed on the spectral reflectance information to extract the reflectance ratio and absorption peak position at a specific wavelength, forming the first set of stain feature information; Feature analysis is performed on the contact mechanical response information to calculate the mean, variance, and high-frequency components of the force signal during the scratching process, forming a second set of stain feature information; When the first set of stain feature information is similar to the spectral features of a preset first type of known stain, and the second set of stain feature information is similar to the mechanical response features of a preset second type of known stain, the judgment logic based on multidimensional feature space mapping is activated. The judgment logic based on multidimensional feature space mapping includes: Map the first set of stain feature information and the second set of stain feature information to their respective feature spaces, and calculate the first feature distance between the first set of stain feature information and the first type of known stains, and the second feature distance between the second set of stain feature information and the second type of known stains. A pre-defined stain association rule set is introduced, which defines the potential association or mutual exclusion relationships between different stain types in terms of physical and optical properties; Based on the first feature distance, the second feature distance, and the preset stain association rule set, the combination probability of the first type of known stain and the second type of known stain is evaluated, and the stain type of the area to be explored is identified according to the evaluation results, thereby inferring the physical properties of the area to be explored.

[0032] Among these methods, the reflectance spectral data acquired through spectral sensors can identify characteristic absorption peaks and reflectance changes of specific chemical bonds or molecular structures. For example, organic matter typically has specific absorption peaks in the infrared region, while metal oxides may exhibit unique reflectance curves in the visible or ultraviolet regions. By calculating the reflectance ratios at these specific wavelengths, the relative content of different substances can be quantified, while the positions of the absorption peaks directly indicate the chemical composition of the substances. These extracted spectral features collectively constitute the first set of stain characteristic information, aiming to accurately characterize the chemical composition of stains from an optical perspective.

[0033] By analyzing the mechanical feedback data generated when a cleaning device comes into contact with stains on a glass surface, the physical properties of the stains can be determined. The mean of the force signal can reflect the overall hardness or adhesion of the stain; the variance can characterize the uniformity, roughness, or stickiness of the stain surface; while high-frequency components may reveal the microstructure, particle size, or brittleness of the stain. These mechanical characteristics collectively constitute the second set of stain characteristic information, the purpose of which is to quantify the mechanical properties of the stain at a physical level.

[0034] Mapping the first and second sets of stain feature information to their respective feature spaces aims to standardize and abstract feature data from different sources to facilitate quantitative comparison. Calculating the first feature distance between the first set of stain feature information and the first type of known stains, and the second feature distance between the second set of stain feature information and the second type of known stains, involves using mathematical methods (such as Euclidean distance, Mahalanobis distance, etc.) to quantify the similarity between the stain to be investigated and known stains in their respective feature spaces; a smaller distance indicates a higher similarity.

[0035] A pre-defined set of stain association rules is introduced, defining the potential associations or mutual exclusions between different stain types in terms of physical and optical properties. This can be understood as a logical rule base based on expert knowledge or historical data. For example, rules might define oily stains as typically having specific infrared absorption peaks and exhibiting low friction and stickiness upon contact, or mineral deposits as typically having high hardness and exhibiting high reflectivity under specific spectra. The purpose of introducing these rules is to provide a higher level of semantic understanding and logical reasoning capabilities based on feature distance calculation, in order to resolve ambiguities that may arise from independent analysis of single or multiple features.

[0036] This application employs refined feature extraction from spectral reflectance and contact mechanical response information to form a first and a second set of stain feature information, enabling a comprehensive characterization of stains from both chemical composition and physical structure dimensions. This multi-dimensional and refined feature extraction allows the system to more accurately quantify the similarity to known stains by calculating distances in their respective feature spaces when faced with stains possessing similar single features but different overall properties. Furthermore, a stain association rule set is introduced, allowing the system to logically evaluate the probability of different feature combinations using pre-defined physical and optical associations or mutual exclusion relationships. This judgment logic, combining quantitative feature distance and qualitative association rules, effectively solves the misjudgment problem that may occur in traditional methods when dealing with complex, mixed, or ambiguous stain features, significantly improving the accuracy and robustness of stain identification.

[0037] In some embodiments, the steps of mapping the first set of stain feature information and the second set of stain feature information to their respective feature spaces, and calculating the first feature distance between the first set of stain feature information and the first type of known stains, and the second feature distance between the second set of stain feature information and the second type of known stains, include: The spectral reflectance data in the first set of stain feature information is locally smoothed, and the force signal data in the second set of stain feature information is low-pass filtered to obtain the pre-processed first set of stain feature information and the pre-processed second set of stain feature information. Based on the ambient light intensity of the current work area and the self-diagnostic status of the sensor cables, the reliability of the current data from the vision sensor and force feedback sensor is evaluated in real time. Based on data reliability, the first set of pre-processed stain feature information and the second set of pre-processed stain feature information are mapped to their respective feature spaces, and the weights of the first set of pre-processed stain feature information and the second set of pre-processed stain feature information in the feature space are dynamically adjusted. A confidence-based distance correction factor is introduced, which is calculated based on the degree of overlap and distribution density between the stain feature information to be investigated and the known stain feature information in the feature space. When the feature distance between the stain to be investigated and two known stains is close, the feature distance is fine-tuned according to the distance correction factor to obtain the corrected feature distance. Based on the corrected feature distance, the first feature distance between the first set of stain feature information and the first type of known stain, and the second feature distance between the second set of stain feature information and the second type of known stain are calculated.

[0038] Specifically, the spectral reflectance data in the first set of stain feature information refers to the raw data reflecting the optical properties of the stain, acquired through a spectral sensor. Local smoothing processing is applied, such as using moving averages, Savitzky-Golay filtering, or wavelet denoising, to eliminate random noise and glitches in the data, highlighting the true trend and characteristic peaks and valleys of the spectral curve, thereby improving the accuracy of subsequent feature analysis. The force signal data in the second set of stain feature information refers to the dynamic response data such as scratching force and friction force recorded by the force sensor when the cleaning mechanism comes into contact with the glass surface. Low-pass filtering processing is applied, such as using a Butterworth filter or a Gaussian filter, to filter out high-frequency noise and instantaneous impacts, retaining stable mechanical response characteristics reflecting the physical properties of the stain, and avoiding noise interference with mechanical feature extraction. Real-time evaluation of the reliability of the current data from the vision sensor and force feedback sensor can be performed based on the ambient light intensity of the current work area and the self-diagnostic status of the sensor cables. For example, the reliability of visual sensor data may decrease when ambient light intensity is too low or too high; the reliability of force feedback sensor data will also be affected when self-diagnostic anomalies such as open circuits, short circuits, or signal attenuation occur in the sensor cable. By monitoring these parameters, the current working state of the sensor can be quantified, providing a basis for subsequent data processing. Based on the assessed data reliability, the weights of the first and second sets of preprocessed stain feature information in the feature space are dynamically adjusted. For example, when the reliability of visual sensor data is low, the weight of the first set of stain feature information (spectral features) can be appropriately reduced, while the weight of the second set of stain feature information (mechanical features) can be increased, and vice versa. This dynamic weight adjustment mechanism aims to ensure that the system can focus more on data sources with higher reliability under different operating conditions, thereby improving the overall recognition accuracy. In addition, a confidence-based distance correction factor is introduced, which is calculated based on the degree of overlap and distribution density of the stain feature information to be explored and the known stain feature information in the feature space. For example, if the feature points of the stain to be investigated highly overlap with and are densely distributed with the feature clusters of a known stain type in the feature space, it indicates a high similarity between the two, and the correction factor may approach 1. Conversely, if the overlap is low or the distribution is sparse, the correction factor may be small, indicating a low similarity or uncertainty. This distance correction factor aims to quantify the confidence of feature matching and provide a basis for subsequent distance fine-tuning. When the feature distances of the stain to be investigated and two known stains are close, the feature distances are fine-tuned according to the aforementioned distance correction factor. For example, if the feature distances of the stain to be investigated and known stains A and B are d_A and d_B respectively, and d_A ≈ d_B, then the distance correction factor can be used to weight or adjust d_A and d_B to more accurately reflect the actual similarity and avoid misjudgments caused by small distance differences.

[0039] This application effectively addresses issues such as noise interference, data reliability fluctuations, and difficulties in distinguishing similar stains in feature mapping and distance calculation by introducing mechanisms such as data preprocessing, real-time reliability assessment, dynamic weight adjustment, and distance correction. Specifically, local smoothing of spectral reflectance data and low-pass filtering of force signal data effectively remove noise from the original data, resulting in purer and more stable extracted stain features, thus laying the foundation for accurate feature mapping. Simultaneously, real-time assessment of sensor data reliability and dynamic adjustment of feature weights enable the system to intelligently select more reliable feature information for judgment based on actual working conditions, avoiding recognition bias caused by poor data quality from a single sensor. For example, in insufficient lighting conditions, the system reduces the weight of visual features and relies more on mechanical features for judgment, thereby ensuring robustness of recognition. Furthermore, a confidence-based distance correction factor is introduced, and fine-tuning is performed when feature distances are close. This allows the system to make more refined judgments when faced with stains with high feature similarity, by combining the degree of overlap and distribution density of features in the feature space. This effectively distinguishes stain types that are easily confused in traditional methods, significantly improving the accuracy of identification. It is precisely because of these synergistic effects that this application enables window cleaning machines to identify the types of stains on glass surfaces more accurately and reliably.

[0040] In some embodiments, the step of introducing a preset stain association rule set, which defines the potential association or mutual exclusion relationships between different stain types in terms of physical and optical properties, includes: Obtain environmental parameter information for the current work area, including ambient temperature, humidity, and air pollutant concentration; Based on environmental parameter information, the applicable weight and priority of existing rules in the stain association rule set are dynamically adjusted; When the stain association rule set cannot clearly assess the probability of combinations of known stains of type 1 and known stains of type 2, the stain feature tracing and evolution analysis process is initiated. The stain feature tracing and evolution analysis process includes: In-depth analysis of the first and second sets of stain feature information was performed to identify feature patterns that slightly deviated from known stain types. By combining environmental parameter information, we can infer the physicochemical changes or new pollutant formation pathways that may correspond to the characteristic patterns of subtle deviations. Based on the inference results, temporary association rules for new types of stains are generated and incorporated into the stain association rule set.

[0041] Specifically, the window cleaning machine acquires data such as ambient temperature, humidity, and airborne pollutant concentration in the work area in real time or near real time through integrated environmental sensors or external data interfaces. These environmental parameters are crucial for understanding the formation, adhesion, and evolution of stains. For example, high humidity may lead to the spread of certain water-soluble stains, while the concentration of specific pollutants may indicate the presence of a particular type of stain.

[0042] The preset stain association rules are optimized and updated in real time based on current environmental conditions. For example, in high-temperature and high-humidity environments, the weight of rules related to mold or algae growth may be increased; while in areas with severe industrial pollution, the priority of rules related to particulate matter or acid rain traces may be increased. This dynamic adjustment ensures that the stain recognition logic can better adapt to the actual working environment.

[0043] When multiple rules yield contradictory evaluation results, or when no rule effectively covers the current stain characteristics, the system will initiate a stain characteristic tracing and evolution analysis process. This process aims to delve deeper into the potential information of stain characteristics to resolve uncertainties in identification.

[0044] The stain characterization and evolution analysis process includes in-depth analysis of the first set of stain characteristic information (spectral reflectance information) and the second set of stain characteristic information (contact mechanical response information). This analysis aims to identify characteristic patterns that subtly deviate from known stain types, such as small shifts in spectral curves, slight broadening of absorption peaks, or unusual fluctuations in the mechanical response signal. These subtle deviations are often early signs of physicochemical changes in the stain or the formation of new contaminants.

[0045] By combining environmental parameter information, we can infer the physicochemical changes or new pollutant formation pathways that may correspond to the characteristic patterns of subtle deviations. For example, if spectral characteristics show signs of organic matter degradation and the ambient temperature is high, it may be inferred to be decomposition products caused by microorganisms. If the mechanical response exhibits abnormal viscosity and specific chemical pollutants are present in the air, it may be inferred to be the formation of new polymer stains.

[0046] Therefore, based on the inference results, temporary association rules for novel stains are generated and incorporated into the stain association rule set. This means that the system has the ability to learn and adapt, and can transform the identified features of novel stains and their association with the environment into new rules, thereby expanding its stain recognition library and improving its ability to identify unknown stains.

[0047] This application addresses the applicability issue of static rule sets in changing environments by introducing environmental parameter information and dynamically adjusting the weights and priorities of stain association rules based on this information. When environmental conditions change, the physicochemical properties of stains also change. For example, the spectral reflectance characteristics and mechanical responses of the same stain may differ under different temperatures and humidity levels. By acquiring environmental parameters in real time and dynamically adjusting rule weights, the system can make the stain identification logic more closely reflect the current actual situation, thereby improving the accuracy of the assessment.

[0048] When the stain association rule set cannot clearly assess the stain type, this application identifies subtle deviations from known stain types through in-depth analysis of the first and second sets of stain feature information. These deviations, combined with environmental parameters, are used to infer potential physicochemical changes or the formation pathways of novel pollutants, thereby avoiding identification failures or misjudgments due to an incomplete rule set. By generating temporary association rules for novel stains and incorporating them into the rule set, this application enables the system to continuously learn and adapt to new stain types, effectively addressing the limitations of traditional methods in handling unknown or evolving stains.

[0049] In some embodiments, the steps of evaluating the combination probability of a first type of known stain and a second type of known stain based on a first feature distance, a second feature distance, and a preset stain association rule set, and identifying the stain type of the area to be explored based on the evaluation results, thereby inferring the physical properties of the area to be explored, include: Based on the first feature distance and the second feature distance, the rules in the stain association rule set that are related to the combination probability of the first type of known stains and the second type of known stains are filtered; Based on the source reliability of the filtered rules, the update time of the rules, and the current reliability of the feature information on which the rules depend, a dynamic confidence weight is assigned to each filtered rule. Calculate the strength of the probability of supporting or opposing a combination for each filtered rule, and combine it with dynamic confidence weights to weight all filtered rules to obtain a confidence score for the probability of the combination. When the confidence score of the combination probability is lower than the preset judgment threshold, the key features related to the filtered rules in the first group of stain feature information and the second group of stain feature information are subjected to difference amplification processing. Based on the pre-set decision tree, the system makes judgments based on the key features after amplification of differences, identifies the type of stain in the area to be explored, and infers the physical properties of the area to be explored.

[0050] Specifically, the purpose of filtering is to focus on the rules most relevant to the stain to be identified, avoiding interference from irrelevant rules. For example, a distance threshold can be set to select only the association rules corresponding to known stain types that are within a certain distance from the features of the stain to be detected.

[0051] To improve the accuracy of the evaluation, this application assigns a dynamic confidence weight to each selected rule. This weight is determined based on the reliability of the rule's source, its update time, and the current reliability of the feature information upon which the rule relies. For example, rules published by authoritative institutions and recently updated have higher source reliability and therefore a higher weight; conversely, if the sensor data upon which the rule relies currently exhibits anomalies or noise, the reliability of that feature information is lower, and the rule's weight will be reduced.

[0052] Based on this, the strength of the probability of each filtered rule supporting or opposing a combination is calculated, and combined with dynamic confidence weights, all filtered rules are weighted to obtain a comprehensive combination probability confidence score. This score quantifies the degree to which the current stain feature information supports a specific stain combination.

[0053] When the confidence score of the combination probability is lower than the preset judgment threshold, it indicates that the current recognition result may have uncertainty or ambiguity. Difference amplification processing aims to highlight subtle feature differences that may be ignored or difficult to distinguish under low confidence conditions, thus providing a clearer basis for subsequent judgments. For example, for spectral reflectance information, the reflectance curve of a specific band can be locally amplified or subjected to second derivative analysis to highlight minute absorption peaks or reflection valleys; for contact mechanical response information, the instantaneous rate of change of the force signal can be analyzed to capture subtle resistance changes during scratching.

[0054] By combining a pre-set decision tree with key features amplified after difference processing, the system identifies the type of stain in the area to be investigated and infers the physical properties of the area. This decision tree can be pre-trained based on a large amount of stain sample data and includes multiple decision paths and branches, enabling more accurate classification based on amplified feature differences.

[0055] This application effectively addresses the problem of insufficient recognition accuracy due to information uncertainty in complex or ambiguous stain recognition scenarios by introducing dynamic confidence weights and a difference amplification processing mechanism. Specifically, firstly, stain association rules are filtered based on first and second feature distances, ensuring the relevance of subsequent evaluations and avoiding interference from irrelevant rules. Secondly, dynamic confidence weights are assigned to the filtered rules, allowing the evaluation process to fully consider the reliability, timeliness, and quality of the feature information upon which the rules depend, thus enabling the combined probability confidence score to more accurately reflect the actual situation. This dynamic weighting mechanism allows the system to adaptively adjust to rules from different sources, with different update frequencies, and with sensor data of varying quality, improving the robustness of the evaluation. When the combined probability confidence score falls below a preset threshold, difference amplification processing is initiated. This process enhances key features, making previously indistinguishable subtle differences significant, thus providing a clearer and more discriminative input for the decision tree's judgment. This mechanism enables the system to delve deeper into potential discriminative information when faced with stains that have high feature similarity but low distinguishability, avoiding misjudgments caused by feature ambiguity. Ultimately, combined with a pre-set decision tree, it can make more accurate judgments based on these key features that have undergone difference amplification, thereby achieving high-precision identification of stain types and accurate inference of physical properties.

[0056] Suppose a window cleaning robot detects an area to be inspected on the glass surface. After analyzing spectral reflectance and contact mechanical response information, it initially determines that the area may contain a combination of oil stains or limescale. At this point, the system will filter rules related to the probability of a combination of oil stains and limescale from a set of stain association rules based on a first feature distance (e.g., the distance between the spectral characteristics of the area to be inspected and the known spectral characteristics of oil stains) and a second feature distance (e.g., the distance between the mechanical response characteristics of the area to be inspected and the known mechanical response characteristics of limescale).

[0057] For example, rule A is selected: oil stains and scale often occur together in humid environments; rule B is selected: oil stains typically have high reflectivity, and scale typically has high hardness. The system assigns a dynamic confidence weight to rule A based on its source (e.g., a regional environmental report issued by a weather bureau), update time (e.g., updated within the last week), and the reliability of the current ambient humidity sensor data. Similarly, rule B is assigned another dynamic confidence weight based on its source (e.g., a materials science database), update time, and the reliability of the spectral sensor and force feedback sensor data.

[0058] Subsequently, the system calculates the strength of the probability that rule A and rule B support or oppose the combination of oil stains and scale, and performs weighted processing based on their respective dynamic confidence weights to obtain a confidence score for the combination probability. Assuming this score is 0.45, which is lower than the preset judgment threshold of 0.6, this indicates that the system has a low confidence level in its judgment of oil stains + scale.

[0059] At this point, the system will initiate difference amplification processing. For the spectral reflectance data in the first set of stain characteristic information, second-derivative analysis is performed to amplify the subtle differences in absorption peaks that oil and scale may exhibit at specific wavelengths. For example, oil may have a smooth absorption in a certain wavelength band, while scale may have a sharp absorption peak in the same band; second-derivative analysis can reveal these differences more clearly. For the force signal data in the second set of stain characteristic information, instantaneous rate of change analysis is performed to amplify the subtle fluctuations in the force signal during scraping. For example, oil may cause a stable scraping force, while scale may cause slight jumps or vibrations in the scraping force; instantaneous rate of change analysis can capture these differences.

[0060] Finally, combined with a pre-defined decision tree, which may include judgment logic such as identifying a mixed stain dominated by scale if second-derivative analysis shows a sharp absorption peak in a specific band and instantaneous rate-of-change analysis shows a slight jump in the force signal. Based on the key features after difference amplification processing, the decision tree will make a judgment and ultimately identify the type of stain in the area to be investigated, such as an oil-water mixed stain dominated by scale, and infer that its physical properties are a mixture of hard adhering substances and sticky substances.

[0061] In some embodiments, when the confidence score of the combined probability is lower than a preset judgment threshold, the step of performing difference amplification processing on the key features related to the filtered rules in the first group of stain feature information and the second group of stain feature information includes: Adjust the intensity and scope of the difference amplification process based on the low confidence level of the combination probability confidence score; Based on the potential conflict stain types indicated by the filtered rules, a preset differential amplification processing strategy is implemented. Then, based on this strategy, and the adjusted intensity and range of the differential amplification processing, differential amplification processing is applied to the spectral reflectance data in the first set of stain feature information and the force signal data in the second set of stain feature information. For the spectral reflectance data in the first set of stain feature information, second derivative analysis or high power function processing is performed to amplify the subtle differences in spectral features; For the force signal data in the second set of stain characteristic information, instantaneous rate of change analysis or frequency domain energy distribution mapping is performed to amplify the differences in contact mechanical response.

[0062] Specifically, when the confidence score for the probability of stain combinations assessed by the system is low, it indicates significant uncertainty in the current identification results. To more effectively address this uncertainty, the intensity of subsequent difference amplification processing and the feature dimensions of focus need to be dynamically adjusted based on the specific value of this low confidence score. For example, the lower the score, the greater the uncertainty, so the intensity of difference amplification processing can be increased accordingly, and the range of amplified features can be broadened to uncover deeper discriminative features.

[0063] Based on the potential conflicting stain types indicated by the filtered rules, the pre-set difference amplification processing strategy means that when identifying stain types, if the filtered association rules point to multiple potentially similar or conflicting stain types (for example, oil stains and scale may overlap in certain spectral or mechanical characteristics), the system will pre-set specific difference amplification processing schemes according to these potentially conflicting stain types. These strategies are designed to address the difficulty in distinguishing between different stain types, aiming to highlight their key differences. For example, for oil stains and scale, it may be necessary to amplify the differences in absorption of the spectrum in a specific infrared band, or the differences in the viscous characteristics of the mechanical response in the initial stage of scratching.

[0064] The core operation of this scheme is to perform differential amplification processing on the spectral reflectance data in the first set of stain feature information and the force signal data in the second set of stain feature information. For the spectral reflectance data, second-order derivative analysis or high-power function processing can be used. Second-order derivative analysis can effectively highlight subtle changes in inflection points and peaks / valleys in the spectral curve, which often correspond to differences in the molecular structure or surface morphology of the material, thus amplifying subtle differences in spectral features. High-power function processing, through nonlinear transformation, can widen the gap between originally similar reflectance values, making them easier to distinguish in the feature space. For the force signal data, instantaneous rate of change analysis or frequency domain energy distribution mapping can be used. Instantaneous rate of change analysis can capture abrupt changes or rapid fluctuations in the force signal during scratching, which are often related to the hardness, adhesion, or particle structure of the stain. Frequency domain energy distribution mapping can convert the force signal from the time domain to the frequency domain, analyze the energy distribution of different frequency components, thereby revealing the inherent vibration or damping characteristics of the stain in the contact mechanical response, amplifying the differences in the contact mechanical response.

[0065] This application effectively addresses the potential blindness and inefficiency of difference amplification processing when the confidence score for combination probability is low by introducing refined control over the process. Specifically, when the system identifies significant uncertainty in stain type, it first dynamically adjusts the intensity and scope of the difference amplification processing based on the low confidence level of the confidence score, ensuring that the processing intensity matches the identification difficulty and avoiding excessive or insufficient amplification. Secondly, by analyzing the potential conflicting stain types indicated by the filtered rules, a targeted difference amplification strategy is preset, making the processing more targeted and concentrating resources on amplifying the key features that best distinguish these conflicting stains. For example, for spectral reflectance data, second-order derivative analysis or high-power function processing can highlight subtle changes in spectral curves that are difficult to detect with the naked eye, making previously similar spectral features appear significantly different after processing. For force signal data, instantaneous rate of change analysis or frequency domain energy distribution mapping can reveal the unique fingerprint of stains in their contact mechanical response, such as the frictional force fluctuation patterns or energy dissipation characteristics generated by different stains during scratching, thus providing stronger distinguishing ability in the mechanical dimension. It is precisely because of this multi-dimensional, adaptive, and refined difference amplification processing that the system is able to effectively extract discriminative information when faced with complex and ambiguous stain features, providing clearer and more reliable input for subsequent decision tree judgments.

[0066] In some embodiments, the step of adjusting the intensity and scope of the difference amplification process based on the low confidence level of the combination probability confidence score includes: Obtain local environmental parameters of the current work area, including local wind speed, local temperature, and local light intensity; Based on local environmental parameters, analyze the degree of influence of local microclimate on the physicochemical properties of stains on glass surfaces; Based on the degree of impact, the confidence score of the combination probability is corrected by local microclimate to obtain the corrected confidence score; Based on the numerical range of the corrected confidence scores, the low confidence level is divided into multiple levels, and the intensity coefficient and feature dimension range of the difference amplification processing are preset for each level. Based on the deviation between the corrected confidence score and the preset threshold, the intensity of the difference amplification process is dynamically calculated through nonlinear function mapping, and the priority of the feature dimensions of the difference amplification process is determined according to the sign of the deviation. Based on the trend of the revised confidence score over time, the future level of low confidence is predicted, and the intensity and scope of the difference amplification process are adjusted in advance based on the prediction results.

[0067] Specifically, acquiring local environmental parameters of the current work area refers to collecting environmental data such as local wind speed, local temperature, and local light intensity in real time through sensors integrated on the window cleaning machine or deployed near the work area. These parameters are crucial for understanding the physicochemical state of stains on the glass surface. For example, high wind speeds may accelerate stain drying or introduce new particles, high temperatures may change the viscosity of some organic stains, and light intensity directly affects the quality of spectral reflectance information acquisition.

[0068] Analyzing the impact of local environmental parameters on the physicochemical properties of glass surface stains can be understood as using a pre-established environmental impact model or empirical knowledge base to assess the potential changes in the optical properties (e.g., reflectivity, absorption peaks) and mechanical properties (e.g., adhesion, hardness) of specific stain types (e.g., oil, dust, scale). The aim is to quantify the potential perturbation of stain characteristics by environmental factors.

[0069] The confidence score of the combination probability is adjusted by local microclimate based on the degree of influence to obtain the adjusted confidence score. For example, the original confidence score of the combination probability can be weighted, shifted or nonlinearly transformed according to the degree of influence obtained from the analysis to eliminate or reduce the bias caused by environmental factors, so that the confidence score can more accurately reflect the inherent characteristics of the stain.

[0070] Based on the numerical range of the corrected confidence scores, low confidence levels are divided into multiple grades, and a preset intensity coefficient and feature dimension range for differential amplification processing are assigned to each grade. For example, low confidence levels can be divided into slightly uncertain, moderately uncertain, and highly uncertain grades, with different amplification coefficients (e.g., 1.2x, 1.5x, 2.0x) and prioritized amplified feature dimensions (e.g., specific bands of the spectrum or high-frequency components of mechanical signals) set for each grade. The aim is to achieve fine-grained control of differential amplification processing.

[0071] Based on the deviation between the corrected confidence score and a preset threshold, the intensity of the difference amplification process is dynamically calculated using a non-linear function mapping. The priority of the feature dimensions for difference amplification is determined according to the sign of the deviation. For example, when the corrected confidence score is closer to the threshold but still below it, finer adjustments may be needed, and the non-linear function provides this flexibility. When the deviation is negative (i.e., the confidence score is far below the threshold), a larger amplification may be needed, prioritizing the feature dimensions that best distinguish the types of potential conflict stains.

[0072] Based on the trend of the corrected confidence score over time, the future level of low confidence is predicted, and the intensity and scope of the difference amplification process are adjusted in advance based on the prediction results. This can be understood as using historical data and machine learning models (such as time series prediction models) to analyze the dynamic changes in confidence scores, identify potential trends or periodicities, and thus predict the degree of low confidence before it occurs and adjust the difference amplification strategy in advance, achieving proactive optimization of stain identification.

[0073] This application significantly improves the adaptability and foresight of differential amplification processing by introducing the perception and analysis of local environmental parameters and combining them with time series prediction. Specifically, when the confidence score of the combined probability is low, it no longer relies solely on the static low confidence level, but first obtains environmental parameters such as local wind speed, local temperature, and local light intensity. These parameters are used to analyze the influence of local microclimate on the physicochemical properties of stains on the glass surface, thereby correcting the original confidence score for local microclimate. It is precisely because of this correction that the confidence score can more accurately reflect the true characteristics of the stains, avoiding misjudgments caused by environmental disturbances. On this basis, by dividing the numerical range of the corrected confidence score into multiple levels and presetting the intensity coefficient and feature dimension range of differential amplification processing for each level, fine-grained hierarchical control of differential amplification processing is achieved. The intensity of differential amplification processing is dynamically calculated through nonlinear function mapping, and the priority of feature dimensions is determined according to the positive or negative value of the deviation, enabling differential amplification processing to be flexibly adjusted according to the actual degree of uncertainty, ensuring the effective amplification of key features. Meanwhile, by analyzing the changing trend of the corrected confidence score over time, the system predicts the future level of low confidence and adjusts the intensity and scope of the difference amplification processing in advance based on the prediction results. This enables the system to anticipate and preprocess potential identification difficulties, allowing for more appropriate amplification strategies to be adopted before the stain characteristics become blurred or complex, effectively avoiding identification lag.

[0074] In some embodiments, the step of predicting the future degree of low confidence based on the changing trend of the modified confidence score over time includes: Identify the area where the window cleaning machine is currently operating. The area identification includes information on the building facade type and floor intervals. Based on the region identifier, extract the historical confidence score time series change pattern that matches the current region identifier from the pre-stored region-specific trend library; Match the current revised confidence score with the extracted region-specific trend patterns; Based on the matching results, the trend of the current corrected confidence score over time is locally corrected. Based on the corrected trend, predict its future low confidence level.

[0075] Specifically, identifying the current area of ​​operation for the window cleaning machine refers to obtaining the physical location information of the machine's current operation and associating it with a pre-defined area classification system. An area identifier can be understood as a geographical or environmental label, containing information such as the building facade type and the type of glass used (e.g., curtain walls, regular windows, floor-to-ceiling windows), as well as floor level information. Glass types include different thicknesses of glass, frosted glass, tinted / coated glass, and double-glazed windows. Floor level information includes low-rise (1-5 floors), mid-rise (6-20 floors), and high-rise (above 20 floors). This information can be obtained through the window cleaning machine's own positioning system, pre-defined building information model data, or manual input. Its purpose is to provide accurate contextual information for subsequent trend pattern matching. Based on the area identifier, historical confidence score time-series change patterns matching the current area identifier are extracted from a pre-stored area-specific trend database. This can be understood as the system maintaining a database containing a large amount of historical data, storing typical patterns or trends of corrected confidence scores changing over time under different area identifiers. Once the current work area identifier is obtained, the system will query the database to find the historical change pattern that best matches the current area identifier. For example, if the current work area is a high-rise glass curtain wall, the system will extract the historical confidence score change pattern for this type of area. These patterns may reflect the impact of factors unique to this area, such as wind pressure, sunlight, and pollutant deposition, on the confidence level of stain identification.

[0076] Matching the current corrected confidence score with the extracted region-specific trend pattern involves comparing the real-time time-series data of the corrected confidence score with historical patterns extracted from a trend database. This matching can be achieved through various statistical or machine learning methods, such as time-series similarity measures, correlation analysis, or pattern recognition algorithms. The aim is to assess the degree of agreement between the current confidence score trend and typical historical patterns, thereby providing a basis for subsequent local corrections.

[0077] Based on the matching results, a local correction is made to the trend of the current revised confidence score over time. This means that if a deviation is found between the current trend and historical patterns during the matching process, the current trend is adjusted according to the magnitude and nature of the deviation. For example, if the current trend shows that the confidence score is decreasing faster than historical patterns, but the matching results indicate that the region is indeed prone to rapid declines under specific conditions, the parameters of the prediction model can be fine-tuned to better reflect the actual situation in the region. This local correction aims to eliminate potential biases in the general model, making the prediction results closer to reality.

[0078] Predicting future low-confidence levels based on corrected trends involves using a predictive model to infer future confidence scores based on locally corrected time-series trends, thereby predicting when and where low-confidence situations might occur. The aim is to provide forward-looking guidance for window cleaning machine operation scheduling and cleaning strategy adjustments, ensuring preventative measures are taken before stain identification confidence levels may decrease.

[0079] This application incorporates the concept of regional identifiers to integrate the physical environmental factors of window cleaning machine operations into the confidence score prediction model. Specifically, by identifying the building facade type and floor range information of the current operation, the system can obtain the unique environmental context related to that area. Given the significant differences in microclimate, pollutant types, and deposition rates across different areas, these regional identifiers are used to extract the best-matching historical confidence score time-series variation patterns from a pre-stored regional-specific trend library. Since these historical patterns reflect the true laws governing stain evolution and confidence fluctuations within a specific area, matching the current corrected confidence score with these regional-specific patterns effectively identifies the similarities and differences between the current trend and historical experience. Based on this, the current trend is locally corrected according to the matching results, enabling the prediction model to adapt to the unique environmental conditions of the current operation area and avoiding prediction biases that may arise from general models. In this way, the prediction model is no longer applied in a one-size-fits-all manner but can be personalized according to the specific operation area, thereby significantly improving the accuracy and reliability of predicting future low-confidence levels.

[0080] In some embodiments, the step of predicting the future degree of low confidence based on the changing trend of the modified confidence score over time includes: Obtain information on structural defects on the glass surface, including the distribution of microcracks and the degree of material aging; Based on information on structural defects on the glass surface, a defect impact factor is constructed, which quantifies the degree to which structural defects alter the stain adhesion and evolution mechanism. The trend of the defect impact factor and the modified confidence score in the time series is fused to obtain the trend of the fused confidence score. Nonlinear feature extraction is performed on the trend of confidence score changes after fusion to identify non-periodic abnormal patterns related to structural defects. Based on non-periodic anomaly patterns, the parameters of the prediction model are adjusted to adapt to nonlinear, non-periodic anomaly fluctuations caused by structural defects and to predict their future low confidence levels.

[0081] Specifically, obtaining information on structural defects on the glass surface refers to scanning and analyzing the glass surface using equipment such as high-resolution visual sensors, ultrasonic detectors, or laser scanners to obtain information such as the distribution, depth, and width of microcracks, as well as the degree of degradation of the glass material. This information can serve as an important basis for assessing the overall health of the glass surface.

[0082] Constructing a defect impact factor can be understood as calculating a quantitative index based on the acquired information about structural defects on the glass surface, using a pre-set mathematical model or expert system. This index reflects the potential impact of structural defects on aspects such as the adhesion, diffusion rate, chemical reactivity, and optical reflection / absorption characteristics of stains on the glass surface. For example, more severe microcracks may lead to easier penetration and adhesion of stains, thereby altering their spectral reflectance and mechanical response characteristics.

[0083] This method integrates the time-series trends of defect impact factors with the adjusted confidence scores. For example, it can combine quantified defect impact factors with historical confidence score trend data using methods such as weighted averaging, neural network fusion, or Kalman filtering. The aim is to ensure that the predictive model considers both time trends and the impact of inherent glass surface defects on the uncertainty of stain identification.

[0084] Nonlinear feature extraction can be performed on the trend of confidence score changes after fusion. Techniques such as wavelet transform, empirical mode decomposition (EMD), or deep learning models (such as recurrent neural networks) can be used to identify anomalous patterns in the fused data that do not conform to regular periodic changes and are closely related to structural defects. These non-periodic anomalous patterns may indicate special stain behavior or identification challenges caused by defects.

[0085] Based on non-periodic anomaly patterns, the parameters of the prediction model can be adjusted. For example, the weights, biases, or activation functions of the prediction model (such as the ARIMA model or LSTM model) can be dynamically modified. The aim is to enable the prediction model to better adapt to the nonlinear and irregular confidence fluctuations caused by structural defects in glass, thereby improving the accuracy of predictions for future low-confidence levels.

[0086] This application, by incorporating information on structural defects on the glass surface and fusing it with the time-series trend of the corrected confidence score, enables a more comprehensive assessment of the complexity of stain identification. The time-series prediction in the basic approach may struggle to capture the nonlinear changes in stain adhesion and evolution mechanisms caused by glass microcracks or aging. By constructing a defect impact factor, the degree to which structural defects alter stain behavior is quantified, and this factor is integrated into the trend of the confidence score, allowing the prediction model to perceive these potential physicochemical effects. Furthermore, nonlinear feature extraction from the fused trend identifies aperiodic anomaly patterns associated with structural defects, patterns that are difficult to detect using traditional linear or periodic analyses. Finally, the prediction model parameters are adjusted based on these aperiodic anomaly patterns, enabling the model to adapt to the nonlinear, aperiodic fluctuations caused by structural defects, thereby significantly improving the prediction accuracy for future low-confidence levels.

[0087] like Figure 2 As shown in the embodiments of this application, a multimodal sensor data fusion processing system for a window cleaning machine is also disclosed, including: Visual information acquisition module 1 is used to acquire visual information about the glass surface; Feature extraction and comparison module 2 is used to extract features from visual information, compare the extracted features with preset features, and determine the area to be explored based on the comparison results; The operation control module 3 is used to pause the current operation and start the fine exploration process in response to the determination of the area to be explored; The cleaning mechanism control module 4 is used to respond to the fine detection process to move the cleaning mechanism to the area to be detected, so that the cleaning mechanism contacts the area to be detected with a controlled force, and obtains the spectral reflectance information and contact mechanical response information of the area to be detected. The information processing and identification module 5 is used to combine spectral reflectance information and contact mechanical response information to infer the physical properties of the area to be explored and to identify the type of stain in the area to be explored.

[0088] The system provided in this application, through its modular design, achieves automated and intelligent identification of stains on glass surfaces, effectively solving the problem of inaccurate stain identification by existing window cleaning machines in complex environments, and improving the efficiency and quality of cleaning operations.

[0089] The above are merely embodiments of this application and are not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A method for fusion processing of multimodal sensor data from a window cleaning machine, characterized in that, include: Obtain visual information about the glass surface; Visual information is used to extract features, and the extracted features are compared with preset features. The area to be explored is determined based on the comparison results. In response to the identification of the area to be explored, the current operation is suspended and a detailed exploration process is initiated, which includes: The cleaning unit is moved to the area to be explored, and the cleaning unit contacts the area with a controlled force, and the spectral reflectance information and contact mechanical response information of the area to be explored are obtained; By combining spectral reflectance information and contact mechanical response information, the physical properties of the area to be explored can be inferred, and the type of stain in the area to be explored can be identified.

2. The method for multimodal sensor data fusion processing of a window cleaning machine according to claim 1, characterized in that, The steps of combining spectral reflectance information and contact mechanical response information to infer the physical properties of the area to be explored and to identify the type of stain in the area to be explored include: Feature analysis is performed on the spectral reflectance information to extract the reflectance ratio and absorption peak position at a specific wavelength, forming the first set of stain feature information; Feature analysis is performed on the contact mechanical response information to calculate the mean, variance, and high-frequency components of the force signal during the scratching process, forming a second set of stain feature information; When the first set of stain feature information is similar to the spectral features of a preset first type of known stain, and the second set of stain feature information is similar to the mechanical response features of a preset second type of known stain, the judgment logic based on multi-dimensional feature space mapping is activated: Map the first set of stain feature information and the second set of stain feature information to their respective feature spaces, and calculate the first feature distance between the first set of stain feature information and the first type of known stains, and the second feature distance between the second set of stain feature information and the second type of known stains. Based on the first feature distance, the second feature distance, and the preset stain association rule set, the combination probability of the first type of known stain and the second type of known stain is evaluated, and the stain type of the area to be explored is identified according to the evaluation results, thereby inferring the physical properties of the area to be explored.

3. The method for multimodal sensor data fusion processing of a window cleaning machine according to claim 2, characterized in that, The steps of mapping the first group of stain feature information and the second group of stain feature information to their respective feature spaces, and calculating the first feature distance between the first group of stain feature information and the first type of known stains, and the second feature distance between the second group of stain feature information and the second type of known stains, include: The spectral reflectance data in the first set of stain feature information is locally smoothed, and the force signal data in the second set of stain feature information is low-pass filtered to obtain the pre-processed first set of stain feature information and the pre-processed second set of stain feature information. Based on the ambient light intensity of the current work area and the self-diagnostic status of the sensor cables, the reliability of the current data from the vision sensor and force feedback sensor is evaluated in real time. Based on data reliability, the first set of pre-processed stain feature information and the second set of pre-processed stain feature information are mapped to their respective feature spaces, and the weights of the first set of pre-processed stain feature information and the second set of pre-processed stain feature information in the feature space are dynamically adjusted. The distance correction factor is calculated based on the degree of overlap and distribution density between the feature information of the stain to be investigated and the feature information of the known stain in the feature space; When the feature distance between the stain to be investigated and two known stains is close, the feature distance is fine-tuned according to the distance correction factor to obtain the corrected feature distance. Based on the corrected feature distance, the first feature distance between the first set of stain feature information and the first type of known stain, and the second feature distance between the second set of stain feature information and the second type of known stain are calculated.

4. The method for multimodal sensor data fusion processing of a window cleaning machine according to claim 2, characterized in that, Also includes: Update the preset stain association rule set as follows: Obtain environmental parameter information for the current work area; Based on environmental parameter information, the applicable weight and priority of existing rules in the stain association rule set are dynamically adjusted; When the stain association rule set cannot clearly assess the probability of combinations of known stains of type 1 and known stains of type 2, the stain feature tracing and evolution analysis process is initiated: In-depth analysis of the first and second sets of stain feature information was performed to identify feature patterns that slightly deviated from known stain types. By combining environmental parameter information, we can infer the physicochemical changes or new pollutant formation pathways corresponding to the characteristic patterns of subtle deviations. Based on the inference results, temporary association rules for new types of stains are generated and incorporated into the stain association rule set.

5. The method for multimodal sensor data fusion processing of a window cleaning machine according to claim 2, characterized in that, The steps of evaluating the probability of a combination of a first type of known stain and a second type of known stain based on a first feature distance, a second feature distance, and a preset stain association rule set, and identifying the stain type of the area to be explored based on the evaluation results, thereby inferring the physical properties of the area to be explored, include: Based on the first feature distance and the second feature distance, the rules in the stain association rule set that are related to the combination probability of the first type of known stains and the second type of known stains are filtered; Based on the source reliability of the filtered rules, the update time of the rules, and the current reliability of the feature information on which the rules depend, a dynamic confidence weight is assigned to each filtered rule. Calculate the strength of the probability of supporting or opposing a combination for each filtered rule, and combine it with dynamic confidence weights to weight all filtered rules to obtain a confidence score for the probability of the combination. When the confidence score of the combination probability is lower than the preset judgment threshold, the key features related to the filtered rules in the first group of stain feature information and the second group of stain feature information are subjected to difference amplification processing. Based on the pre-set decision tree, the system makes judgments based on the key features after amplification of differences, identifies the type of stain in the area to be explored, and infers the physical properties of the area to be explored.

6. The method for multimodal sensor data fusion processing of a window cleaning machine according to claim 5, characterized in that, The step of performing difference amplification processing on key features related to the filtered rules in the first group of stain feature information and the second group of stain feature information when the confidence score of the combination probability is lower than the preset judgment threshold includes: Adjust the intensity and scope of the difference amplification process based on the low confidence level of the combination probability confidence score; Based on the potential conflict stain types indicated by the filtered rules, a differential amplification treatment strategy is preset; Based on the preset differential amplification processing strategy, and the adjusted intensity and range of differential amplification processing, differential amplification processing is performed on the spectral reflectance data in the first group of stain feature information and the force signal data in the second group of stain feature information.

7. The method for multimodal sensor data fusion processing of a window cleaning machine according to claim 6, characterized in that, The step of adjusting the intensity and scope of the difference amplification processing based on the low confidence level of the combination probability confidence score further includes: Obtain local environmental parameters for the current work area; Based on local environmental parameters, analyze the degree of influence of local microclimate on the physicochemical properties of stains on glass surfaces; Based on the degree of impact, the confidence score of the combination probability is corrected by local microclimate to obtain the corrected confidence score; Based on the trend of the revised confidence score over time, predict its future level of low confidence. Based on the prediction results, the intensity and scope of the difference amplification processing are adjusted in advance.

8. The method for multimodal sensor data fusion processing of a window cleaning machine according to claim 7, characterized in that, The step of predicting the future level of low confidence based on the changing trend of the corrected confidence score over time includes: Identify the area currently being cleaned by the window cleaning machine; Based on the region identifier, extract the historical confidence score time series change pattern that matches the current region identifier from the pre-stored region-specific trend library; Match the current revised confidence score with the extracted region-specific trend patterns; Based on the matching results, the trend of the current corrected confidence score over time is locally corrected. Based on the corrected trend, predict its future low confidence level.

9. The method for multimodal sensor data fusion processing of a window cleaning machine according to claim 7, characterized in that, The step of predicting the future level of low confidence based on the changing trend of the corrected confidence score over time includes: Obtain information on structural defects on the glass surface; Based on information on structural defects on the glass surface, a defect impact factor is constructed, which quantifies the degree to which structural defects alter the stain adhesion and evolution mechanism. The trend of the defect impact factor and the modified confidence score in the time series is fused to obtain the trend of the fused confidence score. Nonlinear feature extraction is performed on the trend of confidence score changes after fusion to identify non-periodic abnormal patterns related to structural defects. Based on non-periodic anomaly patterns, the parameters of the prediction model are adjusted to adapt to nonlinear, non-periodic anomaly fluctuations caused by structural defects and to predict their future low confidence levels.

10. A multimodal sensor data fusion processing system for a window cleaning machine, characterized in that, include: The visual information acquisition module is used to acquire visual information about the glass surface; The feature extraction and comparison module is used to extract features from visual information, compare the extracted features with preset features, and determine the area to be explored based on the comparison results. The operation control module is used to pause the current operation and start the fine exploration process in response to the determination of the area to be explored; The cleaning mechanism control module is used to respond to the fine detection process to move the cleaning mechanism to the area to be detected, so that the cleaning mechanism contacts the area to be detected with a controlled force, and acquires the spectral reflectance information and contact mechanical response information of the area to be detected; The information processing and identification module is used to combine spectral reflectance information and contact mechanical response information to infer the physical properties of the area to be explored and to identify the type of stain in the area to be explored.

Images