A Multimodal Fusion-Based Intelligent Perception Method for Inspection Robot Dogs
By using multimodal fusion and state resilience potential field analysis, the problems of temporal correlation and cross-cycle state evolution of multimodal data in inspection robot dogs were solved, enabling accurate identification and stability judgment of equipment status and improving the accuracy and robustness of anomaly identification.
Patent Information
- Application Number
- CN202611310781.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-08-27
- Publication Date
- 2026-11-17
AI Technical Summary
In the current inspection robot dog, the temporal correlation of multimodal data and the cross-cycle state evolution relationship are not effectively integrated in the identification of equipment operation status, resulting in inaccurate judgment of equipment operation stability and lag in anomaly identification.
A multimodal fusion and state-toughness potential field evolution analysis method is adopted. By collecting equipment identification, visible light images, infrared thermal imaging, vibration and acoustic data, a state expression sequence is constructed, state switching relationship and swing spectrum are generated, structural constraint evolution analysis is performed, and equipment state stability assessment results are generated.
It enables more accurate identification and comprehensive characterization of equipment status, improves the ability to characterize the dynamic evolution characteristics of equipment during operation, and enhances the accuracy and stability of anomaly identification under complex working conditions.
Smart Images

Figure CN123116147A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of intelligent sensing for inspection, and in particular to an intelligent sensing method for inspection robot dogs based on multimodal fusion. Background Technology
[0002] In existing technologies, inspection robot dogs typically collect equipment operation data through multimodal sensors and perform state recognition and anomaly detection based on single features such as images, vibrations, and acoustics. However, the temporal correlation between different modes and the cross-cycle state evolution relationship are not effectively integrated, resulting in insufficient characterization of the continuous changes in equipment operation stability and difficulty in reflecting the dynamic evolution characteristics of the state during continuous inspection.
[0003] In addition, existing methods mostly rely on static feature statistics or local time window analysis, lacking a unified modeling mechanism for state switching behavior, round-trip changes, and structural correlations. This makes it difficult to accurately characterize the coupling relationship between state fluctuations and structural changes, which can easily lead to problems such as inaccurate stability judgment and delayed anomaly identification under complex working conditions. Summary of the Invention
[0004] One objective of this invention is to propose an intelligent perception method for inspection robot dogs based on multimodal fusion. This invention is based on multimodal fusion and state resilience potential field evolution analysis method to realize the intelligent perception and evaluation of cross-cycle stability changes of equipment state by inspection robot dogs. It has the advantages of more accurate state recognition, more comprehensive evolution characterization and more reliable stability judgment.
[0005] A method for intelligent perception of a patrol robot dog based on multimodal fusion according to an embodiment of the present invention includes the following steps: The inspection robot dog collects equipment identification data, visible light image data, infrared thermal imaging data, vibration data, and acoustic data during continuous inspection cycles, and preprocesses them to generate a standardized inspection dataset. Multimodal state features are extracted from a standardized inspection dataset, and a state representation sequence is constructed based on each multimodal state feature. Extract the temporal transition between state expressions according to the time sequence corresponding to the continuous inspection cycle, and generate a set of state switching relationships; Based on the set of state transition relationships, identify the state round-trip paths formed between state expressions, and aggregate them according to the correlation between multiple state round-trip paths to generate a set of state swing zones; A state swing spectrum is constructed based on the state switching relationships, state switching directions, and state return situations among the state expressions within each state swing region. Structural constraint evolution analysis of the state sway spectrum is performed based on the state toughness potential field to generate the results of state toughness distribution change. Based on the changes in the state resilience distribution, the system generates equipment state stability assessment results and outputs intelligent sensing results.
[0006] Optionally, the equipment identification data includes equipment number, equipment type, and inspection time; the visible light image data includes equipment surface image; the infrared thermal imaging data includes equipment thermal imaging image; the vibration data includes equipment vibration signal; the acoustic data includes equipment operating sound signal; and the preprocessing specifically includes time alignment, missing data completion, anomaly removal, and standardization.
[0007] Optionally, the generation of the state representation sequence specifically includes: Read the standardized inspection dataset, extract visual state features from visible light image data, thermal state features from infrared thermal imaging data, vibration state features from vibration data, and acoustic state features from acoustic data to generate a multimodal state feature set. Based on the inspection time, read the multimodal state feature set, and combine the visual state features, thermal state features, vibration state features and acoustic state features corresponding to the same inspection time to generate a state expression set. Extract the inspection time corresponding to each state expression in the state expression set, and sort the multiple state expressions according to the inspection time order to generate the state expression arrangement result; Based on the arrangement of state expressions, establish the sequential order of multiple state expressions to generate a state expression sequence.
[0008] Optionally, the generation of the state transition relationship set specifically includes: Based on the order of the state representation sequence, adjacent state representations are extracted sequentially to generate multiple state transition units; Read the preceding and following state expressions in each state transition unit, extract the state transition process from the preceding state expression to the following state expression, and generate a set of state transition records. Perform a classification and merging process on multiple state transition records in the state transition record set, grouping state transition records with the same preceding and subsequent state expressions into the same type of state transition records, and generating state transition statistics results; Based on the various state transition records and their occurrence frequency in the state transition statistics, establish the state switching relationship between the preceding state expression and the following state expression, and generate multiple state switching relationships; Multiple state transition relationships are aggregated to generate a set of state transition relationships.
[0009] Optionally, the generation of the set of state swing regions specifically includes: Read multiple state transition relationships in the state transition relationship set, identify state transition relationships in which the preceding state expression and the following state expression are mutually reversed, determine them as state round-trip relationships, and generate a state round-trip relationship set; Based on the set of state round trip relationships, extract the state representation and state transition direction corresponding to each state round trip relationship, and generate multiple state round trip paths; Statistically analyze the state representation composition of each state round-trip path, group state round-trip paths containing the same state representation into the same type of state round-trip path, and generate state round-trip path classification results. Based on the various types of state round-trip paths in the state round-trip path classification results, extract the state representations involved in each type of state round-trip path and generate multiple state swing regions; Multiple state swing regions are aggregated to generate a set of state swing regions.
[0010] Optionally, the construction of the state swing spectrum specifically includes: Read multiple state swing regions from the set of state swing regions, extract the state representations contained in each state swing region and the corresponding state transition relationships, and generate a set of state swing relationships. Based on the multiple state transition relationships in the state swing relationship set, the occurrence frequency of each state transition relationship is counted, and the corresponding state transition direction is extracted to generate state swing statistics results. By combining the statistical results of state swing, we can identify the state return process formed by each state expression returning to the original state expression after one or more state transitions, count the corresponding number of returns and the return span, and generate the state return information. Based on the statistical results of state swing and the state return status, the state switching relationship, state switching direction, occurrence frequency and state return status corresponding to the state expression in each state swing zone are collected and processed to generate multiple state swing results. Multiple state swing results are aggregated according to their corresponding state swing regions to generate a state swing spectrum.
[0011] Optionally, the generation of the state toughness distribution change results specifically includes: Extract the set of state representations and the set of state switching relationships corresponding to each state swing zone from the state swing spectrum, and sort the state switching occurrence process in the set of state switching relationships according to the inspection time order to obtain the state switching sequence. Based on the state expression in the state switching sequence, the time interval of its repeated occurrence in different inspection cycles is statistically analyzed, and the state dwell time sequence is formed according to the chronological relationship. At the same time, the state dwell time information in each inspection cycle is summarized. By combining the state dwell time series and state dwell duration information, the state dwell intensity of each state expression is calculated. The state dwell intensity is determined by the proportional relationship between the state dwell duration and the time interval between two adjacent occurrences, and the state intensity set is obtained by summarizing them. The state representation is mapped to state nodes, and the state retention strength is assigned to the corresponding state node attribute. At the same time, the state switching relationship is transformed into a structural association relationship between state nodes, forming a state resilience potential field structure. Periodic consistency correction is performed on the structural correlation in the state-resilient potential field, and the state node attributes are adjusted and updated based on the return count and return span of each state expression to obtain the updated state-resilient potential field structure. A cycle-by-cycle comparative analysis was conducted on the state toughness potential field structure corresponding to different inspection cycles to obtain the change range of attributes of each state node and the change range of structural correlation, thus forming the result of state toughness distribution change.
[0012] Optionally, the generation of the intelligent sensing result specifically includes: Read the changes in state node attributes and structural relationships corresponding to each state swing zone in the state resilience distribution change results, and summarize and organize them according to the inspection cycle to form state change statistics. Based on the statistical results of state changes, the direction of change of state node attributes and the direction of change of structural association are calculated for each state swing zone. The direction of change of state node attributes is determined by the trend of change of state retention intensity, and the direction of change of structural association is determined by the trend of change of state switching relationship. Based on the degree of consistency between the direction of change of state node attributes and the direction of change of structural association, stability division is performed on each state swing region to obtain the state stability division result. Based on the state stability classification results, different state swing zones are merged and analyzed to form the overall state stability assessment results of the equipment. Based on the overall stability assessment results of the equipment, intelligent sensing output information is generated and output to the maintenance terminal of the inspection robot dog.
[0013] The beneficial effects of this invention are: Compared with existing technologies, this invention introduces a continuous modeling mechanism of state expression sequence and state switching relationship based on multimodal inspection data. Furthermore, it uses state swing regions and state swing spectra to structurally summarize the repeatedly changing state behavior during equipment operation. This enables state information that was originally scattered in multi-source data such as images, vibrations, and acoustics to achieve temporal correlation and structural fusion under a unified state expression framework, thereby effectively improving the overall ability to characterize the dynamic evolution characteristics of equipment operation.
[0014] Meanwhile, by constructing a state resilience potential field structure, this invention unifies the mapping between state retention strength and state switching relationship, and combines it with the state resilience distribution change analysis across inspection cycles to achieve continuous tracking and quantitative expression of state stability change trends. This enables the equipment state to no longer be limited to single-cycle static judgment, but to reflect its evolution path and stability fluctuation process in the time dimension, thereby improving the continuity and reliability of state identification.
[0015] Furthermore, this invention further utilizes consistency analysis based on changes in state node attributes and structural correlation to achieve stability division of state swing zones and overall equipment status assessment. This enables joint perception from three levels: "state change trend, structural evolution trend, and consistency judgment." Compared to traditional methods that rely on single features or local statistics, this invention significantly improves the accuracy of anomaly identification and the robustness of stability judgment under complex working conditions, and has stronger engineering adaptability and practical application value. Attached Figure Description
[0016] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of an intelligent perception method for inspection robot dogs based on multimodal fusion proposed in this invention; Figure 2 This is a diagram showing the state swing region formation relationship of an intelligent perception method for inspection robot dogs based on multimodal fusion proposed in this invention. Detailed Implementation
[0017] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.
[0018] refer to Figures 1-2 A method for intelligent perception in a patrol robot dog based on multimodal fusion includes the following steps: The inspection robot dog collects equipment identification data, visible light image data, infrared thermal imaging data, vibration data, and acoustic data during continuous inspection cycles, and preprocesses them to generate a standardized inspection dataset. Multimodal state features are extracted from a standardized inspection dataset, and a state representation sequence is constructed based on each multimodal state feature. Extract the temporal transition between state expressions according to the time sequence corresponding to the continuous inspection cycle, and generate a set of state switching relationships; Based on the set of state transition relationships, identify the state round-trip paths formed between state expressions, and aggregate them according to the correlation between multiple state round-trip paths to generate a set of state swing zones; A state swing spectrum is constructed based on the state switching relationships, state switching directions, and state return situations among the state expressions within each state swing region. Structural constraint evolution analysis of the state sway spectrum is performed based on the state toughness potential field to generate the results of state toughness distribution change. Based on the changes in the state resilience distribution, the system generates equipment state stability assessment results and outputs intelligent sensing results.
[0019] In this embodiment, the equipment identification data includes the equipment number, equipment type, and inspection time; the visible light image data includes images of the equipment surface; the infrared thermal imaging data includes thermal imaging images of the equipment; the vibration data includes equipment vibration signals; the acoustic data includes equipment operating sound signals; and the preprocessing specifically includes time alignment, missing data completion, anomaly removal, and standardization. The generation of standardized inspection datasets specifically includes: The inspection robot performs periodic inspections on approximately 860 devices, covering about three inspection cycles per day. Each cycle lasts about 2.5 hours, and the size of the multimodal raw data remains between 15GB and 18GB. Device identification data, visible light image data, infrared thermal imaging data, vibration data, and acoustic data are first aggregated into a unified acquisition channel. An inspection timeline is established through a unified timestamp mapping mechanism, enabling the multi-source data to form a synchronous correspondence in the time dimension. The time alignment error is controlled within approximately 0.2 seconds, thereby generating an initial time-aligned data set. An integrity scan is performed on the multimodal data in the time-aligned data set, identifying data units with a modality missing ratio of approximately 6% to 8%. This data is then interpolated and reconstructed using historical data from the same device in adjacent inspection cycles to eliminate data gaps. Partial compensation was achieved, and the overall data integrity rate was improved to approximately 97% after processing, thus forming a complete multimodal dataset. The complete multimodal dataset then entered the consistency detection process. By comparing the multimodal distribution characteristics of the same equipment in adjacent inspection cycles, approximately 1.5% to 2% of data records deviating from the normal distribution range were selected and removed. This eliminated the impact of abnormal disturbances at the data structure level, forming a de-anomaly multimodal dataset. The de-anomaly multimodal dataset then underwent standardization processing. Visible light images were adjusted to a resolution range of 1280×720, infrared thermal imaging data was mapped to a temperature normalization range of 0 to 1, vibration signals underwent frequency domain energy standardization, and acoustic signals underwent spectral energy normalization conversion, thereby constructing a standardized inspection dataset.
[0020] In this embodiment, the generation of the state representation sequence specifically includes: Read the standardized inspection dataset, extract visual state features from visible light image data, thermal state features from infrared thermal imaging data, vibration state features from vibration data, and acoustic state features from acoustic data to generate a multimodal state feature set. Based on the inspection time, read the multimodal state feature set, and combine the visual state features, thermal state features, vibration state features and acoustic state features corresponding to the same inspection time to generate a state expression set. Extract the inspection time corresponding to each state expression in the state expression set, and sort the multiple state expressions according to the inspection time order to generate the state expression arrangement result; Based on the arrangement of state expressions, establish the sequential order of multiple state expressions to generate a state expression sequence.
[0021] In this embodiment, the generation of the state transition relationship set specifically includes: Based on the order of the state representation sequence, adjacent state representations are extracted sequentially to generate multiple state transition units; Read the preceding and following state expressions in each state transition unit, extract the state transition process from the preceding state expression to the following state expression, and generate a set of state transition records. Perform a classification and merging process on multiple state transition records in the state transition record set, grouping state transition records with the same preceding and subsequent state expressions into the same type of state transition records, and generating state transition statistics results; Based on the various state transition records and their occurrence frequency in the state transition statistics, establish the state switching relationship between the preceding state expression and the following state expression, and generate multiple state switching relationships; The state transition relationship is used to record the transition between equipment state expressions perceived by the inspection robot during continuous inspection, so as to reflect the development process of equipment state from one state expression to another. By establishing the state transition relationship, the actual transition path between equipment states can be identified, providing a basis for subsequent state round-trip relationship identification, state swing zone generation and state swing spectrum construction. Multiple state transition relationships are aggregated to generate a set of state transition relationships.
[0022] In this embodiment, the generation of the state swing region set specifically includes: Read multiple state transition relationships in the state transition relationship set, identify state transition relationships in which the preceding state expression and the following state expression are mutually reversed, determine them as state round-trip relationships, and generate a state round-trip relationship set; Based on the set of state round trip relationships, extract the state representation and state transition direction corresponding to each state round trip relationship, and generate multiple state round trip paths; The generation of the state round-trip path specifically includes: Based on multiple state round trip relationships in the state round trip relationship set, the corresponding state expressions and state transition directions are extracted respectively; the connection order between multiple state expressions is determined according to the state transition direction, and the multiple state expressions are associated according to the connection order to form a state round trip path; The state round trip path is used to record the round trip switching between the equipment state expressions perceived by the inspection robot dog during continuous inspection, so as to reflect whether there is repeated change in equipment state, and to provide a basis for subsequent identification of areas with repeated changes in equipment state and generation of state swing areas. Statistically analyze the state representation composition of each state round-trip path, group state round-trip paths containing the same state representation into the same type of state round-trip path, and generate state round-trip path classification results. State round trip path classification is used to centrally organize state round trip paths involving the same state expression in order to identify the state expressions that frequently appear during repeated changes in the device state; when multiple state round trip paths involve the same state expression, they are identified as state round trip paths of the same type; the state round trip path classification results are used to reflect the composition of state expressions involved in repeated changes in the device state, providing a basis for the subsequent generation of state swing zones. Based on the various types of state round-trip paths in the state round-trip path classification results, extract the state representations involved in each type of state round-trip path and generate multiple state swing regions; The generation of the state swing region specifically includes: The various types of state round trip paths in the state round trip path classification results are processed separately, the state expressions involved in each type of state round trip path are extracted, and the state expressions involved are summarized and organized to generate the corresponding state swing regions; The state swing region is used to record the state expressions involved when the inspection robot dog perceives repeated changes in the equipment state during continuous inspection, so as to reflect the situation where the equipment state frequently switches back and forth, and to provide a basis for the subsequent construction of the state swing spectrum. Multiple state swing regions are aggregated to generate a set of state swing regions.
[0023] In this embodiment, the construction of the state swing spectrum specifically includes: Read multiple state swing regions from the set of state swing regions, extract the state representations contained in each state swing region and the corresponding state transition relationships, and generate a set of state swing relationships. Based on the multiple state transition relationships in the state swing relationship set, the occurrence frequency of each state transition relationship is counted, and the corresponding state transition direction is extracted to generate state swing statistics results. The frequency of occurrence reflects the frequency of the corresponding state switching relationship within a continuous inspection cycle; the state switching direction reflects the development direction of the equipment state between different state expressions; together, they are used to characterize the state change pattern during the equipment state swing process. By combining the statistical results of state swing, we can identify the state return process formed by each state expression returning to the original state expression after one or more state transitions, count the corresponding number of returns and the return span, and generate the state return information. The status return information is used not only to record when the device status returns to its original state expression, but also to reflect whether the device status can be maintained in the current state expression. When the device status frequently returns to its original state expression, it indicates that the device status is fluctuating continuously between multiple state expressions and is difficult to maintain stably in a certain state expression. The number of returns reflects the persistence of the device status instability phenomenon, and the return span reflects the range of state changes involved in the device status instability process. The status return information can identify the process of the device status developing from stable to unstable, providing a basis for the inspection robot to detect changes in device status stability in advance and to construct the subsequent state swing spectrum. When the equipment status frequently returns to its original state, it indicates that the equipment status is difficult to maintain and that there is a continuous fluctuation in the equipment's operating status. This fluctuation reflects that the equipment is deviating from a stable operating state and may be in the process of adjusting its operating status or forming an anomaly. By identifying the equipment's unstable state, the inspection robot can detect changes in the equipment's operating status before the equipment forms a stable abnormal state, thereby achieving early detection of the equipment's state change process. Based on the statistical results of state swing and the state return status, the state switching relationship, state switching direction, occurrence frequency and state return status corresponding to the state expression in each state swing zone are collected and processed to generate multiple state swing results. State swing results are used to uniformly record the state switching relationships, state switching directions, occurrence counts, and state return status within the same state swing zone, reflecting the state changes and return status of the equipment within that state swing zone. By uniformly organizing multiple state change information, a complete state swing situation for the corresponding state swing zone can be formed, providing a basis for subsequent state swing spectrum construction. Since the equipment state instability process usually manifests as continuous switching and repeated return between multiple state expressions, it is difficult to fully reflect the equipment state changes based on a single state switching relationship. By uniformly recording the state switching relationships, state switching directions, occurrence counts, and state return status, a complete state swing situation for the corresponding state swing zone can be formed, thereby reflecting the stability of the equipment state within that state swing zone. Multiple state swing results are aggregated according to their corresponding state swing regions to generate a state swing spectrum. The state swing spectrum is used to uniformly express the state swing situation corresponding to multiple state swing zones of the equipment. By comprehensively reflecting the state switching and state return situations between different state expressions of the equipment, it forms an expression of the overall state stability of the equipment, providing a basis for subsequent analysis of changes in equipment state stability.
[0024] In this embodiment, the generation of the state toughness distribution change result specifically includes: Extract the set of state representations and the set of state switching relationships corresponding to each state swing zone from the state swing spectrum, and sort the state switching occurrence process in the set of state switching relationships according to the inspection time order to obtain the state switching sequence. Based on the state expression in the state switching sequence, the time interval of its repeated occurrence in different inspection cycles is statistically analyzed, and the state dwell time sequence is formed according to the chronological relationship. At the same time, the state dwell time information in each inspection cycle is summarized. The generation of state dwell time series specifically includes: The time index is extracted from the two adjacent occurrence positions of each state expression in the state switching sequence to obtain the time distribution position of the state expression in different inspection cycles. The time distribution positions are sorted according to the time sequence to form the state occurrence interval sequence, and the state retention time sequence is generated accordingly. State retention time series is used to characterize the time interval variation of the same state expression recurring in consecutive inspection cycles. It is obtained by sequentially arranging the time differences between two adjacent occurrences of the state expression in different inspection cycles. It is used to characterize the recurring rhythmic features of the state expression, thereby transforming the discrete occurrence behavior of the state expression into a time-regular sequence structure, and providing a time rhythmic basis for subsequent calculation of state retention intensity and construction of state toughness potential field. The generation of status dwell time information specifically includes: The interval identification process is performed on the continuous occurrence intervals corresponding to each state expression in the state switching sequence to determine the continuous existence segment of each state expression in a single inspection cycle. The duration of the continuous existence segment is calculated to obtain the state dwell time. The state dwell time in all inspection cycles is summarized to generate state dwell time information. State dwell time information is used to characterize the set of time lengths during which the same state expression is maintained within a single inspection cycle. It is used to reflect the dwell stability of the state within a local time interval. It is obtained by quantitative analysis of the dwell time of the state expression within a single inspection cycle. It is used to distinguish between instantaneous fluctuating states and continuously stable states, and to provide basic stability attribute inputs for the subsequent construction of state resilience potential field. By combining the state dwell time series and state dwell duration information, the state dwell intensity of each state expression is calculated. The state dwell intensity is determined by the proportional relationship between the state dwell duration and the time interval between two adjacent occurrences, and the state intensity set is obtained by summarizing them. The generation of the state dwell intensity set specifically includes: The time interval sequence of each state expression is read from the state dwell time series and arranged in chronological order to form a state interval sequence group. The dwell time data of the same state expression in each inspection cycle is extracted from the state dwell time information and arranged in the order of the inspection cycle to form a state dwell time sequence group. For each state expression, the dwell time in the corresponding state dwell time sequence group is matched and calculated with the time interval between two adjacent occurrences in the state interval sequence group to generate a state ratio sequence. The state ratio sequence is numerically normalized to generate a state dwell intensity value. The state dwell intensity values corresponding to all state expressions are summarized to form a state dwell intensity set. The state dwell intensity set is used to characterize the distribution of the continuous dwell capacity of each state expression in different inspection cycles. It is obtained by calculating the ratio of the state dwell time to the time interval corresponding to each state expression. It is used to transform the temporal behavior of the state expression into a comparable stability quantification index and serve as the node attribute input parameter in the process of constructing the state resilience potential field to support the subsequent state resilience distribution analysis and state stability assessment. The state representation is mapped to state nodes, and the state retention strength is assigned to the corresponding state node attribute. At the same time, the state switching relationship is transformed into a structural association relationship between state nodes, forming a state resilience potential field structure. The generation of state-flexible potential field structures specifically includes: The process involves reading the set of state expressions and the set of state transition relationships, arranging the state expressions according to the inspection time order to form a state expression sorting sequence; structurally locating the state transition relationships between adjacent state expressions in the sorting sequence to determine the connection correspondence between each state expression, thus forming a state structure association relationship; extracting the retention intensity value corresponding to each state expression from the state retention intensity set, and matching the retention intensity value with the corresponding state expression to form a state attribute mapping sequence; merging the state structure association sequence and the state attribute mapping sequence to form a unified structural expression that includes state expressions, state attributes, and state structure association relationships; and constructing a state resilience potential field structure based on the unified structural expression, where state expressions serve as state nodes, state retention intensity serves as state node attributes, and state structure association relationships serve as the connection relationships between state nodes. The state resilience potential field structure is used to characterize the dynamic stability constraint distribution formed by the state expression based on the time dwell behavior and state switching behavior during the inspection process. The state stability response changes are triggered by the switching behavior between state expressions, and the constraint strength of the switching behavior is dynamically adjusted by combining the continuous dwell ability of the state expression, thereby forming a structured representation model for describing the state stability evolution constraint relationship. State transition relationships are used to characterize the dynamic transformation behavior between state expressions, while structural association relationships are used to characterize the state organization structure built based on state transition relationships, and to describe the overall structured connection form between state expressions; Periodic consistency correction is performed on the structural correlation in the state-resilient potential field, and the state node attributes are adjusted and updated based on the return count and return span of each state expression to obtain the updated state-resilient potential field structure. The generation of the updated state-resilient potential structure specifically includes: The state transition relationships in the state resilience potential field are grouped according to the inspection cycle. The difference in the number of times the same state expression occurs in adjacent inspection cycles is statistically analyzed, and this difference is compared with the average number of times the state transition relationship occurs in adjacent cycles to generate a state transition intensity change value. The number of times a state expression returns to its own state during continuous state transitions is cumulatively statistically analyzed to generate a return count statistic. The number of state transition steps experienced by a state expression each time it transitions from its current state to another state and then returns to that state is recorded sequentially, and the summation and average of all recorded steps are calculated to generate a return span value. Based on the return count statistic and the return span value, the state retention intensity is numerically corrected, where the return count statistic is used as... The state retention intensity is repeatedly regressed and corrected. The returned span value is used to correct the fluctuation range of the state retention intensity, generating corrected state node attribute values. The specific calculation method is as follows: the return count statistics are multiplied by the state retention intensity to obtain the regression influence, and the regression influence is summed with the state retention intensity to generate the regression update value; the returned span value is divided by the state retention intensity to obtain the fluctuation coefficient, and the state retention intensity is multiplied by the fluctuation coefficient to generate the fluctuation update value; the regression update value and the fluctuation update value are summed to obtain the corrected state node attribute values; the state switching intensity change value and the corrected state node attribute values are synchronously written into the state resilience potential field structure to obtain the updated state resilience potential field structure. The updated state resilience potential field structure is used to characterize the structured distribution result of state expression based on state switching behavior and state stagnation characteristics under cross-inspection cycle conditions. It is formed by periodic consistency correction of state switching relationship and updating and adjusting state node attributes. It is used to unify the state structure expression under different inspection cycles and serve as the basic structural carrier for state resilience distribution change analysis and state stability assessment. This calculation method is adopted because state representation involves two opposing mechanisms during state transitions: repetitive regression and fluctuating diffusion. The return frequency statistics characterize the frequency with which a state representation repeatedly returns to its own state during multiple state transitions. This frequency is positively correlated with state stability. Therefore, a multiplicative approach is used to enhance the mapping of state retention intensity, thereby strengthening the contribution of repetitive regression to stability. The return span value characterizes the number of state transition steps a state representation takes from leaving its current state to returning to it. This number of steps reflects the degree of state deviation diffusion and is negatively correlated with state stability. Therefore, an inverse proportional approach is used to constrain the mapping of state retention intensity, thereby reducing the impact of excessive state diffusion on stability. By fusing the enhanced mapping results and the constrained mapping results, a comprehensive characterization of state stability is achieved, resulting in the corrected state node attribute values. A cycle-by-cycle comparative analysis was conducted on the state toughness potential field structure corresponding to different inspection cycles to obtain the change range of attributes of each state node and the change range of structural correlation, thus forming the result of state toughness distribution change. The generation of the results of the change in state toughness distribution specifically includes: The process involves: reading the state resilience potential field structure corresponding to different inspection cycles; extracting the state node attribute values corresponding to the same state expression cycle by cycle; forming a state node attribute sequence according to the inspection cycle order; calculating the difference between adjacent inspection cycles for each state node attribute sequence to obtain the change in state node attributes; summarizing the attribute changes of all state expressions to form a set of state node attribute change amplitudes; reading the state switching relationships corresponding to different inspection cycles; statistically analyzing the occurrence of state switching relationships corresponding to the same state expression cycle by cycle; calculating the difference between adjacent inspection cycles to obtain the change in structural correlation relationships; summarizing these to form a set of structural correlation relationship change amplitudes; performing a corresponding matching calculation between the set of state node attribute change amplitudes and the set of structural correlation relationship change amplitudes to generate state resilience distribution change values; the corresponding matching calculation method is as follows: pairing the attribute changes of the same state expression with the structural correlation relationship changes item by item; and summarizing and arranging the state resilience distribution change values corresponding to all state expressions to generate the state resilience distribution change results. The results of the state toughness distribution change are used to characterize the comprehensive evolution of the state node attribute changes and state switching relationship changes in the state toughness potential field structure between different inspection cycles. It is obtained by correspondingly calculating the set of state node attribute change amplitudes and the set of structural correlation change amplitudes. It is used to reflect the stability change trend and structural evolution characteristics of the state expression under cross-cycle conditions, and to provide a basic analytical basis for equipment state stability assessment.
[0025] In this embodiment, the generation of intelligent sensing results specifically includes: Read the changes in state node attributes and structural relationships corresponding to each state swing zone in the state resilience distribution change results, and summarize and organize them according to the inspection cycle to form state change statistics. Based on the statistical results of state changes, the direction of change of state node attributes and the direction of change of structural association are calculated for each state swing zone. The direction of change of state node attributes is determined by the trend of change of state retention intensity, and the direction of change of structural association is determined by the trend of change of state switching relationship. The generation of the direction of change of state node attributes and the direction of change of structural association specifically includes: The changes in state node attributes within the same state swing zone in the state change statistics are sorted periodically to form a state node attribute change sequence. The differences between state node attribute values in adjacent periods within this sequence are calculated to obtain the state node attribute change difference. The direction of state node attribute change is determined based on the sign of the difference: a difference greater than zero indicates an upward trend, a difference less than zero indicates a downward trend, and a difference equal to zero indicates a stable trend. Similarly, the changes in structural correlation are sorted according to the inspection cycle to form a structural correlation change sequence. The differences between the occurrences of structural correlations in adjacent periods within this sequence are calculated to obtain the structural correlation change difference. The direction of structural correlation change is determined based on the sign of the difference: a difference greater than zero indicates an strengthening trend, a difference less than zero indicates a weakening trend, and a difference equal to zero indicates a stable trend. Based on the degree of consistency between the direction of change of state node attributes and the direction of change of structural association, stability division is performed on each state swing region to obtain the state stability division result. The generation of state stability partitioning results specifically includes: For the same state swing region in the state change statistics, the direction of change of state node attributes and the direction of change of structural association are matched state by state to form a set of direction matching pairs. Each state expression in the set of direction matching pairs is compared and judged, and the number of state expressions with the same direction of change of state node attributes and structural association is counted. The total number of state expressions is counted, and the ratio of the same number to the total number is calculated to obtain the direction consistency ratio. The direction consistency ratios corresponding to all state swing regions are sorted to generate a direction consistency ratio sequence. The ratios corresponding to each state swing region in the direction consistency ratio sequence are compared one by one. Based on whether the ratio is greater than the median of the direction consistency ratios of all state swing regions, the state swing regions are divided into a first state swing region set and a second state swing region set. The first and second state swing region sets are output as the state stability partitioning results. The first set of state swing regions is used to characterize the set of state swing regions where the direction of change of state node attributes and the direction of change of structural relationship are consistent. It is used to reflect the region where state stability and structural evolution change synchronously, thus serving as the stability support region in the determination of state stability. The second set of state swing regions is used to characterize the set of state swing regions where the direction of change of state node attributes and the direction of change of structural relationship are inconsistent. It is used to reflect the region where state stability and structural evolution deviate, serving as the fluctuation or instability analysis region in the determination of state stability. Based on the state stability classification results, different state swing zones are merged and analyzed to form the overall state stability assessment results of the equipment. Based on the overall stability assessment results of the equipment, intelligent sensing output information is generated and output to the maintenance terminal of the inspection robot dog.
[0026] Example 1: To verify the feasibility of this invention in practice, it was applied to an equipment inspection scenario in an industrial park. The inspection covered approximately 860 key pieces of equipment, including distribution cabinets, transformers, fans, and motors. The inspection robot performed periodic inspections of the equipment according to a fixed inspection route, completing three rounds of full-coverage inspections daily, with each round lasting approximately 2.5 hours. During equipment operation, multimodal data was generated simultaneously, including images, infrared thermal imaging, vibration, and acoustic signals, with an overall data volume of approximately 18 GB / day.
[0027] During the inspection, different equipment may exhibit abnormal temperature rise, increased vibration, and abnormal noise under the influence of load changes and ambient temperature fluctuations. For example, during the load increase phase of a certain motor, its infrared temperature gradually rose from 52℃ to 67℃, while the vibration spectrum energy increased by about 35% in the 120Hz–180Hz range, and the high-frequency band of the acoustic signal showed a significant increase.
[0028] Based on the above data, time synchronization processing is performed on different modal signals to unify the image features, thermal features, vibration features, and acoustic features within the same time window, and combine them into state expression units. Based on the continuous inspection sequence, approximately 1200 state expressions are generated per day for each device to describe the continuous change trajectory of its operating status.
[0029] As the state sequence progresses, the changing relationships between adjacent states are gradually extracted and formed into state transition records. In actual operating data, approximately 48,000 state transition relationships contain repetitive structures. For example, during a slight overload phase, the same equipment often exhibits a repetitive switching pattern of "normal state → slight temperature rise → normal state." In these repetitive switching structures, some state expressions exhibit a backtracking phenomenon, meaning that the state returns to its original state after experiencing multiple intermediate states. Taking a certain transformer as an example, its temperature rise state exhibits a backtracking phenomenon on average once every 6 switching steps in different inspection cycles, with approximately 14 backtracking events per day. This behavior is particularly evident during periods of equipment load fluctuation.
[0030] Based on the return structure, after classifying paths with similar state compositions, approximately 320 types of state swing zones can be obtained. Among them, about 65 types are concentrated in the region where temperature rise and vibration change synchronously. These regions often correspond to the alternating operation of equipment under long-term light load and intermittent impact load.
[0031] Further statistical analysis was conducted on the frequency of state transitions within each swing zone, and a state swing spectrum was constructed by combining the behavioral characteristics of the state returning to its original state. For example, during 7 consecutive days of operation, the frequency of the "high temperature-normal-high temperature" cycle pattern of a certain distribution cabinet increased from 11 times per day to 19 times per day, while the return path span gradually increased, indicating that the range of state fluctuations expanded.
[0032] In the cross-period analysis, mapping the state representation to the state resilience potential field structure revealed significant differences among different devices within the continuous inspection cycle. For example, the state retention intensity of a certain wind turbine decreased from 0.55 to 0.31 within 3 days, while the structural switching frequency increased by approximately 46%, exhibiting a clear unstable trend. In contrast, another motor, during the load stabilization phase, saw its state retention intensity increase from 0.42 to 0.67, with a corresponding gradual decrease in the structural switching frequency, indicating a trend towards stability.
[0033] By introducing correction mechanisms for the number of return trips and the return span, the impact of continuous return trips on stability can be further amplified, while the interference of long-path fluctuations on state stability can be weakened. For example, when a device's number of return trips increases from 8 to 19 during high-load operation, the stability index increases by approximately 21%; however, when its return span increases from 4.2 to 9.8, the fluctuation correction term decreases by approximately 18%, ultimately resulting in a stability evaluation result that more closely reflects the actual operating state.
[0034] Finally, by comparing the consistency differences in different state swing zones, the overall stability distribution of the equipment is obtained. In practical applications, this method can advance the abnormal state identification time by approximately 1.8 to 3.2 hours, while reducing the false alarm rate from approximately 12.4% to approximately 5.1%, demonstrating stronger stability judgment capabilities under complex operating conditions.
[0035] Table 1 Comparison of Equipment Status Change Sensing Performance
[0036] As can be seen from Table 1, the method of the present invention has a stronger cross-cycle expression capability in terms of equipment state evolution analysis compared with traditional threshold detection methods and traditional multimodal feature fusion methods. Its performance improvement does not come from the enhancement of a single feature, but from the unified modeling mechanism of state resilience potential field on state dwell behavior, switching behavior and structural correlation.
[0037] The anomaly detection lead time reaches 1.9 to 2.8 hours, which is an improvement of about 0.6 to 1.3 hours compared with traditional methods. This is because traditional methods mainly rely on instantaneous feature changes for trigger judgment, while this invention constructs a continuous state evolution link through a set of state switching relationships and classifies repeated switching behaviors in combination with state swing zones. This allows the anomaly evolution process to gradually emerge in the process of multiple state return accumulation, so that the anomaly trend can be captured in advance before obvious feature mutations occur.
[0038] The accuracy of state stability recognition reaches 0.88, an improvement of approximately 0.07 compared to traditional multimodal fusion methods. This improvement stems from the introduction of state dwell intensity as a state node attribute, which eliminates the reliance on single-frame or single-cycle features for state stability recognition. Instead, it characterizes cross-cycle dwell capability changes through state dwell time series and dwell duration information, thereby enhancing the ability to recognize slowly evolving states. The false alarm rate of this invention is reduced to 0.06, a decrease of approximately 0.04 to 0.08 compared to traditional methods. The key reason for this is the introduction of structural correlation constraints into the state resilience potential field structure, ensuring that non-continuous state switching caused by short-term fluctuations does not affect the overall stability judgment. Simultaneously, the state dwell intensity is modified by combining the number of returns and the return span, distinguishing between occasional and continuous returns in terms of influence weight, thereby reducing the probability of false positives.
[0039] Regarding the false negative rate, this invention reduces it to 0.07, a decrease of approximately 0.05 to 0.11 compared to traditional methods. This is because the state swing spectrum structurally aggregates the state return paths, enabling weak anomaly return behaviors that were originally scattered across different time segments to be uniformly expressed and accumulated for analysis. This improves the ability to capture progressive anomaly evolution paths, achieving a consistency recognition rate of 0.86, a significant improvement compared to traditional methods. The core reason for this is the introduction of a dual-dimensional determination mechanism that considers the direction of change in state node attributes and the direction of change in structural relationships. This allows state changes to be characterized simultaneously from both the intensity and structural dimensions, and the consistency comparison identifies cooperative or divergent relationships, thereby improving the ability to distinguish complex state evolution patterns.
[0040] Furthermore, in the results of changes in state resilience distribution, the comparison of the state resilience potential field structure across cycles allows the differences in state evolution between different inspection cycles to be expressed in a unified structural form. The number of returns is used to characterize the frequency of repeated state dwell times, and the return span is used to characterize the extent of the state return path expansion. Together, these two factors make the state dwell intensity correction results closer to the dynamic fluctuation characteristics in actual operation, thereby improving the reliability and robustness of the equipment state stability assessment results.
[0041] In summary, the method of this invention, through a continuous evolution path of "state switching relationship modeling, state swing zone aggregation, state swing spectrum construction, state resilience potential field mapping, and cross-cycle consistency analysis", enables equipment state analysis to shift from single-point judgment to cross-cycle structured evolution analysis, exhibiting stronger state identification and anomaly detection capabilities under complex operating conditions.
[0042] The above are merely preferred embodiments of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.
Claims
1. A method for intelligent perception in a patrol robot dog based on multimodal fusion, characterized in that, Includes the following steps: The inspection robot dog collects equipment identification data, visible light image data, infrared thermal imaging data, vibration data, and acoustic data during continuous inspection cycles, and preprocesses them to generate a standardized inspection dataset. Multimodal state features are extracted from a standardized inspection dataset, and a state representation sequence is constructed based on each multimodal state feature. Extract the temporal transition between state expressions according to the time sequence corresponding to the continuous inspection cycle, and generate a set of state switching relationships; Based on the set of state transition relationships, identify the state round-trip paths formed between state expressions, and aggregate them according to the correlation between multiple state round-trip paths to generate a set of state swing zones; A state swing spectrum is constructed based on the state switching relationships, state switching directions, and state return situations among the state expressions within each state swing region. Structural constraint evolution analysis of the state sway spectrum is performed based on the state toughness potential field to generate the results of state toughness distribution change. Based on the changes in the state resilience distribution, the system generates equipment state stability assessment results and outputs intelligent sensing results.
2. The intelligent perception method for inspection robot dogs based on multimodal fusion according to claim 1, characterized in that, The equipment identification data includes the equipment number, equipment type, and inspection time; the visible light image data includes images of the equipment surface; the infrared thermal imaging data includes thermal images of the equipment; the vibration data includes equipment vibration signals; the acoustic data includes equipment operating sound signals; and the preprocessing specifically includes time alignment, missing data completion, anomaly removal, and standardization.
3. The intelligent perception method for inspection robot dogs based on multimodal fusion according to claim 1, characterized in that, The generation of the state representation sequence specifically includes: Read the standardized inspection dataset, extract visual state features from visible light image data, thermal state features from infrared thermal imaging data, vibration state features from vibration data, and acoustic state features from acoustic data to generate a multimodal state feature set. Based on the inspection time, read the multimodal state feature set, and combine the visual state features, thermal state features, vibration state features and acoustic state features corresponding to the same inspection time to generate a state expression set. Extract the inspection time corresponding to each state expression in the state expression set, and sort the multiple state expressions according to the inspection time order to generate the state expression arrangement result; Based on the arrangement of state expressions, establish the sequential order of multiple state expressions to generate a state expression sequence.
4. The intelligent perception method for inspection robot dogs based on multimodal fusion according to claim 1, characterized in that, The generation of the state transition relationship set specifically includes: Based on the order of the state representation sequence, adjacent state representations are extracted sequentially to generate multiple state transition units; Read the preceding and following state expressions in each state transition unit, extract the state transition process from the preceding state expression to the following state expression, and generate a set of state transition records. Perform a classification and merging process on multiple state transition records in the state transition record set, grouping state transition records with the same preceding and subsequent state expressions into the same type of state transition records, and generating state transition statistics results; Based on the various state transition records and their occurrence frequency in the state transition statistics, establish the state switching relationship between the preceding state expression and the following state expression, and generate multiple state switching relationships; Multiple state transition relationships are aggregated to generate a set of state transition relationships.
5. The intelligent perception method for inspection robot dogs based on multimodal fusion according to claim 1, characterized in that, The generation of the set of state swing regions specifically includes: Read multiple state transition relationships in the state transition relationship set, identify state transition relationships in which the preceding state expression and the following state expression are mutually reversed, determine them as state round-trip relationships, and generate a state round-trip relationship set; Based on the set of state round trip relationships, extract the state representation and state transition direction corresponding to each state round trip relationship, and generate multiple state round trip paths; Statistically analyze the state representation composition of each state round-trip path, group state round-trip paths containing the same state representation into the same type of state round-trip path, and generate state round-trip path classification results. Based on the various types of state round-trip paths in the state round-trip path classification results, extract the state representations involved in each type of state round-trip path and generate multiple state swing regions; Multiple state swing regions are aggregated to generate a set of state swing regions.
6. The intelligent perception method for inspection robot dogs based on multimodal fusion according to claim 1, characterized in that, The construction of the state swing spectrum specifically includes: Read multiple state swing regions from the set of state swing regions, extract the state representations contained in each state swing region and the corresponding state transition relationships, and generate a set of state swing relationships. Based on the multiple state transition relationships in the state swing relationship set, the occurrence frequency of each state transition relationship is counted, and the corresponding state transition direction is extracted to generate state swing statistics results. By combining the statistical results of state swing, we can identify the state return process formed by each state expression returning to the original state expression after one or more state transitions, count the corresponding number of returns and the return span, and generate the state return information. Based on the statistical results of state swing and the state return status, the state switching relationship, state switching direction, occurrence frequency and state return status corresponding to the state expression in each state swing zone are collected and processed to generate multiple state swing results. Multiple state swing results are aggregated according to their corresponding state swing regions to generate a state swing spectrum.
7. The intelligent perception method for inspection robot dogs based on multimodal fusion according to claim 1, characterized in that, The generation of the state toughness distribution change results specifically includes: Extract the set of state representations and the set of state switching relationships corresponding to each state swing zone from the state swing spectrum, and sort the state switching occurrence process in the set of state switching relationships according to the inspection time order to obtain the state switching sequence. Based on the state expression in the state switching sequence, the time interval of its repeated occurrence in different inspection cycles is statistically analyzed, and the state dwell time sequence is formed according to the chronological relationship. At the same time, the state dwell time information in each inspection cycle is summarized. By combining the state dwell time series and state dwell duration information, the state dwell intensity of each state expression is calculated. The state dwell intensity is determined by the proportional relationship between the state dwell duration and the time interval between two adjacent occurrences, and the state intensity set is obtained by summarizing them. The state representation is mapped to state nodes, and the state retention strength is assigned to the corresponding state node attribute. At the same time, the state switching relationship is transformed into a structural association relationship between state nodes, forming a state resilience potential field structure. Periodic consistency correction is performed on the structural correlation in the state-resilient potential field, and the state node attributes are adjusted and updated based on the return count and return span of each state expression to obtain the updated state-resilient potential field structure. A cycle-by-cycle comparative analysis was conducted on the state toughness potential field structure corresponding to different inspection cycles to obtain the change range of attributes of each state node and the change range of structural correlation, thus forming the result of state toughness distribution change.
8. The intelligent perception method for inspection robot dogs based on multimodal fusion according to claim 1, characterized in that, The generation of the intelligent perception results specifically includes: Read the changes in state node attributes and structural relationships corresponding to each state swing zone in the state resilience distribution change results, and summarize and organize them according to the inspection cycle to form state change statistics. Based on the statistical results of state changes, the direction of change of state node attributes and the direction of change of structural association are calculated for each state swing zone. The direction of change of state node attributes is determined by the trend of change of state retention intensity, and the direction of change of structural association is determined by the trend of change of state switching relationship. Based on the degree of consistency between the direction of change of state node attributes and the direction of change of structural association, stability division is performed on each state swing region to obtain the state stability division result. Based on the state stability classification results, different state swing zones are merged and analyzed to form the overall state stability assessment results of the equipment. Based on the overall stability assessment results of the equipment, intelligent sensing output information is generated and output to the maintenance terminal of the inspection robot dog.
