Intelligent Inspection Method for Quadruped Robots Based on Multimodal Dynamic Fusion

Abstract

This application discloses an intelligent inspection method for quadruped robots based on multimodal dynamic fusion, which improves the reliability of inspection results. The method includes: collecting raw multimodal inspection data using a sensor array mounted on the quadruped robot; extracting temporal features and dynamically adjusting weights from the multimodal inspection data using a long short-term memory network to obtain target multimodal inspection data; fusing the target multimodal inspection data to generate fused data; and generating inspection results based on the fused data.

Description

Technical Field

[0001] This application relates to the field of intelligent robot technology, and in particular to an intelligent inspection method for quadruped robots based on multimodal dynamic fusion. Background Technology

[0002] With the rapid development of industrial and internet technologies, the requirements for quadruped robots in the field of intelligent inspection are gradually increasing, especially in complex and ever-changing environments where they need to achieve efficient and accurate data collection and analysis. Multimodal inspection data fusion technology has been widely applied in the field of quadruped robot intelligent inspection. However, existing multimodal inspection data fusion methods typically use fixed weights to fuse data from different modalities, which is insufficient when dealing with complex and ever-changing inspection environments. For example, under poor lighting conditions, the quality of data collected by visual sensors will significantly decrease, but a fixed-weight fusion method may still assign a high weight to visual data, leading to deviations in the final fusion result and reduced accuracy of feature extraction. In high-temperature and high-humidity environments, the reliability of data from temperature and humidity sensors is affected. If fusion continues according to fixed weights, it may interfere with the judgment of the true state of the equipment, thus affecting the reliability of the quadruped robot's inspection results. Furthermore, with the widespread application of quadruped robots in industrial inspection, public safety, and emergency rescue, their flexibility and mobility enable them to enter densely populated or open environments to perform inspection tasks for extended periods and in all directions. However, the multimodal sensor data (such as images, temperature, humidity, vibration, etc.) generated during the inspection process must be transmitted in real time to the remote monitoring center via wireless link for subsequent feature extraction, fusion analysis, and fault determination. Because these environments often lack physical isolation protection, wireless data streams are highly vulnerable to eavesdropping, tampering, and replay attacks: Eavesdropping risk: Attackers can passively intercept data packets with insufficient encryption strength or improper key management, thereby recovering sensitive information; Tampering risk: Unauthenticated data packets can be tampered with and forwarded, leading to system misjudgments or incorrect control commands; Replay risk: Highly repetitive multimodal data packets (such as periodic frame sequences and fixed-format messages) are more easily injected by attackers, affecting the reliability and real-time performance of model decisions. Summary of the Invention

[0003] This application provides an intelligent inspection method for quadruped robots based on multimodal dynamic fusion, which is used to solve the technical problems mentioned in the background art above.

[0004] The present invention adopts the following technical solution.

[0005] The first aspect of this invention discloses an intelligent inspection method for quadruped robots based on multimodal dynamic fusion, comprising: collecting raw multimodal inspection data through a sensor array mounted on the quadruped robot; extracting temporal features and dynamically adjusting weights of the multimodal inspection data based on a long short-term memory network to obtain target multimodal inspection data; fusing the target multimodal inspection data to generate fused data; and generating inspection results based on the fused data.

[0006] The second aspect of this invention discloses a quadruped robot intelligent inspection system based on multimodal dynamic fusion, used to execute the method described in the first aspect, the system comprising: a data acquisition unit, a weight dynamic adjustment unit, a fusion processing unit, and a result classification unit;

[0007] The data acquisition unit is used to acquire raw multimodal inspection data through the sensor array mounted on the quadruped robot;

[0008] The weight dynamic adjustment unit is used to extract temporal features and dynamically adjust weights of multimodal inspection data based on the long short-term memory network to obtain target multimodal inspection data.

[0009] The fusion processing unit is used to fuse target multimodal inspection data to generate fused data.

[0010] The result classification unit is used to generate inspection results based on the fused data.

[0011] A third aspect of the present invention discloses a terminal, including a processor and a storage medium; characterized in that:

[0012] The storage medium is used to store instructions;

[0013] The processor is configured to operate according to the instructions to perform the steps of the method described in the first aspect.

[0014] A fourth aspect of the present invention discloses a computer-readable storage medium having a computer program stored thereon, characterized in that the program, when executed by a processor, implements the steps of the method described in the first aspect.

[0015] Compared with the prior art, this application has the following advantages:

[0016] (1) This invention deeply integrates the random activation mechanism of the neural network itself with traditional symmetric encryption algorithms to form a dual protection architecture of "dynamic encryption-neural perturbation". Specifically, each round of data encryption not only relies on a preset invertible matrix sequence (such as the round function of AES or ChaCha20), but also superimposes random activation values ​​and forgetting factors from the gating units in LSTM (or other RNN / CNN) networks. In this way, when an attacker attempts to crack the ciphertext, they not only need to brute-force the encryption algorithm, but also must completely reproduce the neural network topology, weights and instantaneous gating state at the corresponding time. Otherwise, even if they obtain the ciphertext and plaintext pair, they cannot deduce the encryption mapping relationship of the next round, thus minimizing the feasibility of known plaintext attacks and frequency analysis. More importantly, this invention eliminates the influence of the asynchronicity of the neural network output at different time steps on the consistency of the decision during the generation of the "transition result". Traditional RNNs often result in slightly different hidden states for the same input under varying iteration counts and computational latency, affecting subsequent inspection results. This invention, however, merges equivalent computational units and reconstructs time steps, and standardizes random activations before encryption. This allows the remote end to directly reconstruct the target features consistent with the local one without rerunning the entire neural network after decrypting the "transitional result." This mechanism completely eliminates inconsistencies in decisions caused by temporal drift or differences in execution environments. Through these technical means, this invention significantly improves security while ensuring end-to-end real-time encryption, and also considers the efficiency of computing resources and network bandwidth utilization, fully meeting the high reliability and low latency requirements of quadruped robots in various complex and variable inspection scenarios.

[0017] (2) First, the original multimodal inspection data is collected through the sensor array mounted on the quadruped robot; then, based on the long short-term memory network, the time-series features of the multimodal inspection data are extracted and the weights are dynamically adjusted to obtain the target multimodal inspection data; then, the target multimodal inspection data is fused to generate fused data; finally, the inspection results are generated based on the fused data. In this way, the time-series characteristics of each modal inspection data can be utilized to dynamically allocate weights according to the importance of each modal inspection data in different time periods, and the fused data of each modal inspection data after the weights are reallocated can be fused to generate fused data. This can combine the advantages of multiple modal inspection data, overcome the limitations of single modal inspection data, and improve the pertinence and effectiveness of data processing, making the inspection results of the quadruped robot more accurate and reliable. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] Figure 1 This is a schematic diagram of an embodiment of an intelligent inspection method for quadruped robots based on multimodal dynamic fusion in this application;

[0020] Figure 2 This is a schematic diagram of an embodiment of a quadruped robot intelligent inspection system based on multimodal dynamic fusion in this application.

[0021] Figure 3 This is an example diagram of the computational unit of the fully connected layer in this application.

[0022] Figure 4A This is a simplified example diagram of the LSTM network in this application.

[0023] Figure 4B for Figure 4A A simplified example diagram of an LSTM network after merging nodes to form computational iterative units.

[0024] Figure 5A This is a simplified example diagram of another LSTM network in this application.

[0025] Figure 5B for Figure 5A A simplified example diagram of an LSTM network after merging nodes to form computational iterative units.

[0026] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0027] In the following description, specific details such as particular system architectures and techniques are set forth for illustrative purposes and not for limitation, in order to provide a thorough understanding of the embodiments of this application. However, those skilled in the art will understand that this application may also be implemented in other embodiments without these specific details. In other instances, detailed descriptions of well-known systems, apparatuses, circuits, and methods have been omitted so as not to obscure the description of this application with unnecessary detail.

[0028] This application discloses an intelligent inspection method and system for quadruped robots based on multimodal dynamic fusion, which can improve the reliability of inspection results.

[0029] Please see Figure 1 As shown, one embodiment of the intelligent inspection method for quadruped robots based on multimodal dynamic fusion in this application includes:

[0030] 101. Collect raw multimodal inspection data through the sensor array mounted on the quadruped robot;

[0031] In this embodiment, the quadruped robot is equipped with a sensor array consisting of various sensors, including vision sensors, temperature sensors, gas sensors, humidity sensors, and vibration sensors, the specific types of which are not limited here. These sensors are installed in different parts of the quadruped robot to ensure the comprehensiveness and accuracy of data acquisition. The vision sensor can be installed on the head or top of the quadruped robot to capture environmental image data; the temperature and humidity sensors can be placed on the surface of the quadruped robot's shell near the inspection target to more directly perceive changes in ambient temperature and humidity; the gas sensor is responsible for real-time monitoring of the concentration of combustible gases, volatile organic compounds, etc. in the environment, and can immediately alert personnel or the system when abnormal concentrations are detected; the vibration sensor can be fixed on the bottom of the quadruped robot near the ground to detect vibration spectrum data during equipment operation. The data from different sensors complement each other, jointly providing multi-dimensional state perception and safety assurance for hazardous chemical scenarios. During the startup process of the quadruped robot, the initial settings of each sensor need to be completed first, including adjusting parameters such as sampling frequency and data format. Subsequently, the data acquisition task of the sensor array is initiated. During inspection, vision sensors continuously capture image information of the environment, temperature sensors record changes in ambient temperature, humidity sensors monitor air humidity levels, and vibration sensors acquire vibration data of the equipment's operating status in real time. During data acquisition, data from different sensors are integrated according to timestamps to generate a complete multimodal data stream. For example, when a vision sensor captures an image of the equipment's exterior at a certain moment, the temperature and humidity sensors simultaneously record the temperature and humidity values ​​at that moment, while the vibration sensor outputs the corresponding vibration spectrum data. This data, after being timestamped, forms raw multimodal data available for subsequent processing.

[0032] This invention relates to a quadruped robot primarily used for intelligent inspection of hazardous chemicals. In chemical plant tank areas, the robot patrols regularly, relying on gas sensors to detect gas leaks (such as a continuous increase in methane concentration); simultaneously, it uses visual sensors to capture images of liquid leaks at pipe joints or valves, visually locating the leak site based on the images; for environmental monitoring, temperature and humidity sensors collect real-time ambient climate data, indicating potential chemical reaction risks if the temperature is too high or humidity changes abruptly; abnormal vibration sensing can determine abnormal operation of pump stations or compressors, issuing early warnings. If any sensor data reaches a preset threshold, the system immediately triggers an alarm and records on-site video, achieving comprehensive leak detection, environmental monitoring, and anomaly warning.

[0033] Understandably, in different practical application scenarios, quadruped robots can be equipped with sensors with specific functions according to different needs. For example, when using quadruped robots for industrial equipment inspection, they can be equipped with vision sensors, infrared sensors, and sound sensors simultaneously. The vision sensor collects images of the operating status of the equipment under inspection, observing whether the components are loose or worn; the infrared sensor monitors the temperature of key parts of the equipment to prevent damage due to overheating; and the sound sensor collects the sound signals of the equipment during operation, analyzing sound characteristics to determine whether the equipment is operating normally. As another example, when using quadruped robots for power facility inspection, they can be equipped with vision sensors, infrared sensors, lidar sensors, and gas sensors simultaneously. The vision sensor on the quadruped robot collects images of the power equipment's appearance, such as whether transformers and insulators have cracks, damage, or discoloration on their surfaces; the infrared sensor detects the temperature of various parts of the equipment to determine if overheating is present; the lidar sensor measures the distance and spatial relationship between equipment to ensure correct installation and no collision risk; and the gas sensor monitors the concentration of gases such as SF6 around the power equipment to prevent gas leaks and safety accidents.

[0034] 102. Based on the Long Short-Term Memory Network, temporal feature extraction and dynamic weight adjustment are performed on multimodal inspection data to obtain target multimodal inspection data;

[0035] Long Short-Term Memory (LSTM) networks are a special type of recurrent neural network (RNN) designed to address the vanishing or exploding gradient problems faced by traditional RNNs when processing long sequences of data. Their core advantage lies in their ability to effectively capture long-term dependencies in sequential data, making them widely used in natural language processing, speech recognition, and time series prediction. In this embodiment, after collecting raw multimodal inspection data via a sensor array mounted on a quadruped robot, the collected raw multimodal inspection data is input into the LSTM network according to its time sequence. The LSTM network selectively forgets past information and retains important current information through the collaborative work of forget gates, input gates, and output gates, thereby extracting the temporal features of each modality in the raw multimodal inspection data. Simultaneously, based on the extracted temporal features, the weights of different modal inspection data are dynamically adjusted to generate the target multimodal inspection data. For example, when inspecting power equipment, if the trend of temperature data changes is highly correlated with equipment failure within a certain period of time, the Long Short-Term Memory (LSTM) network will automatically increase the weight of temperature data and correspondingly reduce the weight of other modal inspection data at that moment, in order to highlight important information and improve the relevance and effectiveness of data processing.

[0036] 103. The multimodal inspection data of the target is fused to generate fused data;

[0037] In this embodiment, after acquiring the target multimodal inspection data, the different modal inspection data within the target multimodal inspection data can first be formatted and time-aligned. For example, image data is converted to a fixed size and data type; numerical data is standardized in units and precision. Time alignment uses timestamp matching to ensure that each modal data corresponds one-to-one in the time dimension, guaranteeing the validity of the fusion result. Then, the different modal inspection data within the target multimodal inspection data are fused to obtain fused data. This fused data integrates the advantages of multiple modal inspection data, providing a more comprehensive and accurate reflection of the actual condition of the inspection target. For example, fusing visual image data, temperature data, and distance data allows for the acquisition of equipment appearance information from images, the determination of overheating based on temperature data, and the determination of spatial relationships of the equipment based on distance data, thus providing richer information for subsequent analysis.

[0038] 104. Generate inspection results based on the fused data and send them to the remote terminal.

[0039] Based on pre-set data analysis models and judgment rules, the system analyzes, processes, and compares the fused data to determine whether there are any abnormalities in the inspection targets and generates corresponding inspection result reports. For example, when the temperature value in the fused data exceeds a preset threshold, and the surface of the visual image display device shows abnormal phenomena such as discoloration, it is determined that the device may be faulty. The inspection result report records in detail the type, location, and severity of the abnormality so that staff can take appropriate maintenance measures in a timely manner.

[0040] Understandably, compared to step 102, step 104 has lower computational overhead and is mainly responsible for further optimizing and validating the fused data. The data analysis module can use either lightweight neural networks or classic classifiers such as SVM, using a preset rule base as the classification standard to judge the fused features and generate the final inspection results.

[0041] In this embodiment, raw multimodal inspection data is first collected using a sensor array mounted on a quadruped robot. Then, based on a long short-term memory network, temporal features are extracted and weights are dynamically adjusted from the multimodal inspection data to obtain target multimodal inspection data. Next, the target multimodal inspection data is fused to generate fused data. Finally, inspection results are generated based on the fused data. This approach leverages the time-series characteristics of each modal inspection data point, dynamically assigning weights according to the importance of each modal inspection data point in different time periods. The reweighted modal inspection data is then fused to generate fused data, thus combining the advantages of multiple modal inspection data points, overcoming the limitations of single-modal inspection data, and improving the targeting and effectiveness of data processing, making the quadruped robot's inspection results more accurate and reliable.

[0042] In some embodiments, step 102 may include: constructing a multi-layer long short-term memory network, wherein the input layer of the long short-term memory network receives the original multimodal inspection data; learning from historical inspection data through the hidden layers of the long short-term memory network to extract the temporal features of each original modal inspection data; calculating the weight distribution of each original modal inspection data based on the temporal features; and performing weighted processing on each original modal inspection data according to the weight distribution to generate target multimodal inspection data.

[0043] Specifically, the raw multimodal inspection data includes various single-modal inspection data such as visual image data, LiDAR point cloud data, inertial measurement unit data, and force and tactile sensor data. Before being input into the Long Short-Term Memory (LSTM) network, it needs to undergo preprocessing operations such as normalization to convert it into a numerical sequence in a unified format. The multi-layer LSM network can be composed of two or three hidden layers stacked together, each layer containing several units. Higher layers can capture more abstract temporal features. The hidden layers work collaboratively through forget gates, input gates, and output gates. The forget gate determines to discard irrelevant information from historical states, the input gate filters and updates the effective features of the current time step, and the output gate generates the output based on the current state and input. During the extraction process, single-modal temporal features can be extracted, such as the trajectory of objects moving in consecutive frames and the trend of illumination changes in the visual modality, and the temporal fluctuations of acceleration and angular velocity of a quadruped robot in the inertial sensor. Simultaneously, cross-modal temporal correlation features can also be extracted, such as analyzing the synchronization between the sparsity of the LiDAR point cloud and the blurriness of the visual image, and correlating the peak value of the motor current with the curve of joint angle changes. The bottom layer extracts local temporal features, while the top layer integrates these features to form a global temporal pattern. The temporal feature vectors output from the hidden layer are mapped to weight coefficients for each original modality of the inspection data through a fully connected layer and activation function, resulting in a weight distribution. These weights are dynamically adjusted based on different scenarios: normal inspection, complex environments, and fault warning scenarios. The weights of each original modality change accordingly to highlight key modalities. For example, in low light conditions, the weight of the visual sensor data may decrease, while the weights of the temperature and humidity sensors may increase. Finally, the original modality inspection data is weighted according to the weight coefficients to obtain the target multimodal inspection data. This allows for real-time dynamic adjustment of the weights of each modality based on different environments and task scenarios, avoiding the limitations of fixed-weight fusion, improving the robustness of the quadruped robot in complex scenarios, and enhancing inspection efficiency and quality.

[0044] In some embodiments, the original modal inspection data may include: infrared temperature curves, vibration spectra, and acoustic signals; then the weight distribution of the corresponding target multimodal inspection data can be: temperature weight 0.6, vibration weight 0.3, and acoustic weight 0.1, generating a 3D weighted feature vector [0.6T, 0.3V, 0.1S].

[0045] In some embodiments, step 103 may include: inputting target multimodal inspection data into a fully connected layer (FC layer); performing nonlinear mapping on the target multimodal data using a nonlinear activation function; and adding the nonlinearly mapped target multimodal data element by element to generate fused data.

[0046] Because LSTM networks inherently handle temporal dependencies and can capture long-short-term correlations, step 102 essentially replaces the pre-processor portion of the neural network—equivalent to the pre-processor portion of the fully connected (FC) layer—with an LSTM network. It should be understood that the FC layer in step 103 is not equivalent to a fully connected layer in an LSTM network; it should actually be a standard convolutional neural network, such as the FC layer of a CNN, to break down the inconsistency in feature dimensions of the multimodal inspection data and map them to the same feature space through linear transformation and activation functions. In essence, this FC layer is a lightweight to medium-weight matrix multiplication network.

[0047] In some embodiments, step 104 may include: generating inspection results by combining a preset rule base.

[0048] Specifically, this rule base is constructed based on different inspection scenarios and needs, combined with industry standards and expert experience. It includes the normal range of various inspection indicators and rules for judging anomalies. For example, in the scenario of power equipment inspection, the rule base sets normal thresholds for parameters such as temperature, current, and voltage of power equipment. When relevant parameters in the fused data exceed the threshold range, the rule base will determine that the equipment is abnormal and mark and explain it in the inspection results. Understandably, the remote terminal can ultimately present the inspection results to the user in an intuitive form such as reports and charts, making it convenient for the user to understand the inspection situation. For example, in the inspection of industrial plants, a quadruped robot collects images of the equipment's appearance through a camera and collects temperature data through a temperature sensor. Then, it judges whether the equipment has potential faults based on the rule base and generates a detailed inspection report to feed back to the staff. Through the above method, stable transmission of fused data by the quadruped robot and the generation of inspection results based on scientific rules can be achieved, effectively improving the efficiency of intelligent inspection and the reliability of the generated inspection results. It should be understood that step 104 is essentially adding a low-level reasoning layer on the basis of the preset rule base.

[0049] In some embodiments, because quadruped robots are adapted to various scenarios, which are often characterized by continuous inspections in densely populated or open environments, there is a risk that transmitted data may be eavesdropped on, tampered with, or replayed when transmitted over a wireless link. Therefore, those skilled in the art should employ lightweight, reversible encryption algorithms to provide end-to-end protection for all data streams to be transmitted (whether target multimodal inspection data or inspection results).

[0050] It should be understood that since quadruped robots need to continuously send data out in real time, conventional encryption methods are similar to lightweight encryption algorithms, such as AES and ChaCha20. Lightweight encryption algorithms essentially transform the data to be encrypted through a series of invertible matrix transformations to obtain the encrypted ciphertext. The decryption process involves sequentially applying the inverses of these matrices to recover the plaintext.

[0051] It is worth noting that multimodal inspection data or results often have a high degree of repetition—such as periodic frame sequences, similar environmental features, and fixed-format data packets—which can lead to predictable repeating blocks in the encrypted ciphertext pattern, thereby increasing the risk of frequency analysis and known-plaintext attacks; at the same time, repeated data streams are also more easily exploited by replay attacks.

[0052] In some embodiments, target multimodal inspection data can be sent to the remote terminal instead of the inspection results, and then the remote terminal executes steps 103 and 104. However, it should be noted that since steps 103 and 104 are lightweight algorithms, their essence can actually be traced back to... Figure 3 In the steps shown, the result of the calculation unit y depends on the weighted sum of calculation units x1, x2, and x3, where a1, a2, and a3 represent weights, and b represents the bias. Since the value of y is either 1 or 0, a truncation needs to be set, which is usually 0.5: when y is greater than 0.5, y is 1; otherwise, it is 0. This simple linear truncation operation may be less complex than matrix operations, which means that the increase in difficulty from target multimodal inspection data to inspection results may not even be linear.

[0053] Therefore, it is necessary to extract transition results from the complex LSTM network as much as possible to replace the target multimodal inspection data or inspection results. That is, the transition results are transmitted from the quadruped robot to the remote terminal; these transition results are obtained by the Long Short-Term Memory network through temporal feature extraction and dynamic weight adjustment of the multimodal inspection data. Correspondingly, at the remote terminal, based on the transition results, execution continues to obtain the target multimodal inspection data and further the inspection results. The key consideration is maintaining the matching of various parameters between the quadruped robot and the remote terminal, such as weights, biases, activation functions, etc.

[0054] It's important to note that LSTM networks differ from strictly feedforward networks (such as CNNs); in LSTM, each time step of the computational unit is independent. Figure 3For example, if the current time step x1=1, the LSTM network may determine the result beforehand, even if the inputs x2 and x3 are not ready. Assuming y=(0.8*x1+0.1*x2+0.1*x3>0.5)?1:0, it can be understood that regardless of the specific values ​​of x2 and x3, y will directly output 1.

[0055] by Figure 4A For example, x1 and x2 can represent the original multimodal inspection data, while y can represent the target multimodal inspection data. Assume that initially, the time step for both x1 and x2 is 20 ns, and that each calculation's time step is 1 ns. Also assume that x1 iterates 30 times in the first loop and x2 iterates 20 times in the second loop. Then, in the first iteration, the time steps for computational units 11 and 21 are both 21 ns; in the second iteration, their time steps are 25 ns and 24 ns respectively, and so on. In the last iteration, the time steps for nodes 14 and 22 are 20 + 4 * 30 = 140 ns and 20 + 3 * 20 = 80 ns respectively. Therefore, in long-term iterations, node 31 may only be affected by node 34, and not by node 14.

[0056] To eliminate this potential problem while maximizing the complexity of the transition result, the specific process for determining the transition result is as follows: First, in the Long Short-Term Memory network, computational units with iterative relationships need to be merged to form computational iteration units, such as nodes 100, 200, and 300 in the following text.

[0057] Specifically, in Figure 4A In the algorithm, nodes {11,12,13,14} can be merged into a single node 100 to form a computational iteration unit, while nodes {21,22,23} can be merged into a single node 200, and nodes {31,32,33,34} can be merged into a single node 300. The merged nodes inherit the direction of the edges from the other nodes, such as... Figure 4B As shown. For example, node 100 has two directed edges: x1->100 and 100->31.

[0058] Secondly, based on all the nodes after forming the computational iteration unit, for example... Figure 4B For all nodes, a traceable time step sequence is generated along the directed edges.

[0059] It should be noted that if there are multiple original multimodal inspection data or target multimodal inspection data, then it is necessary to additionally set a unique node pointing to all x or being pointed to by all y. For example... Figure 4B As shown, since there are multiple x (i.e., x1 and x2), a virtual x node needs to be set up to point to x1 and x2.

[0060] In the time step sequence, node x has a time step of 1, node x1 has a time step of 2, node 100 has a time step of 3, and so on, forming a time step sequence with node numbers {x, x1, 100, 300, 41, 51, y}. Since there are still other nodes not included in the time step sequence, backtracking is necessary, resulting in the time step sequence becoming {x, x1, 100, 300, 41, 51, y, y, 51, 52, 52, 41}. It should be noted that when each node appears a second time, it means that all the nodes it points to have appeared in the time step sequence; backtracking continues. This process is repeated for each node. Figure 4B For example, the traceable time step sequence is {x,x1,100,300,41,51,y,y,51,52,52,41,43,53,53,54,54,43,42,42,300,100,x1,x2,200,200,x2,x}.

[0061] Understandably, without merging computational units that have iterative relationships, it is impossible to obtain a time-step sequence (e.g., ...). Figure 4A (No time step sequence exists). The number of time step sequences can be multiple, but different time step sequences will yield the same result for the choice of the transition outcome described below.

[0062] Finally, the transition result is determined based on the time step sequence.

[0063] Specifically, the transition result is a subset of bridges for all first-level iteration results. A first-level iteration result refers to the result (i.e., directed edge) between the core computational iteration unit and its subordinate first-level nodes. A core computational iteration unit is one that contains no other internal computational iteration units but includes the entire output (i.e., node y), such as 300. Understandably, if it cannot contain the entire output (which is usually impossible), the transition result is empty. Subordinate first-level nodes refer to nodes that do not have other nested nodes between them and the core computational iteration unit. For example, node 51 has a nested node 41 between it and the core computational iteration unit; nodes 53 and 54 have a nested node 43 between them and the core computational iteration unit; and so on. Figure 4B In this context, the subordinate first-level nodes can only be 41, 42, and 43; therefore, all first-level iteration results are {300->41; 300->42; 300->43}. Figure 4B In the problem, the three edges 300->41, 300->42, and 300->43 form a bridge subset, defined as one that splits all nodes into two disconnected parts (left and right). It's easy to understand that a bridge subset might not exist. Therefore, the transition result might be empty.

[0064] Furthermore, to ensure the transition result is not empty as much as possible, a more preferred subset of bridges is provided. Here, a more preferred subset of bridges refers to a subset where both the left and right disconnected parts contain the core computational iteration unit, thereby greatly increasing the complexity of plaintext attacks. In some embodiments, if a computational iteration unit contains the core computational iteration unit and is exactly contained within a node of a single non-computational iteration unit, then the transition result is the result between the node of the single non-computational iteration unit and the computational iteration unit (i.e., a directed edge).

[0065] Understandably, self-containment means that the computational iteration unit can be the core computational iteration unit itself.

[0066] refer to Figure 5A and Figure 5B ,at this time, Figure 5B The traceable time step sequence is {x,x1,100,6,300,41,51,y,y,51,52,52,41,43,53,53,54,54,43,42,42,300,6,100,x1,x2,200,200,x2,x}. At this point, the computational iteration unit is exactly contained within a single non-computational iteration unit node (i.e., node 6), therefore, the transition result is 6->300.

[0067] Please see Figure 2 As shown, one embodiment of the quadruped robot intelligent inspection system based on multimodal dynamic fusion of this application includes:

[0068] The acquisition unit 201 is used to acquire raw multimodal inspection data through the sensor array mounted on the quadruped robot;

[0069] The weight dynamic adjustment unit 202 is used to extract temporal features and dynamically adjust weights of multimodal inspection data based on the long short-term memory network to obtain target multimodal inspection data.

[0070] The fusion processing unit 203 is used to fuse the multimodal inspection data of the target to generate fused data.

[0071] The result classification unit 204 is used to generate inspection results based on the fused data.

[0072] Furthermore, the system also includes a transmission unit and a transition result calculation unit;

[0073] The transmission unit is used to transmit the transition result from the quadruped robot end to the remote terminal; wherein, the transition result is the transition result obtained by the long short-term memory network extracting temporal features and dynamically adjusting weights of the multimodal inspection data;

[0074] The transition result calculation unit is used in the long short-term memory network to merge calculation units with iterative relationships to form a calculation iteration unit; based on all nodes after the calculation iteration unit is formed, a traceable time step sequence is generated along the directed edges; and based on the time step sequence, the transition result is determined.

[0075] As a carrier of multimodal intelligent inspection, quadruped robots possess high mobility and environmental adaptability in complex terrains, effectively replacing the limitations of traditional wheeled or tracked robots on irregular ground such as chemical plants and tank farms. The following discussion focuses on three aspects: mechanical structure, motion control, and environmental adaptation.

[0076] In this invention, each leg of the quadruped robot typically employs a three-degree-of-freedom design, including the thigh joint, knee joint, and ankle joint, enabling the robot to perform complex movements such as obstacle crossing and climbing. The thigh joint can withstand a torque of 500 Nm, the knee joint can withstand 300 Nm, and the ankle joint is designed with a thin servo motor and harmonic reducer, with a total stroke of ±45°.

[0077] In addition, to ensure stable walking on different terrains (such as concrete, gravel, and muddy surfaces), the system is designed with multiple gait modes, including quadrupedal diagonal gait, continuous gait, stride, and lateral gait. On flat ground, the quadrupedal diagonal gait (two feet moving in tandem, stride length 0.4m, cadence 2Hz) is typically preferred, while on loose or sloping terrain, the system switches to continuous gait (single support phase, stride length 0.2m, cadence 1Hz) to improve stability and ground contact area.

[0078] The system measures attitude information in real time using a chassis IMU (six-axis gyroscope + three-axis accelerometer) and combines this with foot tactile sensors to collect ground contact force (typical range 0–500 N) to perform ZMP dynamic balance calculations on the real-time gait. When the system detects that the robot's forward tilt angle exceeds 10° or that the foot contact force is uneven (the difference between the left and right sides exceeds 50N), it will immediately adjust the movement trajectory of each leg, shifting the center of gravity to a stable support surface to ensure that the robot does not tip over or become unstable.

[0079] In this way, the time series characteristics of each modal inspection data can be utilized to dynamically allocate weights according to the importance of each modal inspection data in different time periods. The reweighted modal inspection data can then be fused to generate fused data. This allows for the integration of the advantages of multiple modal inspection data, overcomes the limitations of single modal inspection data, and improves the targeting and effectiveness of data processing, making the inspection results of the quadruped robot more accurate and reliable.

[0080] In some embodiments, the weight dynamic adjustment unit 202 may be specifically used to: construct a multi-layer long short-term memory network, wherein the input layer of the long short-term memory network receives the original multimodal inspection data; extract the temporal features of each original modal inspection data by learning from the historical inspection data through the hidden layer of the long short-term memory network; calculate the weight distribution of each original modal inspection data based on the temporal features; and perform weighted processing on each original modal inspection data according to the weight distribution to generate target multimodal inspection data.

[0081] In some embodiments, the fusion processing unit 203 may be specifically used to: input multimodal inspection data into a fully connected layer; perform nonlinear mapping on the target multimodal data using a nonlinear activation function; and add the nonlinearly mapped target multimodal data element by element to generate fused data.

[0082] In some embodiments, the result classification unit 204 may be specifically used to: combine a preset rule base and generate inspection results based on a lightweight algorithm, such as the SVM algorithm.

[0083] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0084] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0085] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0086] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0087] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

Claims

1. A quadruped robot intelligent inspection method based on multi-modal dynamic fusion, characterized in that, The system comprises: An array of sensors carried by a quadruped robot collects raw multi-modal inspection data; A long short-term memory network is used to extract time-series features from the raw multi-modal inspection data and dynamically adjust the weights, to obtain a transition result and transmit it to a remote terminal by the quadruped robot; The target multi-modal inspection data is obtained at the remote terminal through the transition result; The target multi-modal inspection data is fused to generate fusion data; An inspection result is generated based on the fusion data; The transition result specifically comprises: in the long short-term memory network, the calculation units with iterative relationships are merged to form a calculation iteration unit; based on all the nodes after the formation of the calculation iteration unit, a traceable time step sequence is generated along the directed edges; and based on the time step sequence, the transition result is determined; 2. The method according to claim 1, wherein, The transition result is a bridge subset of all first-level iteration results; the first-level iteration result refers to the result between the core calculation iteration unit and its subordinate first-level nodes; the core calculation iteration unit refers to a calculation iteration unit that does not contain other calculation iteration units but contains the entire output; the subordinate first-level nodes refer to nodes that do not have other nested nodes between them and the core calculation iteration unit; and the bridge subset is defined as a subset that exactly divides all nodes into two unconnected parts.

3. The method according to claim 1, wherein, The long short-term memory network is used to extract time-series features from the raw multi-modal inspection data and dynamically adjust the weights, specifically comprising: a multi-layer long short-term memory network is constructed, the input layer of the long short-term memory network receives the raw multi-modal inspection data; the time-series features of each raw multi-modal inspection data are extracted through the learning of the historical inspection data by the hidden layer of the long short-term memory network; the weight distribution of each raw multi-modal inspection data is calculated based on the time-series features; and each raw multi-modal inspection data is weighted based on the weight distribution.

4. A quadruped robot intelligent inspection system based on multi-modal dynamic fusion, for executing the method of any one of claims 1-3, characterized in that, If the calculation iteration unit contains the core calculation iteration unit and is contained by a single non-calculation iteration unit node, the transition result is the result between the single non-calculation iteration unit node and the calculation iteration unit; the core calculation iteration unit refers to a calculation iteration unit that does not contain other calculation iteration units but contains the entire output. The system comprises: a collection unit, a weight dynamic adjustment unit, a fusion processing unit, a result classification unit, and a transmission unit; The collection unit is used to collect raw multi-modal inspection data through an array of sensors carried by a quadruped robot; The weight dynamic adjustment unit is configured to perform time sequence feature extraction and weight dynamic adjustment on the original multi-modal inspection data based on the long short-term memory network to obtain a transition result; wherein the transition result is determined by: in the long short-term memory network, merging the calculation units having an iterative relationship to form a calculation iteration unit; based on all nodes after the calculation iteration unit is formed, generating a traceable time step sequence along the directed edge; based on the time step sequence, determining the transition result; the transition result is a bridge subset of all first-level iteration results; the first-level iteration result refers to the result between the core calculation iteration unit and its subordinate first-level nodes; wherein the core calculation iteration unit refers to a calculation iteration unit that does not have other calculation iteration units inside but contains the entire output, and the subordinate first-level nodes refer to nodes that do not have other nested nodes between them and the core calculation iteration unit, and the bridge subset is defined as splitting all nodes into two unconnected parts; The fusion processing unit is located at the remote terminal and is configured to obtain target multi-modal inspection data through the transition result; and perform fusion processing on the target multi-modal inspection data to generate fusion data; The result classification unit is configured to generate an inspection result based on the fusion data; The transmission unit is configured to transmit the transition result from the quadruped robot to the remote terminal.

5. A terminal comprising a processor and a storage medium; characterized in that: The storage medium is configured to store instructions; The processor is configured to operate according to the instructions to perform the steps of the method according to any one of claims 1-3.

6. A computer readable storage medium having stored thereon a computer program, characterized in that The program is executed by the processor to implement the steps of the method according to any one of claims 1-3.

Images