A fault analysis method, system, device and medium for a humanoid robot

Abstract

The application discloses a kind of humanoid robot fault analysis method, system, equipment and medium, wherein, the method obtains the multimodal time series data of humanoid robot within preset time before and after failure, and the fault description data of the humanoid robot, the multimodal time series data include joint time series data and sensing time series data;The multimodal time series data is input into the first model trained in visual conversion, and the visual representation output by the trained first model is obtained;The fault description data is input into the second model trained in text conversion, and the text representation output by the trained second model is obtained;The visual representation and the text representation are analyzed for fault, and the fault analysis result of the humanoid robot is obtained.The method can provide to improve the accuracy of humanoid robot fault analysis.The application relates to the technical field of intelligent robot.

Description

Technical Field

[0001] This invention relates to the field of intelligent robot technology, and in particular to a method, system, device and medium for fault analysis of humanoid robots. Background Technology

[0002] With the development of humanoid robot technology, its application scenarios have expanded from the industrial field (such as high-precision assembly and heavy-duty handling) to the home field (such as intelligent cleaning, companionship, and education). However, since the performance and health of humanoid robots often decline after long-term operation, fault analysis of humanoid robots has become one of the key focuses of relevant personnel.

[0003] Currently, related technologies are usually based on analyzing time-series data collected from humanoid robots in a single modality to determine the faults that occur in the humanoid robots. However, this method often has a high misjudgment rate and the accuracy of fault analysis is not satisfactory.

[0004] Therefore, the problems with the relevant technologies still need to be solved and optimized. Summary of the Invention

[0005] The purpose of this invention is to at least partially solve one of the technical problems existing in the related art.

[0006] Therefore, one objective of this invention is to provide a method, system, device, and medium for fault analysis of humanoid robots, wherein the method can improve the accuracy of fault analysis of humanoid robots.

[0007] To achieve the above-mentioned technical objectives, the technical solutions adopted in the embodiments of this application include:

[0008] In a first aspect, embodiments of this application provide a fault analysis method for a humanoid robot, including:

[0009] Acquire multimodal timing data of the humanoid robot within a preset time period before and after the failure, as well as the failure description data of the humanoid robot. The multimodal timing data includes joint timing data and sensor timing data.

[0010] The multimodal time-series data is input into the trained first model for visual transformation to obtain the visual representation output by the trained first model;

[0011] The fault description data is input into the trained second model for text conversion to obtain the text representation output by the trained second model;

[0012] Fault analysis is performed on the visual representation and the text representation to obtain the fault analysis results of the humanoid robot.

[0013] In addition, the method according to the above embodiments of this application may also have the following additional technical features:

[0014] Furthermore, in one embodiment of this application, the step of inputting the multimodal time-series data into a trained first model for visual transformation to obtain the visual representation output by the trained first model includes:

[0015] Obtain the working scene information of the humanoid robot;

[0016] The multimodal time-series data is subjected to channel-wise image conversion to obtain a first time-series image corresponding to the joint time-series data and a second time-series image corresponding to the sensing time-series data;

[0017] Based on the work scenario information, image features are extracted from the first time-series image and the second time-series image to obtain the visual representation.

[0018] Furthermore, in one embodiment of this application, the step of extracting image features from the first time-series image and the second time-series image based on the work scenario information to obtain the visual representation includes:

[0019] Based on the work scenario information, the first time-series image is segmented and encoded to obtain several first time-series vectors. Each first time-series vector records the feature vector and position encoding vector of the corresponding position of the first time-series image. The position encoding vector of each first time-series vector is different.

[0020] Based on the work scenario information, the second time-series image is segmented and encoded to obtain several second time-series vectors. Each second time-series vector records the feature vector and position encoding vector of the corresponding position of the second time-series image. The position encoding vector of each second time-series vector is different.

[0021] Spatiotemporal fusion feature extraction is performed on all the first temporal vectors and the second temporal vectors to obtain the visual representation.

[0022] Furthermore, in one embodiment of this application, the step of inputting the fault description data into a trained second model for text conversion to obtain the text representation output by the trained second model includes:

[0023] Obtain the working scene information and fault knowledge graph of the humanoid robot;

[0024] Based on the work scenario information, the fault description data is segmented into scenarios to obtain several text segmentation sequences;

[0025] Based on the fault knowledge graph, all the text segmentation sequences are encoded to obtain the text representation.

[0026] Furthermore, in one embodiment of this application, the step of text encoding all the text segmentation sequences according to the fault knowledge graph to obtain the text representation includes:

[0027] Obtain the scene weights corresponding to the work scene information;

[0028] Based on the fault knowledge graph, information encoding is performed on all the text segmentation sequences to obtain the hidden state corresponding to each text segmentation sequence;

[0029] Based on the scene weights, all the hidden states are weighted and fused to obtain the text representation.

[0030] Furthermore, in one embodiment of this application, the step of performing fault analysis on the visual representation and the text representation to obtain the fault analysis result of the humanoid robot includes:

[0031] The visual representation and the text representation are concatenated to obtain the concatenated features;

[0032] Self-attention fusion is performed on the spliced ​​features to obtain attention fusion features;

[0033] Feature analysis is performed on the attention fusion features to obtain the fault analysis results.

[0034] Furthermore, in one embodiment of this application, the method further includes:

[0035] Acquire historical multi-source fault data of humanoid robots;

[0036] Structured knowledge is extracted from the historical multi-source fault data to obtain a fault knowledge graph;

[0037] Based on the fault knowledge graph, the initialized target model is trained to obtain a trained target model, wherein the target model is either the first model or the second model.

[0038] Secondly, embodiments of this application provide a fault analysis system for a humanoid robot, comprising:

[0039] The first processing unit is used to acquire multimodal time-series data of the humanoid robot within a preset time before and after the fault, as well as fault description data of the humanoid robot. The multimodal time-series data includes joint time-series data and sensor time-series data.

[0040] The second processing unit is used to input the multimodal time series data into the trained first model for visual transformation to obtain the visual representation output by the trained first model.

[0041] The third processing unit is used to input the fault description data into the trained second model for text conversion to obtain the text representation output by the trained second model.

[0042] The fourth processing unit is used to perform fault analysis on the visual representation and the text representation to obtain the fault analysis results of the humanoid robot.

[0043] Thirdly, embodiments of this application also provide an electronic device, including:

[0044] At least one processor;

[0045] At least one memory for storing at least one program;

[0046] When the at least one program is executed by the at least one processor, the at least one processor performs the method described above.

[0047] Fourthly, embodiments of this application also provide a computer-readable storage medium storing a processor-executable program, which, when executed by the processor, is used to implement the above-described method.

[0048] The advantages and beneficial effects of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application:

[0049] This application discloses a method, system, device, and medium for fault analysis of a humanoid robot. The method acquires multimodal time-series data of the humanoid robot within a preset time period before and after a fault, as well as fault description data of the humanoid robot. The multimodal time-series data includes joint time-series data and sensor time-series data. The multimodal time-series data is input into a trained first model for visual transformation to obtain a visual representation output by the trained first model. The fault description data is input into a trained second model for text transformation to obtain a text representation output by the trained second model. Fault analysis is performed on the visual representation and the text representation to obtain the fault analysis result of the humanoid robot. This method, by performing visual transformation on the multimodal time-series data and text transformation on the fault modality data, and then performing fault analysis on the transformed visual and text representations, can utilize the correlation between different modal data for fault analysis, which helps reduce the misjudgment rate of humanoid robot fault analysis and thus effectively improves the accuracy of humanoid robot fault analysis. Attached Figure Description

[0050] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the following description is provided with accompanying drawings of the relevant technical solutions in the embodiments of this application or the prior art. It should be understood that the accompanying drawings described below are only for the purpose of clearly illustrating some embodiments of the technical solutions in this application. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0051] Figure 1 A flowchart illustrating a fault analysis method for a humanoid robot provided in an embodiment of this application;

[0052] Figure 2 A schematic diagram of the framework of a fault analysis system for a humanoid robot provided in an embodiment of this application;

[0053] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0054] The embodiments of this application are described in detail below. Examples of these embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and are only used to explain this application, and should not be construed as limiting this application. The step numbers in the following embodiments are set only for ease of explanation, and there is no limitation on the order between the steps. The execution order of each step in the embodiments can be adaptively adjusted according to the understanding of those skilled in the art.

[0055] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0056] Currently, relevant technologies typically analyze time-series data collected from humanoid robots in a single modality to determine the robot's faults. However, since fault-related data includes two core types of data: time-series data (such as joint motor current and real-time signals collected by sensors) and text data (such as user feedback "the robot is not talking" or maintenance personnel's record "the visual lens is blurry"), this approach ignores the correlation between data from different modalities (e.g., "no voice response" may correspond to "sound pickup intensity of 0" in the time-series data of the voice sensor), leading to a high rate of false alarms and unsatisfactory accuracy in fault analysis.

[0057] Furthermore, some related technologies rely on expert experience when analyzing humanoid robot faults (e.g., "a sudden 30% increase in motor current corresponds to a stall fault"). However, expert experience is difficult to reuse systematically, and the transmission and repeatability of fault knowledge for humanoid robots are difficult.

[0058] Furthermore, humanoid robots used in industrial settings prioritize "load capacity" and "motion accuracy" (faults are often due to joint overload or end effector failure), while those used in home settings prioritize "interaction experience" and "mobility" (faults are often due to voice recognition malfunction or chassis jamming). Related technologies have not optimized data collection (e.g., industrial applications are not adapted for high-frequency load data collection) and analysis logic for these different scenarios, resulting in unclear fault localization in industrial settings and low analysis efficiency in home settings.

[0059] It should be noted that the aforementioned related technologies are only used to assist in understanding the technical solutions of this application and do not mean that they belong to the publicly disclosed prior art.

[0060] In view of this, embodiments of this application provide a method, system, device, and medium for fault analysis of humanoid robots. The method performs visual transformation on multimodal time-series data and text transformation on fault modality data, and then performs fault analysis on the transformed visual and text representations. It can utilize the correlation between different modal data for fault analysis, which helps to reduce the misjudgment rate of humanoid robot fault analysis and thus effectively improves the accuracy of humanoid robot fault analysis.

[0061] Furthermore, this method extracts structured knowledge from the historical multi-source fault data of humanoid robots. It can extract the correlation between "component entities, fault types, signal features, and scene events" in the fault knowledge of humanoid robots. Then, it trains the first or second model based on the knowledge in the extracted fault knowledge graph. This can provide high-quality knowledge support for model training, thereby realizing the inheritance and repeatability of fault knowledge of humanoid robots.

[0062] Furthermore, this method performs visual transformation of multimodal time-series data through work scenario information and text transformation through work scenario information. This enables the first model and / or the second model to perform targeted analysis of humanoid robots based on the scene differences of humanoid robots, effectively improving the fault location accuracy of humanoid robots in industrial scenarios and improving the analysis efficiency of humanoid robots in home scenarios.

[0063] Reference Figure 1 In this embodiment of the application, a fault analysis method for a humanoid robot includes:

[0064] Step 110: Obtain multimodal timing data of the humanoid robot within a preset time before and after the fault, as well as fault description data of the humanoid robot. The multimodal timing data includes joint timing data and sensor timing data.

[0065] In this embodiment, the multimodal time series data can be data from a period of time before and after a malfunction occurs when a humanoid robot is working in a work scenario, such as an industrial scenario or a home scenario.

[0066] Specifically, if the work scenario is an industrial scenario, for example, if the fault is "the assembly joint of the right arm No. 3 of the humanoid robot is stuck", then the multimodal timing data can be the joint timing data and sensor timing data from 30 minutes before the fault to 10 minutes after the fault. The joint timing data can be collected through the robot controller's CAN bus based on the Nyquist sampling theorem, such as the current (unit: A), speed (unit: rpm), and torque (unit: N·m) of the motor of the right arm No. 3 joint; while the sensor timing data can be collected through the force sensor based on the Nyquist sampling theorem, and the deviation value of the joint motion trajectory of the right arm No. 3 joint can be obtained through the vision sensor.

[0067] Alternatively, if the work scenario is a home environment, for example, if the fault is "unresponsive voice interaction of the humanoid robot", then the multimodal timing data can be joint timing data and sensor timing data from 60 minutes before the fault to 20 minutes after the fault. The joint timing data can be data such as the rotational speed (unit: rpm) and steering angle (unit: °) of the humanoid robot chassis motor, which are collected based on the Nyquist sampling theorem; while the sensor timing data can be data such as the pickup intensity (unit: dB) of the voice sensor, the voice recognition success rate (unit: %), and the signal-to-noise ratio (unit: dB) of the microphone array, which are collected based on the Nyquist sampling theorem.

[0068] It is understandable that fault description data can be text data corresponding to multimodal time-series data. For example, in an industrial scenario, fault description data can be a text sequence [assembly, 50 parts, right arm joint 3, jamming, abnormal end-effector pressure, E02, overload]. This text sequence can be obtained by filtering redundant representations of the original text data and fault code text using the TF-IDF algorithm. The original text data can be a record from an industrial maintenance worker stating that "right arm joint 3 jammed when assembling the 50th part, briefly recovered after restarting, and the jamming worsened upon reassembly, with an abnormally high end-effector pressure." The fault code text can be a fault code "E02 - Joint motor overload" generated by the humanoid robot controller. The acquisition method for fault description data in a home scenario is similar to that in an industrial scenario and can be easily deduced, so it will not be elaborated further here.

[0069] Step 120: Input the multimodal time series data into the trained first model for visual transformation to obtain the visual representation output by the trained first model;

[0070] In this embodiment, the first model can be a neural network model based on the Transformer encoder architecture. Visual transformation can involve inputting multimodal temporal data into a trained first model, and then using the trained first model to convert the multimodal temporal data into a visual representation.

[0071] In some embodiments, the step of inputting the multimodal temporal data into a trained first model for visual transformation to obtain the visual representation output by the trained first model includes:

[0072] Obtain the working scene information of the humanoid robot;

[0073] The multimodal time-series data is subjected to channel-wise image conversion to obtain a first time-series image corresponding to the joint time-series data and a second time-series image corresponding to the sensing time-series data;

[0074] In this embodiment, the work scene information is used to indicate the work scene of the humanoid robot. Specifically, the work scene information can indicate industrial assembly scene, industrial handling scene, household cleaning scene, household interaction scene, etc. The multi-channel image conversion can be performed in parallel, converting the joint timing data and the sensor timing data separately. For example, in the aforementioned industrial scene of "right arm joint 3 jamming", the current timing data in the joint timing data can be converted into a coordinate axis image (x-axis is time, y-axis is current value), denoted as the first timing image; and the pressure timing data in the sensor timing data can be converted into a coordinate axis image, denoted as the second timing image.

[0075] Based on the work scenario information, image features are extracted from the first time-series image and the second time-series image to obtain the visual representation.

[0076] Further, the step of extracting image features from the first time-series image and the second time-series image based on the work scenario information to obtain the visual representation includes:

[0077] Based on the work scenario information, the first time-series image is segmented and encoded to obtain several first time-series vectors. Each first time-series vector records the feature vector and position encoding vector of the corresponding position of the first time-series image. The position encoding vector of each first time-series vector is different.

[0078] Based on the work scenario information, the second time-series image is segmented and encoded to obtain several second time-series vectors. Each second time-series vector records the feature vector and position encoding vector of the corresponding position of the second time-series image. The position encoding vector of each second time-series vector is different.

[0079] Spatiotemporal fusion feature extraction is performed on all the first temporal vectors and the second temporal vectors to obtain the visual representation.

[0080] In this embodiment, the first image segmentation and encoding can be based on a preset pixel size (such as 4×4 pixels, 8×8 pixels, 16×16 pixels, etc.) to segment the first temporal image, thereby obtaining several pixel blocks of the preset pixel size; then, each pixel block is flattened into a one-dimensional vector, and each one-dimensional vector corresponds to a feature vector at a certain position in the first temporal image; then, a position encoding vector corresponding to the working scene information is added to each one-dimensional vector, thereby obtaining several first temporal vectors. This position encoding vector can be represented as:

[0081]

[0082] in, This is a position-encoded vector; For the position index of the pixel block, Dimension Index , For scene weighting factors (industrial scene) Home use scenario ); This is a dimension parameter, and one possible example of its specific value is 256.

[0083] It is understandable that the content of the second image segmentation and encoding is similar to that of the first image segmentation and encoding, and can be easily deduced by analogy. The spatiotemporal fusion feature extraction can be achieved by learning the management relationship of each first temporal vector through the Transformer encoder architecture of the first model (i.e., learning the correlation relationship of each pixel block) to extract the spatiotemporal fusion features of the multimodal temporal data, denoted as visual representation.

[0084] Step 130: Input the fault description data into the trained second model for text conversion to obtain the text representation output by the trained second model;

[0085] In this embodiment, the second model can be a pre-trained language model containing multiple layers of text encoders, such as a BERT model containing three layers of text encoders. Text conversion can involve inputting fault description data into the trained second model, and then using the trained second model to convert the fault description data into a text representation.

[0086] In some embodiments, inputting the fault description data into a trained second model for text conversion to obtain the text representation output by the trained second model includes:

[0087] Obtain the working scene information and fault knowledge graph of the humanoid robot;

[0088] Based on the work scenario information, the fault description data is segmented into scenarios to obtain several text segmentation sequences;

[0089] In this embodiment of the application, contextualized word segmentation can be based on work scenario information to segment the text description in the fault description data. For example, in an industrial scenario, professional terms such as "assembly error" and "overload limit" are retained, while in a home scenario, everyday expressions such as "unresponsive language" and "stuck sweeping" are retained, thereby obtaining several text segmentation sequences.

[0090] Based on the fault knowledge graph, all the text segmentation sequences are encoded to obtain the text representation.

[0091] Further, the step of encoding all the text segmentation sequences according to the fault knowledge graph to obtain the text representation includes:

[0092] Obtain the scene weights corresponding to the work scene information;

[0093] Based on the fault knowledge graph, information encoding is performed on all the text segmentation sequences to obtain the hidden state corresponding to each text segmentation sequence;

[0094] Based on the scene weights, all the hidden states are weighted and fused to obtain the text representation.

[0095] In this embodiment, the scene weight can be a pre-set static weight parameter corresponding to the work scene, or it can be an adaptively updated dynamic weight parameter. Information encoding can be performed by inputting all text segmentation sequences into a 3-layer text encoder for encoding. Specifically, each layer encoder combines the sub-encoding of entities in the fault knowledge graph (such as the entity "joint motor" in the fault knowledge graph, whose sub-encoding can be [0.8, 0.2, 0, ..., 0]) and learns word associations (such as the association between "joint" and "overload") based on a self-attention mechanism, thereby obtaining the hidden state of each text segmentation sequence.

[0096] It is understandable that weighted fusion can be based on scene weights, performing scene-specific weighted fusion on each hidden state to obtain a text representation corresponding to the fault description data. The vector form of this text representation can be expressed as:

[0097]

[0098] in, This is a text representation vector; For the number of text types, For the first Context weights for text-like objects Let be the hidden state vector of the k-th text class.

[0099] Step 140: Perform fault analysis on the visual representation and the text representation to obtain the fault analysis results of the humanoid robot.

[0100] In this embodiment of the application, the fault analysis can be based on visual representation and text representation to determine the type of fault that occurs in the humanoid robot, and recorded as the fault analysis result of the humanoid robot.

[0101] In some embodiments, the fault analysis of the visual representation and the text representation to obtain the fault analysis results of the humanoid robot includes:

[0102] The visual representation and the text representation are concatenated to obtain the concatenated features;

[0103] Self-attention fusion is performed on the spliced ​​features to obtain attention fusion features;

[0104] Feature analysis is performed on the attention fusion features to obtain the fault analysis results.

[0105] In this embodiment, feature concatenation can be achieved by concatenating (Concat) vector-based visual representations and text representations into concatenated features. Self-attention fusion can be achieved by learning the feature relationships in the concatenated features based on a self-attention mechanism, and then multiplying the learned self-attention weights element-wise with the concatenated features to obtain the attention fusion features. For example, the self-attention weights can be expressed as:

[0106]

[0107] in, For visual first Dimensions and text Attention weights of dimensions For visual representation Dimensional value, For visual representation Dimensional value; For text representation Dimensional value, For text representation Dimensional value; and These are the maximum dimensions of visual representation and textual representation, respectively; This is a dimension parameter, and one possible example of its value is 512; " indicates the dot product operation.

[0108] Understandably, feature analysis can input attention-fused features into a fully connected layer to predict fault probability, and determine the fault type corresponding to the maximum fault probability output by the fully connected layer as the fault analysis result.

[0109] In some embodiments, the method further includes:

[0110] Acquire historical multi-source fault data of humanoid robots;

[0111] Structured knowledge is extracted from the historical multi-source fault data to obtain a fault knowledge graph;

[0112] Based on the fault knowledge graph, the initialized target model is trained to obtain a trained target model, wherein the target model is either the first model or the second model.

[0113] In this embodiment of the application, the historical multi-source fault data can be data such as fault records and experience documents of humanoid robots in working scenarios. For example, the historical multi-source fault data can include several fault records of humanoid robots in industrial scenarios and corresponding expert experience documents, as well as fault records of humanoid robots in home scenarios and corresponding expert experience documents.

[0114] Structured knowledge extraction can be based on knowledge graph technology to extract structured knowledge from historical multi-source fault data, thus obtaining a fault knowledge graph. Various methods exist for constructing such knowledge graphs, which will not be elaborated upon here. Specifically, this embodiment uses the multimodal time-series data from industrial and residential scenarios mentioned in step 110 above as historical multi-source fault data. One possible knowledge representation method for this historical multi-source fault data in the fault knowledge graph is as follows:

[0115] - Entity layer:

[0116] - Component entities: "Right arm joint motor No. 3" (Attributes: Rated current 10A, Rated speed 100rpm), "Voice sensor" (Attributes: Pickup distance 0.5-5m, Pickup intensity range 0-60dB);

[0117] - Fault type entities: "Overload" (attribute: troubleshooting time ≤ 30 minutes), "Signal interruption" (attribute: troubleshooting time ≤ 15 minutes);

[0118] -Scene Entities: "Industrial Assembly Scene" and "Home Care Scene".

[0119] -Relationship layer:

[0120] - "Component-Fault": <Right Arm Joint 3 Motor, Associated, Overload>, <Voice Sensor, Associated, Signal Interruption>;

[0121] - "Fault-Signal": <Overload, corresponding to a sudden current increase of 30%>, <Signal interruption, corresponding to a pickup intensity of 0dB>;

[0122] - "Fault-Scenario": <Overload, Adaptation, Industrial Assembly Scenario>, <Signal Interruption, Adaptation, Home Care Scenario>.

[0123] -Event layer:

[0124] - Industrial event: "Assembly event - Right arm joint 3 movement - Overload - Current surge of 30%";

[0125] -Home incident: "Accompanying incident - Voice interaction - Signal interruption - Sound intensity 0dB".

[0126] Understandably, if the target model is the first model, model training can involve using knowledge related to time-series signals and faults in the fault knowledge graph as training data and inputting it into the first model. The loss value of the first model's output is calculated using a loss function. Then, based on this loss value, the parameters of the first model are updated using a backpropagation algorithm. After several iterations, a trained first model can be obtained. The specific number of iterations can be preset, or training can be considered complete when the test set reaches the required accuracy. The loss function used can be a 0-1 loss function, a squared loss function, an absolute loss function, a logarithmic loss function, a cross-entropy loss function, or a combination of multiple loss functions. In this embodiment, the loss function of the first model is taken as a weighted sum of a multi-class cross-entropy loss function and a knowledge graph regularization term. The loss function of the first model can be expressed as:

[0127]

[0128] in, This is the loss function for the first model; This represents the cross-entropy loss value for multi-class classification. ,in, This refers to the number of knowledge samples (i.e., the amount of training data). The actual fault labels corresponding to the training data; The predicted probability output by the first model; Output the Euclidean distance between the first model's output features and the "signal-fault" association features in the knowledge graph; The parameters are determined through the model training and tuning process.

[0129] It should be noted that if the target model is the second model, the content of the second model is similar to that of the first model mentioned above. It can be simply deduced that the loss function of the second model can be any of the loss functions mentioned above, or it can be the same as the loss function of the first model. This application will not elaborate further here.

[0130] Reference Figure 2 The present application proposes a fault analysis system for a humanoid robot, including...

[0131] The first processing unit 101 is used to acquire multimodal timing data of the humanoid robot within a preset time before and after the fault, as well as fault description data of the humanoid robot. The multimodal timing data includes joint timing data and sensor timing data.

[0132] The second processing unit 102 is used to input the multimodal time series data into the trained first model for visual transformation, and obtain the visual representation output by the trained first model.

[0133] The third processing unit 103 is used to input the fault description data into the trained second model for text conversion, and obtain the text representation output by the trained second model.

[0134] The fourth processing unit 104 is used to perform fault analysis on the visual representation and the text representation to obtain the fault analysis results of the humanoid robot.

[0135] It is understood that the content of the above method embodiments is applicable to this system embodiment. The specific functions implemented in this system embodiment are the same as those in the above method embodiments, and the beneficial effects achieved are also the same as those achieved in the above method embodiments.

[0136] Reference Figure 3 This application also provides an electronic device, including:

[0137] At least one processor 201;

[0138] At least one memory 202 is used to store at least one program;

[0139] When the at least one program is executed by the at least one processor 201, the at least one processor 201 implements the method embodiment described above.

[0140] Similarly, it can be understood that the content of the above method embodiments is applicable to this device embodiment. The specific functions implemented by this device embodiment are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0141] This application also provides a computer-readable storage medium storing a program executable by a processor 201, which, when executed by the processor 201, is used to implement the above-described method embodiments.

[0142] Similarly, the content of the above method embodiments is applicable to the present computer-readable storage medium embodiments. The specific functions implemented by the present computer-readable storage medium embodiments are the same as those of the above method embodiments, and the beneficial effects achieved are also the same as those achieved by the above method embodiments.

[0143] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0144] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above methods.

[0145] In some alternative embodiments, the functions / operations mentioned in the block diagrams may not occur in the order shown in the operation diagrams. For example, depending on the functions / operations involved, two consecutively shown blocks may actually be executed substantially simultaneously, or the blocks may sometimes be executed in reverse order. Furthermore, the embodiments presented and described in the flowcharts of this application are provided by way of example to provide a more comprehensive understanding of the technology. The disclosed methods are not limited to the operations and logic flows presented herein. Alternative embodiments are contemplated in which the order of various operations is changed and sub-operations described as part of a larger operation are executed independently.

[0146] Furthermore, although this application is described in the context of functional modules, it should be understood that, unless otherwise stated to the contrary, one or more of the functions and / or features may be integrated into a single physical device and / or software module, or one or more functions and / or features may be implemented in a separate physical device or software module. It is also understood that a detailed discussion of the actual implementation of each module is unnecessary for understanding this application. Rather, given the properties, functions, and internal relationships of the various functional modules in the apparatus disclosed herein, the actual implementation of the module will be understood within the scope of conventional technology for an engineer. Therefore, those skilled in the art can implement the application set forth in the claims using ordinary techniques without excessive experimentation. It is also understood that the specific concepts disclosed are merely illustrative and not intended to limit the scope of this application, which is determined by the full scope of the appended claims and their equivalents.

[0147] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods in the embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0148] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-including system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device.

[0149] More specific examples (a non-exhaustive list) of computer-readable media include: electrical connections (electronic devices) having one or more wires, portable computer disk drives (magnetic devices), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Furthermore, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in computer memory.

[0150] It should be understood that various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0151] In the foregoing description of this specification, the references to terms such as "one embodiment," "another embodiment," or "some embodiments," etc., indicate that a specific feature, structure, material, or characteristic described in connection with an embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0152] Although embodiments of this application have been shown and described, those skilled in the art will understand that various changes, modifications, substitutions and variations can be made to these embodiments without departing from the principles and spirit of this application, the scope of which is defined by the claims and their equivalents.

[0153] The above is a detailed description of the preferred embodiments of this application, but this application is not limited to the embodiments. Those skilled in the art can make various equivalent modifications or substitutions without departing from the spirit of this application, and these equivalent modifications or substitutions are all included within the scope defined by the claims of this application.

Claims

1. A fault analysis method for a humanoid robot, characterized in that, include: Acquire multimodal timing data of the humanoid robot within a preset time period before and after the failure, as well as the failure description data of the humanoid robot. The multimodal timing data includes joint timing data and sensor timing data. The multimodal time-series data is input into the trained first model for visual transformation to obtain the visual representation output by the trained first model; The fault description data is input into the trained second model for text conversion to obtain the text representation output by the trained second model; Fault analysis is performed on the visual representation and the text representation to obtain the fault analysis results of the humanoid robot; The step of inputting the multimodal temporal data into a trained first model for visual transformation to obtain the visual representation output by the trained first model includes: Obtain the working scene information of the humanoid robot; The multimodal time-series data is subjected to channel-wise image conversion to obtain a first time-series image corresponding to the joint time-series data and a second time-series image corresponding to the sensing time-series data; Based on the work scenario information, image features are extracted from the first time-series image and the second time-series image to obtain the visual representation; The step of inputting the fault description data into the trained second model for text conversion to obtain the text representation output by the trained second model includes: Obtain the working scene information and fault knowledge graph of the humanoid robot; Based on the work scenario information, the fault description data is segmented into scenarios to obtain several text segmentation sequences; Based on the fault knowledge graph, all the text segmentation sequences are encoded to obtain the text representation.

2. The method according to claim 1, characterized in that, The step of extracting image features from the first time-series image and the second time-series image based on the work scenario information to obtain the visual representation includes: Based on the work scenario information, the first time-series image is segmented and encoded to obtain several first time-series vectors. Each first time-series vector records the feature vector and position encoding vector of the corresponding position of the first time-series image. The position encoding vector of each first time-series vector is different. Based on the work scenario information, the second time-series image is segmented and encoded to obtain several second time-series vectors. Each second time-series vector records the feature vector and position encoding vector of the corresponding position of the second time-series image. The position encoding vector of each second time-series vector is different. Spatiotemporal fusion feature extraction is performed on all the first temporal vectors and the second temporal vectors to obtain the visual representation.

3. The method according to claim 1, characterized in that, The step of encoding all the text segmentation sequences according to the fault knowledge graph to obtain the text representation includes: Obtain the scene weights corresponding to the work scene information; Based on the fault knowledge graph, information encoding is performed on all the text segmentation sequences to obtain the hidden state corresponding to each text segmentation sequence; Based on the scene weights, all the hidden states are weighted and fused to obtain the text representation.

4. The method according to claim 1, characterized in that, The fault analysis of the visual representation and the text representation to obtain the fault analysis results of the humanoid robot includes: The visual representation and the text representation are concatenated to obtain the concatenated features; Self-attention fusion is performed on the spliced ​​features to obtain attention fusion features; Feature analysis is performed on the attention fusion features to obtain the fault analysis results.

5. The method according to claim 1, characterized in that, The method further includes: Acquire historical multi-source fault data of humanoid robots; Structured knowledge is extracted from the historical multi-source fault data to obtain a fault knowledge graph; Based on the fault knowledge graph, the initialized target model is trained to obtain a trained target model, wherein the target model is either the first model or the second model.

6. A fault analysis system for a humanoid robot, characterized in that, include: The first processing unit is used to acquire multimodal time-series data of the humanoid robot within a preset time before and after the fault, as well as fault description data of the humanoid robot. The multimodal time-series data includes joint time-series data and sensor time-series data. The second processing unit is used to input the multimodal time series data into the trained first model for visual transformation, and obtain the visual representation output by the trained first model. The third processing unit is used to input the fault description data into the trained second model for text conversion, and obtain the text representation output by the trained second model. The fourth processing unit is used to perform fault analysis on the visual representation and the text representation to obtain the fault analysis results of the humanoid robot; The step of inputting the multimodal temporal data into a trained first model for visual transformation to obtain the visual representation output by the trained first model includes: Obtain the working scene information of the humanoid robot; The multimodal time-series data is subjected to channel-wise image conversion to obtain a first time-series image corresponding to the joint time-series data and a second time-series image corresponding to the sensing time-series data; Based on the work scenario information, image features are extracted from the first time-series image and the second time-series image to obtain the visual representation; The step of inputting the fault description data into the trained second model for text conversion to obtain the text representation output by the trained second model includes: Obtain the working scene information and fault knowledge graph of the humanoid robot; Based on the work scenario information, the fault description data is segmented into scenarios to obtain several text segmentation sequences; Based on the fault knowledge graph, all the text segmentation sequences are encoded to obtain the text representation.

7. An electronic device, characterized in that, include: At least one processor; At least one memory for storing at least one program; When the at least one program is executed by the at least one processor, the at least one processor performs the method as described in any one of claims 1-5.

8. A computer-readable storage medium storing a processor-executable program, characterized in that, The processor-executable program, when executed by the processor, is used to implement the method as described in any one of claims 1-5.

Images