Robot fault self-service diagnosis system based on multi-mode autonomous deep learning

Abstract

The invention, which relates to the technical field of industrial robot fault diagnosis, discloses a robot fault self-service diagnosis system based on multi-modal autonomous deep learning, comprising a data acquisition module, a preprocessing module, a three-dimensional semantic graph construction module, a cross-modal semantic association module, a diagnosis engine module, an interaction module and a localization deployment module. The method has the advantages that a three-dimensional semantic graph neural network topological graph containing geometric features, function tags and symbol tags of components is constructed in the three-dimensional semantic graph construction module based on a robot CAD digital model; a geometric-semantic bidirectional matching mechanism is utilized to calculate geometric feature Euclidean distance and semantic cosine similarity between a text semantic tag and a three-dimensional semantic graph node, cross-modal association between multi-modal data and the graph node is established, and meanwhile, node features are updated through iteration of a graph neural network model, so that multi-modal data are obtained. The problems that multi-source heterogeneous data is difficult to convert into a unified diagnosis feature space and cross-modal semantic linkage is insufficient are solved.

Description

TECHNICAL FIELD

[0001] The application relates to the technical field of industrial robot fault diagnosis, in particular to a robot fault self-diagnosis system based on multi-modal autonomous deep learning. BACKGROUND

[0002] With the deep integration of Industry 4.0 and artificial intelligence technology, the application of robots in the fields of intelligent manufacturing, smart logistics and medical services presents a trend of scaling and complication, and the efficient and stable operation of the robot system puts higher requirements on the fault diagnosis technology - not only the multi-fault types such as mechanical arm joint wear, sensor abnormality and control algorithm failure need to be accurately identified, but also the complex application scenarios such as variable load, multiple working conditions and strong noise need to be adapted; The prior art has certain defects, firstly, the prior art is difficult to integrate multi-source heterogeneous data and establish cross-modal semantic linkage, multi-modal data fusion stays on the surface, and the correlation depth is insufficient, secondly, the traditional diagnosis model relies on manual annotation data updating, and the generalization ability is weak, and it is difficult to adapt to multi-brand robot iteration and individualized fault scenes, therefore, the robot fault self-diagnosis system based on multi-modal autonomous deep learning is proposed. SUMMARY

[0003] The robot fault self-diagnosis system based on multi-modal autonomous deep learning is provided.

[0004] To achieve the above-mentioned purpose, the application provides the following technical scheme: the robot fault self-diagnosis system based on multi-modal autonomous deep learning comprises: A data acquisition module acquires multi-modal data of multi-brand robots by using a standardized interface; A preprocessing module performs format conversion and noise reduction processing on the original data to generate a feature vector in a unified format; A three-dimensional semantic graph construction module constructs a three-dimensional semantic graph neural network based on a robot CAD model, encodes component geometric features, function labels and symbol labels as graph nodes, constructs a "component-interface-constraint" topology graph by taking the rotary pair connection relationship and the fixed pair connection relationship between components as edges, and outputs to a cross-modal semantic association module; A cross-modal semantic association module establishes cross-modal association by using a geometric-semantic bidirectional matching mechanism, establishes cross-modal association between text, graphics and time sequence data and three-dimensional semantic graph nodes, generates a cross-modal diagnosis feature vector, and transmits to a diagnosis engine module; A diagnosis engine module comprises a hybrid neural network and a reinforcement learning component, is used for fusing the cross-modal diagnosis feature vector to generate a diagnosis result, and automatically optimizes the model according to feedback, and outputs to an interaction module; An interaction module is used for analyzing user fault description, visualizing fault components and generating a diagnosis report. Localized deployment module: used for full-process localized deployment to ensure data security and low-latency response.

[0005] As a further solution of the present invention: the multimodal data collected by the data acquisition module includes text maintenance materials in PDF / WORD format, graphic data (CAD digital model, three-dimensional model) in STEP / IGES / STL format, and timing sensor data (torque, temperature, vibration signals) transmitted by real-time communication protocol, and the original data is transmitted to the preprocessing module.

[0006] As a further solution of the present invention: the preprocessing module uses OCR character recognition technology to parse text data, uses the OpenCASCADE open source library to parse the CAD model to extract the geometric features of the component surface curvature and edge contour, voxelizes the STL model to generate a three-dimensional grid feature vector, uses the Kalman filter algorithm to perform noise reduction on the time series data, generates a feature vector in a unified format, and transmits it to the three-dimensional semantic map construction module.

[0007] As a further solution of the present invention: the three-dimensional semantic graph construction module encodes geometric features such as component volume and centroid coordinates, functional labels such as the corresponding system label and fault mode label, and symbolic labels such as associated fault codes into graph nodes; The graph neural network model of the 3D semantic graph building module iteratively updates node features using the following formula: , in, For nodes In the +1 layer feature vector, including geometric features such as volume and centroid coordinates and semantic labels of fault mode probability distribution, For nodes The set of adjacent component nodes, Adjacent nodes In the The feature vector of the layer, is the global topological eigenvector, encoding the rotational and fixed sub-connection constraints between components. 、 It is a learnable weight matrix that realizes the joint representation of component status and topological relationship through neighbor node feature aggregation and global constraint fusion.

[0008] As a further solution of the present invention: the hybrid neural network and reinforcement learning components are included. The hybrid neural network uses an improved BERT text encoder to extract text data features, a lightweight CNN graphic encoder to extract graphic data features, and a bidirectional LSTM time series encoder to extract time series data features. The cross-modal diagnostic feature vector is combined to generate a comprehensive diagnostic result. The reinforcement learning component automatically adjusts the model parameters according to the "valid" and "invalid" signals fed back by the engineer, and outputs the diagnostic report to the interactive module. The hybrid neural network fusion layer of the diagnostic engine module uses a gating mechanism to dynamically adjust the modal weight. For software failures (such as program errors), the system automatically increases the text modal weight to 80%-90% and suppresses the graphic modal and time series modal. For mechanical failures (such as abnormal joint noises), the system increases the sum of the graphic modal and time series modal weights to more than 90%.

[0009] As a further solution of the present invention: the interactive module uses natural language processing technology to parse the text, pictures and code fault descriptions input by the user, and generates context-related dynamic guidance questions based on the fault description keywords (for example, when "robotic arm shakes", the system asks "Is the shake accompanied by abnormal noise?") to supplement the diagnostic information, and highlights the faulty parts in the visual interface based on the three-dimensional semantic graph, superimposes textual maintenance steps and spare parts inventory information, supports converting the diagnostic results into Chinese voice broadcast, English voice broadcast and Japanese voice broadcast, and displays an interactive diagnostic report to the user. The natural language processing function includes generating context-related guidance questions based on the fault description keywords (for example, when "controller alarm" is input, the system asks "Does the alarm code start with ER?").

[0010] As a further solution of the present invention: the local deployment module uses the OPC UA industrial protocol and the Modbus industrial protocol to access the robot controller, and deploys the entire process of data acquisition, model inference and result output on the enterprise local server. Firewall rules are used to limit only the IP address of the industrial robot controller to access the local server data interface, and blockchain smart contracts are used to perform SHA-256 hash storage on the diagnostic log. The VPN channel uses the AES-256 encryption protocol to realize encrypted transmission of model parameters between local servers in different workshops, ensuring data security and millisecond-level response.

[0011] By adopting the above technical solution, compared with the prior art, the beneficial effects of the present invention are: 1. The application constructs a three-dimensional semantic graph neural network topology graph containing component geometric features, function labels and symbol labels based on a robot CAD model in the three-dimensional semantic graph construction module, calculates the Euclidean distance of the text semantic label and the geometric features of the three-dimensional semantic graph node and the semantic cosine similarity by using the geometric-semantic bidirectional matching mechanism, establishes the cross-modal association of the multi-modal data and the graph node, and simultaneously iteratively updates the node features by the graph neural network model, aggregates the neighbor node features and the global topology constraints, solves the problems of difficulty in converting multi-source heterogeneous data into a unified diagnostic feature space and insufficient cross-modal semantic linkage, and finally realizes efficient association of fault description and component positioning, principle analysis, and improves the accuracy and efficiency of robot fault diagnosis.

[0012] 2. The application designs a hybrid neural network and reinforcement learning component in the diagnosis engine module, uses an improved BERT text encoder, a lightweight CNN graph encoder and a bidirectional LSTM time series encoder to extract multi-modal features respectively, generates a comprehensive diagnostic result in combination with the cross-modal diagnostic feature vector, and automatically optimizes the model parameters based on engineer feedback through the reinforcement learning component, solves the problems of dependence on manual annotation data update and insufficient generalization ability of traditional diagnostic models, and without a large amount of manual intervention, the system can adapt to multi-brand robot model iteration and personalized fault scenarios, realizes the autonomous evolution of the diagnostic model, significantly improves the adaptability of the system to new faults and complex working conditions, and reduces the model update cost and dependence on professional experience. BRIEF DESCRIPTION OF DRAWINGS

[0013] Figure 1 The system flowchart in the embodiments of the application. DETAILED DESCRIPTION

[0014] The specific embodiments of the application will be further described below in conjunction with the drawings, and it should be noted that the description of these embodiments is used to help understand the application, but does not constitute a limitation on the application.

[0015] In addition, the technical features involved in each of the embodiments of the application described below can be combined with each other as long as they do not conflict with each other.

[0016] Please refer to the accompanying Figure 1 The robot fault self-diagnosis system based on multi-modal autonomous deep learning comprises: Data acquisition module: multi-modal data of multi-brand robots are acquired by using standardized interfaces; Preprocessing module: format conversion and noise reduction processing are performed on the original data to generate feature vectors in a unified format; Three-dimensional semantic graph construction module: based on the robot CAD model, a three-dimensional semantic graph neural network is constructed, the component geometric features, function labels and symbol labels are encoded as graph nodes, and the "component-interface-constraint" topological graph is constructed by taking the rotary pair connection relationship and the fixed pair connection relationship between components as edges, and is output to the cross-modal semantic association module; Cross-modal semantic association module: a geometric-semantic bidirectional matching mechanism is used to establish cross-modal association, cross-modal association between text, graphics and time sequence data and three-dimensional semantic graph nodes is established, cross-modal diagnostic feature vectors are generated, and are transmitted to the diagnostic engine module; Diagnostic engine module: containing a hybrid neural network and a reinforcement learning component, used for fusing cross-modal diagnostic feature vectors to generate diagnostic results, and automatically optimizing the model according to feedback, and output to the interaction module; Interaction module: used for analyzing user fault description, visualizing fault components and generating diagnostic report; Local deployment module: used for full-process localization deployment, ensuring data security and low-latency response.

[0017] In an embodiment of the present application: the multi-modal data collected by the data acquisition module includes text type maintenance materials in PDF / WORD format, graphic type data (CAD model, three-dimensional model) in STEP / IGES / STL format, and time sequence type sensor data (torque, temperature, vibration signal) transmitted by real-time communication protocol, and the original data is transmitted to the preprocessing module.

[0018] In an embodiment of the present application: the preprocessing module uses OCR character recognition technology to analyze text data, uses OpenCASCADE open source library to analyze CAD model to extract geometric features of component surface curvature and edge contour, voxelizes STL model to generate three-dimensional grid feature vector, and uses Kalman filter algorithm to denoise time sequence data, and generates feature vectors in a unified format, which are then transmitted to the three-dimensional semantic graph construction module.

[0019] In an embodiment of the present application: the three-dimensional semantic graph construction module encodes geometric features such as component volume and centroid coordinates, function labels such as system labels and fault mode labels, and symbol labels such as associated fault codes into graph nodes; The graph neural network model of the three-dimensional semantic graph construction module iteratively updates the node features by the following formula: , Wherein, is the node In the first layer feature vector, which contains geometric features such as volume and centroid coordinates and fault mode probability distribution semantic labels, is the node a set of adjacent component node sets of the component, adjacent nodes in the first eigenvectors of the layer, is a global topology eigenvector, encoding the rotational pair connection constraint and fixed pair connection constraint between components, , is a learnable weight matrix, realizing the joint representation of component state and topology relationship through neighbor node feature aggregation and global constraint fusion.

[0020] In an embodiment of the present application: the cross-modal semantic association module utilizes a geometric-semantic bidirectional matching mechanism to establish cross-modal association. First, the Euclidean distance of the semantic label of the text and the component and the geometric feature of the three-dimensional semantic graph node is calculated. When the Euclidean distance is less than a preset threshold , the cosine similarity of the text fault description and the node fault mode label is calculated using the BERT model. When the cosine similarity is greater than a preset threshold , the association between the text and the component node is established, and the vibration source node in the topology graph is mapped to the time series data abnormal frequency feature using the graph neural network message passing mechanism. After generating the cross-modal diagnostic feature vector, it is transmitted to the diagnostic engine module.

[0021] In an embodiment of the present application: a mixed neural network and reinforcement learning component are included. The mixed neural network uses an improved BERT text encoder to extract text data features, a lightweight CNN graph encoder to extract graph data features, and a bidirectional LSTM time series encoder to extract time series data features. The cross-modal diagnostic feature vector is combined to generate a comprehensive diagnostic result. The reinforcement learning component automatically adjusts the model parameters according to the "effective" and "ineffective" signals fed back by engineers, and outputs the diagnostic report to the interaction module. The mixed neural network fusion layer of the diagnostic engine module adopts a gating mechanism to dynamically adjust the modal weight. For software faults (such as program error), the system automatically increases the text modal weight to 80%-90% and suppresses the graph modal and time series modal. For mechanical faults (such as joint abnormal noise), the system increases the sum of the weights of the graph modal and the time series modal to more than 90%.

[0022] In an embodiment of the application: the interaction module parses user inputted text, picture and code fault description with natural language processing techniques, generates context-related dynamic guiding questions based on fault description keywords (e.g. input "robot arm shaking", the system asks "is the shaking accompanied by abnormal sound?") to supplement diagnostic information, and highlights fault components in the visualization interface based on a three-dimensional semantic graph, superimposes textual maintenance steps and spare parts inventory information, supports converting diagnostic results into Chinese voice broadcast, English voice broadcast and Japanese voice broadcast, and shows users an interactive diagnostic report. Natural language processing functions include generating context-related guiding questions based on fault description keywords (e.g. input "controller alarm", the system asks "is the alarm code started with ER?").

[0023] In an embodiment of the application: the localization deployment module accesses the robot controller using OPCUA industrial protocol and Modbus industrial protocol, deploys data collection, model inference and result output in the enterprise local server, restricts access to the local server data interface only to industrial robot controller IP addresses using firewall rules, uses SHA-256 hash storage of diagnostic logs using blockchain smart contracts, and uses VPN channels to implement encrypted transmission of model parameters between different plant local servers using AES-256 encryption protocol to ensure data security and millisecond-level response.

[0024] Example 1, please refer to the attached Figure 1 , KUKA KR60-3 industrial robot reducer fault diagnosis: 1. Data collection phase Text data: Collect KUKA official PDF format maintenance manual (file size: 2.3MB, containing 127 pages of graphic text description), extract the "KR series reducer maintenance guide" chapter on page 45; Parse the WORD format fault code table (version: V2.0, containing 876 fault records) to obtain the description text of code "E12035" (reducer torque anomaly); Graphic data: Import the STEP format CAD model of KR60-3 (precision: 0.01mm, containing 238 components), extract the three-dimensional geometric data of the reducer assembly (component ID: KR-C017); Get the STL format reducer model (facet number: 12,547, triangulation error <0.05mm) for generating voxelized features; Time series data: Robot controller data acquisition through Profinet protocol, sampling frequency: 100Hz, acquisition duration: 30 minutes, input shaft torque data of reducer (range: 0-300Nm, accuracy ±0.5Nm) acquisition; Vibration sensor data acquisition through EtherCAT protocol (model: B&K4508-B-005, measurement range: 0-20g, frequency response: 0.5-10kHz); 2. Preprocessing stage Text preprocessing: OCR recognition accuracy: 98.7%, using PyTesseract library to process scanned manual images; After text cleaning, the number of valid characters retained: 14256, removing headers and footers and irrelevant symbols; Entity recognition through spaCy, extracting keywords such as "reducer", "torque fluctuation", "bearing wear".

[0025] Graph preprocessing: OpenCASCADE extracts reducer component surface curvature (range: 0.01-10mm -1 ), edge contour point number: 2143; STL model voxelization parameters: voxel size 0.5mm×0.5mm×0.5mm, generating 128×128×128 three-dimensional grid feature vector.

[0026] Time series preprocessing: Kalman filter parameters: process noise covariance =0.01, measurement noise covariance =0.1; After denoising, the signal-to-noise ratio (SNR) of the torque data is improved from 12dB to 25dB; Short-time Fourier transform (STFT) is used for time-frequency analysis, window length: 1024 points, overlap rate: 75%.

[0027] 3. Three-dimensional semantic graph construction stage Node feature encoding: Reducer node (ID: N007) geometric features: volume = 0.023m 3 , centroid coordinates (350, -120, 480) mm; Function label: transmission efficiency (current value 92%, threshold ≥95%), load capacity (280Nm, rated 300Nm); Symbol label: associated fault code E12035 (confidence 0.85), maintenance period (5000 hours, current use 4872 hours); Topology graph construction: Adjacent nodes: motor (N006, revolute pair connection), output shaft (N008, spline connection); Edge weight: set according to connection stiffness, motor → gearbox = 0.85, gearbox → output shaft = 0.92; 4. Cross-modal semantic association stage Geometric-semantic matching: Text keyword "gearbox torque anomaly" and geometric feature Euclidean distance of node N007: 0.07m 3 (threshold =0.1m 3 ); BERT model calculates the cosine similarity between text and node function label: 0.88 (threshold =0.8); Abnormal frequency feature (120Hz, corresponding to gear meshing frequency) in time series data is mapped to N007 node through topological graph.

[0028] 5. Diagnosis reasoning stage Hybrid neural network: Improved BERT encoder: hidden layer dimension 768, attention head number 12, fine-tuning learning rate 2e-5; Lightweight CNN encoder: MobileNetV3 architecture, parameter quantity 2.5M, input image size 224x224; Bidirectional LSTM encoder: hidden layer dimension 256, time step 100, dropout rate 0.2.

[0029] Diagnosis result: Gearbox gear wear probability: 93.2%, bearing clearance probability: 87.5%; Fault reason: input shaft imbalance leads to abnormal stress on gear (time series data shows 1X speed frequency vibration amplitude = 1.2g, threshold ≤0.8g).

[0030] 6. Localization deployment and verification Deployment parameters: Local server configuration: Intel Xeon E5-2620v4 @ 2.10GHz, 16 cores, 64GB RAM, NVIDIA Tesla P4 GPU; Response time: ≤2.3 seconds from data collection to diagnosis report generation; Blockchain storage: Hyperledger Fabric is used, block generation time is 2 seconds, transaction throughput is 200TPS.

[0031] Verification result: After disassembling the reducer, it was found that the gear tooth surface wear depth was 0.12mm on average (standard value ≤0.05mm); After replacing the bearing, the fault disappeared, and the MTTR (mean time to repair) was reduced from 8 hours to 3 hours.

[0032] Example two, please refer to the attached Figure 1 , ABBIRB6700 handling robot tool center point (TCP) offset fault diagnosis: 1. Data collection stage Text data: Parse ABB Robotics Manual PDF (version: 2024.03, 4.7MB), extract chapter 6 "TCP calibration and maintenance"; Get Excel format fault log (contains 12,543 records), filter TCP offset related records 217 in the past 3 months.

[0033] Graphical data: Import IRB6700 IGES format number model (precision: 0.001mm, contains 512 components), focus on tool flange (component ID: AB-T004); Get the actual point cloud data of the tool through the 3D scanner (resolution: 0.1mm, point cloud number: 1280000 points).

[0034] Time series data: Collect 6-axis joint angle data through EtherCAT protocol (sampling rate: 500Hz, accuracy ±0.01°); Force sensor data (model: ATIMini40, measurement range ±500N, sampling rate 1kHz).

[0035] 2. Preprocessing stage Text preprocessing: Extract keywords: "TCP offset", "flange loosening", "tool calibration", etc., build a domain dictionary containing 3,245 terms; Text vectorization: use Sentence-BERT to generate 768-dimensional semantic vectors.

[0036] Graphical preprocessing: Point cloud registration: use ICP algorithm, iteration times 30, registration error ≤0.05mm; Extract tool flange geometric features: flatness 0.03mm, cylindricity 0.02mm.

[0037] Time series preprocessing: Wavelet denoising: select db4 wavelet, decomposition level 5, threshold selection VisuShrink; Feature extraction: Calculate joint angle fluctuation rate (normal range ≤ 0.5° / s, current measurement 1.2° / s).

[0038] 3. Three-dimensional semantic graph construction phase Node feature encoding: Tool flange node (ID: N023) geometric features: diameter 120mm, thickness 25mm; Functional label: positioning accuracy (current value ±0.3mm, standard ±0.1mm), maximum load (250kg, current load 180kg); Symbolic label: associated calibration program number PRG-6700-TCP, last calibration time 2025-03-15.

[0039] Topology graph construction: Adjacent nodes: sixth axis (N022, fixed bolt connection), end effector (N024, quick-change interface connection); Edge weight: set according to connection stiffness and fault propagation probability, N022→N023=0.95, N023→N024=0.88.

[0040] 4. Cross-modal semantic association phase Geometric-semantic matching: Cosine similarity between text description "tool positioning inaccurate" and N023 node functional label: 0.92; Point cloud data shows that the flange flatness is out of tolerance, with an Euclidean distance of 0.04m from the text keyword "flange loose" 3 .

[0041] Time series data mapping: Sixth axis joint angle fluctuation rate anomaly (1.2° / s) is transmitted to N023 node through the topology graph, updating its "positioning accuracy" confidence to 0.96.

[0042] 5. Diagnosis reasoning phase Mixed neural network: Improved BERT encoder: F1-score=0.91 on ABB fault text dataset after fine-tuning; CNN encoder: classification accuracy on tool point cloud data =98.3%; LSTM encoder: recall rate of time series anomaly detection =95.7%.

[0043] Diagnosis result: Tool flange fixed bolt loose probability: 97.8%; Suggested operation: tighten M12 bolt with torque value 120±5N・m (refer to maintenance manual).

[0044] 6. Localized deployment and verification Deployment parameters: Use edge computing gateway (model: Advantech UNO-2483G), 8GB memory, 128GB SSD; Data encryption: AES-256, key update cycle 24 hours; Model inference latency: ≤80ms (single batch data).

[0045] Verification results: Measured flange bolt torque value: minimum 85N·m, maximum 105N·m; After tightening according to the standard torque, the TCP repeat positioning accuracy is improved from ±0.3mm to ±0.08mm, which meets the equipment standards.

[0046] According to the contents of the above embodiments, it can be concluded that by constructing a three-dimensional semantic graph neural network topology graph containing multi-dimensional labels based on the robot CAD digital model in the three-dimensional semantic graph construction module, the geometric-semantic bidirectional matching mechanism is used to establish the association between multimodal data and graph nodes, and the node features are iteratively updated with the help of the graph neural network to aggregate multi-dimensional information. At the same time, a hybrid neural network and reinforcement learning component are designed in the diagnosis engine module to realize multimodal feature extraction, diagnosis result generation and model autonomous optimization, which solves the problems of insufficient semantic linkage of multi-source heterogeneous data and weak generalization ability of traditional diagnosis models. Finally, it realizes the efficient association between fault description and component positioning and principle analysis, improves the accuracy and efficiency of diagnosis, and enables the system to autonomously adapt to the iteration and complex working conditions of multi-brand robots, reduces the cost of model update and dependence on professional experience, and achieves good results of accurate diagnosis and intelligent maintenance of robot faults through the coordinated cooperation of various module technologies.

[0047] Although the present invention is disclosed above with reference to preferred embodiments, this is not intended to limit the present invention. Any person skilled in the art may make possible changes and modifications without departing from the spirit and scope of the present invention. Therefore, any modifications, equivalent variations, and modifications made to the above embodiments in accordance with the technical essence of the present invention without departing from the content of the technical solution of the present invention shall fall within the scope of protection defined by the claims of the present invention.

Claims

1. A robot fault self-service diagnosis system based on multimodal autonomous deep learning, characterized by: The self-service fault diagnosis system includes: Data acquisition module: uses standardized interfaces to collect multi-modal data from robots of multiple brands; Preprocessing module: performs format conversion and noise reduction on the original data to generate feature vectors in a unified format; 3D semantic graph construction module: This module builds a 3D semantic graph neural network based on the robot CAD digital model. It encodes component geometric features, functional labels, and symbolic labels as graph nodes, and constructs a "component-interface-constraint" topological graph using the rotational and fixed sub-connections between components as edges. Cross-modal semantic association module: This module uses a geometric-semantic bidirectional matching mechanism to establish cross-modal associations between text, graphics, and time series data and 3D semantic graph nodes, generating cross-modal diagnostic feature vectors. Diagnostic engine module: This module includes a hybrid neural network and reinforcement learning components, which are used to fuse cross-modal diagnostic feature vectors to generate diagnostic results and automatically optimize the model based on feedback; Interactive module: used to parse user fault descriptions, visualize faulty components, and generate diagnostic reports; Localized deployment module: used for full-process localized deployment to ensure data security and low-latency response.

2. The robot fault self-service diagnosis system based on multimodal autonomous deep learning according to claim 1 is characterized by: The multimodal data collected by the data acquisition module includes text maintenance materials in PDF / WORD format, graphic data in STEP / IGES / STL format, and time series sensor data transmitted by real-time communication protocol.

3. The robot fault self-service diagnosis system based on multimodal autonomous deep learning according to claim 1 is characterized by: The preprocessing module uses OCR character recognition technology to parse text data, uses the OpenCASCADE open source library to parse CAD models to extract the geometric features of component surface curvature and edge contours, voxelizes the STL model to generate a three-dimensional grid feature vector, uses the Kalman filter algorithm to reduce noise on the time series data, generates a feature vector in a unified format, and transmits it to the three-dimensional semantic map construction module.

4. The robot fault self-service diagnosis system based on multimodal autonomous deep learning according to claim 1 is characterized in that: The three-dimensional semantic graph construction module encodes the geometric features of the component volume and center of mass coordinates, the system label to which it belongs, the functional label of the fault mode label, and the associated fault code symbol label into graph nodes; The graph neural network model of the 3D semantic graph building module iteratively updates node features using the following formula: ， in, For nodes In the +1 layer feature vector, For nodes The set of adjacent component nodes, Adjacent nodes In the The feature vector of the layer, is the global topological eigenvector, encoding the rotational and fixed sub-connection constraints between components. 、 is the learnable weight matrix.

5. The robot fault self-service diagnosis system based on multimodal autonomous deep learning according to claim 1 is characterized in that: The cross-modal semantic association module uses the geometric-semantic bidirectional matching mechanism to establish cross-modal associations. First, the Euclidean distance between the semantic labels of texts and components and the geometric features of the three-dimensional semantic graph nodes is calculated. When the Euclidean distance is less than a preset threshold, the cross-modal semantic association module Then, the BERT model is used to calculate the cosine similarity between the text fault description and the node fault mode label. When the cosine similarity is greater than the preset threshold When the text is associated with the component nodes, the graph neural network message passing mechanism is used to map the abnormal frequency characteristics of the time series data to the vibration source nodes in the topological graph to generate a cross-modal diagnostic feature vector.

6. The robot fault self-service diagnosis system based on multimodal autonomous deep learning according to claim 1 is characterized in that: The system includes a hybrid neural network and reinforcement learning components. The hybrid neural network uses an improved BERT text encoder to extract text data features, a lightweight CNN graphic encoder to extract graphic data features, and a bidirectional LSTM time series encoder to extract time series data features, and combines cross-modal diagnostic feature vectors to generate comprehensive diagnostic results. The reinforcement learning component automatically adjusts model parameters based on the "valid" and "invalid" signals fed back by engineers. The hybrid neural network fusion layer of the diagnostic engine module uses a gating mechanism to dynamically adjust the modal weight. For software failures, the system automatically increases the text modal weight to 80%-90% and suppresses the graphic modal and time series modal. For mechanical failures, the system increases the sum of the graphic modal and time series modal weights to more than 90%.

7. The robot fault self-service diagnosis system based on multimodal autonomous deep learning according to claim 1 is characterized in that: The interactive module uses natural language processing technology to parse the text, pictures and code fault descriptions entered by the user, generates context-related dynamic guidance questions based on the fault description keywords to supplement the diagnostic information, and highlights the faulty parts in the visual interface based on the three-dimensional semantic graph, superimposes textual maintenance steps and spare parts inventory information, supports converting the diagnostic results into Chinese, English and Japanese voice broadcasts, and displays interactive diagnostic reports to the user. The natural language processing function includes generating context-related guidance questions based on the fault description keywords.

8. The robot fault self-service diagnosis system based on multimodal autonomous deep learning according to claim 1 is characterized in that: The local deployment module uses the OPC UA industrial protocol and the Modbus industrial protocol to access the robot controller, and deploys the entire process of data collection, model inference, and result output on the enterprise's local server. Firewall rules are used to limit access to the local server data interface to only the IP address of the industrial robot controller. Blockchain smart contracts are used to perform SHA-256 hash storage on the diagnostic log, and VPN channels are used with the AES-256 encryption protocol to achieve encrypted transmission of model parameters between local servers in different workshops, ensuring data security and millisecond-level response.

Images