Maintenance operation guiding method, system and equipment based on intelligent interaction

Abstract

The invention relates to a maintenance operation guiding method, system and equipment based on intelligent interaction, and the method comprises the steps: analyzing the operation intention of a user as at least one type of equipment dismounting, fault removal and knowledge retrieval, and generating a multi-mode guiding strategy; based on a multi-modal guiding strategy, coordinating and controlling a man-machine collaborative operation link; after one job is completed, multi-dimensional verification data is constructed, and when any one-dimensional verification data exceeds a corresponding knowledge graph threshold value, intervention work is activated; and generating a guiding efficiency file which is synchronously mapped with the operation link and comprises a multi-mode guiding strategy triggering log, user response behavior sensing data and a strategy adjustment utility evaluation graph, and binding the guiding efficiency file and the unique identification code of the maintenance object through an encrypted data link and storing the guiding efficiency file and the unique identification code in a preset database. According to the method, the talent cultivation period is remarkably shortened, meanwhile, the maintenance quality stability and the knowledge inheritance efficiency are synchronously improved, and reliable technical support is provided for equipment full-life-cycle management.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of intelligent maintenance, and in particular to a maintenance operation guiding method, system and device based on intelligent interaction. BACKGROUND

[0002] The traditional maintenance operation mode generally has systematic defects in knowledge inheritance and quality control when dealing with complex equipment failures. Since the maintenance process highly depends on the individual experience accumulation of technicians, their skills usually need to be cultivated through long-term practice to master the comprehensive diagnostic ability of mechanical, hydraulic and electrical systems, resulting in a long personnel training cycle. Especially when dealing with complex failures, inexperienced personnel are prone to misjudgment due to the lack of standardized knowledge reference, while the tacit experience of experienced engineers, such as vibration touch threshold and sound spectrum recognition, is difficult to effectively convert into transferable explicit knowledge, further exacerbating the risk of skill gap faced by enterprises.

[0003] In terms of quality control, the existing technology has significant defects due to the lack of experience-oriented and quantitative standards. Since fault diagnosis relies mainly on subjective judgment, the lack of real-time sensing data and quantitative analysis tools support makes the maintenance decision-making process prone to deviation, resulting in insufficient consistency of repair solutions for similar failures. At the same time, key process parameters such as assembly torque gradient and welding temperature range are often set according to the experience of operators, and the ambiguity of execution standards can easily cause performance degradation or secondary damage to repaired components. In addition, when facing scenarios such as repair of new composite materials and troubleshooting of precision electronic systems, traditional manual repair methods inevitably cause microstructure damage due to technical gap, further restricting the reliability of maintenance quality.

[0004] In terms of technology inheritance, the existing mode has structural contradictions of low knowledge transfer efficiency. Since the apprenticeship training method is adopted, the tacit experience of core technology is prone to information loss in the coding process, making it difficult to build a systematic knowledge system. Coupled with the lack of cross-disciplinary skill training path, the integration of complex skills such as mechanical assembly and electronic diagnosis is difficult, forming a knowledge island for technicians, ultimately leading to a disconnection between maintenance capability building and equipment iterative upgrade needs. These defects collectively limit the improvement of maintenance operation efficiency and standardization level. SUMMARY

[0005] (I) Technical problems to be solved

[0006] In view of the above-mentioned shortcomings and deficiencies of the prior art, the present application provides a maintenance operation guiding method, system and device based on intelligent interaction, which solves the technical problems of the traditional maintenance mode that relies too much on individual experience, resulting in large fluctuations in maintenance quality, and the lack of standardized process parameters and effective knowledge inheritance mechanism, further causing systematic defects such as long personnel training cycle, poor repair consistency and technical iteration lag.

[0007] (II) Technical Solution

[0008] To achieve the above object, the main technical solution adopted by the present application comprises:

[0009] In a first aspect, the embodiments of the present application provide a maintenance operation guiding method based on intelligent interaction, comprising:

[0010] According to the user input instruction or voice instruction, the operation intention of the user is analyzed as at least one type of device disassembly, troubleshooting, and knowledge retrieval, and a function module adapted to the intention type is dynamically called to generate a multi-modal guiding strategy;

[0011] Based on the multi-modal guiding strategy, at least one human-machine collaborative operation link is coordinated and controlled, including disassembly action sequence guidance, collaborative action sequence arrangement of intelligent feeding device, fault reasoning decision, assembly compatibility verification, and retrieval information feedback of intelligent maintenance board;

[0012] After completing one operation in the operation link, multi-dimensional verification data is constructed according to the fusion analysis result of multi-modal sensing data, and when any dimensional verification data exceeds the corresponding knowledge graph threshold, intervention work is activated;

[0013] A guiding efficiency archive containing multi-modal guiding strategy trigger log, user response behavior sensing data, and strategy adjustment utility evaluation graph is generated and stored in a preset database through an encrypted data chain and a unique identification code of the maintenance object.

[0014] Optionally, according to the user input instruction or voice instruction, the operation intention of the user is analyzed as at least one type of device disassembly, troubleshooting, and knowledge retrieval, and a function module adapted to the intention type is dynamically called to generate a multi-modal guiding strategy, comprising:

[0015] According to the received user input instruction or real-time captured voice instruction stream, the operation intention type is determined as at least one of device disassembly, troubleshooting, and knowledge retrieval;

[0016] In response to the operation intention category, a preset function module cluster containing a three-dimensional disassembly guiding module, a fault reasoning decision module, and a knowledge semantic association module is dynamically called;

[0017] Based on the output result of the called function module, a multi-modal guiding strategy adapted to the operation intention category is generated, and the strategy comprises:

[0018] The device disassembly and assembly guidance strategy is to call a three-dimensional disassembly and assembly guidance module to generate a device disassembly and assembly instruction set containing disassembly component topology decomposition display, disassembly sequence, assembly sequence, intelligent rack bin coding, and mechanical arm trajectory planning parameters, wherein the mechanical arm end effector switches tool clamping mode based on the disassembly / installation stage, and any disassembly and assembly process jump is triggered through collected gesture data or voice data;

[0019] The troubleshooting guidance strategy is to call a fault reasoning decision module to construct a troubleshooting instruction set containing maintenance operation chain with disassembly sequence constraints, fault positioning logic tree, and assembly compatibility verification rules, wherein the maintenance operation chain generates process interlocking conditions between the disassembly stage, maintenance stage, inspection stage, and installation stage by associating a historical case library; the fault positioning logic tree contains a minimum maintainable unit matching algorithm based on device topology and an associated part replacement priority sequence;

[0020] The knowledge retrieval guidance strategy is to call a knowledge semantic association module to generate multi-modal knowledge data matching the physical maintenance scene space.

[0021] Optionally, based on the multi-modal guidance strategy, the human-machine collaborative work link that coordinates control at least one of the following: disassembly and assembly action sequence guidance, collaborative action sequence arrangement of intelligent feeding devices, fault reasoning decision, assembly compatibility verification, and retrieval information feedback of intelligent maintenance boards, includes:

[0022] When the multi-modal guidance strategy is the device disassembly and assembly guidance strategy, a three-dimensional disassembly and assembly guidance module is called to load a device structure decomposition model and map the spatial topology relationship of parts in the intelligent maintenance board, real-time collect user gestures or voice instruction streams in a pre-set operation area, activate the disassembly stage or the installation stage when the feature vectors of the gesture or voice instruction stream match the pre-set trigger conditions, display disassembly step decomposition graphics and text guidance, associated tool three-dimensional model, and current step part parameters in the intelligent maintenance board during the disassembly process, and real-time update the data collected by the wireless measurement tool participating in the disassembly work to the intelligent maintenance board, while real-time monitoring the disassembly action and disassembly sequence of the user through an industrial camera, control the mechanical arm to grab the parts when a part separation event is detected, and perform access operations according to the mechanical arm trajectory planning path obtained by analyzing the intelligent rack bin coding; and during the assembly process, display assembly step decomposition graphics and text guidance, associated tool three-dimensional model, and current step part parameters in the intelligent maintenance board, and real-time update the data collected by the wireless measurement tool to the intelligent maintenance board, while real-time monitoring the assembly action and assembly sequence of the user through an industrial camera, control the mechanical arm to grab the parts when a part assembly is detected, and perform access operations according to the mechanical arm trajectory planning path obtained by analyzing the intelligent rack bin coding;

[0023] When the multi-modal guidance strategy is a troubleshooting guidance strategy, a fault reasoning decision module is called to generate a candidate troubleshooting set containing disassembly sequence constraint conditions, minimum repairable unit positioning rules, and associated part replacement priority sequences from a historical fault case database in response to user input fault phenomenon text; based on the user-selected candidate troubleshooting scheme, a maintenance operation chain with process interlocking of the serial disassembly stage, maintenance stage, inspection stage, and installation stage is generated; at the same time, a tool list three-dimensional diagram corresponding to the current step, associated component disassembly animation, and process parameter comparison table uploaded by the wireless measurement tool are generated in the intelligent maintenance board; when it is detected that the installation stage is completed, an assembly compatibility check based on the historical maintenance data is triggered, which contains the matching degree verification of the installed component hash value and the equipment topology database and the tolerance accumulation simulation calculation of the adjacent part interface parameters;

[0024] When the multi-modal guidance strategy is a knowledge retrieval guidance strategy, a knowledge semantic association module is called to generate a three-dimensional structure decomposition graph that is spatially mapped to the physical maintenance scene, and a multimedia index of the maintenance procedure is associated to form dynamic annotation information.

[0025] Optionally, a fault reasoning decision module is called to generate a candidate troubleshooting set containing disassembly sequence constraint conditions, minimum repairable unit positioning rules, and associated part replacement priority sequences from a historical fault case database in response to user input fault phenomenon text, which includes:

[0026] Based on the syntax rule engine and the Bi-LSTM+CRF joint model, fault knowledge extraction is performed on historical maintenance cases containing structured data, fault associated entities, and topological dependency relationships, wherein the syntax rule engine is configured to perform dependency syntax analysis on fault case text in historical maintenance cases according to a pre-configured device disassembly sequence constraint condition template to extract structured data containing fault phenomena, mandatory procedures, and prohibited operations; the Bi-LSTM+CRF joint model is configured to perform sequence labeling on unstructured text in historical maintenance cases to simultaneously extract fault associated entities and topological dependency relationships;

[0027] The extracted fault knowledge is injected into a Neo4j graph database to generate a multi-dimensional fault constraint graph containing fault nodes, procedure constraint edges, minimum repairable unit nodes, and part replacement priority weights;

[0028] In response to user input fault phenomenon retrieval instructions, multi-hop reasoning is performed in the multi-dimensional fault constraint graph to generate a minimum cut set that satisfies the disassembly sequence constraint conditions as the first candidate subset;

[0029] The LDA topic probability model is used to perform semantic clustering on historical maintenance cases that are not involved in fault knowledge extraction. The central topic distribution vector of each cluster is generated. The fault phenomenon text input by the user is converted into a query topic distribution vector. The cosine similarity between the query topic distribution vector and the central topic distribution vector of each cluster is calculated as the topic relevance. The historical maintenance cases contained in the clusters with cosine similarity higher than the preset cosine threshold are selected to form the second candidate subset.

[0030] The case texts of the second candidate subset are parsed using the HanLP word segmenter to identify new entities not registered in the multidimensional fault constraint graph. Based on the hierarchical relationships of the equipment topology, the superordinate concepts of the new entities are derived and inheritance constraints are established to complete the ontology-driven dynamic knowledge completion of the case subset.

[0031] Inject the established ontology inheritance relationship constraints back into the multi-dimensional fault constraint graph to generate an incremental version graph. Perform secondary reasoning optimization on the first candidate subset based on the incremental graph.

[0032] According to the multi-hop reasoning results and the topic relevance, the union of the first candidate subset and the second candidate subset of the secondary reasoning optimization is ranked by confidence, and a set of candidate troubleshooting solutions with priority labels is output.

[0033] Optionally, when the multimodal guidance strategy is a knowledge retrieval guidance strategy, the knowledge semantic association module is called to generate a three-dimensional structural decomposition map mapped to the physical maintenance scene space, and the multimedia index of the maintenance procedures is associated to form dynamic annotation information including:

[0034] Based on the spatial coordinate data of the physical maintenance scene, the knowledge semantic association module is called to generate a three-dimensional structural decomposition map that includes the parts disassembly hierarchy identification and assembly topology dependency relationship mapped with the spatial coordinates;

[0035] The LDA topic probability model is used to cluster the historical maintenance case texts in the historical maintenance case library, extract keyword clusters as semantic labels, and index associations are established between the labels and the multimedia resources corresponding to the process nodes in the three-dimensional structure decomposition map. The multimedia resources include one or more combinations of text analysis documents, fault feature images, and operation demonstration videos.

[0036] Based on user interaction actions or tool proximity detection results, a dynamic annotation box is superimposed on the target part node of the 3D structural decomposition diagram. The annotation box is embedded with interactive controls of associated multimedia indexes for real-time retrieval and display of multimedia resources.

[0037] Optionally, after completing a task in the task chain, multi-dimensional verification data is constructed based on the fusion analysis results of the multimodal sensor data. When the verification data of any dimension exceeds the corresponding knowledge graph threshold, the activation intervention work includes:

[0038] synchronously, a multi-source perception data space-time correlation matrix is constructed in response to the completion signal of a single job in the job link;

[0039] Based on the process constraint rules of the preset device topology structure, the correlation matrix is dimensionally reduced, and a process completeness index is extracted as a topological matching degree of the executed steps and corresponding processes;

[0040] By calculating the Mahalanobis distance between the torque / displacement measurement sequence and the tolerance band reference value in the preset knowledge graph, the assembly precision deviation degree is obtained;

[0041] The semantic features of the voice instructions and the action features of the user gestures are analyzed, and the safety specification compliance rate is calculated as the proportion of compliant operation actions combined with the behavior specification items in the preset safety protocol library;

[0042] According to the process completeness index, the assembly precision deviation degree, and the safety specification compliance rate, a multi-dimensional verification vector is obtained;

[0043] The multi-dimensional verification vector is matched with the process threshold in the preset knowledge graph. When the process completeness index is lower than the process procedure integrity threshold, or the assembly precision deviation degree exceeds the cumulative simulation value of the tolerance, or the safety specification compliance rate does not meet the minimum standard of the safety protocol, an intervention work including intelligent maintenance board information projection and voice correction instruction broadcast is triggered.

[0044] Optionally, a guidance efficiency archive including a multi-modal guidance strategy trigger log, user response behavior sensing data, and a strategy adjustment utility evaluation graph is generated, and is stored in a preset database through an encrypted data chain and a unique identification code of the maintenance object, including:

[0045] Based on the timestamp alignment result of the device running state snapshot at the multi-modal guidance strategy trigger time and the user operation event stream, a guidance strategy trigger log including strategy type encoding, trigger space coordinates, and parameter adjustment information is generated;

[0046] Synchronously aggregate the user gesture motion trajectory data captured by the industrial camera, the torque / displacement measurement sequence uploaded by the wireless measurement tool, and the semantic analysis result stream of the voice instruction, and generate a user response behavior sensing data set with action trigger time and response time delay difference value labels;

[0047] By inputting the process completeness index change rate before and after strategy adjustment, the assembly precision deviation degree convergence gradient, and the safety specification compliance rate increment into the pre-trained reinforcement learning model, a strategy adjustment utility evaluation graph is generated, with strategy optimization suggestions as nodes and utility evaluation values as edge weights.

[0048] Based on the guidance strategy trigger log, user response behavior sensing dataset and strategy adjustment utility evaluation atlas, a guidance performance archive is constructed, the archive data is block encrypted based on a hash chain generated based on the serial number of the maintenance object, and the encrypted block is bound with multiple sub-identification codes obtained by decomposing the serial number, and the historical maintenance cases are stored.

[0049] In a second aspect, an embodiment of the present application provides a maintenance operation guidance system based on intelligent interaction, comprising:

[0050] A guidance strategy output module is configured to parse the operation intention of the user into at least one type of device disassembly, troubleshooting and knowledge retrieval according to the user input instruction or voice instruction, and dynamically call a function module adapted to the intention type to generate a multi-modal guidance strategy.

[0051] An operation link execution module is configured to coordinate and control at least one human-machine collaborative operation link including disassembly action sequence guidance, collaborative action sequence arrangement of the intelligent feeding device, fault reasoning decision, assembly compatibility verification and retrieval information feedback of the intelligent maintenance board based on the multi-modal guidance strategy.

[0052] An intervention work module is configured to construct multi-dimensional verification data according to the fusion analysis result of the multi-modal sensing data after one operation in the to-be-completed operation link, and activate the intervention work when any dimensional verification data exceeds the corresponding knowledge graph threshold.

[0053] A data storage module is configured to generate a guidance performance archive including the multi-modal guidance strategy trigger log, user response behavior sensing data and strategy adjustment utility evaluation atlas which are synchronized with the operation link, and store the archive in a preset database through an encrypted data chain and the unique identification code of the maintenance object.

[0054] In a third aspect, an embodiment of the present application provides a maintenance operation guidance device based on intelligent interaction, comprising: a hardware execution unit including an intelligent maintenance board, an industrial camera, a mechanical arm, an intelligent shelf, an operation workbench, a wireless measurement tool and a voice microphone; an intelligent feeding device including an intelligent shelf, a mechanical arm, an electric control box and a workbench; and a server configured to execute the method as described above.

[0055] Optionally, the intelligent shelf is built-in with multiple bins, the server controls the step motor to drive the bin circulation queue to move, and triggers an infrared positioning signal when the target bin reaches the discharging position; the mechanical arm is configured with six-axis freedom, responds to the infrared positioning signal to perform a bin grabbing action, and transfers the bin to the magnetic positioning groove of the operation workbench; the electric control box is built-in with a CAN bus communication interface, synchronously receives the instructions issued by the server and the torque / displacement measurement sequence uploaded by the wireless measurement tool.

[0056] (III) Beneficial Effects

[0057] The beneficial effects of the present application are: the intelligent adaptation and dynamic optimization mechanism of the multi-modal guidance strategy of the present application effectively solves the key defects of the traditional maintenance mode. First, the semantic analysis module identifies the user's operation intention in real time, and automatically matches the function module based on the intention type, so that the guidance strategy of the device disassembly, fault diagnosis and other scenes is accurately adapted to the maintenance demand, thereby significantly improving the pertinence and accuracy of the operation guidance. On this basis, the closed-loop control architecture of the man-machine cooperative operation link is adopted, the disassembly action guidance, the feeding device cooperation and other core process are standardized, the operation deviation is corrected in real time through multi-dimensional sensing feedback, thereby ensuring the strict consistency of the maintenance process parameter execution, and the quality fluctuation risk caused by experience difference is greatly reduced.

[0058] At the same time, based on the threshold determination mechanism of multi-dimensional verification data, the multi-modal correction intervention is triggered immediately when abnormal operation is detected, the composite guidance of information labeling and voice prompt is realized, the real-time quality control of the maintenance process is realized, the operation error diffusion is blocked in real time, and the process compliance rate of the key process (such as torque gradient control, component assembly compatibility) is ensured. More importantly, by synchronously constructing the encrypted binding guidance efficiency file, the strategy optimization track and operation response characteristics are recorded completely, not only the traceable iterative data source is provided for the maintenance knowledge base, but also the decision model is continuously optimized through machine learning, and then the positive cycle of self-evolution of maintenance experience is formed.

[0059] The synergistic effect of the above technical features not only significantly shortens the talent training cycle, but also ensures the stability of maintenance quality and the synchronous improvement of knowledge inheritance efficiency, providing reliable technical support for the whole life cycle management of equipment. BRIEF DESCRIPTION OF DRAWINGS

[0060] Figure 1 The flowchart of the method provided for the embodiment of the present application is shown;

[0061] Figure 2 The specific flowchart of step S1 of the method provided for the embodiment of the present application is shown;

[0062] Figure 3 The specific flowchart of step S2 of the method provided for the embodiment of the present application is shown;

[0063] Figure 4 The device structure decomposition model of the method provided for the embodiment of the present application is shown;

[0064] Figure 5 The device disassembly schematic diagram of the method provided for the embodiment of the present application is shown;

[0065] Figure 6 The device fault elimination schematic diagram of the method provided for the embodiment of the present application is shown;

[0066] Figure 7 The specific flowchart of S22 of the method provided by the embodiment of the application is shown in the figure;

[0067] Figure 8 The generation flowchart of the multi-dimensional fault constraint graph of the method provided by the embodiment of the application is shown in the figure;

[0068] Figure 9 The generation flowchart of the candidate troubleshooting set of the method provided by the embodiment of the application is shown in the figure;

[0069] Figure 10 The specific flowchart of S23 of the method provided by the embodiment of the application is shown in the figure;

[0070] Figure 11 The specific flowchart of S3 of the method provided by the embodiment of the application is shown in the figure;

[0071] Figure 12 The specific flowchart of S4 of the method provided by the embodiment of the application is shown in the figure;

[0072] Figure 13 The composition diagram of the maintenance operation guiding device provided by the embodiment of the application is shown in the figure. DETAILED DESCRIPTION

[0073] In order to better explain the application and facilitate understanding, the application is described in detail through specific embodiments in combination with the accompanying drawings.

[0074] As shown in the figure, Figure 1 The maintenance operation guiding method based on intelligent interaction provided by the embodiment of the application comprises the following steps: according to a user input instruction or a voice instruction, analyzing the operation intention of the user into at least one type of device disassembly, fault elimination and knowledge retrieval, dynamically calling a function module adapted to the intention type to generate a multi-modal guiding strategy; based on the multi-modal guiding strategy, coordinating and controlling at least one human-machine collaborative operation link comprising a disassembly action sequence guide, a collaborative action sequence arrangement of an intelligent feeding device, a fault reasoning decision, an assembly compatibility verification and a retrieval information feedback of an intelligent maintenance board; after completing one operation in the operation link, constructing multi-dimensional verification data according to the fusion analysis result of multi-modal sensing data, activating an intervention work when any dimension verification data exceeds a corresponding knowledge graph threshold; generating a guiding efficiency archive comprising a multi-modal guiding strategy trigger log, user response behavior sensing data and a strategy adjustment utility evaluation graph which is synchronously mapped with the operation link, and storing the archive into a preset database through an encrypted data chain and a unique identification code of a maintenance object.

[0075] The intelligent adaptation and dynamic optimization mechanism of the multi-modal guidance strategy of the application effectively solves the key defects of the traditional maintenance mode. First, the semantic analysis module identifies the user's operation intention in real time, and automatically matches the function module based on the intention type, so that the guidance strategy of device disassembly, fault diagnosis and other scenes is accurately adapted to the maintenance demand, thereby significantly improving the pertinence and accuracy of the operation guide. On this basis, the closed-loop control architecture of the man-machine cooperative operation link is adopted to standardize the key working procedures such as disassembly action guidance and feeding device cooperation, and the operation deviation is corrected in real time through multi-dimensional sensing feedback, thereby ensuring the strict consistency of the maintenance process parameter execution and greatly reducing the quality fluctuation risk caused by experience difference.

[0076] At the same time, based on the threshold determination mechanism of multi-dimensional verification data, the multi-modal correction intervention is triggered immediately when abnormal operation is detected, and through the composite guidance of information labeling and voice prompt, the real-time quality control of the maintenance process is realized, the operation error diffusion is blocked in real time, and the process compliance rate of key working procedures (such as torque gradient control and component assembly compatibility) is ensured. More importantly, by synchronously constructing the encrypted binding guidance efficiency file, the strategy optimization track and operation response characteristics are recorded completely, not only providing traceable iterative data source for the maintenance knowledge base, but also continuously optimizing the decision model through machine learning, thereby forming a positive cycle of self-evolution of maintenance experience.

[0077] The synergistic effect of the above technical features significantly shortens the talent training cycle, while ensuring the stability of maintenance quality and the synchronous improvement of knowledge inheritance efficiency, providing reliable technical support for the whole life cycle management of equipment.

[0078] In order to better understand the above technical solutions, the exemplary embodiments of the application will be described in more detail below with reference to the accompanying drawings. Although the exemplary embodiments of the application are shown in the drawings, it should be understood that the application can be implemented in various forms and should not be limited by the embodiments described herein. On the contrary, these embodiments are provided to enable a clearer, more thorough understanding of the application and to convey the complete scope of the application to those skilled in the art.

[0079] Specifically, the embodiment of the application provides a maintenance operation guidance method based on intelligent interaction, which comprises:

[0080] S1, according to the user input instruction or voice instruction, the operation intention of the user is analyzed as at least one type of device disassembly, fault elimination, knowledge retrieval, and the function module adapted to the intention type is dynamically called to generate a multi-modal guidance strategy.

[0081] Further, as Figure 2 shown, step S1 comprises:

[0082] S11, determine an operation intention type as at least one of device disassembly, troubleshooting, and knowledge retrieval based on a received user input instruction or a real-time captured voice instruction stream.

[0083] S12, dynamically invoke a preset functional module cluster containing a three-dimensional disassembly guidance module, a fault reasoning decision module, and a knowledge semantic association module in response to the operation intention category.

[0084] S13, generate a multi-modal guidance strategy adapted to the operation intention category based on the output result of the invoked functional module.

[0085] The strategy includes:

[0086] The device disassembly guidance strategy is to invoke the three-dimensional disassembly guidance module to generate a device disassembly instruction set containing disassembly component topology structure decomposition display, disassembly sequence, assembly sequence, intelligent shelf bin coding, and mechanical arm trajectory planning parameters, wherein the mechanical arm end effector switches tool clamping mode based on the disassembly / installation stage, and triggers any disassembly process jump through collected gesture data or voice data;

[0087] The troubleshooting guidance strategy is to invoke the fault reasoning decision module to construct a troubleshooting instruction set containing maintenance operation chain, fault positioning logic tree, and assembly compatibility verification rules constrained by disassembly sequence, wherein the maintenance operation chain generates process interlocking conditions between the disassembly stage, maintenance stage, inspection stage, and installation stage by associating the historical case library; the fault positioning logic tree contains a minimum maintainable unit matching algorithm based on device topology structure and an associated part replacement priority sequence;

[0088] The knowledge retrieval guidance strategy is to invoke the knowledge semantic association module to generate multi-modal knowledge data matched with the physical maintenance scene space.

[0089] S2, based on the multi-modal guidance strategy, coordinate control of at least one of the human-machine collaborative work link containing disassembly action sequence guidance, coordinated action sequence arrangement of intelligent feeding device, fault reasoning decision, assembly compatibility verification, and retrieval information feedback of intelligent maintenance board.

[0090] Further, as shown in Figure 3 Step S2 includes:

[0091] S21, when the multi-modal guidance strategy is the equipment disassembly and assembly guidance strategy, calling the three-dimensional disassembly and assembly guidance module, loading the equipment structure decomposition model in the intelligent maintenance board and mapping the spatial topological relationship of the parts, collecting the user gestures or voice instruction stream in the preset operation area in real time, when the feature vector of the gesture or voice instruction stream matches the preset trigger condition, activating the disassembly stage or the assembly stage; in the disassembly process, displaying the disassembly step decomposition graphic guide, the associated tool three-dimensional model and the current step part parameters in the intelligent maintenance board, and updating the data collected by the wireless measurement tool to the intelligent maintenance board in real time, and simultaneously monitoring the disassembly action and the disassembly sequence of the user in real time through the industrial camera, when detecting the part separation event, controlling the mechanical arm to grab the part and performing the access operation according to the mechanical arm trajectory planning path obtained by analyzing the intelligent shelf bin code; and in the assembly process, displaying the assembly step decomposition graphic guide, the associated tool three-dimensional model and the current step part parameters in the intelligent maintenance board, and updating the data collected by the wireless measurement tool to the intelligent maintenance board in real time, and simultaneously monitoring the assembly action and the assembly sequence of the user in real time through the industrial camera, when detecting that the part is assembled in place, controlling the mechanical arm to grab the part and performing the access operation according to the mechanical arm trajectory planning path obtained by analyzing the intelligent shelf bin code.

[0092] In an embodiment, referring to Figure 4 It can be known that when the equipment disassembly and assembly guidance strategy is executed, the three-dimensional disassembly and assembly guidance module loads the equipment structure decomposition model in the intelligent maintenance board, and specifically provides visual cognitive support for the internal structure of the equipment through the VE distribution pump structure diagram, wherein the parts are displayed in the form of decomposition connection and are labeled with unique numbers, and a matching table accurately corresponds the numbers and the part names, helping the user to quickly establish the spatial topological relationship and the part function association.

[0093] As Figure 5 shown, after the user enters the equipment disassembly module through the home page entrance, the interface presents the visual map of the detachable part, supporting quick positioning based on part position retrieval. After triggering the "start disassembly" instruction, the process guidance state is entered, and whether to enter the next operation step is dynamically judged by real-time analyzing the feature vector of the user gesture or voice instruction. During the disassembly process, the intelligent maintenance board synchronously displays the graphic guide, the tool three-dimensional model and the associated part information, and the measurement data (such as actual torque value, size accuracy) of the wireless torque wrench or vernier caliper is transmitted to the board in real time for visual comparison through the acquisition terminal. At the same time, the assembly action and the assembly sequence of the user are monitored in real time through the industrial camera, when the industrial camera captures the part separation event, the mechanical arm is automatically controlled to carry the disassembled part to the specified bin position of the intelligent shelf according to the preset trajectory.

[0094] The user gesture analysis described above relies on the use of a YOLOV5 model deployed on a Windows system to realize maintenance scene recognition, uses its high-precision detection capability to capture hand key point coordinates and motion trajectories in real time, generates a gesture feature vector and matches it with a pre-defined instruction library; in special cases (such as mobile terminal deployment or computing resource limited environment), switch to the lightweight NanoDet model, deploy it on the mobile terminal Android system to perform maintenance scene recognition, realize low-delay gesture analysis through its optimized GFocalLoss loss function, and ensure consistent cross-platform instruction response consistency. Through the dynamic adaptation of the two models, the application conditions of multiple environments and multiple terminals are met.

[0095] The user voice recognition described above is based on the voice recognition engine of the Xunfei SDK. After converting the user voice instruction into text, it is mapped to the pre-defined operation instruction set (such as "pause disassembly" "switch tool model") through the intent classification model, and the environmental noise interference is filtered through the semantic error correction mechanism built-in the SDK, to ensure the accuracy of instruction execution in complex maintenance scenarios. The voice recognition result is interacted with the robot arm control module and the intelligent maintenance board data interface in real time through REST API, forming a closed-loop control link.

[0096] The trajectory of the robot arm is generated by analyzing the spatial mapping relationship of the shelf code, to ensure the optimality of the access path. Specifically, first, each bin unit of the intelligent shelf is built-in with a unique code identifier, and the code is embedded with structured data such as bin three-dimensional coordinates, volume threshold and adjacent bin topological relationship; when the robot arm needs to perform access operation, based on the type of parts triggered in the disassembly / installation stage, the target bin code is obtained through radio frequency identification or visual two-dimensional code scanning, then the spatial mapping analysis module is called to convert the code information into a three-dimensional spatial coordinate set in the robot arm base coordinate system, and the pre-set shelf obstacle point cloud data is loaded to construct a dynamic environment model. Then, input the analyzed target coordinates and the current pose of the robot arm into the kinematics model, use the RRT*(rapidly-exploring random tree star) algorithm based on time optimization to generate a collision-free path, and fuse joint acceleration constraints and energy consumption weight factors during trajectory interpolation, to ensure that the robot arm end effector completes displacement at the minimum time-energy cost. During execution, the industrial camera continuously collects point cloud data of the actual work scene, compares the pre-set environment model with the physical space deviation in real time through SLAM technology, if the bin displacement or temporary obstacle intrusion is detected, the path re-planning mechanism is triggered, the obstacle motion trend is predicted based on Kalman filtering and the trajectory parameters are dynamically adjusted, to finally realize the real-time optimality closed-loop control of the access path.

[0097] S22, when the multi-modal guidance strategy is a troubleshooting guidance strategy, calling a fault reasoning decision module, in response to the user input fault phenomenon text, calling a historical fault case database to generate a candidate troubleshooting set containing disassembly sequence constraint conditions, minimum repairable unit positioning rules and associated part replacement priority sequences; based on the user selected candidate troubleshooting scheme, generating a maintenance operation chain of process interlocking of the serial disassembly stage, maintenance stage, inspection stage and installation stage; at the same time, generating a tool list three-dimensional diagram corresponding to the current step, associated parts disassembly animation and process parameter comparison table uploaded by the wireless measurement tool in the intelligent maintenance board; when it is detected that the installation stage is completed, triggering the assembly compatibility verification based on the matching degree verification of the installed component hash value and the equipment topology database and the tolerance accumulation simulation calculation of the adjacent part interface parameters.

[0098] Reference Figure 6 In the installation stage, the disassembly process is reconstructed with reverse logic, the assembly step decomposition guide is adapted to the tool model dynamically, the user is guided to complete the component assembly in order, and whether to enter the next operation step is dynamically judged by real-time analysis of the feature vector of the user gesture or voice instruction. During the assembly process, the industrial camera continuously monitors the user's assembly action and assembly sequence, and when it is detected that the component is assembled in place, the mechanical arm is triggered to grab the next part to be assembled from the intelligent shelf, and the access path is planned based on the shelf coding analysis result. In addition, the key assembly parameters (such as bolt pre-tightening force, fit clearance) are uploaded to the board in real time through the wireless measurement tool, and are automatically checked with the process standard value to form a closed-loop quality control link. In the whole process of disassembly and installation, multi-modal guidance resources such as graphics and video are always synchronized with the operation steps to ensure the standardized execution of maintenance work.

[0099] Further, as Figure 7 , Figure 8 and Figure 9 , the "calling a fault reasoning decision module, in response to the user input fault phenomenon text, calling a historical fault case database to generate a candidate troubleshooting set containing disassembly sequence constraint conditions, minimum repairable unit positioning rules and associated part replacement priority sequences" in step S22 includes:

[0100] S221, based on the syntax rule engine and the Bi-LSTM+CRF joint model, the historical maintenance cases are implemented fault knowledge extraction containing structured data, fault associated entities and topological dependency relationship, wherein the syntax rule engine is configured to: according to the pre-configured device disassembly sequence constraint condition template, the dependency syntax analysis of the fault case text in the historical maintenance case is carried out, the structured data containing fault phenomenon, forced process and disabled operation is extracted, and the causal order conforming to the physical assembly or maintenance is ensured; the Bi-LSTM+CRF joint model is configured to sequence label the unstructured text in the historical maintenance case, and synchronously extract the fault associated entity and the topological dependency relationship. The Bi-LSTM+CRF joint model is a combination of bidirectional long short-term memory network and conditional random field. The bidirectional long short-term memory network captures the bidirectional semantic dependency of the text sequence through the context information, and the conditional random field is used to optimize the global sequence consistency of the entity label, avoiding illegal label transfer (such as “fault reason” label cannot be connected with “disabled operation”).

[0101] S222, the extracted fault knowledge is injected into the Neo4j graph database, and a multi-dimensional fault constraint graph containing fault nodes (such as “bearing abnormal sound”), process constraint edges (such as “disassembly step 2 → must be executed after the completion of the previous step 1”), minimum maintainable unit nodes (such as “gearbox gear set”) and part replacement priority weight (such as “gear shaft replacement priority = high”) is generated.

[0102] S223, in response to the fault phenomenon retrieval instruction input by the user, multi-hop reasoning is performed in the multi-dimensional fault constraint graph, and the minimum cut set satisfying the disassembly sequence constraint condition is generated as the first candidate subset. The multi-hop reasoning of multi-level traversal along the connection edge in the multi-dimensional fault constraint graph screens out the minimum cut set that meets the device disassembly sequence constraint, that is, the maintenance unit combination that covers all related faults and has the minimum process conflict, as the first candidate subset.

[0103] S224, the LDA topic probability model is used to perform semantic clustering on the historical maintenance cases which do not participate in fault knowledge extraction, and the center theme distribution vector of each cluster is generated. The fault phenomenon text input by the user is converted into a query theme distribution vector, the cosine similarity between the query theme distribution vector and the center theme distribution vector of each cluster is calculated as the theme correlation degree, and the historical maintenance cases contained in the cluster with cosine similarity higher than the preset threshold are selected to form the second candidate subset.

[0104] S225, parse the case text of the second candidate subset through the HanLP tokenizer, identify new entities (such as unregistered part names) that are not registered in the multi-dimensional fault constraint graph, and according to the hierarchical relationship of the device topology (such as "engine → cylinder → piston ring"), deduce the upper concept of the new entity (such as "piston ring belongs to the cylinder subsystem") and establish inheritance relationship constraints (such as "piston ring fault is automatically associated with cylinder detection") to complete the ontology-driven dynamic knowledge completion of the case subset.

[0105] S226, the established ontology inheritance relationship constraint is injected into the multi-dimensional fault constraint graph to generate an incremental version graph, and the first candidate subset is optimized based on the incremental graph.

[0106] S227, according to the multi-hop reasoning result and the theme correlation degree, the union of the first candidate subset and the second candidate subset optimized by secondary reasoning is sorted by confidence, and the candidate troubleshooting scheme set with priority annotation is output.

[0107] S23, when the multi-modal guidance strategy is the knowledge retrieval guidance strategy, a knowledge semantic association module is called to generate a three-dimensional structure decomposition graph mapped with a physical maintenance scene space, and a multimedia index of a maintenance procedure is associated to form dynamic annotation information. The knowledge retrieval module integrates detailed knowledge bases of equipment structure, working principle, maintenance procedure, and typical faults, and presents them in the form of multi-modal such as text analysis document, fault feature picture, and operation demonstration video, helping users to deeply learn the device structure and maintenance logic. Users can enter the module by clicking the "knowledge retrieval" button on the home page, input keywords, and link multiple selection filtering conditions (equipment structure, maintenance procedure, typical fault), triggering intelligent three-dimensional scene guidance and dynamic knowledge annotation.

[0108] Further, as shown in Figure 10 , step S23 comprises:

[0109] S231, based on the spatial coordinate data of the physical maintenance scene, a knowledge semantic association module is called to generate a three-dimensional structure decomposition graph containing part disassembly level identification and assembly topology dependency relationship mapped with spatial coordinates.

[0110] S232, the historical maintenance case texts in the historical maintenance case library are clustered through the LDA topic probability model, the keyword clusters are extracted as semantic labels, and the labels are indexed and associated with the multimedia resources of the corresponding process nodes in the three-dimensional structure decomposition graph. The multimedia resources include one or more combinations of text analysis documents, fault feature pictures, and operation demonstration videos.

[0111] S233, superimpose a dynamic annotation frame on the target part node of the three-dimensional structure decomposition graph according to the user interaction action or tool proximity detection result, and embed an interactive control of the associated multimedia index in the annotation frame, which is used to call and display the multimedia resource in real time.

[0112] S3, after one task in the task chain is completed, multi-dimensional verification data is constructed according to the fusion analysis result of the multi-modal sensing data, and when any dimensional verification data exceeds the corresponding knowledge graph threshold, intervention work is activated.

[0113] Further, as shown in Figure 11 , step S3 includes:

[0114] S31, in response to the completion signal of a single task in the task chain, time-space synchronization is performed on the user gesture motion trajectory data captured by the industrial camera, the torque / displacement measurement sequence uploaded by the wireless measurement tool, and the semantic analysis result stream of the voice instruction, and a multi-source perception data time-space correlation matrix is constructed. Time stamp alignment and space coordinate system conversion are used to generate a multi-source perception data time-space correlation matrix, as shown in:

[0115] Time stamp Gesture trajectory coordinates Torque value Displacement Semantic instruction ​ (x1,y1,z1) 45.2 0.12 None [t2] (x2, y2, z2) 48.7 0.15 "Fastening complete"

[0116] S32, based on the process constraint rules of the preset device topology structure, the correlation matrix is dimensionally reduced, and a process completeness index is extracted as the topological matching degree of the executed step and the corresponding process.

[0117] The process completeness index is used to quantify the topological matching degree of the executed task step and the preset process procedure, which needs to consider three types of indexes, including step integrity, timing correctness, and dependency compliance. The formula is defined as follows:

[0118]

[0119] In the formula, C CI ∈[0,1], N completed is the number of actual completed process steps, N total is the total number of steps required by the process procedure, the default weight step integrity weight w1=0.4, N ordered is the number of adjacent step pairs executed in correct order (for example, in steps A→B→C, A→B and B→C are correct orders), N pairs is the total number of adjacent step pairs defined by the process procedure (if the total number of steps is n, then N pairs =n-1), the default weight timing correctness weight w2=0.3, N dependency_met is the number of steps that meet all the prerequisites (for example, step C needs to be executed after steps A and B are completed, if both A and B are completed, then 1 is counted, otherwise 0 is counted), N dependencyThe total number of steps defined in the process procedure for dependency relationship, the default weight dependency relationship compliance weight w3=0.3.

[0120] S33, the assembly precision deviation is obtained by calculating the Mahalanobis distance of the torque / displacement measurement sequence and the tolerance band reference value in the preset knowledge graph. The assembly precision deviation formula is:

[0121]

[0122] In the formula, X is the measurement data vector, μ is the reference mean, S is the covariance matrix, and T is the transpose.

[0123] S34, the semantic features of the voice instruction and the action features of the user gesture are analyzed, and the behavior specification items in the preset safety protocol library are combined to calculate the safety specification compliance rate as the proportion of compliant operation actions. The safety specification compliance rate=the number of compliant actions / the total number of actions×100%.

[0124] S35, according to the process completeness index, the assembly precision deviation, and the safety specification compliance rate, a multi-dimensional verification vector is obtained, such as V=[process completeness index, assembly precision deviation, safety specification compliance rate].

[0125] S36, the multi-dimensional verification vector is matched with the process threshold in the preset knowledge graph. When the process completeness index is lower than the process procedure integrity threshold, or the assembly precision deviation exceeds the cumulative simulation value of the tolerance, or the safety specification compliance rate does not reach the minimum standard of the safety protocol, the intervention work including the projection of the intelligent maintenance board information and the broadcast of the voice correction instruction is triggered.

[0126] Specifically, the minimum threshold of the process completeness index is 0.9, the maximum allowable value of the assembly precision deviation is 1.2, and the minimum standard of the safety specification compliance rate is 95%.

[0127] Therefore, the pre-judgment logic is:

[0128] Condition 1: if the process completeness index<0.9→determined as "process step skipping or sequence error";

[0129] Condition 2: if the assembly precision deviation>1.2→determined as "assembly out-of-tolerance risk";

[0130] Condition 3: if the safety specification compliance rate<95%→determined as "operation safety hazard".

[0131] When the process completeness index is lower than the process procedure integrity threshold, or the assembly precision deviation exceeds the cumulative simulation value of the tolerance, or the safety specification compliance rate does not reach the minimum standard of the safety protocol, the intervention work including the projection of the intelligent maintenance board information and the broadcast of the voice correction instruction is triggered.

[0132] S4, generate a guidance performance archive containing a multi-modal guidance policy trigger log, user response behavior sensor data, and a policy adjustment utility evaluation graph that is synchronized with the job link, and store it in a preset database through an encrypted data chain and a unique identification code of the repair object.

[0133] Further, as shown in Figure 12 Step S4 includes:

[0134] S41, based on the timestamp alignment result of the device running state snapshot at the multi-modal guidance policy trigger time and the user operation event stream, generate a guidance policy trigger log containing policy type code, trigger space coordinates, and parameter adjustment information.

[0135] S42, synchronize and aggregate user gesture motion trajectory data captured by industrial cameras, torque / displacement measurement sequences uploaded by wireless measurement tools, and semantic analysis results of voice instructions, to generate a user response behavior sensor data set with action trigger time and response time delay difference value annotations.

[0136] S43, input the process completeness index change rate, assembly precision deviation convergence gradient, and safety specification compliance rate increase before and after policy adjustment into the pre-trained reinforcement learning model to generate a policy adjustment utility evaluation graph with policy optimization suggestions as nodes and utility evaluation values as edge weights.

[0137] S44, based on the guidance policy trigger log, user response behavior sensor data set, and policy adjustment utility evaluation graph, construct a guidance performance archive, use a hash chain generated based on the serial number of the repair object to block encrypt the archive data, then XOR bind the encrypted blocks with multiple sub-identification codes obtained by decomposing the serial number, and store the historical repair cases.

[0138] In an embodiment, when the multi-modal guidance policy is triggered, the device running state snapshot (including motor current, temperature, vibration spectrum) is intercepted in real time; the user operation event stream (such as tool usage record, interface click action) is captured, and the timestamps of the device state snapshot and the operation event are aligned through a high-precision clock service; a guidance policy trigger log containing policy type code, trigger space coordinates, and parameter adjustment information is generated.

[0139] Synchronize and aggregate user gesture motion trajectory data captured by industrial cameras, torque / displacement measurement sequences uploaded by wireless measurement tools, and semantic analysis results of voice instructions, and according to the guidance policy trigger time (T0) and the user's actual response time (T1), calculate the response time delay difference value (Δt = T1-T0) to generate a user response behavior sensor data set with action trigger time and response time delay difference value annotations.

[0140] Then, the process completeness index change rate before and after the strategy adjustment, the assembly precision deviation convergence gradient, and the safety specification compliance rate increase are input into the pre-trained PPO reinforcement learning model to generate a strategy adjustment utility evaluation atlas, wherein the nodes are strategy optimization suggestions, and the edge weights are utility evaluation values (weight = 0.87 indicates that the suggestion can improve the assembly precision convergence efficiency by 87%).

[0141] The trigger log, the sensor data set, and the evaluation atlas are merged according to a timeline to construct a JSON-LD format guidance performance archive; a hash chain is generated based on the serial number of the maintenance object through a SHA-256 algorithm; the archive data is segmented into 128 KB blocks, respectively encrypted by AES-256, and the hash value is written into the block header; the serial number is decomposed into four sub-identification codes (SN1-SN4), and distributed storage is performed in the database after performing an XOR operation with the encrypted blocks.

[0142] The embodiment realizes full-link tracking and optimization feedback of the performance data of the maintenance guidance process, generates an encrypted performance archive bound to the unique identification code of the maintenance object by integrating the multi-modal guidance strategy trigger log, the user response behavior sensor data, and the strategy adjustment utility evaluation result, and supports maintenance quality traceability and adaptive strategy optimization.

[0143] Additionally, the embodiment of the present application provides a maintenance operation guidance system based on intelligent interaction, comprising:

[0144] The guidance strategy output module is configured to analyze the operation intention of the user into at least one type of device disassembly, troubleshooting, and knowledge retrieval according to the user input instruction or voice instruction, and dynamically call a function module adapted to the intention type to generate a multi-modal guidance strategy.

[0145] The operation link execution module is configured to coordinate and control at least one human-machine collaborative operation link including a disassembly action sequence guidance, a collaborative action sequence arrangement of an intelligent feeding device, a fault reasoning decision, an assembly compatibility verification, and a retrieval information feedback of an intelligent maintenance board based on the multi-modal guidance strategy.

[0146] The intervention work module is configured to construct multi-dimensional verification data according to the fusion analysis result of the multi-modal sensor data after one operation in the to-be-completed operation link, and activate the intervention work when any dimension verification data exceeds the corresponding knowledge graph threshold.

[0147] The data storage module is configured to generate a guidance performance archive containing the multi-modal guidance strategy trigger log, the user response behavior sensor data, and the strategy adjustment utility evaluation atlas, which is synchronously mapped with the operation link, and stored in a preset database by binding the encrypted data chain with the unique identification code of the maintenance object.

[0148] Further, the embodiment of the present application provides a maintenance operation guiding device based on intelligent interaction, referring to Figure 13 , comprising: a hardware execution unit, including an intelligent maintenance board, an industrial camera, a mechanical arm, an intelligent shelf, an operation station rack, a wireless measurement tool and a voice microphone; an intelligent feeding device, including an intelligent shelf, a mechanical arm, an electric control box and a workbench; and a server, used for executing the method as described above. Wherein the intelligent shelf is built-in with a plurality of bins, the server controls the step motor to drive the bin circulating queue to move, and triggers an infrared positioning signal when the target bin reaches the discharging position; the mechanical arm is configured with six-axis freedom, and responds to the infrared positioning signal to perform a bin grabbing action, and transfers the bin to the magnetic attraction type positioning groove of the operation station rack; the electric control box is built-in with a CAN bus communication interface, and synchronously receives the work order instruction issued by the server and the torque / displacement measurement sequence uploaded by the wireless measurement tool.

[0149] Wherein, the parameters of the mechanical arm include: the mechanical arm body adopts a six-axis design, with an arm span of 0.82 meters and an end load of 8KG; the robot gripper is arranged at the work station clamping jaw, satisfying the grabbing of the material box, flange and pump cover; the compliant drag teaching, joint quick change, easy teaching and simple programming greatly shorten the integration period. The complete open collaboration ecology provides rich IO and communication interfaces to meet user integration development. The high-sensitivity collision detection function does not require fence isolation, and is more secure and reliable. It is also configured with 6 degrees of freedom, ±360° single joint motion range, spherical workspace range, supports arbitrary angle installation method, ultra-lightweight, compact structure.

[0150] And the structure of the intelligent shelf is a material box circulating mechanism, a cover, a hoof, and a dust cover; the control of the intelligent shelf: external shaft driving is adopted, and is uniformly controlled by the robot teach box; the size of the intelligent shelf: length x width x height = 850 x 550 x 1045, the height of the material taking port is 800; the bin number of the intelligent shelf: a total of 11 bins, each bin can accommodate two material boxes (155 x 155 x 54 / material box). Two fixed large material boxes are externally hung, used for containing large materials such as pump bodies and pump covers. Satisfy the containing of demonstration water pump parts (17 parts). Make a mark at the hoof position when debugging is completed, and confirm that the hoof position has no change before each use.

[0151] When disassembling, the mechanical arm uses the gripper to grab the box containing the parts to the loading area on the workbench surface according to the instructions. When assembling, the mechanical arm uses the gripper to grab the empty box to the intelligent shelf according to the instructions. The specific implementation steps are as follows: manually program the intelligent shelf according to the assembly process requirements, so that the intelligent shelf can send the corresponding box to the material outlet according to the steps; the intelligent shelf stores the parts according to the program setting, and can automatically send the corresponding parts to the material outlet according to the program setting; when disassembling, the mechanical arm uses the gripper to grab the box containing the parts from the intelligent shelf outlet to the loading area on the workbench surface according to the instructions. When disassembling, the action is opposite; when assembling, the mechanical arm uses the gripper to grab the empty box to the intelligent shelf according to the instructions. When disassembling, the action is opposite; the loading area is provided with two discharge positions, one of which is used as a buffer to save the occupation of the disassembly and assembly time by the artificial.

[0152] In addition, a portable wireless acquisition terminal is also introduced, which can connect wireless torque wrenches and wireless measuring tools such as wireless calipers, and can collect and upload the key measurement data required in the maintenance process in real time, ensuring the accuracy and immediacy of the data. This series of intelligent design not only helps maintenance personnel to troubleshoot and accurately disassemble and assemble, but also significantly improves the overall efficiency of maintenance operations, effectively enhances the maintenance quality, and realizes the overall optimization and intelligent upgrading of the maintenance process

[0153] Preferably, the above-mentioned equipment operating environment is:

[0154]

[0155]

[0156]

[0157] Those skilled in the art will appreciate that embodiments of the application can be provided as methods, systems or computer program products. Accordingly, the application can be embodied in the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the application can be embodied in the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk memory, CD-ROM, optical memory, etc.) having computer usable program code embodied therein.

[0158] The present application is described with reference to flowcharts and / or block diagrams of methods, apparatus (systems) and computer program products according to embodiments of the application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams can be implemented by computer program instructions, and the combination of flows and / or blocks in the flowcharts and / or block diagrams.

[0159] While the preferred embodiments of the application have been described, those skilled in the art will recognize that many modifications and variations of this preferred embodiment are possible without departing from the spirit or scope of the present application. Therefore, it is intended that the scope of the application be defined by the following claims and their equivalents.

[0160] Obviously, many modifications and variations of the present application are possible in light of the above teachings. It is, therefore, to be understood that within the scope of the appended claims and their equivalents, the application can be practiced otherwise than as specifically described.

Claims

1. A maintenance operation guidance method based on intelligent interaction, characterized in that: include: Based on user input commands or voice commands, the system analyzes the user's operation intention as at least one of the following types: device disassembly and assembly, troubleshooting, and knowledge retrieval. The system dynamically calls the function module adapted to the intention type to generate a multimodal guidance strategy. Based on a multimodal guidance strategy, coordinated control of a human-machine collaborative operation link includes at least one of the following: disassembly and assembly action sequence guidance, collaborative action sequence arrangement of intelligent loading devices, fault reasoning and decision-making, assembly compatibility verification, and retrieval information feedback of intelligent maintenance dashboards; After completing a task in the job chain, multi-dimensional verification data is constructed based on the fusion analysis results of multimodal sensor data. When the verification data of any dimension exceeds the corresponding knowledge graph threshold, intervention work is activated; Generate a guidance effectiveness file that is synchronously mapped with the operation link and includes multimodal guidance strategy trigger logs, user response behavior sensor data, and strategy adjustment utility evaluation maps, and store it in a preset database through an encrypted data link and binding it to the unique identification code of the maintenance object.

2. The maintenance operation guidance method based on intelligent interaction according to claim 1, characterized in that: Based on user input commands or voice commands, the user's operation intention is analyzed as at least one of the following types: device disassembly and assembly, troubleshooting, and knowledge retrieval. The function module adapted to the intention type is dynamically called to generate a multimodal guidance strategy, including: Parsing and determining, based on received user input commands or real-time captured voice command streams, the operation intention type is at least one of device disassembly and assembly, troubleshooting, and knowledge retrieval; In response to the operation intention category, dynamically call the preset functional module cluster including the three-dimensional disassembly and assembly guidance module, the fault reasoning decision module and the knowledge semantic association module; Based on the output results of the called functional modules, a multimodal guidance strategy adapted to the operation intention category is generated. The strategy includes: The equipment assembly and disassembly guidance strategy involves calling a three-dimensional assembly and disassembly guidance module to generate an equipment assembly and disassembly instruction set that includes a decomposition display of the disassembled component topology, a disassembly sequence, an assembly sequence, smart shelf bin codes, and robot arm trajectory planning parameters. The robot arm's end effector switches tool gripping modes based on the disassembly / installation phase, and triggers jumps to any assembly and disassembly process using collected gesture or voice data. The fault elimination guidance strategy involves calling a fault reasoning decision module to construct a maintenance operation chain containing assembly and disassembly sequence constraints, a fault location logic tree, and a set of troubleshooting instructions based on assembly compatibility verification rules. The maintenance operation chain generates process interlock conditions between the disassembly, maintenance, inspection, and installation phases by associating with a historical case library. The fault location logic tree includes a minimum maintainable unit matching algorithm based on the equipment topology and an associated parts replacement priority sequence. The knowledge retrieval guidance strategy is to call the knowledge semantic association module to generate multimodal knowledge data that matches the physical maintenance scene space.

3. The maintenance operation guidance method based on intelligent interaction according to claim 2, characterized in that: Based on a multimodal guidance strategy, the coordinated control of a human-machine collaborative operation chain includes at least one of the following: guidance of assembly and disassembly action sequences, collaborative action sequence arrangement of intelligent loading devices, fault reasoning and decision-making, assembly compatibility verification, and retrieval information feedback of intelligent maintenance dashboards. The chain includes: When the multimodal guidance strategy is the equipment disassembly and assembly guidance strategy, the three-dimensional disassembly and assembly guidance module is called, the equipment structure decomposition model is loaded into the intelligent maintenance board and the spatial topological relationship of the parts is mapped, and the user gestures or voice command streams in the preset operation area are collected in real time. When the feature vector of the gesture or voice command stream matches the preset trigger condition, the disassembly stage or installation stage is activated; during the disassembly process, the disassembly step decomposition graphic guidance, the associated tool three-dimensional model and the current step part parameters are displayed in the intelligent maintenance board, and the data collected by the wireless measurement tools involved in the disassembly work are updated to the intelligent maintenance board in real time. At the same time, the user's disassembly movement is monitored in real time through the industrial camera. The operation and disassembly sequence is determined. When a component separation event is detected, the robotic arm is controlled to grab the component and perform access operations according to the robotic arm trajectory planning path obtained by parsing the smart shelf bin code. In addition, during the assembly process, the assembly step decomposition graphic guidance, the associated tool 3D model and the current step part parameters are displayed on the smart maintenance board, and the data collected by the wireless measurement tool is updated to the smart maintenance board in real time. At the same time, the user's assembly actions and assembly sequence are monitored in real time through the industrial camera. When it is detected that the component is assembled in place, the robotic arm is controlled to grab the component and perform access operations according to the robotic arm trajectory planning path obtained by parsing the smart shelf bin code. When the multimodal guidance strategy is a troubleshooting guidance strategy, the fault reasoning decision module is called to respond to the fault phenomenon text input by the user and call the historical fault case database to generate a candidate troubleshooting set containing disassembly sequence constraints, minimum repairable unit positioning rules and related parts replacement priority sequences; based on the candidate troubleshooting scheme selected by the user, a maintenance operation chain with process interlocking of the serial disassembly phase, maintenance phase, inspection phase and installation phase is generated; at the same time, a three-dimensional diagram of the tool list corresponding to the current step, an animation of the disassembly of related parts and a comparison table of process parameters uploaded by the wireless measurement tool are generated in the intelligent maintenance dashboard; when the installation phase is detected to be completed, an assembly compatibility check based on historical maintenance data is triggered, including the matching verification of the hash value of the installed parts with the equipment topology database and the tolerance accumulation simulation calculation of the interface parameters of adjacent parts; When the multimodal guidance strategy is a knowledge retrieval guidance strategy, the knowledge semantic association module is called to generate a three-dimensional structural decomposition map that is spatially mapped to the physical maintenance scene, and the multimedia index of the maintenance procedures is associated to form dynamic annotation information.

4. The maintenance work guidance method based on intelligent interaction according to claim 3, characterized in that: The fault reasoning decision module is called to respond to the fault phenomenon text input by the user and call the historical fault case database to generate a candidate fault elimination set containing disassembly sequence constraints, minimum repairable unit positioning rules and associated parts replacement priority sequence, including: Based on a grammatical rule engine and a Bi-LSTM+CRF joint model, fault knowledge extraction is implemented for historical maintenance cases, including structured data, fault-related entities, and topological dependencies. The grammatical rule engine is configured to perform dependency syntax analysis on the fault case text in historical maintenance cases based on a pre-configured equipment disassembly and assembly sequence constraint template, extracting structured data including fault phenomena, mandatory processes, and prohibited operations. The Bi-LSTM+CRF joint model is configured to perform sequence annotation on the unstructured text in historical maintenance cases and simultaneously extract fault-related entities and topological dependencies. The extracted fault knowledge is injected into the Neo4j graph database to generate a multi-dimensional fault constraint graph containing fault nodes, process constraint edges, minimum repairable unit nodes, and part replacement priority weights; In response to a fault phenomenon retrieval instruction input by a user, multi-hop reasoning is performed in the multi-dimensional fault constraint graph to generate a minimum cut set that satisfies the disassembly sequence constraint as a first candidate subset; The LDA topic probability model is used to perform semantic clustering on historical maintenance cases that are not involved in fault knowledge extraction. The central topic distribution vector of each cluster is generated. The fault phenomenon text input by the user is converted into a query topic distribution vector. The cosine similarity between the query topic distribution vector and the central topic distribution vector of each cluster is calculated as the topic relevance. The historical maintenance cases contained in the clusters with cosine similarity higher than the preset cosine threshold are selected to form the second candidate subset. The case texts of the second candidate subset are parsed using the HanLP word segmenter to identify new entities not registered in the multidimensional fault constraint graph. Based on the hierarchical relationships of the equipment topology, the superordinate concepts of the new entities are derived and inheritance constraints are established to complete the ontology-driven dynamic knowledge completion of the case subset. Inject the established ontology inheritance relationship constraints back into the multi-dimensional fault constraint graph to generate an incremental version graph. Perform secondary reasoning optimization on the first candidate subset based on the incremental graph. According to the multi-hop reasoning results and the topic relevance, the union of the first candidate subset and the second candidate subset of the secondary reasoning optimization is ranked by confidence, and a set of candidate troubleshooting solutions with priority labels is output.

5. The maintenance work guidance method based on intelligent interaction according to claim 3, characterized in that: When the multimodal guidance strategy is a knowledge retrieval guidance strategy, the knowledge semantic association module is called to generate a three-dimensional structural decomposition map that is spatially mapped to the physical maintenance scene, and the multimedia index of the maintenance procedures is associated to form dynamic annotation information including: Based on the spatial coordinate data of the physical maintenance scene, the knowledge semantic association module is called to generate a three-dimensional structural decomposition map that includes the parts disassembly hierarchy identification and assembly topology dependency relationship mapped with the spatial coordinates; The LDA topic probability model is used to cluster the historical maintenance case texts in the historical maintenance case library, extract keyword clusters as semantic labels, and index associations are established between the labels and the multimedia resources corresponding to the process nodes in the three-dimensional structure decomposition map. The multimedia resources include one or more combinations of text analysis documents, fault feature images, and operation demonstration videos. Based on user interaction actions or tool proximity detection results, a dynamic annotation box is superimposed on the target part node of the 3D structural decomposition diagram. The annotation box is embedded with interactive controls of associated multimedia indexes for real-time retrieval and display of multimedia resources.

6. The maintenance work guidance method based on intelligent interaction according to claim 3, characterized in that: After completing a task in the job chain, multi-dimensional verification data is constructed based on the fusion analysis results of multimodal sensor data. When the verification data of any dimension exceeds the corresponding knowledge graph threshold, the activation intervention work includes: In response to the completion signal of a single task in the operation chain, the user gesture motion trajectory data captured by the industrial camera, the torque / displacement measurement sequence uploaded by the wireless measurement tool, and the semantic parsing result stream of the voice command are synchronized in time and space to construct a spatiotemporal correlation matrix of multi-source perception data; Based on the process constraint rules of the preset equipment topology structure, the dimension of the correlation matrix is ​​reduced, and the process completeness index is extracted as the topological matching degree between the executed steps and the corresponding process; The assembly accuracy deviation is obtained by calculating the Mahalanobis distance between the torque / displacement measurement sequence and the tolerance band reference value in the preset knowledge graph; Analyze the semantic features of voice commands and the action features of user gestures, combine them with the behavioral specification items in the preset security protocol library, and calculate the safety specification compliance rate as the proportion of compliant operation actions; A multi-dimensional verification vector is obtained based on the process completeness index, assembly accuracy deviation, and safety specification compliance rate; The multi-dimensional verification vector is matched with the process threshold in the preset knowledge graph. When the process completeness index is lower than the process procedure completeness threshold, or the assembly accuracy deviation exceeds the tolerance cumulative simulation value, or the safety specification compliance rate does not meet the minimum standard of the safety protocol, intervention work including intelligent maintenance board information projection and voice correction instruction broadcast is triggered.

7. The maintenance work guidance method based on intelligent interaction according to claim 6, characterized in that: Generate a guidance effectiveness profile that is synchronized with the operation link and includes multimodal guidance strategy trigger logs, user response behavior sensor data, and strategy adjustment effectiveness evaluation maps. This profile is then bound to the unique identification code of the maintenance object through an encrypted data link and stored in a preset database, including: Based on the alignment of the device operating status snapshot at the time of multimodal guidance policy triggering and the timestamp of the user operation event stream, a guidance policy triggering log containing the policy type code, trigger space coordinates, and parameter adjustment information is generated. Synchronously aggregate user gesture motion trajectory data captured by industrial cameras, torque / displacement measurement sequences uploaded by wireless measurement tools, and the semantic parsing result stream of voice commands to generate a user response behavior sensing dataset with annotations of the difference between action trigger time and response delay; By inputting the process completeness index change rate, assembly precision deviation convergence gradient, and safety specification compliance rate increase before and after the strategy adjustment into the pre-trained reinforcement learning model, a strategy adjustment utility evaluation graph is generated with strategy optimization suggestions as nodes and utility evaluation values ​​as edge weights. A guidance effectiveness archive is constructed based on the guidance policy trigger log, user response behavior sensor dataset, and policy adjustment utility evaluation map. The archive data is encrypted in blocks using a hash chain generated based on the serial number of the maintenance object. The encrypted block is then XOR-bound with multiple sub-identification codes decomposed from the serial number to store historical maintenance cases.

8. A maintenance operation guidance system based on intelligent interaction, characterized in that: include: A guidance strategy output module is used to interpret the user's operation intention as at least one of device disassembly and assembly, troubleshooting, and knowledge retrieval based on user input instructions or voice commands, and dynamically call the function module adapted to the intention type to generate a multimodal guidance strategy; An operation link execution module is used to coordinate and control a human-machine collaborative operation link including at least one of the following: disassembly and assembly action sequence guidance, collaborative action sequence arrangement of intelligent loading devices, fault reasoning and decision-making, assembly compatibility verification, and retrieval information feedback of intelligent maintenance dashboards based on a multimodal guidance strategy; The intervention work module is used to construct multi-dimensional verification data based on the fusion analysis results of multimodal sensor data after a job in the job chain is completed. When the verification data of any dimension exceeds the corresponding knowledge graph threshold, the intervention work is activated; The data storage module is used to generate a guidance effectiveness file that is synchronously mapped with the operation link and includes a multimodal guidance strategy trigger log, user response behavior sensor data, and a strategy adjustment utility evaluation map, and is stored in a preset database through an encrypted data link and bound to the unique identification code of the maintenance object.

9. A maintenance operation guidance device based on intelligent interaction, characterized in that: include: Hardware execution units, including smart maintenance boards, industrial cameras, robotic arms, smart shelves, workstations, wireless measuring tools, and voice microphones; Intelligent loading device, including intelligent shelves, robotic arms, electric control boxes and workbenches; And, a server, configured to execute the method according to any one of claims 1-7.

10. The maintenance work guidance device based on intelligent interaction according to claim 9, characterized in that: The smart shelf has multiple built-in silos. The server controls the stepper motor to drive the silo to move in a circular queue. When the target silo reaches the discharge position, an infrared positioning signal is triggered. The robotic arm is equipped with six axes of freedom, responds to infrared positioning signals to perform silo grabbing actions, and transfers the silo to the magnetic positioning slot of the workstation platform; The electric control box has a built-in CAN bus communication interface, which synchronously receives the instructions sent by the server and the torque / displacement measurement sequence uploaded by the wireless measurement tool.

Images