Equipment maintenance guidance method and device, electronic equipment and storage medium

Abstract

The invention discloses an equipment maintenance guidance method and device, electronic equipment and a storage medium. The method comprises the following steps: acquiring environment image data containing target equipment through a camera and acquiring attitude data of a target object through a sensor; determining the spatial position and direction of the target object based on the environment image data and the attitude data; determining an equipment virtual maintenance scheme of the target equipment based on the environment image data; and mapping the equipment virtual maintenance scheme to a virtual coordinate system of the wearable equipment based on the spatial position and direction so as to visually display the equipment virtual maintenance scheme, and guiding the target object to perform maintenance operation on the target equipment based on a visualization effect. According to the method, the equipment virtual maintenance scheme is accurately mapped into the virtual coordinate system, visual display of the equipment virtual maintenance scheme is realized, the target object is guided to carry out maintenance operation on the target equipment, the maintenance work is more efficient, accurate and convenient, and the maintenance efficiency is improved.

Description

Technical Field

[0001] The present invention relates to the field of augmented reality technology, and in particular to an equipment maintenance guidance method, device, electronic equipment and storage medium. Background Art

[0002] With the development of science and technology and the widespread application of various equipment, more and more attention has been paid to the maintenance of equipment. Among the many devices, the maintenance and operation of complex systems are particularly important and challenging. A complex system refers to a system with a large number of interacting elements, complex and uncertain internal correlations, and nonlinear overall behavior. For example, the inertial navigation system (INS) plays an important role in modern navigation technology and is widely used in aviation, navigation, military and autonomous driving. Due to the complexity and high precision requirements of complex systems, their repair and maintenance require highly specialized technicians. At the maintenance site, technicians may face various unexpected problems and need to consult a large number of manuals and documents, which leads to prolonged maintenance time and low maintenance efficiency. Summary of the Invention

[0003] This application proposes an equipment maintenance guidance method, device, electronic device and storage medium to solve the problem that technicians may face various unexpected problems at the maintenance site and need to consult a large number of manuals and documents, resulting in prolonged maintenance time and low maintenance efficiency.

[0004] In a first aspect, a device maintenance guidance method is provided, which is applied to a wearable device, wherein the wearable device is provided with a camera and a sensor, and includes:

[0005] Acquiring environmental image data including the target device through the camera and acquiring posture data of the target object through the sensor;

[0006] determining a spatial position and orientation of the target object based on the environmental image data and the posture data;

[0007] determining a virtual equipment maintenance plan for the target equipment based on the environmental image data;

[0008] The virtual maintenance plan for the device is mapped to a virtual coordinate system of the wearable device based on the spatial position and direction to visualize the virtual maintenance plan for the device, and the target object is guided to perform maintenance operations on the target device based on the visualization effect.

[0009] In a second aspect, a device maintenance guidance device is provided, which is applied to a wearable device, wherein the wearable device is provided with a camera and a sensor, including:

[0010] an acquisition module, configured to acquire environmental image data containing a target device through the camera and acquire posture data of the target object through the sensor;

[0011] a first determining module, configured to determine a spatial position and orientation of the target object based on the environmental image data and the posture data;

[0012] A second determining module is configured to determine a virtual equipment maintenance plan for the target equipment based on the environmental image data;

[0013] A display module is used to map the virtual maintenance plan of the device to the virtual coordinate system of the wearable device based on the spatial position and direction, so as to visualize the virtual maintenance plan of the device and guide the target object to perform maintenance operations on the target device based on the visualization effect.

[0014] Optionally, the first determining module further includes a time synchronization submodule, and the time synchronization submodule is configured to:

[0015] Time-synchronizing the environmental image data and the posture data;

[0016] The spatial position and orientation of the target object are determined based on the time-synchronized environment image data and posture data.

[0017] Optionally, the second determining module includes:

[0018] an extraction submodule, configured to extract a device image of the target device from the environment image data;

[0019] an identification submodule, configured to identify device status information of the target device based on the device image;

[0020] The determination submodule is used to determine the virtual maintenance plan of the equipment based on the equipment status information.

[0021] Optionally, the identification submodule includes:

[0022] an extraction unit, configured to extract features from the device image to obtain device feature information;

[0023] an identification unit, configured to identify components of the target device and states of the components based on the device feature information;

[0024] The first determining unit is configured to determine the device status information according to components of the target device and statuses of the components.

[0025] Optionally, the determining submodule includes:

[0026] an acquiring unit, configured to acquire an actual maintenance plan for the target device based on the device status information;

[0027] a decomposition unit, configured to decompose the actual maintenance plan into steps to obtain a maintenance step sequence;

[0028] The second determining unit is configured to determine the virtual maintenance plan for the equipment based on the maintenance step sequence.

[0029] Optionally, the device further includes an execution module, wherein the execution module is configured to:

[0030] receiving command request information from the target object;

[0031] Execute the operation corresponding to the command request information.

[0032] Optionally, the device further includes a prediction module, wherein the prediction module is configured to:

[0033] Obtaining device operating status data of the target device;

[0034] predicting fault information of the target device based on the device operation data;

[0035] An equipment early warning plan is determined based on the fault information.

[0036] In a third aspect, an electronic device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above-mentioned device maintenance guidance method when executing the computer program.

[0037] In a fourth aspect, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the above-mentioned equipment maintenance guidance method are implemented.

[0038] The present application provides an equipment maintenance guidance method, apparatus, electronic device, and storage medium, which obtain environmental image data containing a target device through a camera and obtain posture data of a target object through a sensor; determine the spatial position and orientation of the target object based on the environmental image data and the posture data; determine a device virtual maintenance plan for the target device based on the environmental image data; map the device virtual maintenance plan to a virtual coordinate system of the wearable device based on the spatial position and orientation, so as to visualize the device virtual maintenance plan, and guide the target object to perform maintenance operations on the target device based on the visualization effect. In the equipment maintenance guidance scheme provided by the present application, the fused data obtained by data fusion of the environmental image data and the posture data accurately reflects the spatial position and orientation of the target object, thereby achieving precise positioning of the spatial position and orientation of the target object; the device virtual maintenance plan determined based on the environmental image data, combined with the spatial position and orientation, is accurately mapped to the virtual coordinate system, achieving visualization of the device virtual maintenance plan, so as to guide the target object to perform maintenance operations on the target device, reducing the professional skills requirements of the target object, reducing the risk of misoperation, making maintenance work more efficient, accurate, and convenient, and improving maintenance efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0040] Figure 1 A diagram illustrating the application environment of the equipment maintenance guidance method provided in an embodiment of the present application;

[0041] Figure 2 A flowchart of the equipment maintenance guidance method provided in an embodiment of the present application;

[0042] Figure 3 A structural block diagram of the equipment maintenance guidance device provided in an embodiment of the present application;

[0043] Figure 4 This is a structural block diagram of the electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0044] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of them. All other embodiments derived by ordinary technicians in this field based on the embodiments of the present invention without creative work are within the scope of protection of the present invention.

[0045] In addition, described feature, structure or characteristic can be combined in one or more embodiments in any suitable manner.In the following description, many specific details are provided so as to provide a full understanding of the embodiments of the present application. However, it will be appreciated by those skilled in the art that the technical scheme of the present application can be put into practice without one or more of the specific details, or other methods, components, devices, steps etc. can be adopted. In other cases, known methods, devices, implementations or operations are not shown or described in detail to avoid blurring the various aspects of the application.

[0046] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0047] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0048] The equipment maintenance guidance method provided by the embodiment of the present invention can be applied in Figure 1In an application environment. The electronic device 110 is provided with a camera and a sensor, and the electronic device 110 communicates with the server 120 through a network 130. The electronic device 110 can obtain environmental image data containing a target device through the camera and obtain posture data of the target object through the sensor; determine the spatial position and orientation of the target object based on the environmental image data and the posture data; determine a virtual maintenance plan for the target device based on the environmental image data; map the virtual maintenance plan for the device to the virtual coordinate system of the wearable device based on the spatial position and orientation to visualize the virtual maintenance plan for the device, and guide the target object to perform maintenance operations on the target device based on the visualization effect, and display the plan through the electronic device 110. In the present invention, the fused data obtained by fusing environmental image data and posture data accurately reflects the spatial position and orientation of the target object, achieving precise positioning of the spatial position and orientation of the target object. The virtual maintenance plan for the device determined based on the environmental image data is accurately mapped to a virtual coordinate system in combination with the spatial position and orientation, achieving a visual display of the virtual maintenance plan for the device to guide the target object in performing maintenance operations on the target device. This reduces the professional skills required of the target object and the risk of misoperation, making maintenance work more efficient, accurate, and convenient, thereby improving maintenance efficiency. The electronic device 110 may be a wearable device, including but not limited to VR glasses and helmets. The present invention is described in detail below using specific embodiments. Optionally, an application app is provided on the wearable device, and a data caching mechanism can be implemented in the application app to store key data (such as maintenance manuals and fault case libraries) in the local storage space of the wearable device, enabling the wearable device to continue operating using the data cached in the local storage space even when there is no network connection. Optionally, when the network is restored, local data is automatically synchronized with server data to ensure data consistency. Optionally, user maintenance data (such as maintenance records) and application data (i.e., maintenance manuals, fault case libraries, etc.) are stored on the server 120 so that they can be synchronized to the user's wearable device when needed. Optionally, application data can be updated regularly, including updates to the maintenance manual and fault case library. When the wearable device is online, the latest data is automatically pushed to the user's device to implement a data synchronization mechanism. The application APP uses a local database to store key data, such as a SQLite database or a Realm database. During the maintenance process, the local database data can be indexed for quick data retrieval. For example, if the target object determines that the fault type is motor heating, the pre-stored equipment maintenance virtual plan can be searched in the local database by inputting motor heating and displayed. For frequently accessed data (such as fault types with a high frequency), it can be cached in memory to increase access speed.For some computing tasks that do not require server participation, they can be executed locally to reduce dependence on the network.

[0049] See also Figure 2 As shown, Figure 2 This is a flow chart of a device maintenance guidance method provided by an embodiment of the present invention. This embodiment is illustrated by applying it to a wearable device, which is equipped with a camera and a sensor. The device maintenance guidance method includes the following steps:

[0050] S101: Acquire environmental image data containing a target device through the camera and acquire posture data of a target object through the sensor.

[0051] The sensors may include an accelerometer sensor, a gyroscope sensor, and an ambient light sensor, among others.

[0052] The environmental image data refers to an image of the target object in the on-site environment of the target device. The target object may be a technician repairing the target device. The posture data may be data indicating changes in the target object's posture relative to a virtual coordinate system in the wearable device. The posture data may be head posture data, i.e., data indicating changes in the target object's head posture relative to the virtual coordinate system in the wearable device.

[0053] The posture data can be expressed in the form of six degrees of freedom (6DoF), including three spatial rotation angles, namely pitch angle, roll angle, and yaw angle, and three spatial displacements, namely X, Y, and Z coordinates.

[0054] For example, assuming the wearable device is a helmet, the target subject wears the helmet. The helmet's camera captures environmental image data of the target device, and sensors capture head pose data of the target subject. Head pose estimation can be performed based on this head pose data, thereby estimating the target subject's head position and orientation. This environmental image data can be overlaid with a virtual repair plan for the equipment, enabling visual guidance through the helmet's display interface.

[0055] S102: Determine the spatial position and direction of the target object based on the environmental image data and the posture data.

[0056] Among them, the spatial position can refer to the specific coordinates of the target object in its field of view space, that is, the specific coordinates in the virtual coordinate system of the wearable device. The coordinates can be expressed in the form of a vector. For example, the spatial position can be expressed as (x, y, z), where x represents the position of the target object in the horizontal direction, y represents the position of the target object in the vertical direction, and z represents the position of the target object in the depth direction. The direction refers to the orientation of the target object in its field of view space. For example, the direction can be expressed as (Y, P, R), where Y is the yaw angle, which represents the rotation angle around the Z axis and represents the orientation of the target object on the horizontal plane; P is the pitch angle, which represents the rotation angle around the Y axis and represents the degree of up and down tilt of the target object; R is the roll angle, which represents the rotation angle around the X axis and represents the degree of left and right tilt of the target object.

[0057] In one embodiment, the image data and the posture data are input into a Kalman filter as observation values ​​to estimate the spatial position and orientation of the target object (such as the head of the target object).

[0058] In one embodiment, a deep learning model (such as a convolutional neural network, a recurrent neural network, etc.) is used to learn the relationship between image data and posture data to output the spatial position and orientation of the target object.

[0059] In one embodiment, determining the spatial position and orientation of the target object based on the environmental image data and the posture data includes:

[0060] Time-synchronizing the environmental image data and the posture data;

[0061] The spatial position and orientation of the target object are determined based on the time-synchronized environment image data and posture data.

[0062] The environmental image data and the posture data can be synchronized in time based on the timestamps in the environmental image data and the posture data. The timestamp synchronization ensures the temporal consistency between the posture data collected by the sensor and the environmental image data. Then, a deep learning model (such as a convolutional neural network, a recurrent neural network, etc.) is used to fuse the posture data with the environmental image data to accurately determine the spatial position and orientation of the target object (such as the head of the target object).

[0063] S103: Determine a virtual maintenance plan for the target device based on the environmental image data.

[0064] Among them, environmental image data can be used to superimpose with the virtual maintenance plan of the equipment and displayed to the target object through wearable devices.

[0065] A virtual equipment maintenance plan refers to a maintenance plan for a target equipment that can be displayed to a target object through a wearable device.

[0066] The device components of the target device and the actual maintenance plans corresponding to the device components can be identified according to the environmental image data, and a virtual maintenance plan for the device can be generated based on the actual maintenance plan.

[0067] AI algorithm models, such as neural networks, machine learning, and deep learning, can be trained using data from equipment maintenance manuals, equipment failure case libraries, and equipment manuals to generate models for generating virtual equipment maintenance plans. Equipment maintenance manuals provide standard procedures and methods for equipment maintenance and serve as a crucial reference for training AI algorithm models to generate virtual equipment maintenance plans. Equipment failure case libraries contain historical maintenance records and fault resolutions, which are crucial for AI algorithm model learning and predicting failure patterns. For example, a large amount of inertial navigation system maintenance case and fault data can be collected. This collected inertial navigation system maintenance case and fault data can then be cleaned, annotated, and formatted to generate a equipment failure case library. Equipment manuals provide technical specifications and operating instructions for target equipment, helping computer vision technology accurately identify equipment components. Combining data from the equipment maintenance manual and fault case library, the equipment virtual maintenance plan generation model can determine the fault type of the target equipment component and generate a virtual maintenance plan. In this way, after acquiring environmental image data, the trained equipment virtual maintenance plan generation model can be fed into the data processing to output a virtual equipment maintenance plan.

[0068] For example, the training process of the equipment virtual maintenance plan generation model is as follows:

[0069] Collect maintenance cases of different types of inertial navigation systems, including fault descriptions, maintenance steps, maintenance results, etc., as well as collect sensor data of the inertial navigation system (such as infrared thermal imagers collecting motor images of the inertial navigation system), log files, alarm information, etc. Then pre-process the collected maintenance cases, sensor data, log files, alarm information and other data. The pre-processing operations include data cleaning, that is, removing invalid and redundant data to ensure data quality; data labeling, that is, labeling the maintenance cases, marking the key operations and results of each step. Data formatting, that is, converting the data into a format suitable for the AI ​​algorithm model, such as images, sequence data, etc., so as to train the AI ​​algorithm model and obtain a model for generating a virtual maintenance plan for the equipment.

[0070] Furthermore, the generated virtual maintenance plan for the equipment can be compared and verified with the actual maintenance case to evaluate its accuracy and effectiveness, and based on the verification results, the model parameters can be adjusted to optimize the model performance.

[0071] For example, the environmental image data includes an image of the motor of the inertial navigation system captured by an infrared thermal imager. By inputting the environmental image data into the equipment virtual maintenance plan generation model for training, a virtual maintenance plan for the equipment can be generated, including: Step 1: Turn off the equipment power; Step 2: Disassemble the motor housing; Step 3: Check whether the motor heat sink is blocked; Step 4: Clean the heat sink; Step 5: Reinstall the motor housing; Step 6: Turn on the equipment power and test the motor operation. Furthermore, a conditional generative adversarial network (CGAN) can be used to convert the text description of the above maintenance steps into corresponding images or animations. For example, Image 1 shows how to turn off the equipment power. Image 2: Shows how to disassemble the motor housing. Image 3: Shows how to check and clean the heat sink. Image 4: Shows how to reinstall the motor housing and test the equipment.

[0072] Optionally, after each maintenance is completed, the target object may be prompted to provide feedback so as to optimize the generated virtual maintenance plan for the equipment.

[0073] Optionally, the equipment virtual maintenance plan generation model can be continuously updated and optimized using new maintenance records.

[0074] In one embodiment, the environment image data includes image data of equipment components. The images of the equipment components may be pre-processed, including denoising, cropping, and standardization.

[0075] In one embodiment, determining a virtual equipment maintenance plan based on the environmental image data includes:

[0076] extracting a device image of the target device from the environment image data;

[0077] identifying device status information of the target device based on the device image;

[0078] A virtual maintenance plan for the equipment is determined based on the equipment status information.

[0079] For example, the convolutional neural network (CNN) in the equipment virtual maintenance plan generation model is used to identify the various components of the inertial navigation system:

[0080]

[0081] Wherein, I is a device image of a target device extracted from environmental image data, and the device image includes a certain component of an inertial navigation system. is the identified component category, such as "accelerometer," "gyroscope," etc. The support vector machine (SVM) or decision tree (Decision Tree) in the equipment virtual maintenance plan generation model is used to identify the component fault type and the corresponding fault description (i.e., equipment status information).

[0082]

[0083] Where F is the input fault feature, representing the fault feature vector extracted from the device image. It includes quantitative indicators such as the accelerometer reading fluctuation amplitude and temperature change rate. For example, the feature value corresponding to "large accelerometer reading fluctuation" is It is the predicted fault type and the corresponding fault description, such as "sensor fault" and "power supply fault".

[0084] The recurrent neural network (RNN) or transformer in the equipment virtual maintenance plan generation model is used to generate an equipment virtual maintenance plan corresponding to the identified faulty component.

[0085]

[0086] Where X is the input fault description and component information, such as "accelerometer abnormality, unstable readings", and X is the combination of text description and component feature vector, which is converted into sequence input through word embedding technology and can be expressed as It is a generated virtual maintenance plan for the device, for example, "1. Disconnect the power supply; 2. Remove the accelerometer; 3. Check the connection lines; 4. Replace the accelerometer."

[0087] In one embodiment, identifying the device status information of the target device based on the device image includes:

[0088] Extracting features from the device image to obtain device feature information;

[0089] identifying components of the target device and states of the components based on the device characteristic information;

[0090] The device status information is determined according to components of the target device and statuses of the components.

[0091] The device feature information refers to quantitative or descriptive feature data used to describe various characteristics and states of the components of the device, such as edges, textures, shapes, etc.

[0092] The neural network can be used to extract features from the device image to obtain device feature information, and the clustering algorithm can be used to classify the device status information to obtain the component name of the target device and the status corresponding to the component name. In this way, the device status information can be obtained based on the component name and the status of each component.

[0093] In one embodiment, determining the virtual maintenance plan for the device based on the device status information includes:

[0094] Obtaining an actual maintenance plan for the target device based on the device status information;

[0095] Decomposing the actual maintenance plan into steps to obtain a maintenance step sequence;

[0096] The virtual maintenance plan for the equipment is determined based on the maintenance step sequence.

[0097] Based on the equipment status information, the actual maintenance plan corresponding to the fault status (i.e., fault type) of the component in the equipment status information can be obtained from the equipment maintenance manual or the fault case library. Then, based on the standard operating procedures of the maintenance manual and combined with the historical step sequence in the fault case library, the actual maintenance plan is operationally decomposed so that each step corresponds to a maintenance operation, and a maintenance step sequence is obtained, which is used as a virtual maintenance plan for the equipment.

[0098] Optionally, detailed operation prompts for each step can be generated, including required tools, operation methods, precautions, etc., so that the decomposed steps and the operation prompts for each step can be used as a virtual maintenance plan for the equipment.

[0099] S104: Mapping the virtual maintenance plan of the device to a virtual coordinate system of the wearable device based on the spatial position and direction to visualize the virtual maintenance plan of the device, and guiding the target object to perform maintenance operations on the target device based on the visualization effect.

[0100] By establishing a virtual coordinate system on a wearable device (such as AR glasses, helmets, etc.), the virtual coordinate system corresponds to the spatial position and orientation of the target object. The virtual maintenance plan of the equipment (such as maintenance step prompts, fault diagnosis information) is mapped to the virtual coordinate system to ensure that the virtual maintenance plan of the equipment is aligned with the real environment in the field of view of the target object (i.e., the field of view space). The virtual coordinate system (the virtual coordinate system is a virtual, global coordinate system used to calculate the position and orientation of the target device in the virtual environment) is dynamically aligned with the real environment coordinate system (the real environment coordinate system is usually based on the earth's ellipsoid model or geographic coordinate system, used to describe the position of objects on the earth's surface) through SLAM (Simultaneous Localization and Mapping) technology.

[0101] For example, imagine a technician wearing AR glasses walks up to a piece of industrial equipment requiring repair. The AR glasses use a camera to identify visual markers on the equipment (such as a QR code) and, combined with IMU and gesture data, determine the equipment's spatial position and orientation. A virtual coordinate system is established within the AR glasses, corresponding to the target equipment's spatial position and orientation.

[0102] The virtual device maintenance plan is as follows: Step 1: Turn off the device power. Text prompt: "Turn off the device power." Graphic prompt: A red arrow points to the device's power switch.

[0103] Step 2: Remove the device housing. Text prompt: "Remove the device housing." Image prompt: A blue arrow points to the screws on the device housing.

[0104] Step 3: Check for faulty components. Text prompt: "Check for faulty components." Graphic prompt: A green arrow points to the location of the faulty component.

[0105] Step 4: Replace the faulty part. Text prompt: "Replace the faulty part." Graphic prompt: A yellow arrow points to the new part.

[0106] Step 5: Reinstall the device housing. Text: "Reinstall the device housing." Image: A blue arrow points to the screws on the device housing.

[0107] Step 6: Turn on the device and test it. Text prompt: "Turn on the device and test it." Graphic prompt: A red arrow points to the device's power switch.

[0108] AR glasses map the generated virtual device repair plan to a virtual coordinate system and display the information in the virtual device repair plan in the technician's field of view. For example, when the technician looks at the device's power switch, the AR glasses display the text prompt "Turn off device power" and a red arrow in the field of view. When the technician looks at the screws on the device casing, the AR glasses display the text prompt "Remove device casing" and a blue arrow in the field of view. When the technician looks at a faulty component, the AR glasses display the text prompt "Check faulty component" and a green arrow in the field of view.

[0109] Optionally, the display position and direction of the information in the virtual maintenance plan of the device are adjusted in real time according to the changes in the posture data to ensure that the virtual maintenance place of the device is always accurately positioned in the correct position in the user's field of view.

[0110] In one embodiment, the position and orientation of the virtual maintenance plan for the equipment in the virtual coordinate system can be updated in real time according to changes in the posture data. For example, the position and orientation of the virtual information can be updated in real time as the head posture changes.

[0111] In this embodiment, a virtual coordinate system allows the virtual repair plan to be precisely aligned with the target device, providing intuitive visual guidance. As the technician moves and changes perspective, the virtual information updates in real time, ensuring consistent alignment with the target device. Technicians can intuitively view repair steps and troubleshooting information, reducing time spent searching for repair manuals and improving repair efficiency. Intuitive visual cues allow technicians to more accurately execute repair steps and reduce operational errors.

[0112] In one embodiment, the method further comprises:

[0113] receiving command request information from the target object;

[0114] Execute the operation corresponding to the command request information.

[0115] The command request information may be information generated when the target object interacts through a user interface set on the wearable device.

[0116] The user interface on the wearable device can be used to display the status of the current maintenance task and provide necessary operation prompts. The app application on the user interface supports multi-language interface to meet the needs of users in different regions.

[0117] The user interface features a high-definition video call button, allowing on-site maintenance personnel to communicate with remote experts in real time by clicking the button. Video calls support picture-in-picture mode, allowing experts to provide guidance while remotely observing the site. Furthermore, remote experts can mark key points in the video stream and use AR technology to overlay the marked information onto the maintenance personnel's (i.e., the target object) field of view in real time, allowing them to quickly respond to the expert's instructions and improve maintenance efficiency.

[0118] In response to the start operation of the remote sharing button on the user interaction interface, maintenance manuals, drawings and other files can be remotely shared to facilitate expert guidance on on-site maintenance.

[0119] Through this user interface, maintenance personnel can interact with the device through voice, gestures, touch, and other methods, such as consulting a repair manual or requesting remote support. The wearable device also supports natural language processing, allowing users to ask questions in spoken form.

[0120] For example, maintenance personnel can interact with wearable devices through voice commands. In response to the voice command, such as asking "System, what is the status of the device?", the system uses natural language processing technology to parse the voice, recognize the intention of the maintenance personnel, and provide corresponding feedback. Specifically, a third-party voice recognition library, such as CMU Sphinx, Google SpeechtoText, etc., can be integrated into the APP on the user interaction interface. The user interaction interface includes a voice input button and a feedback display area. When the target object (maintenance personnel) clicks the voice input button, the voice recognition library is activated to start recording, and the recorded data is sent to the server for voice recognition processing. The voice recognition service is deployed on the server, which receives the recorded data, uses the voice recognition algorithm to parse the recorded data, recognizes the intention of the maintenance personnel, and provides feedback corresponding to the intention, and displays the feedback on the user interaction interface, such as "The device status is normal."

[0121] For example, maintenance personnel can interact with AR glasses or helmets through specific gestures, such as waving or pointing. Gestures are recognized through sensor data, and corresponding operations are performed based on the gestures, such as zooming in or out of the virtual maintenance plan for the equipment displayed on the glasses or helmet. Specifically, a gesture recognition library, such as Gesture Recognizer, is integrated into the APP. The user interaction interface includes gesture operation prompts and feedback display areas. When the maintenance personnel perform gesture operations, the gesture image is captured by the camera. The gesture image data is sent to the server for gesture recognition processing, and the gesture recognition algorithm is used to parse the image data, identify the user's gesture intention, and perform corresponding operations based on the recognized gesture intention, and perform corresponding operations according to the execution results.

[0122] For example, maintenance personnel can interact with the wearable device by touching the screen or buttons. In response to touch events, functions such as querying maintenance manuals or requesting remote support are provided.

[0123] For example, maintenance personnel can ask questions in spoken form, such as "System, which parts need to be checked today?" In response to the voice command, natural language processing technology is used to parse the spoken language, understand the user's question, and provide a corresponding answer.

[0124] When a voice command from a maintenance worker is received, the system uses natural language processing technology to parse the command and extract key information (such as the device status) from the command. Based on the parsed results, a response is generated, which may include text, images, or voice. The response is displayed to the maintenance worker through a user interface or output via voice. If the maintenance worker requests a specific action (such as requesting remote support), the corresponding process is initiated in response to the action. Optionally, feedback from the maintenance worker on the response can be collected and used to optimize the natural language processing model and user interface.

[0125] In one embodiment, the method further comprises:

[0126] Obtaining device operating status data of the target device;

[0127] predicting fault information of the target device based on the device operation data;

[0128] An equipment early warning plan is determined based on the fault information.

[0129] Equipment operating status data can include historical maintenance records. By analyzing these records, we can identify patterns or sequences that may lead to failures, such as failure patterns. For example, if a component has been replaced in the past three repairs, we can determine that it is about to fail again. Machine learning algorithms can also be used to analyze historical maintenance records to identify failure patterns and frequencies, thereby predicting the likelihood of future failures.

[0130] Equipment operating status data can include real-time monitoring data, including temperature, pressure, vibration, and other data, to identify abnormal trends and other fault information. For example, a rapid increase in the target equipment temperature over a short period of time may indicate a cooling system failure. Real-time monitoring data can also include data collected by various sensors, such as accelerometers and gyroscopes. By analyzing this data, equipment status can be monitored in real time.

[0131] Device operating status data also includes operating environment factors, including ambient light, humidity, dust, and other factors that may affect the normal operation of the device. For example, under extreme temperature or humidity conditions, it is judged that the device may malfunction and other fault information.

[0132] Equipment operating status data can include usage frequency and duration. By analyzing the frequency and duration of equipment usage, wear or fatigue-related faults can be predicted. For example, if equipment is running continuously for a long time, bearing or gear fatigue faults may be predicted.

[0133] Equipment operating status data includes maintenance records. By analyzing equipment maintenance and servicing records, it is possible to identify faults caused by improper maintenance. For example, if equipment has not been maintained for an extended period of time, it may be possible to predict faults such as lubrication system failures.

[0134] By analyzing the on-site environment images captured by the camera, auxiliary fault diagnosis is performed to identify abnormal conditions of equipment components.

[0135] Virtual equipment maintenance plans include early warning information, which can be displayed directly to maintenance personnel. By analyzing real-time monitoring data and environmental factors collected by sensors and cameras, the system can predict equipment status and generate predictions. When a failure is predicted, a warning notification (voice, text, etc.) is sent to maintenance personnel, enabling them to take appropriate preventive measures based on the warning, such as inspecting the equipment or replacing parts. Optionally, maintenance recommendations can be generated based on the prediction results, including regular inspections and replacement of wearing parts.

[0136] When the prediction results indicate that the equipment is operating normally, but regular inspections are recommended to prevent potential problems, the generated equipment early warning plan includes a regular inspection schedule and a list of regular inspection items.

[0137] When a component in the structure table is predicted to be about to fail, the generated equipment early warning plan includes a recommendation to replace the component and provides information on the replacement steps and required tools.

[0138] When the prediction results indicate that equipment performance has degraded or there are multiple fault points, the generated equipment early warning plan includes recommendations for in-depth maintenance, including comprehensive inspections and replacement of multiple components.

[0139] When the prediction results indicate that a serious equipment failure is imminent and may affect equipment safety or production, the generated equipment early warning plan includes immediately initiating emergency maintenance procedures and giving priority to handling the failure.

[0140] The above is the equipment maintenance guidance process for this application.

[0141] As described above, the present application provides a device maintenance guidance method, device, electronic device and storage medium, which obtains environmental image data containing a target device through the camera and obtains posture data of the target object through the sensor; determines the spatial position and orientation of the target object based on the environmental image data and the posture data; determines a device virtual maintenance plan for the target device based on the environmental image data; maps the device virtual maintenance plan to the virtual coordinate system of the wearable device based on the spatial position and orientation, so as to visualize the device virtual maintenance plan, and guides the target object to perform maintenance operations on the target device based on the visualization effect. In the device maintenance guidance scheme provided by the present application, the fused data obtained by data fusion of the environmental image data and the posture data accurately reflects the spatial position and orientation of the target object, thereby achieving accurate positioning of the spatial position and orientation of the target object; the device virtual maintenance plan determined based on the environmental image data, combined with the spatial position and orientation, accurately maps the device virtual maintenance plan to the virtual coordinate system, and realizes visualization of the device virtual maintenance plan to guide the target object to perform maintenance operations on the target device, reducing the requirements for the professional skills of the target object, reducing the risk of misoperation, making maintenance work more efficient, accurate and convenient, and improving maintenance efficiency.

[0142] It should be understood that the order of execution of the steps in the above embodiments does not necessarily mean the order of execution. The order of execution of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0143] In one embodiment, a device maintenance guidance device is provided, which corresponds one-to-one to the device maintenance guidance method in the above embodiment. Figure 3 As shown, the device maintenance guidance device is applied to a wearable device, wherein the wearable device is provided with a camera and a sensor, and the device maintenance guidance device includes:

[0144] An acquisition module 201 is configured to acquire environmental image data including a target device through the camera and acquire posture data of the target object through the sensor;

[0145] A first determining module 202 is configured to determine a spatial position and orientation of the target object based on the environment image data and the posture data;

[0146] A second determining module 203 is configured to determine a virtual maintenance plan for the target device based on the environmental image data;

[0147] The display module 204 is used to map the virtual maintenance plan of the device to the virtual coordinate system of the wearable device based on the spatial position and direction, so as to visualize the virtual maintenance plan of the device and guide the target object to perform maintenance operations on the target device based on the visualization effect.

[0148] In the equipment maintenance guidance scheme provided by the present application, environmental image data including a target device is obtained by the camera, and posture data of the target object is obtained by the sensor; the spatial position and orientation of the target object are determined based on the environmental image data and the posture data; a virtual equipment maintenance plan for the target device is determined based on the environmental image data; the virtual equipment maintenance plan is mapped to a virtual coordinate system of the wearable device based on the spatial position and orientation, so as to visualize the virtual equipment maintenance plan, and guide the target object to perform maintenance operations on the target device based on the visualization effect. In the equipment maintenance guidance scheme provided by the present application, the fused data obtained by data fusion of the environmental image data and the posture data accurately reflects the spatial position and orientation of the target object, thereby achieving precise positioning of the spatial position and orientation of the target object; the virtual equipment maintenance plan determined based on the environmental image data is accurately mapped to the virtual coordinate system in combination with the spatial position and orientation, so as to achieve visual display of the virtual equipment maintenance plan, so as to guide the target object to perform maintenance operations on the target device, thereby reducing the requirements for the professional skills of the target object, reducing the risk of misoperation, making maintenance work more efficient, accurate and convenient, and improving maintenance efficiency.

[0149] Optionally, the first determining module further includes a time synchronization submodule, and the time synchronization submodule is configured to:

[0150] Time-synchronizing the environmental image data and the posture data;

[0151] The spatial position and orientation of the target object are determined based on the time-synchronized environment image data and posture data.

[0152] Optionally, the second determining module includes:

[0153] an extraction submodule, configured to extract a device image of the target device from the environment image data;

[0154] an identification submodule, configured to identify device status information of the target device based on the device image;

[0155] The determination submodule is used to determine the virtual maintenance plan of the equipment based on the equipment status information.

[0156] Optionally, the identification submodule includes:

[0157] an extraction unit, configured to extract features from the device image to obtain device feature information;

[0158] an identification unit, configured to identify components of the target device and states of the components based on the device feature information;

[0159] The first determining unit is configured to determine the device status information according to components of the target device and statuses of the components.

[0160] Optionally, the determining submodule includes:

[0161] an acquiring unit, configured to acquire an actual maintenance plan for the target device based on the device status information;

[0162] a decomposition unit, configured to decompose the actual maintenance plan into steps to obtain a maintenance step sequence;

[0163] The second determining unit is configured to determine the virtual maintenance plan for the equipment based on the maintenance step sequence.

[0164] Optionally, the device further includes an execution module, wherein the execution module is configured to:

[0165] receiving command request information from the target object;

[0166] Execute the operation corresponding to the command request information.

[0167] Optionally, the device further includes a prediction module, wherein the prediction module is configured to:

[0168] Obtaining device operating status data of the target device;

[0169] predicting fault information of the target device based on the device operation data;

[0170] An equipment early warning plan is determined based on the fault information.

[0171] In one embodiment, an electronic device is provided, wherein the internal structure of the electronic device can be as follows: Figure 4 As shown. The electronic device includes a processor, memory, network interface, display screen, and input device connected via a system bus. The processor of the electronic device is used to provide computing and control capabilities. The memory of the electronic device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer program in the non-volatile storage medium. The network interface of the electronic device is used to communicate with an external server via a network connection. When the computer program is executed by the processor, it implements the functions or steps of an equipment maintenance guidance method.

[0172] In one embodiment, an electronic device is provided, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the following steps are implemented:

[0173] Acquire environmental image data containing a target device through a camera and acquire posture data of the target object through a sensor; determine the spatial position and orientation of the target object based on the environmental image data and the posture data; determine a virtual maintenance plan for the target device based on the environmental image data; map the virtual maintenance plan for the device to a virtual coordinate system of the wearable device based on the spatial position and orientation to visualize the virtual maintenance plan for the device, and guide the target object to perform maintenance operations on the target device based on the visualization effect.

[0174] In this embodiment, the fused data obtained by fusing the environmental image data and the posture data accurately reflects the spatial position and direction of the target object, thereby realizing precise positioning of the spatial position and direction of the target object. The virtual maintenance plan of the equipment determined based on the environmental image data, combined with the spatial position and direction, accurately maps the virtual maintenance plan of the equipment to the virtual coordinate system, realizing the visualization display of the virtual maintenance plan of the equipment to guide the target object to perform maintenance operations on the target equipment, reducing the requirements for the professional skills of the target object, reducing the risk of misoperation, making the maintenance work more efficient, accurate and convenient, and improving maintenance efficiency.

[0175] In one embodiment, a computer-readable storage medium is provided. The computer-readable storage medium stores a computer program. When the computer program is executed by a processor, the following steps are implemented:

[0176] Acquire environmental image data containing a target device through a camera and acquire posture data of the target object through a sensor; determine the spatial position and orientation of the target object based on the environmental image data and the posture data; determine a virtual maintenance plan for the target device based on the environmental image data; map the virtual maintenance plan for the device to a virtual coordinate system of the wearable device based on the spatial position and orientation to visualize the virtual maintenance plan for the device, and guide the target object to perform maintenance operations on the target device based on the visualization effect.

[0177] In this embodiment, the fused data obtained by fusing the environmental image data and the posture data accurately reflects the spatial position and direction of the target object, thereby realizing precise positioning of the spatial position and direction of the target object. The virtual maintenance plan of the equipment determined based on the environmental image data, combined with the spatial position and direction, accurately maps the virtual maintenance plan of the equipment to the virtual coordinate system, realizing the visualization display of the virtual maintenance plan of the equipment to guide the target object to perform maintenance operations on the target equipment, reducing the requirements for the professional skills of the target object, reducing the risk of misoperation, making the maintenance work more efficient, accurate and convenient, and improving maintenance efficiency.

[0178] It should be noted that the above functions or steps that can be implemented by the computer-readable storage medium or electronic device can be referred to the relevant descriptions on the server side and the client side in the aforementioned method embodiment. To avoid repetition, they will not be described one by one here.

[0179] Those skilled in the art will appreciate that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program. The computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM) or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), memory bus (Rambus) direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

[0180] Those skilled in the art will clearly understand that for the sake of convenience and brevity of description, only the division of the above-mentioned functional units and modules is used as an example. In actual applications, the above-mentioned functions can be distributed and completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0181] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention, and should all be included in the scope of protection of the present invention.

Claims

1. A method for guiding equipment maintenance, characterized in that: Applied to a wearable device, the wearable device is provided with a camera and a sensor, and the method includes: Acquiring environmental image data including the target device through the camera and acquiring posture data of the target object through the sensor; determining a spatial position and orientation of the target object based on the environmental image data and the posture data; determining a virtual equipment maintenance plan for the target equipment based on the environmental image data; The virtual maintenance plan for the device is mapped to a virtual coordinate system of the wearable device based on the spatial position and direction to visualize the virtual maintenance plan for the device, and the target object is guided to perform maintenance operations on the target device based on the visualization effect.

2. The equipment maintenance guidance method according to claim 1, characterized in that: The determining the spatial position and orientation of the target object based on the environmental image data and the posture data includes: Time-synchronizing the environmental image data and the posture data; The spatial position and orientation of the target object are determined based on the time-synchronized environment image data and posture data.

3. The equipment maintenance guidance method according to claim 1, characterized in that: Determining a virtual maintenance plan for the target device based on the environmental image data includes: extracting a device image of the target device from the environment image data; identifying device status information of the target device based on the device image; A virtual maintenance plan for the equipment is determined based on the equipment status information.

4. The equipment maintenance guidance method according to claim 3, characterized in that: The identifying device status information of the target device based on the device image includes: Extracting features from the device image to obtain device feature information; identifying components of the target device and states of the components based on the device characteristic information; The device status information is determined according to components of the target device and statuses of the components.

5. The equipment maintenance guidance method according to claim 3, characterized in that: The determining of the virtual maintenance plan of the device based on the device status information includes: Obtaining an actual maintenance plan for the target device based on the device status information; Decomposing the actual maintenance plan into steps to obtain a maintenance step sequence; The virtual maintenance plan for the equipment is determined based on the maintenance step sequence.

6. The equipment maintenance guidance method according to claim 1, characterized in that: The method further comprises: receiving command request information from the target object; Execute the operation corresponding to the command request information.

7. The equipment maintenance guidance method according to claim 1, characterized in that: The method further comprises: Obtaining device operating status data of the target device; predicting fault information of the target device based on the device operation data; An equipment early warning plan is determined based on the fault information.

8. An equipment maintenance guidance device, characterized in that: Applied to a wearable device, the wearable device is provided with a camera and a sensor, and the method includes: an acquisition module, configured to acquire environmental image data containing a target device through the camera and acquire posture data of the target object through the sensor; A first determination module is configured to determine a spatial position and orientation of the target object based on the environmental image data and the posture data; A second determining module is configured to determine a virtual equipment maintenance plan for the target equipment based on the environmental image data; A display module is used to map the virtual maintenance plan of the device to the virtual coordinate system of the wearable device based on the spatial position and direction, so as to visualize the virtual maintenance plan of the device and guide the target object to perform maintenance operations on the target device based on the visualization effect.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the steps of the equipment maintenance guidance method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the equipment maintenance guidance method according to any one of claims 1 to 7 are implemented.

Images