Remote maintenance assisting method and device and computer readable storage medium

Abstract

The invention relates to a remote maintenance assisting method and device and a computer readable storage medium, and belongs to the technical field of remote maintenance, and the method comprises the steps that a head-mounted device collects identification information and a current image of a to-be-maintained device, and sends the identification information to a management platform; the head-mounted device obtains a three-dimensional model sent by the management platform, wherein the three-dimensional model comprises device operation data, maintenance operation steps and maintenance marks of the to-be-maintained device; rendering the three-dimensional model to obtain a three-dimensional model image of the three-dimensional model under the current visual angle of an operator; superposing the three-dimensional model image and the current image to obtain a superposed image of the to-be-maintained equipment; and displaying the superimposed image at a corresponding position of a current user interface, so that an operator maintains the to-be-maintained equipment according to the superimposed image. The method has the effect of improving the equipment maintenance efficiency and accuracy.

Description

Technical Field

[0001] The present invention relates to the technical field of remote maintenance, and in particular to a remote maintenance auxiliary method, device and computer-readable storage medium. Background Art

[0002] With the continuous advancement of industrial technology, the complexity and integration of modern industrial equipment have increased dramatically, and many industrial equipment are installed in remote factories.

[0003] The repair and maintenance of existing equipment mostly depends on the experience of the repairer, and the efficiency of the repair mostly depends on the proficiency. When new or unfamiliar equipment needs to be repaired, the maintenance operation on site is guided by the maintenance manual or remote telephone.

[0004] However, this method of relying on text descriptions or remote verbal guidance has many disadvantages: on the one hand, the process of consulting the manual is cumbersome and time-consuming, making it difficult to quickly locate the problem; on the other hand, remote communication is limited by the accuracy and timeliness of information transmission, which often leads to misunderstanding of on-site problems or improper implementation of solutions, which may cause further damage to the equipment or extend downtime, making it difficult to meet the needs of efficiency and accuracy. Summary of the invention

[0005] The technical problem to be solved by the present invention is to provide a remote maintenance assistance method, device and computer-readable storage medium, aiming to solve at least one of the above technical problems.

[0006] The technical solution of the present invention to solve the above technical problems is as follows:

[0007] In a first aspect, the present application provides a remote maintenance assistance method, which is applied to a head-mounted device and adopts the following technical solution:

[0008] A remote maintenance assistance method, applied to a head-mounted device, comprising:

[0009] Collect identification information of the equipment to be repaired and the current image from the operator's current perspective, and send the identification information to the management platform;

[0010] Acquire the three-dimensional model sent by the management platform, wherein the three-dimensional model includes equipment operation data, maintenance operation steps and maintenance marks of the equipment to be repaired;

[0011] Rendering the three-dimensional model to obtain a three-dimensional model image of the three-dimensional model under the current viewing angle of the operator;

[0012] Superimposing the three-dimensional model image with the current image to obtain a superimposed image of the equipment to be repaired;

[0013] The superimposed image is displayed at a corresponding position of the current user interface, so that the operator can perform maintenance on the equipment to be maintained according to the superimposed image.

[0014] The beneficial effect of the present invention is that by superimposing the three-dimensional model of the device to be repaired with the current image collected in real time, the operator can intuitively see the internal structure of the device, the maintenance operation steps and the specific location that needs to be repaired on the display interface of the head-mounted device, thereby avoiding the operation errors or omissions that may be caused by the non-intuitive maintenance manual in the traditional maintenance method, and greatly improving the maintenance efficiency;

[0015] The 3D model provides a stereoscopic view of the equipment, making the complex internal structure visual. For novice or inexperienced operators, this intuitive display method can significantly reduce the difficulty of maintenance and increase the success rate of maintenance.

[0016] Based on the above technical solution, the present invention can also be improved as follows.

[0017] Furthermore, during the process of the operator repairing the equipment to be repaired, the method further includes:

[0018] In response to the triggering action of the operator on the superimposed image, acquiring interaction information, the interaction information being any one of gesture sensing information, facial sensing information, voice information, key selection information or touch screen selection information;

[0019] The display content in the overlay image is adjusted based on the interaction information.

[0020] The beneficial effect of adopting the above further scheme is that different interaction methods are suitable for different scenarios and needs, allowing operators to interact with the system through gestures, face, voice, buttons or touch screens, etc., and can control the display of information more naturally without interrupting the maintenance process to find or switch different operation methods, thereby improving the fluency and convenience of the operator experience.

[0021] Furthermore, before displaying the superimposed image at a corresponding position of the current user interface, the method further includes:

[0022] Acquire historical behavior information of the operator, the historical behavior information including the number of times each maintenance function in the head mounted device worn by the operator is used, the number of times each operation area in the user interface is operated, and the number of times each interactive information is used, the operation area being a page area for executing a corresponding set function;

[0023] Based on the historical behavior information, determining the first priority of each maintenance function, the second priority of each operation area and the preference mode corresponding to the operator in the head mounted device, wherein the preference mode represents the arrangement and display mode of each interface element on the user interface selected by the operator based on the preference;

[0024] Determine an interface layout based on the first priority of each maintenance function, the second priority of each operation area and the preference mode, wherein the interface layout represents an arrangement and display mode of interface elements corresponding to each maintenance function and interface elements corresponding to each operation area on the user interface;

[0025] Based on the current image, determining environmental information of the device to be repaired, wherein the environmental information includes a lighting environment;

[0026] Based on the environment information, determining a display level, wherein the display level represents the clarity of the user interface display in different environments;

[0027] Based on the display level and the interface layout, a current user interface of the head mounted device is generated.

[0028] The beneficial effect of adopting the above further scheme is that by collecting and analyzing the operator's historical behavior information, the operator's preferences and operating habits can be understood. This method can intelligently place the most commonly used functions and operating areas in the positions where the operator can most easily reach or focus his or her sight based on the maintenance function priority and operating area priority determined based on the operator's historical behavior information. This layout method reduces the time the operator spends searching for and switching functions, and improves operating efficiency. The display level of the user interface can be adjusted in real time through the environmental information determined by the current image to ensure the clarity and readability of the user interface in different environments.

[0029] Furthermore, the method further comprises:

[0030] In response to a triggering action of the operator for remote diagnosis, sending a remote diagnosis request for the device to be repaired to the management platform;

[0031] After establishing a connection with at least one target remote expert's terminal, sharing the current user interface, and acquiring remote operation instruction information sent by each target remote expert's terminal, wherein the remote operation instruction information is any one or a combination of virtual marking, voice guidance, gesture guidance, graphic guidance and text guidance;

[0032] The remote operation instruction information is displayed at a corresponding position of the head mounted device, so that the operator can operate the device to be repaired according to the remote operation instruction.

[0033] The beneficial effect of adopting the above further solution is that the operator can send a remote diagnosis request to the management platform, the remote expert establishes a connection, and by sharing the current user interface, the remote expert can view the operator's operating environment and the status of the equipment to be repaired in real time, thereby providing more accurate and targeted guidance. At the same time, this real-time sharing also promotes the collaboration between the operator and the remote expert, allowing both parties to jointly solve problems and improve maintenance efficiency. Remote experts can provide remote operation instruction information through a variety of methods such as virtual marking, voice guidance, gesture guidance, graphic guidance and text guidance, making the guidance more intuitive and easy to understand, reducing the difficulty of operation.

[0034] In the second aspect, the present application provides a remote maintenance assistance method, which is applied to a management platform and adopts the following technical solutions:

[0035] A remote maintenance assistance method is applied to a management platform, the method comprising:

[0036] Obtain identification information of the device to be repaired sent by the head mounted device;

[0037] Based on the identification information, determining a three-dimensional model of the equipment to be repaired, wherein the three-dimensional model includes equipment operation data, maintenance operation steps and maintenance marks of the equipment to be repaired;

[0038] The three-dimensional model is sent to the head-mounted device so as to render the three-dimensional model through the head-mounted device to obtain a three-dimensional model image of the three-dimensional model under the current perspective of the operator, the three-dimensional model image is superimposed on the current image to obtain a superimposed image of the equipment to be repaired, and the superimposed image is displayed at a corresponding position of the current user interface, so that the operator can repair the equipment to be repaired according to the superimposed image.

[0039] The beneficial effects of the present invention are as follows: by matching the corresponding three-dimensional model in the preset model database through the identification information of the equipment to be repaired, the management platform can provide the maintenance operator with a three-dimensional model, which contains key information such as equipment operation data, maintenance operation steps and maintenance marks, and provides the operator with accurate maintenance guidance. The operator can intuitively view the internal structure, fault location and maintenance method of the equipment, thereby reducing misjudgment and misoperation and improving the accuracy of maintenance. Through the remote maintenance assistance method, the operator does not need to carry paper drawings or remote verbal guidance from technicians, avoiding repeated work and additional costs caused by inaccurate information transmission.

[0040] Furthermore, the method for generating the three-dimensional model of the equipment to be repaired comprises:

[0041] Acquire image information of the equipment to be repaired and equipment operation data of each monitoring position, where the monitoring position is an installation position of a monitoring sensor for monitoring the equipment to be repaired;

[0042] Extracting structural features from the image information, and inputting the extracted structural features into a preset neural network model to construct an initial model of the equipment to be repaired, wherein the initial model represents the appearance, structural layout, and relative positional relationship between components of the equipment to be repaired;

[0043] Analyze the equipment operation data at each monitoring location to obtain the fault location and fault type of the equipment to be repaired;

[0044] Based on the fault location, generating a maintenance mark of the equipment to be repaired;

[0045] Based on the fault type, determining maintenance operation steps for the equipment to be repaired;

[0046] The equipment operation data, maintenance operation steps and maintenance marks of each monitoring position are associated with the initial model to obtain a three-dimensional model of the equipment to be maintained.

[0047] The beneficial effect of adopting the above further scheme is: by extracting the structural features of the image information and inputting the extracted structural features into the neural network model, an initial model that is highly close to the actual equipment can be constructed, and data analysis of the equipment operation data at each monitoring location can be performed to accurately identify the fault location and type. Based on the fault location and type, the generated maintenance mark and the determined maintenance operation steps are directly associated with the corresponding position of the three-dimensional model. This targeted guidance method enables operators to quickly locate the fault point and understand the specific maintenance operation steps, thereby improving maintenance efficiency and quality.

[0048] Further, determining the maintenance operation steps of the equipment to be maintained based on the fault type includes:

[0049] Based on the fault type of the equipment to be repaired, determining whether the equipment to be repaired has historical maintenance information of the fault type, wherein the historical maintenance information includes operator operation information and operator feedback results;

[0050] If there is no historical maintenance information of the fault type, determining the maintenance operation steps of the equipment to be repaired based on the fault type in a preset operation database; or sending an operation request to a remote expert system based on the fault type of the equipment to be repaired to obtain the maintenance operation steps fed back by the remote expert;

[0051] If there is historical maintenance information for this fault type, the historical maintenance information is analyzed through a preset machine learning model to generate a new maintenance plan, and the maintenance operation steps of the equipment to be repaired are determined based on the new maintenance plan.

[0052] The beneficial effect of adopting the above further solution is that, in the absence of historical maintenance information, the maintenance operation steps can be automatically obtained from the preset operation database or remote expert system to provide timely and accurate guidance for maintenance personnel. In the case of historical maintenance information, the historical maintenance information can be analyzed through a preset machine learning model to generate a more optimized and targeted new maintenance plan.

[0053] Furthermore, the method further comprises:

[0054] Obtaining a remote diagnosis request for the device to be repaired, sent by an operator through the head mounted device;

[0055] Based on the remote diagnosis request, determine the device type, fault type and fault degree of the device to be repaired;

[0056] Based on the device type and fault type of the device to be repaired, matching a remote expert group corresponding to the device type and fault type;

[0057] Based on the working status, remote times and fault degree of the equipment to be repaired of each remote expert in the remote expert group, at least one target remote expert is determined, and a connection invitation is sent to the terminal of each target remote expert so that the terminal of each target remote expert establishes a connection with the head-mounted device of the operator.

[0058] The beneficial effect of adopting the above further solution is that the operator can obtain professional remote technical support through the remote diagnosis request sent by the head-mounted device. The management platform can match and process the remote expert group corresponding to the device to be repaired according to the device type and fault type of the device to be repaired. When determining the target remote expert, the management device considers multiple factors such as the expert's working status, the number of remote visits, and the degree of fault of the device to be repaired, so as to optimize the allocation of expert resources, ensure the efficient use of expert resources, avoid resource waste, and enable operators to obtain the expert support that best meets their needs.

[0059] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0060] An electronic device comprises a memory and a processor, wherein the memory stores a computer program that can be loaded by the processor and execute the remote maintenance assistance method according to any one of the first aspect and the second aspect.

[0061] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0062] A computer-readable storage medium stores a computer program that can be loaded by a processor and execute the remote maintenance assistance method according to any one of the first aspect and the second aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 A schematic diagram of a flow chart of a remote maintenance assistance method of the present invention applied to a head mounted device;

[0064] Figure 2 A flowchart of a remote maintenance assistance method of the present invention applied to a management platform;

[0065] Figure 3 The figure is a structural block diagram of an electronic device of the present invention. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0067] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0068] An embodiment of the present application provides a remote maintenance assistance method applied to a head-mounted device, and the method is implemented by a head-mounted device. The head-mounted device can be smart glasses, a smart head-mounted display device, a smart helmet, and a smart head-mounted near-eye device, etc. This embodiment is not specifically limited.

[0069] Figure 1 The present invention is a flowchart of a remote maintenance assistance method applied to a head-mounted device.

[0070] like Figure 1 As shown, the main process of the method is described as follows (steps S101 to S104):

[0071] Step S101, collecting identification information and current image of the equipment to be repaired, and sending the identification information to the management platform;

[0072] Step S102, obtaining a three-dimensional model sent by the management platform, wherein the three-dimensional model includes equipment operation data, maintenance operation steps and maintenance marks of the equipment to be maintained;

[0073] Step S103, rendering the three-dimensional model to obtain a three-dimensional model image of the three-dimensional model under the current viewing angle of the operator;

[0074] Step S104, superimposing the three-dimensional model image with the current image to obtain a superimposed image of the equipment to be repaired;

[0075] Step S105: displaying the superimposed image at a corresponding position of the current user interface, so that an operator can perform maintenance on the equipment to be maintained according to the superimposed image.

[0076] In the embodiment of the present application, the head-mounted device is integrated with a camera, various types of sensors, a communication module and a display screen, etc. The head-mounted device can collect the identification information of the device to be repaired by identifying the QR code, barcode or RFID tag on the device to be repaired through the camera to obtain the unique identification information of the device to be repaired. In addition, the head-mounted device can also be connected to the same network as the device to be repaired through the communication module to identify the unique identification information of the device to be repaired, but is not limited to this.

[0077] The head-mounted device uses a camera to collect the current image of the device to be repaired, which includes the appearance of the device, the fault location and the surrounding environment. After that, the collected device identification information and the current image are sent to the remote management platform via a wireless network, which can be Wi-Fi, 4G / 5G, etc.

[0078] After the remote management platform recognizes the identification information, the head-mounted device obtains the 3D model sent by the management platform. The head-mounted device uses image processing technology to identify the device outline and device feature points in the current image, and then aligns these features with the corresponding parts of the 3D model image under the operator's current perspective to ensure the consistency of the two in spatial position. After that, the aligned 3D model is superimposed on the current image in a semi-transparent or set color form to form a superimposed image. The user interface is displayed on the display screen of the head-mounted device. The superimposed image is displayed at the preset image setting position in the current user interface.

[0079] Operators can intuitively see the internal structure of the device, maintenance steps and specific locations that need maintenance on the display interface of the head-mounted device, thus avoiding operational errors or omissions that may be caused by the non-intuitive maintenance manual in traditional maintenance methods, greatly improving maintenance efficiency; the three-dimensional model provides a three-dimensional view of the device, making the complex internal structure visual. For novice or inexperienced operators, this intuitive display method can significantly reduce the difficulty of maintenance and increase the success rate of maintenance.

[0080] Optionally, during the process of the operator repairing the equipment to be repaired, the method further includes:

[0081] In response to a triggering action of an operator for displaying the superimposed image, acquiring interaction information, wherein the interaction information is any one of gesture sensing information, facial sensing information, voice information, key selection information, or touch screen selection information;

[0082] The display content in the overlay image is adjusted based on the interaction information.

[0083] In the above implementation, the operator can interact according to the interactive actions to adjust the actual content of the display, such as highlighting the parts to be operated, displaying the next operation prompt, moving the maintenance operation steps in the superimposed image to other display areas, etc., but is not limited to this.

[0084] Taking gesture interaction as an example, the head-mounted device obtains gesture sensing information, then confirms the gesture sensing information, collects gesture images corresponding to the gesture sensing information after confirmation, performs feature recognition on the gesture image through image recognition technology, extracts gesture features from the gesture image, and matches the gesture features with each gesture reference feature in the database, wherein each gesture reference feature corresponds to an action instruction; if the action instruction is matched, the display content in the superimposed image is adjusted based on the action instruction. The gesture image can be a picture or a video, and image recognition and feature matching are public methods in the art, which will not be described in detail in this embodiment.

[0085] During the process of the operator repairing the equipment to be repaired, the head-mounted device also collects the operator's operation screen and operator feedback data in real time, and sends the operation screen and operator feedback data to a remote management platform, so that the management platform can monitor the repair progress, evaluate the repair quality, and optimize the repair plan.

[0086] Optionally, before displaying the superimposed image at a corresponding position of the current user interface, the method further includes (steps Sa to Sf):

[0087] Step Sa, obtaining the historical behavior information of the operator, the historical behavior information including the number of times each maintenance function in the head mounted device worn by the operator is used, the number of times each operation area in the user interface is operated, and the number of times each interactive information is used, the operation area being a page area for executing a corresponding set function;

[0088] Step Sb, based on the historical behavior information, determining the first priority of each maintenance function, the second priority of each operation area and the preference mode corresponding to the operator in the head mounted device, wherein the preference mode represents the arrangement and display mode of each interface element on the user interface selected by the operator based on preference;

[0089] Step Sc, determining an interface layout based on the first priority of each maintenance function, the second priority of each operation area and the preference mode, wherein the interface layout represents an arrangement and display mode of interface elements corresponding to each maintenance function and interface elements corresponding to each operation area on the user interface;

[0090] Step Sd, determining the environmental information of the equipment to be repaired based on the current image, wherein the environmental information includes a lighting environment;

[0091] Step Se, determining a display level based on the environmental information, wherein the display level represents the clarity of the user interface display in different environments;

[0092] Step Sf, generating a current user interface of the head mounted device based on the display level and the interface layout.

[0093] In the above embodiment, the head-mounted device records and stores the operator's historical behavior information in real time, and determines the operator's first priority of each maintenance function, the second priority of each operating area and the preference mode in the head-mounted device based on the statistics of the number of times each maintenance function is used, the number of operations of each operating area and the number of times each interactive information (such as gestures, voice, touch screen, etc.) is used in a set time period.

[0094] For example, each maintenance function is first sorted according to the number of times each maintenance function is used, and the first priority of each maintenance function is determined based on the first sorting result. The more times a maintenance function is used, the greater the first priority of the maintenance function.

[0095] Data analysis algorithms or machine learning models are used to analyze the operator's preference mode, which may be a gesture interaction mode, a voice command mode, etc.

[0096] The environmental information of the equipment to be repaired, such as lighting conditions, spatial layout, etc., is obtained in real time through the camera or sensor of the head-mounted device. Based on the environmental information, the display level in the current environment is determined. The display level ensures that the operator can clearly see the superimposed images and related information in different environments. The content of the current user interface will be dynamically updated according to the operator's real-time operation and environmental changes to ensure that the operator can always obtain accurate and useful information.

[0097] Optionally, the method further includes:

[0098] In response to a triggering action of the operator for remote diagnosis, sending a remote diagnosis request for the device to be repaired to the management platform;

[0099] After establishing a connection with at least one target remote expert's terminal, sharing the current user interface, and obtaining remote operation instruction information sent by each target remote expert's terminal, wherein the remote operation instruction information is any one or a combination of virtual marking, voice guidance, gesture guidance, graphic guidance and text guidance;

[0100] The remote operation instruction information is displayed at a corresponding position of the head mounted device, so that the operator can operate the device to be repaired according to the remote operation instruction.

[0101] In the above implementation, the operator triggers a remote diagnosis request through a specific gesture, voice command, button or touch screen operation on the head-mounted device, and the head-mounted device sends the remote diagnosis request for the device to be repaired to the remote management platform, and the remote diagnosis request may include identification information of the device to be repaired. After the management platform sends a connection invitation to the remote expert's terminal based on the remote diagnosis request, the head-mounted device establishes a connection with at least one target remote expert's terminal and shares the current user interface.

[0102] The remote expert can clearly see what the operator sees. And through the two-way communication mechanism between the remote expert terminal and the head-mounted device, the operator and the remote expert can have real-time voice, text or video calls for deeper communication and collaboration. The remote expert can guide the operator in the shared screen through virtual marking (such as highlighting the parts to be operated), voice guidance, gesture guidance (through video demonstration), graphic guidance (such as flow charts, schematic diagrams) and text guidance (such as operating step instructions).

[0103] Another embodiment of the present application provides a remote maintenance assistance method applied to a management platform, which can be executed by an electronic device with data processing capabilities, such as an independent physical server, a cloud server providing cloud computing services, etc.

[0104] Figure 2 The present invention is a flowchart of a remote maintenance assistance method applied to a management platform.

[0105] like Figure 2 As shown, the main process of the method is described as follows (steps S201 to S203):

[0106] Step S201, obtaining identification information of the device to be repaired sent by the head mounted device;

[0107] Step S202, based on the identification information, determining a three-dimensional model of the equipment to be repaired, wherein the three-dimensional model includes equipment operation data, maintenance operation steps and maintenance marks of the equipment to be repaired;

[0108] Step S203: Send the three-dimensional model to the head-mounted device so as to render the three-dimensional model through the head-mounted device to obtain a three-dimensional model image of the three-dimensional model under the current perspective of the operator, superimpose the three-dimensional model image with the current image to obtain a superimposed image of the equipment to be repaired, and display the superimposed image at a corresponding position of the current user interface, so that the operator can repair the equipment to be repaired according to the superimposed image.

[0109] In an embodiment of the present application, a database query statement (such as SQL) is used to search for a three-dimensional model record matching the device identification information in a model database.

[0110] In an embodiment of the present application, specifically, a method for generating a three-dimensional model of a device to be repaired includes:

[0111] First, image information of the equipment to be repaired and equipment operation data of each monitoring position are obtained, where the monitoring position is an installation position of a monitoring sensor used to monitor the equipment to be repaired;

[0112] Secondly, structural features are extracted from the image information, and the extracted structural features are input into a preset neural network model to construct an initial model of the equipment to be repaired, wherein the initial model represents the appearance, structural layout and relative positional relationship between components of the equipment to be repaired;

[0113] Afterwards, the equipment operation data at each monitoring location is analyzed to obtain the fault location and fault type of the equipment to be repaired;

[0114] Afterwards, based on the fault location, a maintenance mark of the device to be repaired is generated; based on the fault type, a maintenance operation step of the device to be repaired is determined;

[0115] Finally, the equipment operation data, maintenance operation steps and maintenance marks of each monitoring position are associated with the initial model to obtain a three-dimensional model of the equipment to be maintained.

[0116] In the above implementation, each device to be repaired is equipped with a high-precision camera and a multimodal sensor to capture the operating data of the device to be repaired and the image information of multiple angles and key components in real time; then, the acquired image information is subjected to preprocessing operations such as denoising, contrast enhancement, and cropping to improve the image quality. Computer vision technology is used to extract structural features from the preprocessed image to extract the three-dimensional structure and appearance of the device to be repaired. The extracted structural features are input into the neural network model to obtain an initial model, which can be a convolutional neural network model.

[0117] The equipment operation data of each monitoring location is compared with the corresponding monitoring threshold to determine the fault location and fault type. The fault location and fault type of the equipment operation data of each monitoring location can also be identified using machine learning algorithms, but is not limited to this.

[0118] Associating the equipment operation data, maintenance operation steps and maintenance marks of each monitoring location with the initial model specifically includes: adding maintenance marks to corresponding components or locations according to the fault location information in the three-dimensional space of the initial model; binding the maintenance operation steps to the fault location in the initial model. This is achieved by adding interactive elements (such as buttons and links) to the model, and the corresponding maintenance operation steps can be displayed or executed when the user clicks.

[0119] In the data structure of the three-dimensional model, a pointer or link to the relevant equipment operation data is added for each monitoring location, allowing direct access to the real-time or historical operation data of the component when viewing the model.

[0120] Further, the determining of the maintenance operation steps of the equipment to be maintained based on the fault type includes:

[0121] Based on the fault type of the equipment to be repaired, determining whether the equipment to be repaired has historical maintenance information of the fault type, wherein the historical maintenance information includes operator operation information and operator feedback results;

[0122] If there is no historical maintenance information of the fault type, determining the maintenance operation steps of the equipment to be repaired based on the fault type in a preset operation database; or sending an operation request to a remote expert system based on the fault type of the equipment to be repaired to obtain the maintenance operation steps fed back by the remote expert;

[0123] If there is historical maintenance information for this fault type, the historical maintenance information is analyzed through a preset machine learning model to generate a new maintenance plan, and the maintenance operation steps of the equipment to be repaired are determined based on the new maintenance plan.

[0124] In the above implementation, historical maintenance information matching the current fault type of the equipment to be repaired is queried through a database query statement. If there is no historical maintenance information, the maintenance operation steps predefined for the fault type are used as maintenance operation steps associated with the initial model. Alternatively, if no matching maintenance steps are found in the preset operation database, an operation request is sent to the remote expert system, which analyzes the information and returns the maintenance operation steps recommended by the expert.

[0125] If there is historical maintenance information, a preset machine learning model is used to analyze the historical maintenance information. The model can be based on supervised learning (such as decision trees, random forests, neural networks, etc.) or reinforcement learning, and is used to learn from historical data and generate new maintenance plans. The new maintenance plan is an optimization of the current fault type, an improved maintenance operation procedure, or additional precautions. The preset machine learning model is an existing implementation method and will not be described in detail in this embodiment.

[0126] Optionally, the method further includes:

[0127] Obtaining a remote diagnosis request for the device to be repaired, sent by an operator through the head mounted device;

[0128] Based on the remote diagnosis request, determine the device type, fault type and fault degree of the device to be repaired;

[0129] Based on the device type and fault type of the device to be repaired, matching a remote expert group corresponding to the device type and fault type;

[0130] Based on the working status, remote times and fault degree of the equipment to be repaired of each remote expert in the remote expert group, at least one target remote expert is determined, and a connection invitation is sent to the terminal of each target remote expert so that the terminal of each target remote expert establishes a connection with the head-mounted device of the operator.

[0131] In the above implementation, the remote diagnosis request may include unique identification information of the equipment to be repaired. The management platform determines the equipment type, fault type and fault degree in the three-dimensional model of the equipment to be repaired based on the identification information. The fault degree may be information feedback from the operator, or it may be analyzed by the management platform using a machine learning algorithm.

[0132] Based on the type of equipment and the type of fault, the management platform searches for a matching remote expert group in the remote expert database. The remote expert group consists of experts with specific domain knowledge and experience, and is classified according to the type of equipment and fault type.

[0133] Afterwards, at least one target remote expert is further determined based on the remote expert's working status, such as whether he is online or busy, the number of remote calls in a set time period, and the fault degree of the equipment to be repaired.

[0134] Through the remote diagnosis request sent by the head-mounted device, the operator can get professional remote technical support. The management device takes into account multiple factors such as the expert's working status, the number of remote visits, and the degree of failure of the equipment to be repaired, so as to optimize the allocation of expert resources, so that the operator can get the expert support that best meets their needs, and at the same time, enable remote experts to quickly join the remote session.

[0135] Figure 3 This is a structural block diagram of an electronic device 300 according to an embodiment of the present application.

[0136] like Figure 3 As shown, the electronic device 300 includes a processor 301 and a memory 302 , and may further include one or more of an information input / information output (I / O) interface 303 and a communication component 304 .

[0137] The processor 301 is used to control the overall operation of the electronic device 300 to complete all or part of the steps in the above-mentioned remote maintenance auxiliary method; the memory 302 is used to store various types of data to support the operation of the electronic device 300, and these data may include, for example, instructions for any application or method used to operate on the electronic device 300, and data related to the application. The memory 302 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0138] The I / O interface 303 provides an interface between the processor 301 and other interface modules, and the above-mentioned other interface modules can be keyboards, mice, buttons, etc. These buttons can be virtual buttons or physical buttons. The communication component 304 is used to test the wired or wireless communication between the electronic device 300 and other devices. Wireless communication, such as Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G or 4G, or a combination of one or more of them, so the corresponding communication component 304 can include: Wi-Fi components, Bluetooth components, NFC components.

[0139] The communication bus 305 may include a path to transmit information between the above components. The communication bus 305 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The communication bus 305 may be divided into an address bus, a data bus, a control bus, etc.

[0140] The electronic device 300 can be implemented by one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), controllers, microcontrollers, microprocessors or other electronic components to execute the remote maintenance assistance method given in the above embodiment.

[0141] The computer-readable storage medium provided in the embodiment of the present application is introduced below. The computer-readable storage medium described below and the remote maintenance assistance method described above can be referenced to each other.

[0142] The present application also provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned remote maintenance assistance method are implemented.

[0143] The computer-readable storage medium may include: a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, and other media that can store program codes.

[0144] The terms "comprises," "comprising," or any other variations thereof are intended to cover non-exclusive inclusion such that a process, method, article, or apparatus that includes a list of elements includes not only those elements but also other elements not expressly listed, or also includes elements inherent to such process, method, article, or apparatus.

[0145] The above description is only a preferred embodiment of the present application and an explanation of the technical principles used. Those skilled in the art should understand that the scope of application involved in the present application is not limited to the technical solution formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the aforementioned application concept. For example, the above features are replaced with (but not limited to) technical features with similar functions applied in the present application.

Claims

1. A remote maintenance assistance method, characterized in that: Applied to a head mounted device, the method comprises: Collect identification information of the equipment to be repaired and the current image from the operator's current perspective, and send the identification information to the management platform; Acquire the three-dimensional model sent by the management platform, wherein the three-dimensional model includes equipment operation data, maintenance operation steps and maintenance marks of the equipment to be repaired; Rendering the three-dimensional model to obtain a three-dimensional model image of the three-dimensional model under the current viewing angle of the operator; Superimposing the three-dimensional model image with the current image to obtain a superimposed image of the equipment to be repaired; The superimposed image is displayed at a corresponding position of the current user interface, so that the operator can perform maintenance on the equipment to be maintained according to the superimposed image.

2. A remote maintenance assistance method according to claim 1, characterized in that: During the process of the operator repairing the equipment to be repaired, the method further includes: In response to the triggering action of the operator on the superimposed image, acquiring interaction information, the interaction information being any one of gesture sensing information, facial sensing information, voice information, key selection information or touch screen selection information; The display content in the overlay image is adjusted based on the interaction information.

3. A remote maintenance assistance method according to claim 2, characterized in that: Before displaying the overlay image at the corresponding position of the current user interface, the method further includes: Acquire historical behavior information of the operator, the historical behavior information including the number of times each maintenance function in the head mounted device worn by the operator is used, the number of times each operation area in the user interface is operated, and the number of times each interactive information is used, the operation area being a page area for executing a corresponding set function; Based on the historical behavior information, determining the first priority of each maintenance function, the second priority of each operation area and the preference mode corresponding to the operator in the head mounted device, wherein the preference mode represents the arrangement and display mode of each interface element on the user interface selected by the operator based on the preference; Determine an interface layout based on the first priority of each maintenance function, the second priority of each operation area and the preference mode, wherein the interface layout represents an arrangement and display mode of interface elements corresponding to each maintenance function and interface elements corresponding to each operation area on the user interface; Based on the current image, determining environmental information of the device to be repaired, wherein the environmental information includes a lighting environment; Based on the environment information, determining a display level, wherein the display level represents the clarity of the user interface display in different environments; Based on the display level and the interface layout, a current user interface of the head mounted device is generated.

4. A remote maintenance assistance method according to claim 1, characterized in that: The method further comprises: In response to a triggering action of the operator for remote diagnosis, sending a remote diagnosis request for the device to be repaired to the management platform; After establishing a connection with at least one target remote expert's terminal, sharing the current user interface, and acquiring remote operation instruction information sent by each target remote expert's terminal, wherein the remote operation instruction information is any one or a combination of virtual marking, voice guidance, gesture guidance, graphic guidance and text guidance; The remote operation instruction information is displayed at a corresponding position of the head mounted device, so that the operator can operate the device to be repaired according to the remote operation instruction.

5. A remote maintenance assistance method, characterized in that: Applied to a management platform, the method comprises: Obtain identification information of the device to be repaired sent by the head mounted device; Based on the identification information, determining a three-dimensional model of the equipment to be repaired, wherein the three-dimensional model includes equipment operation data, maintenance operation steps and maintenance marks of the equipment to be repaired; The three-dimensional model is sent to the head-mounted device so as to render the three-dimensional model through the head-mounted device to obtain a three-dimensional model image of the three-dimensional model under the current perspective of the operator, the three-dimensional model image is superimposed on the current image to obtain a superimposed image of the equipment to be repaired, and the superimposed image is displayed at a corresponding position of the current user interface, so that the operator can repair the equipment to be repaired according to the superimposed image.

6. A remote maintenance assistance method according to claim 5, characterized in that: The method for generating a three-dimensional model of the equipment to be repaired comprises: Acquire image information of the equipment to be repaired and equipment operation data of each monitoring position, wherein the monitoring position is an installation position of a monitoring sensor used to monitor the equipment to be repaired; Extracting structural features from the image information, and inputting the extracted structural features into a preset neural network model to construct an initial model of the equipment to be repaired, wherein the initial model represents the appearance, structural layout, and relative positional relationship between components of the equipment to be repaired; Analyze the equipment operation data at each monitoring location to obtain the fault location and fault type of the equipment to be repaired; Based on the fault location, generating a maintenance mark of the equipment to be repaired; Based on the fault type, determining maintenance operation steps for the equipment to be repaired; The equipment operation data, maintenance operation steps and maintenance marks of each monitoring position are associated with the initial model to obtain a three-dimensional model of the equipment to be maintained.

7. A remote maintenance assistance method according to claim 6, characterized in that: The step of determining the maintenance operation steps of the equipment to be maintained based on the fault type includes: Based on the fault type of the equipment to be repaired, determining whether the equipment to be repaired has historical maintenance information of the fault type, wherein the historical maintenance information includes operator operation information and operator feedback results; If there is no historical maintenance information of the fault type, determining the maintenance operation steps of the equipment to be repaired based on the fault type in a preset operation database; or sending an operation request to a remote expert system based on the fault type of the equipment to be repaired to obtain the maintenance operation steps fed back by the remote expert; If there is historical maintenance information for this fault type, the historical maintenance information is analyzed through a preset machine learning model to generate a new maintenance plan, and the maintenance operation steps of the equipment to be repaired are determined based on the new maintenance plan.

8. A remote maintenance assistance method according to claim 5, characterized in that: The method further comprises: Obtaining a remote diagnosis request for the device to be repaired, sent by an operator through the head mounted device; Based on the remote diagnosis request, determine the device type, fault type and fault degree of the device to be repaired; Based on the device type and fault type of the device to be repaired, matching a remote expert group corresponding to the device type and fault type; Based on the working status, remote times and fault degree of the equipment to be repaired of each remote expert in the remote expert group, at least one target remote expert is determined, and a connection invitation is sent to the terminal of each target remote expert so that the terminal of each target remote expert establishes a connection with the head-mounted device of the operator.

9. An electronic device, characterized in that: The electronic device comprises a processor coupled to a memory; the processor is used to execute a computer program stored in the memory so that the electronic device executes the method according to any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that: The method comprises a computer program or an instruction, which, when executed on a computer, causes the computer to execute the method according to any one of claims 1 to 8.

Images