AR (Augmented Reality) human-factor-operation-prevention auxiliary method and AR equipment

Abstract

The invention relates to the technical field of equipment maintenance, in particular to an AR (Augmented Reality) anti-human-factor operation auxiliary method and AR equipment. The invention discloses an AR (Augmented Reality) anti-human-factor operation auxiliary method. The method comprises the following steps of: acquiring operation work order information by using AR equipment; the AR equipment obtains the current position in real time; determining to-be-operated target equipment; generating a guide map according to the current position and the position information of the target equipment to be operated in combination with a factory three-dimensional map; positioning a to-be-operated target device according to the guiding map; the AR equipment scans to-be-operated equipment and judges whether the to-be-operated equipment is to-be-operated target equipment or not; reminding according to a judgment result; receiving an execution state instruction issued by a user; and adjusting the operation work order information according to the execution state instruction until all the to-be-operated target equipment confirms execution. Therefore, the searching efficiency of the to-be-operated target equipment and the accuracy of operating the to-be-operated target equipment are improved, and the possibility of human errors is reduced.

Description

Technical Field

[0001] The present invention relates to the technical field of equipment maintenance, and in particular to an AR auxiliary method for preventing human-induced operation and an AR device. Background Art

[0002] To ensure the stable operation of nuclear power units, the accuracy of the instrumentation and control system operation is crucial. To prevent human errors, operators need to take a variety of preventive measures when handling DCS (distributed control system) and field equipment, such as "three-stage communication" and "star self-check". Among these measures, the guardianship system is one of the most important, but it requires operators to have a high level of skills and familiarity with the environment, and the field environment may cause them greater stress and fatigue.

[0003] Due to the large number of devices, the complicated on-site routes of the devices, and the high similarity between the devices, it is easy to make mistakes in device operation or find it difficult to locate the devices. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide an auxiliary method for AR to prevent human factors from operating.

[0005] The technical solution adopted by the present invention to solve the technical problem is: an AR auxiliary method for preventing human error operation, comprising the following steps:

[0006] Use AR devices to obtain operation work order information;

[0007] Using the AR device to obtain the current location in real time;

[0008] Determine the target device to be operated;

[0009] Generate a guidance map based on the current position and the position information of the target device to be operated in combination with a three-dimensional map of the factory area; locate the target device to be operated according to the guidance map;

[0010] The AR device scans the device to be operated, and determines whether the device to be operated is the target device to be operated;

[0011] Providing reminders based on the judgment results;

[0012] Receive the execution status instruction issued by the user; update the operation work order information according to the execution status instruction until all the target devices to be operated are confirmed to have been executed.

[0013] Preferably, the obtaining of operation work order information by using an AR device includes:

[0014] The AR device receives a work order title;

[0015] The AR device determines whether the operation work order information exists in a work order database according to the work order title;

[0016] If the operation work order information exists, obtain the operation work order information.

[0017] Preferably, the device to be operated includes: image information of the device to be operated displayed on a DCS system interface or a physical device of the device to be operated.

[0018] Preferably, the location information of the target device to be operated includes: location information of the DCS system that displays the device to be operated.

[0019] Preferably, the generating of the guidance map according to the current location information and the location information of the target device to be operated in combination with the three-dimensional map of the factory area includes:

[0020] Acquire the location information of the DCS system where the target device information to be operated is located from the operation work order information;

[0021] Locating the device position of the target device to be operated on the three-dimensional map of the factory area according to the location information of the DCS system;

[0022] Planning a traversable route between the current location and the device location according to a path planning algorithm;

[0023] Receive a route confirmation instruction and determine the route;

[0024] The guidance map is generated according to the travel route.

[0025] Preferably, the acquiring the target device information to be operated from the operation work order information includes:

[0026] Extracting device information of all the target devices to be operated in the operation work order information according to preset rules;

[0027] The target devices to be operated are sorted according to the text structure and / or the order in which the target devices to be operated appear in the operation work order information, and the device information of the target device to be operated that ranks first is used as the target device to be operated information.

[0028] Preferably, the AR device scans the device to be operated, and determines whether the device to be operated is the target device to be operated in combination with the current location information, including:

[0029] The AR device receives a scanning command;

[0030] Capturing image information displayed on the interface of the DCS system according to the scanning command;

[0031] Acquiring information of equipment to be operated in the DCS system by identifying the image information;

[0032] Determine whether the current position is consistent with the device position, and if not, alarm;

[0033] If they are consistent, it is determined whether the information of the device to be operated is consistent with the information of the target device to be operated. If they are consistent, it is considered that the device to be operated is the target device to be operated.

[0034] Preferably, the using the AR device to obtain the current position in real time includes:

[0035] The AR device uses wireless positioning technology to determine the user's location and obtain the current location.

[0036] Preferably, the operation work order information is divided into an unexecuted list and an executed list;

[0037] The receiving an execution status instruction issued by a user; and adjusting the operation work order information according to the execution status instruction, comprises:

[0038] If the execution status instruction is completed, the status of the device to be operated is adjusted to completed execution; and the device to be operated is moved to the executed list.

[0039] An AR device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the AR auxiliary method for preventing human error in operation as described in any one of the above items are implemented.

[0040] The implementation of the present invention has the following beneficial effects:

[0041] The present invention uses an AR device to obtain operation work order information and the current position. By determining the target device to be operated, and using the current position and the location information of the target device to be operated in combination with the three-dimensional map of the factory area, a guidance map is generated. The target device to be operated is then located according to the guidance map. Finally, the device to be operated is scanned by the AR device and the execution status instruction issued by the user is received until all the target devices to be operated are confirmed to have been executed. Thereby, the efficiency of finding the target device to be operated and the accuracy of operating the target device to be operated are improved, and the possibility of human error is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] The present invention will be further described below with reference to the accompanying drawings and embodiments:

[0043] Figure 1 The figure is a flow chart of an auxiliary method for AR to prevent human error in one embodiment. DETAILED DESCRIPTION

[0044] In order to have a clearer understanding of the technical features, purposes and effects of the present invention, specific embodiments of the present invention are now described in detail with reference to the accompanying drawings.

[0045] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present invention. However, it should be clear to those skilled in the art that the present invention may be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present invention.

[0046] An embodiment of the present invention provides an AR auxiliary method for preventing human error operation, such as Figure 1 As shown, the following steps are included:

[0047] Use AR devices to obtain operation work order information.

[0048] Specifically, the image information of the paper public survey is obtained by using an AR device, and the image information is converted into operation work order information; or the operation work order information is obtained from a database through data interaction.

[0049] Use AR devices to obtain current location in real time.

[0050] Specifically, the current position is obtained using a positioning device in the AR device.

[0051] Determine the target device to be operated.

[0052] Specifically, the target device to be operated is determined through the operation work order information.

[0053] Generate a guidance map based on the current location and the location information of the target device to be operated combined with the three-dimensional map of the factory area. Locate the target device to be operated based on the guidance map.

[0054] Specifically, AR devices, such as AR glasses, combined with a three-dimensional map of the factory area, can be used to achieve real-time positioning of staff and equipment to ensure efficient scheduling and management. Combined with GPS or indoor positioning technology, a path planning algorithm can be developed to provide users with the best path from one location to another, reducing maintenance time and costs.

[0055] The AR device scans the device to be operated and determines whether the device to be operated is the target device to be operated.

[0056] Specifically, the AR device uses its built-in camera to obtain the screen content of the device to be operated, including the image or feature mark of the device. The AR device scans the device image to identify the model or logo of the device. The AR device can not only identify the device to be operated, but also ensure that the operator can complete the operation task accurately and efficiently, thereby improving the accuracy and efficiency of the operation.

[0057] Make a reminder based on the judgment result.

[0058] Specifically, if the scanned device is not the target device to be operated, the system will issue an error reminder.

[0059] Receive the execution status instruction issued by the user. Update the operation work order information according to the execution status instruction until all the target devices to be operated are confirmed to have completed the execution.

[0060] Specifically, an execution status instruction is received from a user via an interface, a mobile device, a voice command or other means.

[0061] The present invention guides the user to quickly locate the target device to be operated through the guidance map generated by the AR device, thereby improving the efficiency and accuracy of the search and reducing human errors.

[0062] In some executable embodiments, using an AR device to obtain operation work order information includes:

[0063] The AR device receives the work order title.

[0064] Specifically, the AR device obtains image information of the paper public survey and extracts the work order title from the image information; or obtains the work order title from a database through data interaction.

[0065] The AR device determines whether there is operation work order information in the work order database based on the work order title. If there is operation work order information, the operation work order information is obtained.

[0066] Specifically, the work order title is used to query the operation work order information in the database, and if the corresponding operation work order information is found, the corresponding operation work order information is obtained.

[0067] In some executable embodiments, using an AR device to obtain the current location in real time includes:

[0068] AR devices use wireless positioning technology to determine the user's location and obtain the current location.

[0069] Specifically, wireless positioning technologies include:

[0070] GPS Tracking: GPS tracking uses the Global Positioning System to determine the geographic location of an AR device and is particularly useful in outdoor environments to display location-based content such as navigation instructions or location-based games.

[0071] SLAM tracking (Simultaneous Localization and Mapping): SLAM technology combines camera technology, sensors and special algorithms to detect the position and orientation of AR devices in real time, while creating a map of the environment and accurately locating virtual objects.

[0072] Furthermore, wireless communication technologies also include Wi-Fi, Bluetooth, 5G and Ultra-Wideband (UWB), which are used for data transmission and positioning in AR / VR.

[0073] UWB technology: used for positioning and gesture recognition, providing high-precision positioning and high data transmission rate.

[0074] AR Engine service: provides 6-degree-of-freedom (6DoF) pose calculation, including the position and orientation of objects, and obtains the pose of the device in space in real time.

[0075] Bluetooth positioning technology: By deploying Bluetooth beacons indoors, the user's device receives the signal and measures the strength, estimating the distance between the user and the beacon, thereby improving AR indoor navigation performance.

[0076] In an executable embodiment, the wireless positioning technology includes Bluetooth positioning technology.

[0077] It is understandable that compared with other wireless positioning technologies, Bluetooth positioning technology has the advantages of low power consumption, low cost, good compatibility, strong anti-interference ability, cost-effectiveness, high-precision positioning and simplified deployment. These characteristics make Bluetooth technology particularly suitable for application scenarios such as indoor navigation and personnel tracking.

[0078] In some executable embodiments, the device to be operated includes: image information of the device to be operated displayed on a DCS system interface or a physical device of the device to be operated.

[0079] Specifically, the DCS system interface and the physical device are often used in conjunction with each other. The operator can identify and monitor the DCS system interface through the present invention, and can also directly identify and monitor the device on site.

[0080] In some executable embodiments, the location information of the target device to be operated includes: location information of a DCS system that displays the device to be operated.

[0081] Optionally, if the device to be operated is a physical device, the location information of the target device to be operated is the location information of the physical device.

[0082] In some executable embodiments, a guidance map is generated based on the current location information and the location information of the target device to be operated in combination with a three-dimensional map of the factory area, including:

[0083] Obtain the target device information to be operated from the operation work order information.

[0084] According to the information of the target device to be operated, the location information of the DCS system where the target device to be operated is located is obtained.

[0085] In some scenarios, the information of the target device to be operated includes location information of a DCS system where the target device to be operated is located.

[0086] Furthermore, the location information of the corresponding DCS system can be queried through the device code in the information of the target device to be operated.

[0087] Locate the device location of the target device to be operated on the three-dimensional map of the factory area based on the location information of the DCS system;

[0088] Plan a traversable route between the current location and the device location based on the path planning algorithm.

[0089] Receive a route confirmation instruction and determine the route.

[0090] Generate a guidance map based on the travel route.

[0091] Optionally, extract detailed information of the target device to be operated from the operation work order system, including device number, location description, tasks to be performed, etc. Use high-precision positioning technologies such as Bluetooth AoA, combined with the three-dimensional map of the factory area, to accurately locate the location of the target device to be operated. Apply path planning algorithms such as the A* algorithm, combined with Bluetooth Beacon positioning data, to calculate the optimal path from the current location to the target device location. Receive the operator's route confirmation instructions through the AR device to ensure that the planned route meets the operator's actual travel needs. Once the operator confirms, the system will lock the route and provide navigation guidance for the operator. According to the confirmed route, it is displayed on the AR device to guide the operator to the target device to be operated.

[0092] The present invention can effectively obtain information of the target device to be operated from the operation work order, and use advanced positioning technology and path planning algorithms to provide accurate navigation services for operators, thereby ensuring efficient and accurate operation.

[0093] In some executable embodiments, obtaining information of a target device to be operated from operation work order information includes:

[0094] Extract the device information of all target devices to be operated in the operation work order information according to the preset rules.

[0095] The target devices to be operated are sorted according to the text structure and / or the order in which they appear in the operation work order information, and the device information of the target device to be operated that ranks first is used as the target device to be operated information.

[0096] Optionally, after receiving the operation work order information, the AR device extracts the devices to be operated in the work order according to certain rules (such as regular expressions and rules). After extraction, the AR device sorts the devices according to the structure of the previous text and the order in which the devices appear in the work order. Finally, all the devices that the user needs to operate in this work order and the order of operation are determined.

[0097] In an executable embodiment, AR equipment is used to digitize paper chemical orders. The digitization process is mainly completed through OCR technology: (1) Use table extraction technology to obtain the layout information of the table. (2) Through keyword extraction, obtain the step-related columns of the operation step. (3) According to the extracted steps, digitize the workflow.

[0098] After digitization is completed, AR glasses synchronize the digitized work order to the work order record on the computer through communication technologies such as wifi. The computer saves the operation steps and title of the work order. Later, you can find the corresponding work order in the computer by searching the title, and then import the work order in the computer directly into the AR device.

[0099] In some executable embodiments, the AR device scans the device to be operated, and determines whether the device to be operated is a target device to be operated in combination with the current location information, including:

[0100] The AR device receives the scanning command.

[0101] Capture the image information displayed on the interface of the DCS system according to the scanning command.

[0102] By identifying image information, the information of the equipment to be operated in the DCS system is obtained.

[0103] Determine whether the current position is consistent with the device position. If not, an alarm is issued.

[0104] If they are consistent, it is determined whether the information of the device to be operated is consistent with the information of the target device to be operated. If they are consistent, it is considered that the device to be operated is the target device to be operated.

[0105] Specifically, first, the AR device receives the scanning command through the built-in sensor and camera, and captures the image information of the device to be operated. Then, the image recognition algorithm in AR, such as AKAZE or RANSAC, is used to identify the image information and extract the feature points and descriptors of the device to be operated. Then, through natural language processing technology, such as NLTK, spaCy or Transformers, the text structure in the work order information is analyzed to identify and sort the target devices to be operated. Finally, the identified device information is matched with the target device information to be operated. If they are consistent, the device to be operated is confirmed to be the target device to be operated.

[0106] The present invention can efficiently identify and confirm the target device to be operated through the AR device, thereby improving the accuracy and efficiency of the operation.

[0107] In some scenarios, if the information of the device to be operated belongs to the operation work order, but it is not the device to be currently operated, a skip step reminder will be given.

[0108] Specifically, when it is determined that the target device is not the device to be currently operated, the step skipping reminder function will be automatically triggered to notify the operator that the current operation does not conform to the predetermined process.

[0109] Furthermore, when the devices to be operated are ranked, and the target device is not ranked first among the devices to be operated, a skip operation is prompted.

[0110] Skip operation can reduce unnecessary steps, thereby improving the efficiency of the operation, especially in emergency situations, so that it can respond quickly and take necessary measures. And skip operation provides a flexible response mechanism, allowing operators to quickly adjust the operation steps according to the actual situation to adapt to the changing environment.

[0111] In an practicable embodiment, if the target device to be operated is a physical device, image information of the physical device is captured according to a scanning command. By identifying the image information, the device information to be operated on the physical device is obtained. It is determined whether the device information to be operated is consistent with the target device information to be operated. If they are consistent, the device to be operated is considered to be the target device to be operated.

[0112] Furthermore, in this embodiment, obtaining the device to be operated by recognizing image information includes:

[0113] Perform 3D model recognition based on image information to obtain the device to be operated.

[0114] Optionally, after determining the device to be operated, the AR glasses first determine the user's current location based on Bluetooth positioning. Then, based on the 3D modeling map in the factory (the map contains device information and records the device code of each model, etc.), the specific location of the device to be operated is determined. Finally, several routes are planned for users to choose based on the path planning algorithm, and the map and path are displayed in front of the user to guide the user to the location of the device to be operated.

[0115] In some other executable embodiments, the device to be operated may also be obtained by an interface recognition method, where the image information of the interface of the operating system is captured by an AR device, and the device to be operated is obtained by recognizing the image information of the interface.

[0116] Specifically, the AR device can determine whether the device that the user is going to operate next is correct through 3D model recognition or interface recognition. The specific steps are: (1). The user points the AR glasses camera at the device to be operated. (2). The AR device determines whether the device to be operated is correct through 3D model recognition or interface recognition. (3). If it is correct, the user operates the device. (4). If it is incorrect, the user is reminded to search for the device again.

[0117] Furthermore, the AR device may be AR glasses.

[0118] In some executable embodiments, the operation work order information is divided into an unexecuted list and an executed list.

[0119] It can be understood that dividing the operation work order information into an unexecuted list and an executed list can reduce the occurrence of errors and omissions.

[0120] Receive the execution status instruction issued by the user; adjust the operation work order information according to the execution status instruction, including:

[0121] If the execution status instruction is completed, the status of the device to be operated is adjusted to completed execution; the device to be operated is moved to the executed list.

[0122] Optionally, when receiving an execution status instruction of "execution completed", the AR device will verify the legitimacy of the instruction, then update the status of the device to be operated to "execution completed", remove it from the "unexecuted list", and add it to the "executed list". During this process, the AR device will record the operation log and synchronously update all related databases to ensure the accuracy and work efficiency of work order management.

[0123] Furthermore, after the user completes the operation, he can judge whether his operation is successful. If the operation is successful, he can choose to proceed to the next step. If he thinks there is a problem with his operation, he can choose to re-execute and return to the current step to re-execute.

[0124] Optionally, after the user clicks "Complete" or "Re-execute", the AR glasses update the internal state according to the clicked item. If the user clicks "Complete", the AR glasses consider that the current step has been completed, and put the current execution device into the completed execution device, and move all the remaining execution devices forward one step.

[0125] In a feasible embodiment, after AR recognizes the equipment nameplate, the equipment to be operated is identified, and then AR communicates with the DCS system to obtain the equipment operation information parameters and the current actual operation parameters. Based on the actual operation parameters of the current equipment, the equipment operation risk is given, which can reduce the work risks caused by improper equipment isolation or insufficient actual maintenance conditions.

[0126] An AR device includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps of the AR auxiliary method for preventing human error in operation as described above are implemented.

[0127] The above embodiments only express the preferred implementation modes of the present invention, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the patent scope of the present invention. It should be pointed out that, for ordinary technicians in this field, the above technical features can be freely combined without departing from the concept of the present invention, and several deformations and improvements can be made, which all belong to the protection scope of the present invention. Therefore, all equivalent changes and modifications made to the scope of the claims of the present invention should belong to the coverage of the claims of the present invention.

Claims

1. An AR auxiliary method for preventing human error, characterized in that: The following steps are involved: Use AR devices to obtain operation work order information; Using the AR device to obtain the current location in real time; Determine the target device to be operated; Generate a guidance map based on the current position and the position information of the target device to be operated combined with a three-dimensional map of the factory area; Locating the target device to be operated according to the guidance map; The AR device scans the device to be operated, and determines whether the device to be operated is the target device to be operated; Providing reminders based on the judgment results; Receive the execution status instruction issued by the user; update the operation work order information according to the execution status instruction until all the target devices to be operated are confirmed to have been executed.

2. The auxiliary method according to claim 1, characterized in that: The method of obtaining the operation work order information by using an AR device includes: The AR device receives a work order title; The AR device determines whether the operation work order information exists in a work order database according to the work order title; If the operation work order information exists, obtain the operation work order information.

3. The auxiliary method according to claim 1, characterized in that: The device to be operated includes: image information of the device to be operated displayed on the DCS system interface or a physical device of the device to be operated.

4. The auxiliary method according to claim 3, characterized in that: The location information of the target device to be operated includes: location information of the DCS system that displays the device to be operated.

5. The auxiliary method according to claim 4, characterized in that: The generating of the guidance map according to the current position information and the position information of the target device to be operated in combination with the three-dimensional map of the factory area includes: Acquire information of the target device to be operated from the operation work order information; According to the information of the target device to be operated, acquiring the location information of the DCS system where the target device to be operated is located; Locating the device position of the target device to be operated on the three-dimensional map of the factory area according to the location information of the DCS system; Planning a traversable route between the current location and the device location according to a path planning algorithm; Receive a route confirmation instruction and determine the route; The guidance map is generated according to the travel route.

6. The auxiliary method according to claim 5, characterized in that: The acquiring the target device information to be operated from the operation work order information includes: Extracting device information of all the target devices to be operated in the operation work order information according to preset rules; The target devices to be operated are sorted according to the text structure and / or the order in which the target devices to be operated appear in the operation work order information, and the device information of the target device to be operated that ranks first is used as the target device to be operated information.

7. The auxiliary method according to claim 5, characterized in that: The AR device scans the device to be operated, and determines whether the device to be operated is the target device to be operated, including: The AR device receives a scanning command; The AR device captures image information displayed on the interface of the DCS system according to the scanning command; Acquiring information of equipment to be operated in the DCS system by identifying the image information; Determine whether the current position is consistent with the device position, and if not, alarm; If they are consistent, it is determined whether the information of the device to be operated is consistent with the information of the target device to be operated. If they are consistent, it is considered that the device to be operated is the target device to be operated.

8. The auxiliary method according to claim 1, characterized in that: The using the AR device to obtain the current position in real time includes: The AR device uses wireless positioning technology to determine the user's location and obtain the current location.

9. The auxiliary method according to claim 1, characterized in that: Divide the operation work order information into an unexecuted list and an executed list; The receiving user sends an execution status instruction; Adjusting the operation work order information according to the execution status instruction includes: If the execution status instruction is completed, the status of the device to be operated is adjusted to completed execution; Move the device to be operated to the executed list.

10. An AR device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the program, the steps of the AR anti-human operation auxiliary method as described in any one of claims 1-9 are implemented.

Images