Security facility auditing system, method and device and computer readable storage medium

Through real-time communication between reality-enhanced equipment and terminal equipment, the real-time interaction between experts and inspectors is achieved using motion capture and digital twin technology, solving the problems of long audit time and poor real-time performance of security facilities, and improving the efficiency of inspection and improvement processes and security.

CN119027067APending Publication Date: 2024-11-26YINGJI TECH SERVICE (BEIJING) CO LTD

Patent Information

Application Number
CN202411506803.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-10-28
Publication Date
2024-11-26

AI Technical Summary

Technical Problem

There are problems of long time and poor real-time performance during the audit of existing security facilities, resulting in inefficient inspection and improvement processes.

Method used

Through real-time communication between reality-enhanced devices and terminal devices, the motion capture device is used to capture expert gestures and movements, and the integrated images are generated through digital twin algorithms to realize instant interaction and real-time guidance between experts and inspectors, improving communication efficiency and problem-solving speed.

Benefits of technology

The cycle from problem detection to problem solving is greatly shortened, and the accuracy and safety of operations are improved, especially in complex or emergency situations, which significantly improve response efficiency and decision-making quality.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119027067A_ABST
    Figure CN119027067A_ABST
Patent Text Reader

Abstract

The invention discloses a security facility auditing system, method and device and a computer readable storage medium. Relates to the field of security inspection. The method comprises the following steps: shooting a field image of a security and protection facility in a target field through reality augmentation equipment in a communication process with terminal equipment, and uploading the field image to the terminal equipment in real time; capturing a gesture action made by an expert according to the field image after the field image is audited through a motion capture device, and sending a gesture graph corresponding to the gesture action to a terminal device; a digital twinborn image corresponding to the field image is obtained through the terminal device by using a digital twinborn algorithm; the gesture graph and the digital twin image are fused through the terminal device, and a fused image is generated; and sending the fused image to the reality augmentation equipment for real-time display through the terminal equipment. The technical problems of relatively long time and poor real-time performance of auditing on-site images are solved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of security inspection, and in particular to a security facility audit system, method, device and computer-readable storage medium. Background Art

[0002] During the inspection process, when it is necessary to conduct a safety assessment of the target site, the inspector uses a drone to take a live image of the target site. The drone then sends the live image to the server, and the server forwards the live image to the expert's terminal device. When the expert is idle, the safety of the target site is assessed based on the live image and suggestions for improvement are put forward. The expert then sends the assessment result to the server through the terminal device, and the server sends the assessment result to the inspector's terminal device. The inspector then improves the target site based on the improvement suggestions from the assessment results, thereby improving the safety of the target site. However, since the above-mentioned safety assessment and inspection process is not real-time, it may take a long time from the expert receiving the target site image to the improvement suggestions, and from the inspector receiving the improvement suggestions to the completion of the improvement of the target site. This will lead to problems such as poor real-time performance and long audit duration.

[0003] The publication number is CN117078174A, and the name is digital safety supervision system, medium and device based on AI and big data, including a login module and a single project system. The input end of the login module is connected to the PC and the mobile end; the single project system includes a safety facility management module, a video monitoring module, a construction progress management module, an attendance management module, a construction problem feedback module, and a project general control module; the safety facility management module, the video monitoring module, the construction progress management module, and the attendance management module are all communicated with the face recognition module; the safety facility management module is used to manage the safety facilities of the construction site; the video monitoring module is used to monitor the video of the construction site, and the video monitoring module is communicated with the display screen; the construction progress management module is used to prepare, implement and monitor the construction progress plan.

[0004] The publication number is CN116667533A, and the name is a smart dog inspection system, which is used for joint inspection of substations, including remote inspection business function module, image intelligent analysis function module, smart dog management function module and inspection management function module. The image intelligent analysis function module includes high-quality image production, data set components, target recognition detection and judgment, defect recognition, image discrimination, needle recognition, sample library management and defect alarm.

[0005] With respect to the technical problems in the prior art mentioned above, such as long time for reviewing on-site images and poor real-time performance, no effective solution has been proposed so far. Summary of the invention

[0006] The embodiments of the present application provide a security facility audit system, method, device and computer-readable storage medium to at least solve the technical problems existing in the prior art of long time and poor real-time performance for auditing on-site images.

[0007] According to one aspect of an embodiment of the present application, a security facility audit system is provided, comprising: an augmented reality device, a motion capture device, and a terminal device, wherein the augmented reality device is configured to perform the following operations: capture an on-site image of the security facilities in the target site during communication with the terminal device, and upload the on-site image to the terminal device in real time; and display in real time a fused image corresponding to the on-site image sent by the terminal device, and the motion capture device is configured to capture gestures made by an expert based on the on-site image after reviewing the on-site image, and send a gesture graphic corresponding to the gesture to the terminal device, and the terminal device is configured to perform the following operations: generate a digital twin image corresponding to the on-site image using a digital twin algorithm; fuse the gesture graphic with the digital twin image to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; and send the fused image to the augmented reality device.

[0008] According to another aspect of an embodiment of the present application, a security facility audit method is also provided, including: using an augmented reality device to capture an on-site image of the security facilities in the target site during communication with a terminal device, and uploading the on-site image to the terminal device in real time; using a motion capture device to capture gestures made by an expert based on the on-site image after reviewing the on-site image, and sending a gesture graphic corresponding to the gesture to the terminal device; using a digital twin algorithm to generate a digital twin image corresponding to the on-site image through the terminal device; using the terminal device to fuse the gesture graphic with the digital twin image to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; sending the fused image to the augmented reality device through the terminal device; and displaying the fused image corresponding to the on-site image sent by the terminal device in real time through the augmented reality device.

[0009] According to another aspect of an embodiment of the present application, a security facility auditing device is also provided, including: an image shooting module, used to shoot an on-site image of the security facilities in the target site through an augmented reality device during communication with a terminal device, and upload the on-site image to the terminal device in real time; a motion capture module, used to capture the gesture actions made by an expert based on the on-site image after reviewing the on-site image through a motion capture device, and send the gesture graphics corresponding to the gesture actions to the terminal device; an image generation module, used to generate a digital twin image corresponding to the on-site image through a digital twin algorithm through a terminal device; an image fusion module, used to fuse the gesture graphics with the digital twin image through a terminal device to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphics as the foreground; an image sending module, used to send the fused image to the augmented reality device through the terminal device; and an image display module, used to display the fused image corresponding to the on-site image sent by the terminal device in real time through the augmented reality device.

[0010] According to another aspect of an embodiment of the present application, a security facility auditing device is also provided, comprising: a processor; and a memory, connected to the processor, for providing the processor with instructions for processing the following processing steps: capturing a live image of the security facilities in the target site through an augmented reality device during communication with a terminal device, and uploading the live image to the terminal device in real time; capturing gestures made by an expert based on the live image after reviewing the live image through a motion capture device, and sending a gesture graphic corresponding to the gesture to the terminal device; generating a digital twin image corresponding to the live image through a digital twin algorithm through the terminal device; fusing the gesture graphic with the digital twin image through the terminal device to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; sending the fused image to the augmented reality device through the terminal device; and displaying the fused image corresponding to the live image sent by the terminal device in real time through the augmented reality device.

[0011] According to another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above method are implemented.

[0012] In the embodiment of the present application, the inspector communicates with the expert in real time at the target site through the augmented reality device, and transmits the on-site image of the target site to the expert's terminal device in real time. The expert evaluates and reviews the on-site image in real time through the terminal device, and feeds back the evaluation results to the inspector. Thus, the technical solution realizes the instant interaction between the expert and the on-site inspector through the real-time communication between the augmented reality device and the terminal device, greatly improving the communication efficiency and the speed of problem solving. And the real-time guidance and feedback mechanism reduces the unnecessary waiting time in the traditional process, making the inspection and improvement process more efficient, and shortening the cycle from problem discovery to problem solving. And through AR technology, the inspectors at the target site can obtain more flexible and intuitive operation guidance, especially when dealing with complex or emergency situations. This real-time expert guidance can significantly improve the response efficiency and decision-making quality. In addition, with the help of the fusion of digital twin technology and gesture graphics, the expert can intuitively guide the on-site operation in the virtual scene. This precise visual guidance helps to avoid misunderstandings and improve the accuracy and safety of operations. And then solves the technical problems of long time and poor real-time performance of the on-site image review in the prior art. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 is a schematic diagram of the security facility audit system according to Example 1 of the present application; Figure 2 is a schematic diagram of an augmented reality device according to Embodiment 1 of the present application; Figure 3 is a module schematic diagram of a terminal device according to Embodiment 1 of the present application; Figure 4A is a schematic diagram of a scene image according to Example 1 of the present application; Figure 4B is a schematic diagram of a motion capture device according to Example 1 of the present application; Figure 4C is a schematic diagram of a fused image according to Example 1 of the present application; Figure 5A is another schematic diagram of the motion capture device according to Example 1 of the present application; Figure 5B is another schematic diagram of the fused image according to Example 1 of the present application; Figure 5C is a schematic diagram of superimposed images according to Example 1 of the present application; Figure 5Dis another schematic diagram of the superimposed image according to Embodiment 1 of the present application; Figure 6 is another module schematic diagram of the terminal device according to Embodiment 1 of the present application; Figure 7 is a schematic diagram of the knowledge graph according to Example 1 of the present application; Figure 8 is another module schematic diagram of the augmented reality device according to Embodiment 1 of the present application; Fig.9A is a schematic diagram of comparing the fused image slice and the operated image slice according to Example 1 of the present application; Fig. 9B is another schematic diagram of comparing the fused image slice and the operated image slice according to Example 1 of the present application; Fig. 9C is another schematic diagram of comparing the fused image slice and the operated image slice according to Example 1 of the present application; Fig.9D is another schematic diagram of comparing the fused image slice and the operated image slice according to Example 1 of the present application; Fig.10 It is a sequential flow chart of the security facility audit method according to Example 1 of the present application; Fig.11 It is a flowchart of the security facility audit method according to Example 1 of the present application; Fig.12 is a schematic diagram of a security facility audit device according to Embodiment 2 of the present application; and Fig.13 It is a schematic diagram of the security facility audit device described in Example 3 of the present application. DETAILED DESCRIPTION

[0014] In order to enable those skilled in the art to better understand the technical solutions of the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are only embodiments of a part of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work should fall within the scope of protection of the present application.

[0015] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0016] Example 1 Figure 1 Schematic diagram of the security facility audit system according to this embodiment. Figure 1 As shown, the system includes: an augmented reality device 100, a terminal device 200 and a motion capture device 300, wherein the augmented reality device is configured to perform the following operations: capture on-site images of security facilities in the target site during communication with the terminal device, and upload the on-site images to the terminal device in real time; and display in real time a fused image corresponding to the on-site image sent by the terminal device, wherein the fused image is used to indicate an image with a digital twin image as the background and a gesture graphic as the foreground, and the motion capture device is configured to capture gesture actions made by an expert based on the on-site image after reviewing the on-site image, and send the gesture graphics corresponding to the gesture actions to the terminal device, and the terminal device is configured to perform the following operations: generate a digital twin image corresponding to the on-site image using a digital twin algorithm; fuse the gesture graphic with the digital twin image to generate a fused image; and send the fused image to the augmented reality device.

[0017] Specifically, refer to Figure 1 as well as Figure 2 As shown, the augmented reality device 100 (for example, AR glasses) is arranged at the inspection side, and the terminal device 200 and the motion capture device 300 are arranged at the expert side. And the augmented reality device 100 includes a camera 110, a communication module 120 and a display screen 130.

[0018] During the inspection of the target site by the inspector, the inspector wears the augmented reality device 100 to communicate in real time with the expert's terminal device 200. The augmented reality device 100 captures the on-site image of the security facilities in the target site through the camera 110. The on-site image may be a video image of the target site. The augmented reality device 100 then transmits the on-site image to the expert's terminal device 200 in real time through the communication module 120.

[0019] Further, refer to Figure 3 As shown, the terminal device 200 includes an image transmission module 210, an image processing module 220, and an image fusion module 230. The terminal device 200 receives the on-site image transmitted by the augmented reality device 100 through the image transmission module 210, and then processes the on-site image using a preset digital twin algorithm through the image processing module 220 to generate a digital twin image corresponding to the on-site image. The terminal device 200 then displays the digital twin image.

[0020] For example, refer to Figure 4A As shown, the on-site image is a security facility including multiple interfaces, and the inspection personnel need expert guidance to insert multiple data cables into the corresponding interfaces. Thus, the terminal device 200 receives the on-site image transmitted by the augmented reality device 100 through the image transmission module 210, and then the terminal device 200 processes the on-site image using the preset digital twin algorithm through the image processing module 220 to generate a digital twin image corresponding to the on-site image. The digital twin image is Figure 4A Same as the live image in .

[0021] Furthermore, the expert reviews and evaluates the digital twin image displayed by the terminal device 200, and then makes a gesture corresponding to the digital twin image. The motion capture device 300 captures the expert's gesture in real time and generates a gesture graphic corresponding to the gesture. The motion capture device 300 then transmits the gesture graphic to the terminal device 200 in real time.

[0022] For example, refer to Figure 4B As shown, the expert holds the data cable, and the motion capture device 300 captures the expert's hand gesture of holding the data cable in real time, and generates a gesture graphic corresponding to the gesture. Then the motion capture device 300 transmits the gesture graphic to the terminal device 200 in real time.

[0023] Furthermore, the terminal device 200 receives the gesture graphic through the image transmission module 210, and fuses the gesture graphic with the digital twin image through the image fusion module 230 to generate a fused image. The terminal device 200 sends the fused image to the augmented reality device 100 in real time through the image transmission module 210. For example, referring to Figure 4C As shown, the terminal device 200 uses the gesture graphic of the expert holding the data cable as the foreground through the image fusion module 230, and uses the digital twin image of the security facility with multiple interfaces as the background, and then fuses the gesture graphic with the digital twin image to generate the following image: Figure 4C The fused image shown. The fused image is an image with the digital twin image as the background and the gesture graphic as the foreground. And the fused image is a video image of the entire process of an expert holding a data cable and inserting the data cable into a corresponding interface.

[0024] Furthermore, the augmented reality device 100 receives in real time through the communication module 120 the fused image of the entire process of the expert holding the data cable and inserting the data cable into the corresponding interface, and displays the fused image through the display screen 130. Thus, the inspection personnel insert the data cable of the target site into the corresponding interface of the security facility according to the fused image. Thus, the inspection personnel complete the inspection of the target site under the real-time guidance of the expert.

[0025] For example, refer to Figure 4A As shown, the inspection personnel need expert guidance to unplug the data cable from the interface. The expert makes gestures corresponding to the digital twin image displayed by the terminal device 200. The motion capture device 300 captures the expert's gestures in real time and generates gesture graphics corresponding to the gestures. The motion capture device 300 then transmits the gesture graphics to the terminal device 200 in real time.

[0026] refer to Figure 5A As shown, the expert makes a gesture of holding the data cable to be unplugged, and the motion capture device 300 captures the expert's gesture of holding the data cable to be unplugged in real time, and generates a gesture graphic corresponding to the gesture. The motion capture device 300 then transmits the gesture graphic to the terminal device 200 in real time.

[0027] Furthermore, the terminal device 200 receives the gesture graphic through the image transmission module 210, and fuses the gesture graphic with the digital twin image through the image fusion module 230 to generate a fused image. The terminal device 200 sends the fused image to the augmented reality device 100 in real time through the image transmission module 210. For example, referring to Figure 5B As shown, the terminal device 200 uses the gesture graphic of the expert holding the data cable to be unplugged as the foreground through the image fusion module 230, and uses the digital twin image of the security facility with multiple interfaces as the background, and then fuses the gesture graphic with the digital twin image to generate the following image: Figure 5B The fused image shown. The fused image is an image with the digital twin image as the background and the gesture graphic as the foreground. And the fused image is a video image of the entire process of an expert holding a data cable to be unplugged and unplugging the data cable from the corresponding interface.

[0028] Furthermore, the augmented reality device 100 receives in real time through the communication module 120 a fused image of the entire process of the expert holding the data cable and pulling the data cable out of the corresponding interface, and displays the fused image through the display screen 130. Thus, the inspection personnel pull out the data cable of the target site from the interface corresponding to the security facility according to the fused image. Thus, the inspection personnel complete the inspection of the target site under the real-time guidance of the expert.

[0029] Optionally, the terminal device is further configured to display the fused image.

[0030] Specifically, after the terminal device 200 generates a fused image through the image fusion module 230, the fused image is displayed in real time. Therefore, the expert can adjust the gesture action according to the fused image displayed by the terminal device 200, so as to generate a new fused image in real time.

[0031] Optionally, the terminal device is also configured to perform the following operations: superimpose the gesture graphic with the digital twin image to generate a superimposed image, wherein the superimposed image is used to indicate an image whose transparency of the gesture graphic is greater than that of the digital twin image; and send the superimposed image to the augmented reality device 100.

[0032] Specifically, after the terminal device 200 generates a digital twin image corresponding to the on-site image through the image processing module 220, and the terminal device 200 receives the gesture graphic sent by the motion capture device 300 through the image transmission module 210. The expert chooses to fuse or superimpose the image through the terminal device 200. When the expert selects the superimposition operation through the terminal device 200, the terminal device 200 increases the transparency of the gesture graphic through the image fusion module 230, for example, the transparency is increased to 50%. Therefore, the transparency of the digital twin image is less than the transparency of the gesture graphic. The transparency of the digital twin image is 0. Therefore, the terminal device 200 superimposes the gesture graphic with increased transparency with the digital twin image through the fusion module 230 to generate a superimposed image.

[0033] For example, Figure 5C and Figure 5D A schematic diagram showing superimposed images is shown in FIG. Figure 5C and Figure 5D As shown, the transparency of the expert's gesture graphic is higher than the transparency of the digital twin image of the security facility, so that there is an afterimage of the gesture graphic in the overlapping part of the gesture graphic and the digital twin image.

[0034] In addition, when the expert selects a fusion operation through the terminal device 200 , the terminal device 200 performs a fusion operation on the gesture image and the digital twin image through the image fusion module 230 to generate a fused image.

[0035] As described in the background technology, during the inspection process, when it is necessary to conduct a safety assessment on the target site, the inspector uses a drone to take a live image of the target site. The drone then sends the live image to the server, and the server forwards the live image to the expert's terminal device. When the expert is idle, the safety of the target site is assessed based on the live image and suggestions for improvement are put forward. The expert then sends the assessment result to the server through the terminal device, and the server then sends the assessment result to the inspector's terminal device. The inspector then improves the target site based on the improvement suggestions of the assessment result, thereby improving the safety of the target site. However, since the above-mentioned safety assessment and inspection process is not real-time, it may take a long time from the expert receiving the target site image to the improvement suggestions, and from the inspector receiving the improvement suggestions to the completion of the improvement of the target site. This will lead to problems such as poor real-time performance and long audit duration.

[0036] In view of the technical problems described above, through the technical solution of the embodiment of the present application, the inspectors communicate with the experts in real time at the target site through the augmented reality device, and transmit the on-site image of the target site to the terminal device of the expert in real time. The expert evaluates and reviews the on-site image in real time through the terminal device, and feeds back the evaluation results to the inspectors. Thus, the technical solution realizes the instant interaction between the experts and the on-site inspectors through the real-time communication between the augmented reality device and the terminal device, greatly improving the communication efficiency and the speed of problem solving. And the real-time guidance and feedback mechanism reduces the unnecessary waiting time in the traditional process, making the inspection and improvement process more efficient, and shortening the cycle from problem discovery to problem solving. And through AR technology, the inspectors at the target site can obtain more flexible and intuitive operation guidance, especially when dealing with complex or emergency situations. This real-time expert guidance can significantly improve the response efficiency and decision-making quality. In addition, with the help of the fusion of digital twin technology and gesture graphics, the experts can intuitively guide the on-site operation in the virtual scene. This precise visual guidance helps to avoid misunderstandings and improve the accuracy and safety of operations. And then solve the technical problems of long time and poor real-time performance of the on-site image review in the prior art.

[0037] Optionally, the terminal device is also configured to perform the following operations: process the on-site image using a defect detection model to determine the defect type of defects existing in the security facilities at the target site; determine a set of sub-actions for handling defects based on the defect type through a knowledge graph; and match the fused image with the sub-action set to determine whether the gestures made by the expert based on the on-site image are correct.

[0038] Specifically, refer to Figure 6As shown, the terminal device 200 also includes an image recognition module 240. After receiving the on-site image sent by the augmented reality device 100, the terminal device 200 performs defect detection on the on-site image through the defect detection model set by the image recognition module 240 to determine the defect type of the defect existing in the security facilities at the target site. Figure 4A For the security facility shown, the image recognition module 240 performs defect detection on the corresponding on-site image through the defect detection model, and determines that the defect type of the security facility is: the interface is idle. Therefore, the inspection personnel should install a data cable for the interface.

[0039] The image recognition module 240 of the terminal device 200 is pre-set with a knowledge graph, wherein the knowledge graph includes various facilities, defect types corresponding to the various facilities, and actions and sub-actions to overcome each defect. Figure 7 A portion of the knowledge graph is shown. Figure 7 As shown, for example, the defect types of facility A include: interface idle, interface conflict, and crack. The action to overcome the defect "interface idle" is "insert", that is, insert the data line; the action to overcome the defect "interface conflict" is "pull out", that is, pull out the data line; the action to overcome the defect "crack" is "paste", that is, glue the crack. And each action can be decomposed into corresponding sub-actions, for example, the action "insert" can be decomposed into: "push" and "pinch"; the action "pull out" can be decomposed into: "pull" and "pinch"; the action "paste" can be decomposed into: "clean", "smear" and "lay".

[0040] Furthermore, the image recognition module 240 of the terminal device 200 matches the identified defect type of the corresponding security facility through a preset knowledge graph, thereby determining the action corresponding to the defect type, and then generating a sub-action set for all sub-actions under the action.

[0041] For example, the image recognition module 240 of the terminal device 200 first determines the type of security facilities in the on-site image, and finds facilities of the same type as the security facilities in the knowledge graph, such as facility A. The defect type of the security facilities in the on-site image is: idle interface. Thus, the image recognition module 240 of the terminal device 200 matches the defect type of "idle interface" under facility A in the knowledge graph according to the defect type. Then, the image recognition module 240 determines the action under "idle interface", such as "insert", and generates a sub-action set for the sub-actions under the action, such as the sub-action set including: push and pinch.

[0042] Furthermore, the image recognition module 240 of the terminal device 200 matches the fused image of the expert's corresponding action to deal with the defect with the sub-action set to determine whether the gesture action made by the expert according to the on-site image in the fused image is correct. If it is incorrect, the terminal alarm module 250 of the terminal device 200 will issue an alarm.

[0043] Therefore, this technical solution can effectively verify the correctness of the operation by matching the expert's actual operation (gesture action) with a predefined set of sub-actions, reduce safety hazards caused by human errors, and realize real-time monitoring of expert operations. Once it is found that the operation does not meet the expected standard action, the terminal device can immediately issue an alarm to ensure timely correction of the error and ensure the safety and effectiveness of the operation.

[0044] Optionally, the operation of processing the on-site image using the defect detection model to determine the defect type of the defects existing in the security facilities at the target site includes: processing the on-site image through a 3D CNN model to generate image information; and classifying the image information through a softmax classifier to determine the defect type.

[0045] Specifically, the defect detection model includes a 3D CNN model and a softmax classifier.

[0046] Thus, the image recognition module 240 inputs the scene image into the 3D CNN model. The 3D CNN model includes a 3D convolution layer, a 3D pooling layer, and a fully connected layer. Thus, the 3D convolution layer of the 3D CNN model performs a convolution operation on the scene image to generate a corresponding first feature map. Then, the 3D pooling layer performs a pooling operation on the first feature map to generate a corresponding second feature map. Then, the fully connected layer processes the second feature map to generate a corresponding feature vector (i.e., image information).

[0047] Furthermore, the image recognition module 240 inputs the feature vector (ie, image information) output by the 3D CNN model into a softmax classifier, classifies the feature vector (ie, image information) through the softmax classifier, and outputs a corresponding defect type, such as an idle interface.

[0048] Therefore, this technical solution reduces false alarms caused by environmental interference or other non-critical factors and improves the stability and reliability of the system by utilizing fine feature extraction and accurate classification through the defect detection model.

[0049] Optionally, the fused image is matched with a set of sub-actions to determine whether the gesture action made by the expert based on the on-site image is a correct operation, including: slicing the fused image to generate multiple fused image slices; using an action recognition model to process the multiple fused image slices to determine the expert sub-actions corresponding to the multiple fused image slices; and matching the expert sub-actions with the set of sub-actions to determine whether the expert's gesture action is correct.

[0050] Specifically, the image recognition module 240 of the terminal device 200 performs slicing processing on the fused image, for example, slicing once every 10 frames, thereby generating a plurality of fused image slices.

[0051] The image recognition module 240 is also pre-set with an action recognition model, so that multiple fused image slices are input into the action recognition model, and each fused image slice is subjected to action recognition through the action recognition model, so as to determine the expert sub-actions in each fused image slice, for example, the expert sub-actions are pinching, pulling and pushing, etc.

[0052] More specifically, the action recognition model includes a 3D CNN model and a softmax classifier. It should be noted that the 3D CNN model and the softmax classifier in the action recognition model are pre-trained with action samples and are used to identify action models. The 3D CNN model and the softmax classifier of the defect detection model are pre-trained with sample defects and are used to detect defects.

[0053] And the 3D CNN model of the action recognition model includes a 3D convolution layer, a 3D pooling layer and a fully connected layer. Thus, the 3D convolution layer of the 3DCNN model performs a convolution operation on the fused image slice to generate a corresponding third feature map. Then the 3D pooling layer performs a pooling operation on the third feature map to generate a corresponding fourth feature map. Then the fully connected layer will process the fourth feature map to generate a corresponding feature vector.

[0054] Furthermore, the image recognition module 240 inputs the feature vector output by the 3D CNN model of the action recognition model into the softmax classifier of the action recognition model, classifies the feature vector through the softmax classifier, and outputs the corresponding expert sub-action. One fused image slice may include multiple expert sub-actions.

[0055] Furthermore, the image recognition module 240 matches the expert sub-actions of each fused image slice with the sub-actions in the sub-action set in sequence. When the expert sub-actions of a consecutive number of fused image slices do not match the sub-actions in the sub-action set, it means that the expert's gesture action is incorrect; otherwise, it means that the expert's gesture action is correct.

[0056] Thus, in this technical solution, by comparing the identified expert sub-action with the predefined sub-action set, the correctness of the operation can be effectively verified. When the expert sub-action in a certain number of consecutive fused image slices does not match the standard sub-action, a warning can be issued in time. Thus, this technical solution uses a real-time feedback mechanism to help correct erroneous operations immediately and avoid potential risks.

[0057] Optionally, the augmented reality device is further configured to perform the following operations: recording an operation image of the inspector processing the security facility based on the fused image; and matching the operation image with the expert sub-action of the fused image to determine whether the inspector's operation is correct.

[0058] Specifically, the terminal device 200 sends the fused image to the augmented reality device 100, such as AR glasses. The patrol personnel wearing the augmented reality device 100 can perform corresponding operations on the security facilities in the target site according to the actions of processing the security facilities in the fused image, so that the augmented reality device 100 records the operation image of the patrol personnel processing the security facilities according to the fused image through the camera 110. The operation image is a video image. Figure 8 As shown, the augmented reality device 100 then matches the operation image with the expert sub-action of the fused image through the preset AR module 140 to determine whether the operation of the inspection personnel is correct. In the case of incorrectness, the augmented reality device 100 issues an alarm through the AR alarm module 150.

[0059] Therefore, this technical solution can instantly evaluate the accuracy of the operation by recording the operation images of the inspectors in real time and comparing them with the expert sub-actions in the fused image, ensuring that each step complies with the standard process and reducing human errors. When an incorrect operation is found, the augmented reality device can quickly issue an alarm through the AR alarm module, allowing the inspectors to adjust the operation in time to avoid possible safety hazards or equipment damage.

[0060] Optionally, the operation image is matched with the expert sub-actions of the fused image to determine whether the operation of the inspector is correct, including: slicing the operation image to generate multiple operation image slices; using the action recognition model to process the multiple operation image slices to determine the inspection sub-actions corresponding to the multiple operation image slices; and matching the inspection sub-actions with the expert sub-actions to determine whether the operation of the inspector is correct.

[0061] Specifically, the AR module 140 performs a slicing process on the operation image, for example, slicing once every 10 frames during the recording process until the recording ends, thereby generating a plurality of operation image slices.

[0062] The AR module 140 is also pre-set with a motion recognition model, so that multiple operation image slices are input into the motion recognition model, and each operation image slice is subjected to motion recognition through the motion recognition model, so as to determine the inspection sub-actions of the inspection personnel in each operation image slice, such as the inspection sub-actions of pinching, pulling and pushing, etc.

[0063] More specifically, the action recognition model includes a 3D CNN model and a softmax classifier. It should be noted that the 3D CNN model and the softmax classifier in the action recognition model are pre-trained with action samples and are used to identify action models. The 3D CNN model and the softmax classifier of the defect detection model are pre-trained with sample defects and are used to detect defects.

[0064] And the 3D CNN model of the action recognition model includes a 3D convolution layer, a 3D pooling layer and a fully connected layer. Thus, the 3D convolution layer of the 3DCNN model performs a convolution operation on the operation image slice to generate a corresponding fifth feature map. Then the 3D pooling layer performs a pooling operation on the fifth feature map to generate a corresponding sixth feature map. Then the fully connected layer will process the sixth feature map to generate a corresponding feature vector.

[0065] Further, the AR module 140 inputs the feature vector output by the 3D CNN model of the action recognition model into the softmax classifier of the action recognition model, classifies the feature vector through the softmax classifier, and outputs the corresponding inspection sub-action. One operation image slice includes multiple inspection sub-actions.

[0066] Further, the AR module 140 pre-sets a time window with a predetermined width, and the step length is 1. Thus, the AR module 140 divides the multiple fused image slices according to the time window. Fig.9A As shown, for example, the width of the time window is one third of all fused image slices, so that in the first cycle, the data items in the time window are fused image slices 1 to n (ie, P1 to P n ).

[0067] During the recording process of the operation image, for example, the operation image is sliced ​​once every 10 frames, so that the AR module 140 matches the inspection sub-action in the generated operation image slice 1 (i.e., S1) with the expert sub-actions of each fused image slice in the time window, thereby determining whether the expert sub-action of the fused image slice in the time window contains the inspection sub-action of the operation image slice 1 (i.e., S1).

[0068] Further, refer to Fig. 9BAs shown, the time window moves backward according to the step size, so that the data items in the time window of the second cycle are fused image slices 2~n+1 (i.e., P2~P n+1 ), and during the recording of the operation image, the AR module 140 has generated the operation image slice 2 (i.e., S2), so after identifying the inspection sub-action of the operation image slice 2 (i.e., S2), the AR module 140 matches the inspection sub-action in the operation image slice 2 (i.e., S2) with the expert sub-actions of each fused image slice in the time window, respectively, so as to determine whether the expert sub-action of the fused image slice in the time window contains the inspection sub-action of the operation image slice 2 (i.e., S2). Similarly, the inspection sub-action in each operation image slice is matched with the expert sub-action of the fused image slice in the corresponding time window.

[0069] Furthermore, during the matching process, the AR module 140 counts the number of operating image slices that do not match the fused image slices in the corresponding time window. When the inspection sub-actions in a consecutive number of operating image slices do not match the expert sub-actions in the fused image slices in the corresponding time window, the augmented reality device 100 issues an alarm through the AR alarm module 150 to remind the inspection personnel of the operational error.

[0070] In addition, reference Fig. 9C and Fig.9D As shown, when the time window has moved to the last fused image slice, the time window will no longer move in the subsequent cycles, so that the inspection sub-actions of the operating image slices generated during this period are matched with the expert sub-actions of the fused image slices in the time window.

[0071] Therefore, this technical solution utilizes sliding window technology to enable real-time comparison of the actual actions of the inspectors with the expert sub-actions during the operation. Once several consecutive actions are found to be inconsistent, an alarm can be triggered immediately, thus achieving rapid response and real-time error correction functions.

[0072] In summary, reference Fig.10 As shown in the figure, the sequential process of the security facility audit system to audit the security facilities at the target site is as follows: S1001: the augmented reality device captures a scene image of the security facilities in the target scene during the communication process with the terminal device, and uploads the scene image to the terminal device in real time; S1002: The motion capture device captures the gestures made by the expert based on the on-site image after reviewing the on-site image, and sends the gesture graphics corresponding to the gestures to the terminal device; S1003: The terminal device generates a digital twin image corresponding to the on-site image using a digital twin algorithm; S1004: The terminal device fuses the gesture graphic with the digital twin image, generates a fused image, displays it, and sends the fused image to the augmented reality device; S1005: The terminal device processes the on-site image using the defect detection model to determine the defect type of the defect existing in the security facilities of the target site; S1006: The terminal device determines a sub-action set for handling the defect according to the defect type through the knowledge graph; S1007: The terminal device performs slicing processing on the fused image to generate multiple fused image slices; S1008: The terminal device processes the multiple fused image slices using the action recognition model to determine expert sub-actions corresponding to the multiple fused image slices; S1009: The terminal device matches the expert sub-action with the sub-action set to determine whether the expert's gesture action is correct. If it is incorrect, an alarm is issued; S1010: The augmented reality device displays in real time a fused image corresponding to the on-site image and sent by the terminal device; S1011: The augmented reality device records the operation image of the inspection personnel processing the security facilities based on the fused image; S1012: The augmented reality device slices the operation image to generate a plurality of operation image slices; S1013: The augmented reality device processes the multiple operation image slices using the action recognition model to determine the inspection sub-actions corresponding to the multiple operation image slices; S1014: The augmented reality device matches the inspection sub-action with the expert sub-action to determine whether the inspection personnel's operation is correct. If it is incorrect, the augmented reality device will issue an alarm.

[0073] In addition, according to a second aspect of this embodiment, a security facility audit method is provided, the method comprising: Figure 1 The security facility audit system shown in is implemented. Fig.11 A schematic diagram showing the process of the method is shown in FIG. Fig.11 As shown, the method includes: S1102: photographing a live image of security facilities at a target site through the augmented reality device during communication with the terminal device, and uploading the live image to the terminal device in real time; S1104: Capturing the hand gestures made by the expert based on the on-site image after reviewing the on-site image through the motion capture device, and sending the hand gesture graphics corresponding to the hand gestures to the terminal device; S1106: Generate a digital twin image corresponding to the on-site image by using a digital twin algorithm through a terminal device; S1108: Fusing the gesture graphic with the digital twin image through the terminal device to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; S1110: Sending the fused image to the augmented reality device through the terminal device; and S1112: Displaying the fused image corresponding to the on-site image sent by the terminal device in real time through the augmented reality device.

[0074] Thus, according to the second aspect of this embodiment, the inspectors communicate with the experts in real time at the target site through the augmented reality device, and transmit the on-site image of the target site to the terminal device of the expert in real time. The expert evaluates and reviews the on-site image in real time through the terminal device, and feeds back the evaluation results to the inspectors. Thus, the technical solution realizes the instant interaction between the experts and the on-site inspectors through the real-time communication between the augmented reality device and the terminal device, greatly improving the communication efficiency and the speed of problem solving. And the real-time guidance and feedback mechanism reduces the unnecessary waiting time in the traditional process, making the inspection and improvement process more efficient, and shortening the cycle from problem discovery to problem solving. And through AR technology, the inspectors at the target site can obtain more flexible and intuitive operation guidance, especially when dealing with complex or emergency situations. This real-time expert guidance can significantly improve the response efficiency and decision-making quality. In addition, with the help of the fusion of digital twin technology and gesture graphics, the experts can intuitively guide the on-site operation in the virtual scene. This precise visual guidance helps to avoid misunderstandings and improve the accuracy and safety of operations. And then solve the technical problems of long time and poor real-time performance of the on-site image review in the prior art.

[0075] In addition, according to a third aspect of this embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are implemented.

[0076] Thus, according to this embodiment, the inspectors communicate with the experts in real time at the target site through the augmented reality device, and transmit the on-site image of the target site to the terminal device of the expert in real time. The expert evaluates and reviews the on-site image in real time through the terminal device, and feeds back the evaluation results to the inspectors. Thus, the technical solution realizes the instant interaction between the experts and the on-site inspectors through the real-time communication between the augmented reality device and the terminal device, greatly improving the communication efficiency and the speed of problem solving. And the real-time guidance and feedback mechanism reduces the unnecessary waiting time in the traditional process, making the inspection and improvement process more efficient, and shortening the cycle from problem discovery to problem solving. And through AR technology, the inspectors at the target site can obtain more flexible and intuitive operation guidance, especially when dealing with complex or emergency situations. This real-time expert guidance can significantly improve the response efficiency and decision-making quality. In addition, with the help of the fusion of digital twin technology and gesture graphics, the experts can intuitively guide the on-site operation in the virtual scene. This precise visual guidance helps to avoid misunderstandings and improve the accuracy and safety of operations. And then solve the technical problems of long time and poor real-time performance of the on-site image review in the prior art.

[0077] It should be noted that, for the above-mentioned method embodiments, for the sake of simplicity, they are all described as a series of action combinations, but those skilled in the art should know that the present invention is not limited by the described action sequence, because according to the present invention, certain steps can be performed in other sequences or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by the present invention.

[0078] Through the description of the above implementation methods, those skilled in the art can clearly understand that the method according to the above embodiment can be implemented by means of software plus a necessary general hardware platform, and of course can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, or the part that contributes to the prior art, can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk), and includes a number of instructions for enabling a terminal device (which can be a mobile phone, computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0079] Example 2 Fig.12 FIG. 1 shows a security facility audit device 1200 according to this embodiment, which corresponds to the method described in the first aspect of Embodiment 1. Fig.12As shown, the device 1200 includes: an image shooting module 1210, which is used to shoot a live image of the security facilities in the target site through the augmented reality device during the communication with the terminal device, and upload the live image to the terminal device in real time; a motion capture module 1220, which is used to capture the gestures made by the expert based on the live image after reviewing the live image through the motion capture device, and send the gesture graphics corresponding to the gestures to the terminal device; an image generation module 1230, which is used to generate a digital twin image corresponding to the live image through the terminal device using a digital twin algorithm; an image fusion module 1240, which is used to fuse the gesture graphics with the digital twin image through the terminal device to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphics as the foreground; an image sending module 1250, which is used to send the fused image to the augmented reality device through the terminal device; and an image display module 1260, which is used to display the fused image corresponding to the live image sent by the terminal device in real time through the augmented reality device.

[0080] Thus, according to this embodiment, the inspectors communicate with the experts in real time at the target site through the augmented reality device, and transmit the on-site image of the target site to the terminal device of the expert in real time. The expert evaluates and reviews the on-site image in real time through the terminal device, and feeds back the evaluation results to the inspectors. Thus, the technical solution realizes the instant interaction between the experts and the on-site inspectors through the real-time communication between the augmented reality device and the terminal device, greatly improving the communication efficiency and the speed of problem solving. And the real-time guidance and feedback mechanism reduces the unnecessary waiting time in the traditional process, making the inspection and improvement process more efficient, and shortening the cycle from problem discovery to problem solving. And through AR technology, the inspectors at the target site can obtain more flexible and intuitive operation guidance, especially when dealing with complex or emergency situations. This real-time expert guidance can significantly improve the response efficiency and decision-making quality. In addition, with the help of the fusion of digital twin technology and gesture graphics, the experts can intuitively guide the on-site operation in the virtual scene. This precise visual guidance helps to avoid misunderstandings and improve the accuracy and safety of operations. And then solve the technical problems of long time and poor real-time performance of the on-site image review in the prior art.

[0081] Example 3 Fig.13 The security facility audit device 1300 according to this embodiment is shown, and the device 1300 corresponds to the method according to the first aspect of embodiment 1. Fig.13As shown, the device 1300 includes: a processor 1310; and a memory 1320, which is connected to the processor 1310 and is used to provide the processor 1310 with instructions for processing the following processing steps: capturing a live image of the security facilities in the target site through an augmented reality device during communication with a terminal device, and uploading the live image to the terminal device in real time; capturing gestures made by an expert based on the live image after reviewing the live image through a motion capture device, and sending a gesture graphic corresponding to the gesture to the terminal device; generating a digital twin image corresponding to the live image using a digital twin algorithm through the terminal device; fusing the gesture graphic with the digital twin image through the terminal device to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; sending the fused image to the augmented reality device through the terminal device; and displaying the fused image corresponding to the live image sent by the terminal device in real time through the augmented reality device.

[0082] Thus, according to this embodiment, the inspectors communicate with the experts in real time at the target site through the augmented reality device, and transmit the on-site image of the target site to the terminal device of the expert in real time. The expert evaluates and reviews the on-site image in real time through the terminal device, and feeds back the evaluation results to the inspectors. Thus, the technical solution realizes the instant interaction between the experts and the on-site inspectors through the real-time communication between the augmented reality device and the terminal device, greatly improving the communication efficiency and the speed of problem solving. And the real-time guidance and feedback mechanism reduces the unnecessary waiting time in the traditional process, making the inspection and improvement process more efficient, and shortening the cycle from problem discovery to problem solving. And through AR technology, the inspectors at the target site can obtain more flexible and intuitive operation guidance, especially when dealing with complex or emergency situations. This real-time expert guidance can significantly improve the response efficiency and decision-making quality. In addition, with the help of the fusion of digital twin technology and gesture graphics, the experts can intuitively guide the on-site operation in the virtual scene. This precise visual guidance helps to avoid misunderstandings and improve the accuracy and safety of operations. And then solve the technical problems of long time and poor real-time performance of the on-site image review in the prior art.

[0083] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0084] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0085] In the several embodiments provided in this application, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0086] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0087] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0088] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0089] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A security facility audit system, characterized in that: include: Augmented reality equipment, motion capture equipment and terminal equipment, including The augmented reality device is configured to perform the following operations: During the communication process with the terminal device, taking a scene image of the security facilities in the target scene, and uploading the scene image to the terminal device in real time; as well as displaying in real time the fused image corresponding to the on-site image sent by the terminal device, and The motion capture device is configured to capture the gestures made by the expert based on the live image after reviewing the live image, and send the gesture graphics corresponding to the gestures to the terminal device, and The terminal device is configured to perform the following operations: Generate a digital twin image corresponding to the on-site image using a digital twin algorithm; Fusing the gesture graphic with the digital twin image to generate the fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; as well as The fused image is sent to the augmented reality device.

2. The system according to claim 1, characterized in that The terminal device is also configured to perform the following operations: Processing the scene image using a defect detection model to determine the defect type of the defect existing in the security facilities of the target scene; Determine a sub-action set for handling the defect according to the defect type through a knowledge graph; as well as The fused image is matched with the sub-action set to determine whether the gesture action made by the expert according to the on-site image is correct.

3. The system according to claim 2, characterized in that The operation of processing the scene image by using the defect detection model to determine the defect type of the defect existing in the security facilities of the target scene includes: Processing the scene image through a 3D CNN model to generate image information; and The image information is classified by a softmax classifier to determine the defect type.

4. The system according to claim 3, characterized in that Matching the fused image with the sub-action set to determine whether the gesture action made by the expert according to the on-site image is a correct operation includes: Slicing the fused image to generate a plurality of fused image slices; Processing the plurality of fused image slices using an action recognition model to determine expert sub-actions corresponding to the plurality of fused image slices; and The expert sub-action is matched with the sub-action set to determine whether the expert's gesture action is correct.

5. The system according to claim 4, characterized in that The augmented reality device is also configured to perform the following operations: Recording an operation image of an inspector processing the security facility according to the fused image; and The operation image is matched with the expert sub-action of the fused image to determine whether the operation of the inspection personnel is correct.

6. The system according to claim 5, characterized in that Matching the operation image with the expert sub-action of the fused image to determine whether the operation of the inspection personnel is correct includes: Slicing the operation image to generate a plurality of operation image slices; Processing the plurality of operation image slices using the action recognition model to determine inspection sub-actions corresponding to the plurality of operation image slices; and The inspection sub-action is matched with the expert sub-action to determine whether the inspection personnel's operation is correct.

7. A security facility audit method, characterized in that: include: Using the augmented reality device to capture a live image of security facilities at the target site during communication with the terminal device, and uploading the live image to the terminal device in real time; Capturing, by a motion capture device, a gesture action made by an expert based on the live image after reviewing the live image, and sending a gesture graphic corresponding to the gesture action to the terminal device; Generate a digital twin image corresponding to the on-site image by using a digital twin algorithm through the terminal device; fusing the gesture graphic with the digital twin image through the terminal device to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; Sending the fused image to the augmented reality device through the terminal device; as well as The fused image corresponding to the on-site image and sent by the terminal device is displayed in real time through the augmented reality device.

8. A security facility audit device, characterized in that: include: An image capturing module, used to capture a live image of security facilities in a target site through an augmented reality device during communication with a terminal device, and upload the live image to the terminal device in real time; A motion capture module, used to capture, through a motion capture device, gestures made by an expert based on the live image after reviewing the live image, and send gesture graphics corresponding to the gestures to the terminal device; An image generation module, configured to generate a digital twin image corresponding to the on-site image by using a digital twin algorithm through the terminal device; an image fusion module, used to fuse the gesture graphic with the digital twin image through the terminal device to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; An image sending module, used for sending the fused image to the augmented reality device through the terminal device; as well as An image display module is used to display in real time through the augmented reality device a fused image corresponding to the on-site image and sent by the terminal device.

9. A security facility audit device, characterized in that: include: processor; as well as A memory, connected to the processor, configured to provide the processor with instructions for processing the following processing steps: Using the augmented reality device to capture a live image of security facilities at the target site during communication with the terminal device, and uploading the live image to the terminal device in real time; Capturing, by a motion capture device, a gesture action made by an expert based on the live image after reviewing the live image, and sending a gesture graphic corresponding to the gesture action to the terminal device; Generate a digital twin image corresponding to the on-site image by using a digital twin algorithm through the terminal device; fusing the gesture graphic with the digital twin image through the terminal device to generate a fused image, wherein the fused image is used to indicate an image with the digital twin image as the background and the gesture graphic as the foreground; Sending the fused image to the augmented reality device through the terminal device; as well as The fused image corresponding to the on-site image and sent by the terminal device is displayed in real time through the augmented reality device.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to claim 7 are implemented.

Citation Information

Patent Citations

  • Dynamic gesture identification method based on combined nerve network

    CN107180226A

  • Enhanced interaction manner-based remote maintenance support system and method

    CN108509026A

  • ToF-based gesture recognition control method and control system thereof

    CN111651038A

  • Artificial intelligence fault identification system and method based on transient waveform of power transmission line

    CN113033837A

  • Generation method and device of substation maintenance guidance video and processor

    CN118354033A

Cited By

  • Real-time remote guidance system and method based on online video key information fusion

    CN120050515A