Two-way video fusion interactive AR glasses maintenance system and maintenance method

Abstract

The invention discloses a two-way video fusion interactive AR glasses maintenance system and maintenance method, and relates to the technical field of AR glasses, and the system comprises augmented reality glasses which are used for collecting on-site image data and transmitting the on-site image data to an edge calculation server; the target operation terminal is used for acquiring sketching information and gesture image data and transmitting the sketching information and the gesture image data to the edge computing server; and the edge computing server is used for fusing the field image data, the sketching information and the gesture image data to generate a fused interactive video, and sending the fused interactive video to the augmented reality glasses and the target operation terminal for display. The technical problem that AR glasses in related technologies are poor in interactivity and cannot meet accurate maintenance requirements is solved.

Description

Technical Field

[0001] The present invention relates to the technical field of AR glasses, and in particular to a two-way video fusion interactive maintenance AR glasses system and maintenance method. Background Art

[0002] Currently, maintenance personnel can perform maintenance on relay protection in substations through the use of AR (Augmented Reality) glasses.

[0003] In the related technology, by modeling the maintenance site in advance, process information of various standardized operations can be constructed and uploaded to the server, and then the client requests the server to download the process information of the standardized operation corresponding to the current maintenance operation, and the process information of the standardized operation is played by the client or the speaker of the AR glasses, and the display screen of the AR glasses displays the current step of the process. In this way, maintenance personnel can perform maintenance through the played process information and the current step displayed on the display screen.

[0004] However, the above maintenance plan requires on-site modeling in advance, and the workload of model establishment and maintenance is large. Once changes occur on site, the model needs to be modified. In addition, the current working mode of AR glasses cannot achieve real-time interaction between the front and back ends, cannot meet the complex interactive needs of the site, and the user experience is poor. In addition, the current guidance of AR glasses is based on one-way video, and the maintenance personnel and the background interact through voice, while the relay protection operation requires relatively precise requirements. Operations on pressure plates, terminal blocks, etc. require centimeter-level precision guidance. The work cannot be completed simply relying on the current voice and guidance mode.

[0005] To address the above-mentioned problems, no effective solution has been proposed yet. Summary of the invention

[0006] The embodiments of the present invention provide a two-way video fusion interactive maintenance AR glasses system and maintenance method, so as to at least solve the technical problem that AR glasses in the related art have poor interactivity and cannot meet the precise maintenance requirements.

[0007] According to one aspect of an embodiment of the present invention, a maintenance AR glasses system with two-way video fusion interaction is provided, and the maintenance AR glasses system includes: augmented reality glasses, used to collect on-site image data, and transmit the on-site image data to an edge computing server; a target operation terminal, used to obtain outlining information and gesture image data, and transmit the outlining information and the gesture image data to the edge computing server; the edge computing server is used to fuse the on-site image data, the outlining information and the gesture image data to generate a fused interactive video, and send the fused interactive video to the augmented reality glasses and the target operation terminal for display, wherein the fused interactive video is used to guide a maintenance object wearing the augmented reality glasses to perform maintenance operations.

[0008] Furthermore, the augmented reality glasses include: a camera module for collecting the on-site image data; a data transmission module for transmitting the on-site image data to the edge computing server and the target operation terminal; and a display module for displaying the fused interactive video transmitted by the edge computing server.

[0009] Furthermore, the target operation terminal includes: a touch screen for displaying the on-site image data and receiving the sketching information, wherein the on-site image data includes: multiple on-site images, and the sketching information is information obtained by sketching on the on-site images displayed on the touch screen; and a motion capture module for capturing the gesture image data for guiding the on-site images.

[0010] Furthermore, the edge computing server includes: a data processing module, used to pre-process the received on-site image data to obtain target image data, and extract a guiding action image from the gesture image data; a two-way video fusion module, used to fuse the guiding action image, the outlining information and the on-site image in the on-site image data to generate the fused interactive video, wherein each guiding action image corresponds to one of the on-site images, and the outlining information corresponds to one of the on-site images.

[0011] Furthermore, the step of preprocessing the received on-site image data to obtain target image data includes: deblurring each of the on-site images in the on-site image data to obtain a first image; enhancing each of the first images to obtain a second image; denoising each of the second images to obtain a third image; and error processing each of the third images to obtain the target image data.

[0012] Furthermore, the step of fusing the guiding action image, the outlining information and the live image in the live image data includes: determining a touch area when outlining the outlining information, and matching the touch area with an image area on the live image corresponding to the outlining information; when the area position of the touch area is consistent with the target area position of the target image area, marking the outlining information on the target image area on the live image; determining the image area range indicated by the guiding action image, and overlaying the guiding action image within the image area range on the live image.

[0013] According to another aspect of an embodiment of the present invention, there is also provided a maintenance method for an AR glasses system using the above-mentioned two-way video fusion interaction, comprising: collecting on-site image data, and obtaining outline information and gesture image data corresponding to each on-site image of the on-site image data; preprocessing the on-site image data to obtain target image data, and extracting a guiding action image from the gesture image data; fusing the guiding action image, the outline information and the target on-site image in the target image data to generate a fused interactive video, wherein the fused interactive video is used to guide a maintenance object wearing the augmented reality glasses to perform maintenance operations.

[0014] Furthermore, before acquiring the gesture image data, a capture background area of ​​the motion capture module is determined, and a target object of a preset color is covered on the capture background area.

[0015] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is further provided, wherein the computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute any one of the above-mentioned maintenance methods.

[0016] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement any one of the above-mentioned maintenance methods.

[0017] In the present invention, the maintenance AR glasses system includes: augmented reality glasses, which are used to collect on-site image data and transmit the on-site image data to an edge computing server; a target operation terminal, which is used to obtain outlining information and gesture image data, and transmit the outlining information and gesture image data to an edge computing server; an edge computing server, which is used to fuse the on-site image data, the outlining information and the gesture image data to generate a fused interactive video, and send the fused interactive video to the augmented reality glasses and the target operation terminal for display.

[0018] In the present invention, by repairing the AR glasses system, the on-site image data collected by the augmented reality glasses and the outline information and gesture image data generated by the target operation terminal according to the on-site image data can be transmitted to the edge computing server for processing to generate a fused interactive video, and the fused interactive video augmented reality glasses and the target operation terminal are displayed, which can improve the interactivity between the front and back ends and realize accurate on-site maintenance, thereby solving the technical problem that the AR glasses in the related technology have poor interactivity and cannot meet the precise maintenance needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The drawings described herein are used to provide a further understanding of the present invention and constitute a part of the present invention. The exemplary embodiments of the present invention and their descriptions are used to explain the present invention and do not constitute an improper limitation of the present invention. In the drawings:

[0020] Figure 1 is a schematic diagram of an optional two-way video fusion interactive maintenance AR glasses system according to an embodiment of the present invention;

[0021] Figure 2 is a schematic diagram of an optional fused interactive image according to an embodiment of the present invention;

[0022] Figure 3 is a schematic diagram of an optional maintenance process based on a maintenance AR glasses system according to an embodiment of the present invention;

[0023] Figure 4 is a flowchart of an optional maintenance method for an AR glasses system using two-way video fusion interaction according to an embodiment of the present invention;

[0024] Figure 5 4 is a hardware structure block diagram of an electronic device (or mobile device) used for a maintenance method according to an embodiment of the present invention. DETAILED DESCRIPTION

[0025] In order to enable those skilled in the art to better understand the scheme of the present invention, the technical scheme in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the present invention.

[0026] It should be noted that the terms "first", "second", etc. in the specification and claims of the present invention and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged where appropriate, so that the embodiments of the present invention described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0027] The present invention proposes a two-way video fusion interactive relay protection maintenance AR glasses system, which can achieve two-way real-time interactive guidance through two-way video fusion, and effectively solves the low interaction problem of current AR applications that only use sketching and voice.

[0028] The present invention does not need to implement AR modeling and model maintenance, and can directly start guiding on-site operations based on actual on-site images. Moreover, based on the maintenance AR glasses system in the present invention, experts can use gestures and voice to directly guide front-end operations in the background, and the interaction is direct and intuitive, realizing immersive interaction. In addition, the maintenance AR glasses system in the present invention can also realize background outlining and annotation, and the maintenance object can intuitively see the content of the expert guidance on site.

[0029] The present invention is described in detail below in conjunction with various embodiments.

[0030] Embodiment 1

[0031] According to an embodiment of the present invention, an embodiment of a maintenance AR glasses system with two-way video fusion interaction is provided.

[0032] Figure 1 is a schematic diagram of an optional two-way video fusion interactive maintenance AR glasses system according to an embodiment of the present invention, such as Figure 1 As shown, the AR glasses maintenance system includes: augmented reality glasses, a target operation terminal and an edge computing server.

[0033] Among them, the augmented reality glasses are used to collect on-site image data and transmit the on-site image data to the edge computing server;

[0034] The target operation terminal is used to obtain the outline information and the gesture image data, and transmit the outline information and the gesture image data to the edge computing server;

[0035] The edge computing server is used to fuse on-site image data, outline information and gesture image data to generate a fused interactive video, and send the fused interactive video to the augmented reality glasses and the target operation terminal for display, wherein the fused interactive video is used to guide the maintenance object wearing the augmented reality glasses to perform maintenance operations.

[0036] In an embodiment of the present invention, the maintenance object can wear augmented reality glasses (i.e., AR glasses) to collect on-site image data (i.e., real-time images of the maintenance site). These on-site image data can be displayed in real time on the target operation terminal (i.e., expert operation terminal), so that the background expert can generate guidance information (i.e., sketching information and gesture image data) based on these on-site image data through the target operation terminal, and at the same time can transmit the on-site image data to the edge computing server, so that the edge computing server can process these on-site image data and merge these on-site image data with the guidance information to generate a fused interactive video to guide the maintenance object to perform maintenance.

[0037] In an embodiment of the present invention, a backend expert can, through a target operation terminal, perform outlining and annotation and gesture guidance based on a real-time image of the maintenance site displayed in real time to generate outlining information and gesture image data. In addition, the target operation terminal can transmit the generated outlining information and gesture image data to an edge computing server so that the edge computing server can fuse the outlining information and gesture image data with the real-time image, thereby realizing real-time interactive guidance of the maintenance object.

[0038] In an embodiment of the present invention, the edge computing server is able to fuse on-site image data, outline information, and gesture image data to generate a fused interactive video (the fused interactive video is used to guide the maintenance object wearing augmented reality glasses to perform maintenance operations), and send the fused interactive video to the augmented reality glasses and the target operation terminal for display, so as to realize real-time interactive guidance maintenance between the front and back ends.

[0039] In some optional embodiments, the maintenance object may be an on-site worker, a robot, or mechanical equipment.

[0040] Optionally, the augmented reality glasses include: a camera module for collecting on-site image data; a data transmission module for transmitting on-site image data to an edge computing server and a target operation terminal; and a display module for displaying a fused interactive video transmitted by the edge computing server.

[0041] In an embodiment of the present invention, the augmented reality glasses include: a camera module, a data transmission module and a display module, wherein the camera module can obtain on-site image data of the relay protection operation (that is, the camera module is used to collect on-site image data); the data transmission module can send on-site image data to the edge computing server and the target operation terminal through a 5G or wireless network (that is, the data transmission module is used to transmit on-site image data to the edge computing server and the target operation terminal); the display module can display the fused interactive video results (that is, the fused interactive video) (that is, the display module is used to display the fused interactive video transmitted by the edge computing server).

[0042] Optionally, the target operation terminal includes: a touch screen for displaying on-site image data and receiving sketching information, wherein the on-site image data includes: multiple on-site images, and the sketching information is information obtained by sketching on the on-site image displayed on the touch screen; and a motion capture module for capturing gesture image data for guiding the on-site image.

[0043] In the embodiment of the present invention, the target operation terminal includes: a touch screen and a motion capture module, wherein the touch screen is used to display the on-site image and the fused interactive video result, and receive the operation information such as the expert's sketching on the touch screen, and the motion capture module can extract the expert's operation gesture from the background. Optionally, to ensure the extraction effect, the background can use a green table mat.

[0044] Optionally, the edge computing server includes: a data processing module, used to pre-process the received on-site image data to obtain target image data, and extract guidance action images from the gesture image data; a two-way video fusion module, used to fuse the guidance action image, the outline information and the on-site image in the on-site image data to generate a fused interactive video, wherein each guidance action image corresponds to a on-site image, and the outline information corresponds to a on-site image.

[0045] In an embodiment of the present invention, the edge computing server includes: a data processing module and a two-way video fusion module, which can realize the two-way video fusion function and send it to the display module of the augmented reality glasses and the touch screen of the target operation terminal respectively.

[0046] In this embodiment, since the relay protection operation scene is relatively complex, the data processing module can pre-process the received on-site image data to obtain the target image data. In addition, since the environment of the target operation terminal is relatively stable, but it is necessary to capture a clearer action image, a green table mat can be used to obtain a clear expert guidance action image (that is, the data processing module is used to pre-process the received on-site image data to obtain the target image data, and extract the guidance action image from the gesture image data). After that, the result of superimposing the two videos is sent to the display terminals on both sides (that is, the display module of the augmented reality glasses and the touch screen of the target operation terminal) through the two-way video fusion module.

[0047] In this embodiment, each guiding action image corresponds to one scene image, and each drawing information corresponds to one scene image.

[0048] In this embodiment, the edge computing server uses different image processing strategies for the complex field end (i.e., augmented reality glasses) and the stable expert end (i.e., the target operation terminal), which can effectively reduce the amount of calculation. In addition, the edge computing server can complete the fusion of two-way video using edge computing, which can effectively reduce the amount of data transmission.

[0049] Optionally, the step of preprocessing the received scene image data to obtain target image data includes: deblurring each scene image in the scene image data to obtain a first image; enhancing each first image to obtain a second image; denoising each second image to obtain a third image; and error processing each third image to obtain target image data.

[0050] In an embodiment of the present invention, the data processing module of the edge computing server can perform deblurring, low-light enhancement, noise removal and error processing operations on the scene image data to obtain a clear image (i.e., deblurring the scene image to obtain a first image, enhancing the first image to obtain a second image, denoising the second image to obtain a third image, and error processing the third image to obtain target image data).

[0051] Optionally, the step of fusing the guiding action image, the outlining information and the scene image in the scene image data includes: determining a touch area when outlining the outlining information, and matching the touch area with the image area on the scene image corresponding to the outlining information; when the area position of the touch area is consistent with the target area position of the target image area, marking the outlining information on the target image area on the scene image; determining the image area range indicated by the guiding action image, and overlaying the guiding action image within the image area range on the scene image.

[0052] In an embodiment of the present invention, the edge computing server matches the touch area of ​​the touch screen (i.e., the touch area when the expert is outlining) with the camera acquisition area of ​​the AR glasses (i.e., the image area on the scene image). If the area position of the touch area is consistent with the target area position of the target image area, the outlining information is marked on the target image area on the scene image to achieve the fusion of the outlining information and the scene image. In addition, the image area range indicated by the guiding action image can also be determined, and the guiding action image can be overlaid within the image area range on the scene image. For example, the image area range of the expert's gesture operation on the image captured by the motion capture module is used to enable the edge computing server to fuse the gesture operation within the image area range on the scene image, thereby generating a fused interactive video and sending it to the display module of the AR glasses and the touch screen of the expert operation terminal for display.

[0053] Figure 2 is a schematic diagram of an optional fusion interactive image according to an embodiment of the present invention, such as Figure 2 As shown, an image of a fusion expert's gesture and a field operator's gesture is shown. Figure 2 The two hands in the picture come from an expert and a field operator respectively.

[0054] Figure 3 is a schematic diagram of an optional maintenance process based on the maintenance AR glasses system according to an embodiment of the present invention, such as Figure 3 As shown, the following process is included:

[0055] (1) When needed, on-site relay protection workers can call experts through AR glasses, and interactive guidance can be achieved between the two parties after the call is connected;

[0056] (2) The data transmission module of the AR glasses transmits the data collected by the camera of the AR glasses to the edge computing server;

[0057] (3) The expert operation terminal obtains the expert's touch information on the touch screen and transmits the gesture data obtained by the motion capture module to the edge computing server;

[0058] (4) The edge computing server performs deblurring, low-light enhancement, noise removal, and error processing on the image data transmitted by the data transmission module in sequence, thereby obtaining clear video data on site;

[0059] (5) The edge computing server performs background clearing on the gesture data of the expert operation terminal to send the gesture data after the green screen is removed to the two-way video fusion module;

[0060] (6) The edge computing server matches the touch area of ​​the touch screen with the camera acquisition area of ​​the AR glasses, and sends the drawing operation record on the touch screen to the two-way video fusion module;

[0061] (7) The two-way video fusion module fuses the clear video data, gesture data, and sketching operation records to generate a video fusion result, which is sent to the display module of the AR glasses and to the touch screen of the expert operation terminal through the data transmission module;

[0062] (8) If remote guidance is completed, proceed to (9), otherwise return to (3);

[0063] (9) The remote guidance interaction is terminated, and the remote guidance process data is uploaded to the edge computing server, and the expert guidance ends.

[0064] In the embodiment of the present invention, there is no need to implement AR modeling and model maintenance, and the on-site operation can be directly guided based on the actual on-site image. Moreover, based on the maintenance AR glasses system, experts can use gestures and voice to directly guide the front-end operation in the background, and the interaction is direct and intuitive, realizing immersive interaction. In addition, the maintenance AR glasses system can also realize background outlining and annotation, and the maintenance object can intuitively see the content of the expert guidance on site.

[0065] Embodiment 2

[0066] According to an embodiment of the present invention, an embodiment of a maintenance method for an AR glasses system using two-way video fusion interaction is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in an order different from that shown here.

[0067] Figure 4 is a flowchart of an optional method for repairing an AR glasses system using two-way video fusion interaction according to an embodiment of the present invention. Figure 4 As shown, the method comprises the following steps:

[0068] Step S401 , collecting scene image data, and obtaining the outline information and gesture image data corresponding to each scene image of the scene image data.

[0069] Step S402, pre-processing the scene image data to obtain target image data, and extracting the guiding action image from the gesture image data.

[0070] Step S403, fusing the guidance action image, the outline information and the target scene image in the target image data to generate a fused interactive video, wherein the fused interactive video is used to guide the maintenance object wearing the augmented reality glasses to perform maintenance operations.

[0071] Through the above steps, the on-site image data can be collected, and the delineation information and gesture image data corresponding to each on-site image of the on-site image data can be obtained, the on-site image data can be preprocessed to obtain the target image data, and the guiding action image can be extracted from the gesture image data, and the guiding action image, the delineation information and the target on-site image in the target image data can be fused to generate a fused interactive video. In an embodiment of the present invention, the collected on-site image data can be preprocessed, and the guiding action image can be extracted from the acquired gesture image data, and then the guiding action image, the delineation information and the target on-site image can be fused to generate a fused interactive video, thereby realizing real-time interaction between experts and maintenance objects, improving the interactivity between the front and back ends, and realizing accurate on-site maintenance, thereby solving the technical problem that AR glasses in related technologies have poor interactivity and cannot meet accurate maintenance needs.

[0072] In an embodiment of the present invention, the maintenance object can wear augmented reality glasses to collect on-site image data (i.e., real-time images of the maintenance site). These on-site image data can be displayed in real time on the target operation terminal, so that the background experts can generate guidance information (i.e., sketching information and gesture image data) based on these on-site image data through the target operation terminal, and at the same time, the on-site image data can be transmitted to the edge computing server, so that the edge computing server can process these on-site image data and fuse these on-site image data with the guidance information to generate a fused interactive video to guide the maintenance object to perform maintenance.

[0073] In the embodiment of the present invention, the pre-processing of the scene image data includes: blur processing, weak light enhancement processing, noise removal processing and error processing, so as to obtain a clear scene image.

[0074] Optionally, before acquiring the gesture image data, a capture background area of ​​the motion capture module is determined, and a target object of a preset color is covered on the capture background area.

[0075] In an embodiment of the present invention, the capture background area of ​​the motion capture module can be determined first, and a target object (eg, a table mat, a curtain, etc.) of a preset color (ie, green) can be covered on the capture background area to facilitate obtaining a clear guiding motion image.

[0076] In the embodiment of the present invention, experts can use gestures, outlines, annotations, and voice to directly guide front-end operations in the background, which can achieve immersive interaction. In addition, they can directly start guiding on-site operations based on actual on-site images, without the need to implement AR modeling and model maintenance, effectively reducing the workload.

[0077] The present invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing a program that initializes the following method steps: collecting scene image data, and obtaining the outline information and gesture image data corresponding to each scene image of the scene image data, preprocessing the scene image data to obtain target image data, and extracting the guiding action image from the gesture image data, fusing the guiding action image, the outline information and the target scene image in the target image data to generate a fused interactive video.

[0078] According to another aspect of an embodiment of the present invention, a computer-readable storage medium is provided. The computer-readable storage medium includes a stored computer program. When the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the above-mentioned maintenance method.

[0079] According to another aspect of an embodiment of the present invention, there is also provided an electronic device, comprising one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the above-mentioned maintenance method.

[0080] Figure 5 is a hardware structure block diagram of an electronic device (or mobile device) for a maintenance method according to an embodiment of the present invention. Figure 5 As shown, the electronic device may include one or more processors (e.g., Figure 5 The processor 502a, processor 502b, ..., processor 502n, etc., which may include but are not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA, etc.), a memory 504 for storing data. In addition, it may also include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply and / or a camera. It can be understood by those skilled in the art that Figure 5 The structure shown is only for illustration and does not limit the structure of the above electronic device. Figure 5 More or fewer components as shown, or with Figure 5 Different configurations are shown.

[0081] The serial numbers of the above embodiments of the present invention are only for description and do not represent the advantages or disadvantages of the embodiments.

[0082] The embodiments or examples of the present disclosure are not exhaustive, but are only illustrative of some embodiments or examples, and are not intended to be specific limitations on the scope of protection of the present disclosure. In the absence of contradiction, each step in a certain embodiment or example can be implemented as an independent example, and the steps can be combined arbitrarily. For example, the scheme after removing some steps in a certain embodiment or example can also be implemented as an independent example, and the order of the steps in a certain embodiment or example can be arbitrarily exchanged. In addition, the optional methods or optional examples in a certain embodiment or example can be combined arbitrarily; in addition, the various embodiments or examples can be combined arbitrarily, for example, some or all steps of different embodiments or examples can be combined arbitrarily, and a certain embodiment or example can be combined arbitrarily with the optional methods or optional examples of other embodiments or examples.

[0083] In the above embodiments of the present invention, the description of each embodiment has its own emphasis. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0084] In the several embodiments provided by the present invention, it should be understood that the disclosed technical content can be implemented in other ways. Among them, the device embodiments described above are only schematic. For example, the division of the units can be a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of units or modules, which can be electrical or other forms.

[0085] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple units. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0086] In addition, each functional unit in each embodiment of the present invention may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0087] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, server or network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), mobile hard disk, disk or optical disk, etc. Various media that can store program codes.

[0088] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principle of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A two-way video fusion interactive maintenance AR glasses system, characterized in that: The AR glasses maintenance system comprises: Augmented reality glasses, used to collect on-site image data and transmit the on-site image data to an edge computing server; A target operation terminal is used to obtain the drawing information and the gesture image data, and transmit the drawing information and the gesture image data to the edge computing server; The edge computing server is used to fuse the on-site image data, the outline information and the gesture image data to generate a fused interactive video, and send the fused interactive video to the augmented reality glasses and the target operation terminal for display, wherein the fused interactive video is used to guide the maintenance object wearing the augmented reality glasses to perform maintenance operations.

2. The AR glasses maintenance system according to claim 1, characterized in that: The augmented reality glasses include: A camera module, used for collecting the on-site image data; A data transmission module, used for transmitting the on-site image data to the edge computing server and the target operation terminal; A display module is used to display the fused interactive video transmitted by the edge computing server.

3. The AR glasses maintenance system according to claim 1, characterized in that: The target operation terminal includes: A touch screen, used to display the scene image data and receive the delineation information, wherein the scene image data includes: a plurality of scene images, and the delineation information is information obtained by delineating on the scene image displayed on the touch screen; The motion capture module is used to capture the gesture image data for guiding the on-site image.

4. The AR glasses maintenance system according to claim 1, characterized in that: The edge computing server comprises: A data processing module, used for preprocessing the received on-site image data to obtain target image data, and extracting a guiding action image from the gesture image data; A two-way video fusion module is used to fuse the guiding action image, the outlining information and the live image in the live image data to generate the fused interactive video, wherein each guiding action image corresponds to one of the live images, and the outlining information corresponds to one of the live images.

5. The AR glasses maintenance system according to claim 4, characterized in that: The step of preprocessing the received on-site image data to obtain target image data comprises: Performing a defuzzification process on each of the scene images in the scene image data to obtain a first image; Performing enhancement processing on each of the first images to obtain a second image; Performing denoising processing on each of the second images to obtain a third image; Error processing is performed on each of the third images to obtain the target image data.

6. The AR glasses maintenance system according to claim 4, characterized in that: The step of fusing the guiding action image, the delineation information and the scene image in the scene image data comprises: determining a touch area when outlining the outlining information, and matching the touch area with an image area on the scene image corresponding to the outlining information; In a case where the area position of the touched area is consistent with the target area position of the target image area, marking the delineation information on the target image area on the scene image; The image area range indicated by the guiding action image is determined, and the guiding action image is overlaid within the image area range on the scene image.

7. A maintenance method for an AR glasses system using the two-way video fusion interaction described in claim 1, characterized in that: include: Collecting scene image data, and obtaining delineation information and gesture image data corresponding to each scene image of the scene image data; Preprocessing the on-site image data to obtain target image data, and extracting a guiding action image from the gesture image data; The guiding action image, the sketching information and the target scene image in the target image data are fused to generate a fused interactive video, wherein the fused interactive video is used to guide the maintenance object wearing the augmented reality glasses to perform maintenance operations.

8. The maintenance method according to claim 7, characterized in that: Before acquiring the gesture image data, a capture background area of ​​the motion capture module is determined, and a target object of a preset color is covered on the capture background area.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program, wherein when the computer program is executed, the device where the computer-readable storage medium is located is controlled to execute the maintenance method according to any one of claims 7 to 8.

10. An electronic device, characterized in that: It comprises one or more processors and a memory, wherein the memory is used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the maintenance method described in any one of claims 7 to 8.

Images