Image stabilization method and device based on ar glasses remote guidance, electronic device

Abstract

The application discloses a kind of image stabilization method and device based on AR glasses remote guidance, electronic equipment, it is related to AR glasses technical field, including: when being in interactive guidance mode, determine guidance gesture entry coordinates, obtain guidance gesture data and live video data, generate the initial interactive guidance video of preset time length based on guidance gesture data, live video data and guidance gesture entry coordinates, determine guidance gesture end coordinates when each interactive guidance time length reaches preset time length, determine the latest position coordinates of equipment area, based on the start position coordinates of equipment area in the first frame interactive guidance image in initial interactive guidance video, latest position coordinates, guidance gesture entry coordinates and guidance gesture end coordinates, initial interactive guidance video is handled, and stable interactive guidance video is obtained.The application solves the technical problem that interactive guidance video cannot be kept stable when remote guidance is carried out through AR glasses in the related art.

Description

Technical Field

[0001] This invention relates to the field of AR glasses technology, and more specifically, to an image stabilization method and apparatus, and an electronic device based on remote guidance for AR glasses. Background Technology

[0002] Currently, for the maintenance of relay protection systems in substations, maintenance personnel can use AR (Augmented Reality) glasses. However, during AR remote guidance, the movement of the AR glasses can easily cause the image seen by the maintenance personnel to be unstable, making it difficult for them to adapt and inconvenient for interactive guidance.

[0003] In related technologies, for unstable video images input in real time, key points of the current input image can be extracted first. The current image and the previous frame image are then input into a network model to obtain the optical flow map of the previous frame image. The optical flow map and key points are then combined to generate the mesh vertex motion of the frame. The mesh vertex motion is placed in a sliding window for motion smoothing. After passing through a real-time smoother network, the difference between the mesh motion trajectory before and after smoothing is finally obtained. Then, a stable image and a corresponding fill mask are obtained through homography transformation. The stable image and the fill mask are processed to obtain the filled full-frame image, which is then output.

[0004] However, while the above scheme can output stable images, it requires a large amount of computation, and because it directly processes the images, it is easy to obtain discontinuous frames, which will affect the interactive effect.

[0005] There is currently no effective solution to the above problems. Summary of the Invention

[0006] This invention provides an image stabilization method, apparatus, and electronic device for remote guidance based on AR glasses, to at least solve the technical problem in related technologies that interactive guidance videos cannot be kept stable when providing remote guidance through AR glasses.

[0007] According to one aspect of the present invention, an image stabilization method based on AR glasses for remote guidance is provided, comprising: determining the entry coordinates of a guidance gesture when in an interactive guidance mode; acquiring guidance gesture data and on-site video data, and generating an initial interactive guidance video of a preset duration based on the guidance gesture data, on-site video data, and the entry coordinates of the guidance gesture, wherein the start time of the initial interactive guidance video is the start time of the interactive guidance mode or the start time of each preset duration, and each frame of the interactive guidance image in the initial interactive guidance video includes the equipment area where the target maintenance equipment is located and the guidance gesture; determining the end coordinates of the guidance gesture and the latest position coordinates of the equipment area when the interactive guidance duration reaches the preset duration; and processing the initial interactive guidance video based on the start position coordinates of the equipment area on the first frame of the interactive guidance image in the initial interactive guidance video, the latest position coordinates, the entry coordinates of the guidance gesture, and the end coordinates of the guidance gesture to obtain a stable interactive guidance video.

[0008] Furthermore, the image stabilization method further includes: determining an interactive guidance area; determining to enter the interactive guidance mode when the guidance gesture of the guided object is detected to enter the interactive guidance area; and determining to exit the interactive guidance mode when the guidance gesture of the guided object is detected to leave the interactive guidance area.

[0009] Furthermore, before generating an initial interactive guidance video of a preset duration based on the guidance gesture data, on-site video data, and the guidance gesture entry coordinates, the method further includes: determining the guidance gesture entry time; and determining the starting position coordinates of the equipment area where the target maintenance equipment is located at the guidance gesture entry time based on the on-site video data.

[0010] Further, the step of generating an initial interactive guidance video of a preset duration based on the guidance gesture data, the on-site video data, and the guidance gesture entry coordinates includes: extracting guidance gesture images for each moment within the preset duration from the guidance gesture data, wherein each moment corresponds to a frame image in the on-site video data, and each frame image in the on-site video data contains the device area; superimposing the guidance gesture images at each moment onto the images in the corresponding frames of the on-site video data based on the guidance gesture entry coordinates and the start position coordinates to obtain multiple frames of interactive guidance images; and generating the initial interactive guidance video based on the multiple frames of interactive guidance images.

[0011] Furthermore, after determining the latest position coordinates of the device area, the method further includes: determining the latest guiding gesture entry coordinates on the device area indicated by the latest position coordinates based on the position of the guiding gesture entry coordinates on the device area indicated by the starting position coordinates; and determining the latest guiding gesture end coordinates on the device area indicated by the latest position coordinates based on the position of the guiding gesture end coordinates on the device area indicated by the starting position coordinates.

[0012] Further, the step of processing the initial interactive guidance video to obtain a stable interactive guidance video based on the start position coordinates, the latest position coordinates, the guide gesture entry coordinates, and the guide gesture end coordinates of the device region on the first frame of the initial interactive guidance video includes: determining the overlap area of ​​the device region indicated by the start position coordinates and the device region indicated by the latest position coordinates in the same coordinate system; if the overlap area is less than a preset area threshold, using an interpolation uniformity algorithm, drawing the guide gesture from the guide gesture entry coordinates to the guide gesture end coordinates onto the interval from the guide gesture entry coordinates to the latest guide gesture end coordinates to obtain the stable interactive guidance video.

[0013] Furthermore, after determining the overlapping surface of the device area indicated by the starting position coordinates and the device area indicated by the latest position coordinates in the same coordinate system, the method further includes: if the overlapping area is greater than or equal to a preset area threshold, determining that the initial interactive guidance video is in a stable state; and characterizing the initial interactive guidance video as the stable interactive guidance video.

[0014] According to another aspect of the present invention, an image stabilization device based on AR glasses for remote guidance is also provided, comprising: a first determining unit, configured to determine the entry coordinates of a guidance gesture when in an interactive guidance mode; a generating unit, configured to acquire guidance gesture data and on-site video data, and generate an initial interactive guidance video of a preset duration based on the guidance gesture data, on-site video data, and the entry coordinates of the guidance gesture, wherein the start time of the initial interactive guidance video is the start time of the interactive guidance mode or the start time of each preset duration, and each frame of the interactive guidance image in the initial interactive guidance video includes the equipment area where the target maintenance equipment is located and the guidance gesture; a second determining unit, configured to determine the end coordinates of the guidance gesture and the latest position coordinates of the equipment area when the interactive guidance duration reaches the preset duration; and a processing unit, configured to process the initial interactive guidance video based on the start position coordinates of the equipment area on the first frame of the interactive guidance image in the initial interactive guidance video, the latest position coordinates, the entry coordinates of the guidance gesture, and the end coordinates of the guidance gesture, to obtain a stable interactive guidance video.

[0015] Furthermore, the image stabilization device further includes: a first determining module for determining an interactive guidance area; a second determining module for determining to enter the interactive guidance mode when the guiding gesture of the guided object is detected to enter the interactive guidance area; and a third determining module for determining to exit the interactive guidance mode when the guiding gesture of the guided object is detected to leave the interactive guidance area.

[0016] Furthermore, the image stabilization device further includes: a fourth determining module, used to determine the moment of entry of the guiding gesture before generating an initial interactive guiding video of a preset duration based on the guiding gesture data, on-site video data, and the guiding gesture entry coordinates; and a fifth determining module, used to determine the starting position coordinates of the equipment area where the target maintenance equipment is located at the moment of entry of the guiding gesture based on the on-site video data.

[0017] Further, the generation unit includes: a first extraction module, used to extract the guidance gesture image at each moment within the preset duration from the guidance gesture data, wherein each moment corresponds to a frame image in the on-site video data, and each frame image in the on-site video data contains the device area; a first overlay module, used to overlay the guidance gesture image at each moment onto the image in the corresponding frame of the on-site video data based on the guidance gesture entry coordinates and the start position coordinates, to obtain multiple frames of interactive guidance images; and a first generation module, used to generate the initial interactive guidance video based on the multiple frames of interactive guidance images.

[0018] Furthermore, the image stabilization device further includes: a sixth determining module, configured to, after determining the latest position coordinates of the device area, determine the latest guiding gesture entry coordinates on the device area indicated by the latest position coordinates based on the position of the guiding gesture entry coordinates on the device area indicated by the start position coordinates; and a seventh determining module, configured to determine the latest guiding gesture end coordinates on the device area indicated by the latest position coordinates based on the position of the guiding gesture end coordinates on the device area indicated by the start position coordinates.

[0019] Further, the processing unit includes: an eighth determining module, used to determine the overlapping area of ​​the device area indicated by the start position coordinates and the device area indicated by the latest position coordinates in the same coordinate system; and a first drawing module, used to, when the overlapping area is less than a preset area threshold, use an interpolation uniformity algorithm to draw the guidance gesture from the guide gesture entry coordinate to the guide gesture end coordinate onto the interval from the guide gesture entry coordinate to the latest guide gesture end coordinate, thereby obtaining the stable interactive guidance video.

[0020] Furthermore, the image stabilization device further includes: a ninth determining module, configured to determine that the initial interactive guidance video is in a stable state after determining that the overlapping area of ​​the device area indicated by the starting position coordinates and the device area indicated by the latest position coordinates are in the same coordinate system, provided that the overlapping area is greater than or equal to a preset area threshold; and a first characterization module, configured to characterize the initial interactive guidance video as the stable interactive guidance video.

[0021] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform any of the above-described image stabilization methods based on remote guidance from AR glasses.

[0022] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement any of the above-described image stabilization methods based on AR glasses remote guidance.

[0023] In this invention, when in interactive guidance mode, the coordinates of the guidance gesture entry are determined, guidance gesture data and on-site video data are acquired, and an initial interactive guidance video of a preset duration is generated based on the guidance gesture data, on-site video data and guidance gesture entry coordinates. When the interactive guidance duration reaches the preset duration each time, the coordinates of the guidance gesture end are determined, and the latest position coordinates of the device area are determined. Based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates and guidance gesture end coordinates of the device area on the first frame of the interactive guidance image in the initial interactive guidance video, the initial interactive guidance video is processed to obtain a stable interactive guidance video.

[0024] In this invention, if it is determined that the current state is in interactive guidance mode, the coordinates of the guidance gesture entry are determined, and based on the guidance gesture entry coordinates, the guidance gesture data is superimposed on the on-site video data to generate an initial interactive guidance video of a preset duration. After each interactive guidance duration reaches the preset duration, the coordinates of the guidance gesture end and the latest position coordinates of the equipment area where the target maintenance equipment is located are determined. Then, based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates of the equipment area on the first frame of the interactive guidance image in the initial interactive guidance video, the initial interactive guidance video is processed to obtain a stable interactive guidance video. By processing the interactive guidance video of the preset duration, not only can the amount of computation be reduced, but the continuity of video frames can also be improved, effectively achieving the stability of the interactive guidance video. This solves the technical problem in related technologies where the interactive guidance video cannot maintain stability when providing remote guidance through AR glasses, and achieves the technical effect of maintaining the stability of the interactive guidance video when providing remote guidance through AR glasses. Attached Figure Description

[0025] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this invention, illustrate exemplary embodiments of the invention and are used to explain the invention, but do not constitute an undue limitation of the invention. In the drawings:

[0026] Figure 1 This is a flowchart of an optional image stabilization method based on remote guidance from AR glasses according to an embodiment of the present invention;

[0027] Figure 2 This is a schematic diagram of an optional guide gesture coordinate update according to an embodiment of the present invention;

[0028] Figure 3 This is a schematic diagram of an optional relay-protected AR glasses remote guidance image stabilization system according to an embodiment of the present invention;

[0029] Figure 4This is a schematic diagram of an optional image stabilization device based on remote guidance from AR glasses according to an embodiment of the present invention;

[0030] Figure 5 This is a hardware structure block diagram of an electronic device (or mobile device) for an image stabilization method based on remote guidance of AR glasses, according to an embodiment of the present invention. Detailed Implementation

[0031] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0032] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover a non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0033] This invention proposes an image stabilization method for remote guidance of AR glasses with relay protection. It can stabilize the interactive guidance video when experts provide interactive guidance, without the need for complex video processing algorithms, and with less computation and lower cost.

[0034] In this invention, when an expert's gesture enters an area requiring interactive guidance, the coordinates of the expert's gesture and the coordinates of the area where the equipment to be inspected is located, as seen by the maintenance personnel through AR glasses, can be recorded. Then, by superimposing the expert's gesture onto the on-site video captured by the AR glasses camera in the corresponding frame, an interactive guidance video of a preset duration (e.g., 1 second) can be obtained. An interpolation uniformity algorithm is used to stabilize the interactive guidance video.

[0035] The present invention will now be described in detail with reference to various embodiments.

[0036] Example 1

[0037] According to an embodiment of the present invention, an embodiment of an image stabilization method based on remote guidance of AR glasses is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.

[0038] Figure 1 This is a flowchart of an optional image stabilization method based on remote guidance from AR glasses according to an embodiment of the present invention, such as... Figure 1 As shown, the method includes the following steps:

[0039] Step S101: In interactive guidance mode, determine the coordinates for the guidance gesture.

[0040] Step S102: Obtain guidance gesture data and on-site video data, and generate an initial interactive guidance video of preset duration based on the guidance gesture data, on-site video data, and guidance gesture entry coordinates. The start time of the initial interactive guidance video is the start time of the interactive guidance mode or the start time of each preset duration. Each frame of interactive guidance image in the initial interactive guidance video includes the equipment area where the target maintenance equipment is located and the guidance gesture.

[0041] Step S103: When the duration of each interactive guidance reaches the preset duration, determine the coordinates of the end of the guidance gesture and determine the latest position coordinates of the device area.

[0042] Step S104: Based on the start position coordinates, latest position coordinates, guide gesture entry coordinates, and guide gesture end coordinates of the device region on the first frame of the initial interactive guide video, process the initial interactive guide video to obtain a stable interactive guide video.

[0043] Through the above steps, in the interactive guidance mode, the coordinates of the guidance gesture entry can be determined, guidance gesture data and on-site video data can be acquired, and an initial interactive guidance video of a preset duration can be generated based on the guidance gesture data, on-site video data and guidance gesture entry coordinates. When the interactive guidance duration reaches the preset duration, the coordinates of the guidance gesture end can be determined, and the latest position coordinates of the device area can be determined. Based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates and guidance gesture end coordinates of the device area on the first frame of the interactive guidance image in the initial interactive guidance video, the initial interactive guidance video is processed to obtain a stable interactive guidance video. In this embodiment of the invention, if it is determined that the current state is in interactive guidance mode, the coordinates of the guidance gesture entry are determined, and based on the guidance gesture entry coordinates, the guidance gesture data is superimposed on the on-site video data to generate an initial interactive guidance video of a preset duration. After each interactive guidance duration reaches the preset duration, the coordinates of the guidance gesture end and the latest position coordinates of the equipment area where the target maintenance equipment is located are determined. Then, based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates of the equipment area on the first frame of the interactive guidance image in the initial interactive guidance video, the initial interactive guidance video is processed to obtain a stable interactive guidance video. By processing the interactive guidance video of the preset duration, not only can the amount of computation be reduced, but the continuity of video frames can also be improved, effectively achieving the stability of the interactive guidance video. This solves the technical problem in related technologies where the interactive guidance video cannot maintain stability when providing remote guidance through AR glasses, and achieves the technical effect of maintaining the stability of the interactive guidance video when providing remote guidance through AR glasses.

[0044] The embodiments of the present invention will now be described in detail with reference to the steps described above.

[0045] In this embodiment of the invention, after the relay protection operation begins, experts and maintenance personnel can use AR glasses to provide interactive video guidance.

[0046] In some alternative embodiments, maintenance personnel may be on-site workers, robots, or mechanical equipment.

[0047] In this embodiment of the invention, no dynamic stabilization is performed on the video when the expert does not provide interactive guidance using gestures. For example, if the expert only provides inspection guidance by drawing, there is no need to stabilize the interactive video; however, if the expert provides inspection guidance by gestures, the shaking of the on-site video captured by the AR glasses will cause the expert's gestures to appear shaken on the interactive video, thus requiring stabilization of the interactive video.

[0048] Optionally, the image stabilization method further includes: determining an interactive guidance area; determining to enter an interactive guidance mode when the guidance gesture of the guided object is detected to enter the interactive guidance area; and determining to exit the interactive guidance mode when the guidance gesture of the guided object is detected to leave the interactive guidance area.

[0049] In this embodiment of the invention, when an expert's gesture enters an area requiring interactive guidance, it is determined that the interactive guidance mode has been entered. Specifically, a pre-set interactive guidance area can be determined based on the current relay protection work order. If the expert's gesture is detected to enter the interactive guidance area, it is determined that the interactive guidance mode has been entered; if the expert's gesture is detected to leave the interactive guidance area, it is determined that the interactive guidance mode has been exited.

[0050] In some alternative embodiments, the work order is a pre-planned document outlining the maintenance procedures for a scheduled maintenance operation, including an interactive guidance area for expert use and the maintenance operations for each piece of equipment. This interactive guidance area is a physical space, pre-positioned with green tablecloths to ensure clear guidance gestures can be extracted from captured gesture data.

[0051] Step S101: In interactive guidance mode, determine the coordinates for the guidance gesture.

[0052] In step S101, when it is determined that the current interactive guidance mode is in progress, the coordinates of the guidance gesture can be determined first. That is, when the expert's gesture enters the area that requires interactive guidance, the coordinates of the guidance gesture on the real-time on-site image seen by the maintenance personnel through AR glasses.

[0053] Optionally, before generating an initial interactive guidance video of a preset duration based on the guidance gesture data, on-site video data, and guidance gesture entry coordinates, the method further includes: determining the guidance gesture entry time; and determining the starting position coordinates of the equipment area where the target maintenance equipment is located at the guidance gesture entry time based on the on-site video data.

[0054] In this embodiment of the invention, the moment of the guiding gesture can be recorded to determine the starting position coordinates (which can be determined by the coordinates of the four vertices of the equipment area) of the target maintenance equipment (i.e. the current maintenance setting of the maintenance personnel) at the moment of the guiding gesture from the on-site video data collected by the AR glasses.

[0055] Step S102: Obtain guidance gesture data and on-site video data, and generate an initial interactive guidance video of preset duration based on the guidance gesture data, on-site video data, and guidance gesture entry coordinates. The start time of the initial interactive guidance video is the start time of the interactive guidance mode or the start time of each preset duration. Each frame of interactive guidance image in the initial interactive guidance video includes the equipment area where the target maintenance equipment is located and the guidance gesture.

[0056] In step S102, guidance gesture data can be collected through the motion capture module of the expert working terminal, and on-site video data can be collected through the camera of the AR glasses. Then, the guidance gestures can be superimposed on the on-site video data according to the coordinates of the guidance gestures to generate an initial interactive guidance video.

[0057] In this embodiment of the invention, the stable video duration (i.e., preset duration) can be set according to the inertia duration of neural feedback, thereby ensuring that the stable interactive guidance video feedback to the maintenance personnel's field of vision is continuous and smooth.

[0058] In this embodiment of the invention, the start point of the initial interactive guidance video is either the start point of the interactive guidance mode (i.e., the moment when the expert's gesture enters the area requiring interactive guidance) or the start point of each preset duration (i.e., a segment of the interactive guidance video is extracted for stabilization every preset duration). Each frame of the interactive guidance image in the initial interactive guidance video includes the equipment area where the target maintenance equipment is located and the guidance gesture.

[0059] Optionally, the step of generating an initial interactive guidance video of a preset duration based on guidance gesture data, on-site video data, and guidance gesture entry coordinates includes: extracting guidance gesture images for each moment within the preset duration from the guidance gesture data, wherein each moment corresponds to a frame image in the on-site video data, and each frame image in the on-site video data contains the device area; superimposing the guidance gesture images of each moment onto the images in the corresponding frame of the on-site video data based on the guidance gesture entry coordinates and start position coordinates to obtain multiple frames of interactive guidance images; and generating the initial interactive guidance video based on the multiple frames of interactive guidance images.

[0060] In this embodiment of the invention, guidance gesture images at each moment within a preset time period can be extracted from the guidance gesture data. Each moment corresponds to a frame in the on-site video data (i.e., the guidance gesture image in the guidance gesture data and the frame containing the image in the on-site video data at the same moment are corresponding), and each frame in the on-site video data contains the device area. Then, based on the start position coordinates of the device area, the guidance gesture images are superimposed on the on-site video data captured by the AR glasses camera of the corresponding frame (i.e., based on the guidance gesture entry coordinates and start position coordinates, the guidance gesture image at each moment is superimposed on the image in the on-site video data of the corresponding frame to obtain multiple frames of interactive guidance images). In this embodiment, because the operation time is relatively short, considering the expert's reaction time, the expert believes that the image in the maintenance personnel's AR glasses is relatively stable.

[0061] In this embodiment of the invention, an initial interactive guidance video of a preset duration can be generated based on multiple frames of interactive guidance images, that is, the superimposed video is saved according to the preset duration (e.g., every second).

[0062] Step S103: When the duration of each interactive guidance reaches the preset duration, determine the end coordinates of the guidance gesture and determine the latest position coordinates of the device area.

[0063] In step S103, the superimposed interactive guidance video is saved according to a preset duration. The position coordinates of the guidance gesture on the last frame of the interactive guidance video can be determined as the end coordinates of the guidance gesture, and the latest position coordinates of the device area after each preset duration is determined.

[0064] Optionally, after determining the latest position coordinates of the device area, the method further includes: determining the latest guide gesture entry coordinates in the device area indicated by the latest position coordinates based on the position of the guide gesture entry coordinates in the device area indicated by the start position coordinates; and determining the latest guide gesture end coordinates in the device area indicated by the latest position coordinates based on the position of the guide gesture end coordinates in the device area indicated by the start position coordinates.

[0065] In this embodiment of the invention, since the position of the device area has been switched from the starting position coordinates to the latest position coordinates, the latest guide gesture entry coordinates and the latest guide gesture end coordinates on the device area indicated by the latest position coordinates can be determined based on the guide gesture entry coordinates and the guide gesture end coordinates on the device area indicated by the starting position coordinates. That is, the latest guide gesture entry coordinates on the device area indicated by the latest position coordinates are determined based on the position of the guide gesture entry coordinates on the device area indicated by the starting position coordinates, and the latest guide gesture end coordinates on the device area indicated by the latest position coordinates are determined based on the position of the guide gesture end coordinates on the device area indicated by the starting position coordinates.

[0066] Figure 2 This is a schematic diagram of an optional guide gesture coordinate update according to an embodiment of the present invention. Assume S1 represents the device area indicated by the start position coordinates, and within this device area, the entry and end coordinates of the guide gesture are represented by a and b, respectively; S2 represents the device area indicated by the latest position coordinates, and within this device area, the latest entry and latest end coordinates of the guide gesture are represented by "latest". Figure 2 As shown, the coordinates of the guiding gesture are a and b on S1, and when the position of S1 is switched to the position of S2, the coordinates of the guiding gesture are updated to a1 and b1 on S2.

[0067] Step S104: Based on the start position coordinates, latest position coordinates, guide gesture entry coordinates, and guide gesture end coordinates of the device region on the first frame of the initial interactive guide video, process the initial interactive guide video to obtain a stable interactive guide video.

[0068] In step S104, it can be determined whether the interactive guidance video needs to be stabilized based on the overlapping areas of S1 and S2. If so, the initial interactive guidance video is stabilized to obtain a stable interactive guidance video. If not, the stable interactive guidance video is obtained directly (i.e., the initial interactive guidance video is processed based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates of the device region on the first frame of the interactive guidance image in the initial interactive guidance video to obtain a stable interactive guidance video).

[0069] Optionally, the step of processing the initial interactive guidance video to obtain a stable interactive guidance video based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates of the device area on the first frame of the initial interactive guidance video includes: determining the overlap area of ​​the device area indicated by the start position coordinates and the device area indicated by the latest position coordinates in the same coordinate system; if the overlap area is less than a preset area threshold, using an interpolation uniformity algorithm, drawing the guidance gesture from the guidance gesture entry coordinate to the guidance gesture end coordinate onto the interval from the guidance gesture entry coordinate to the latest guidance gesture end coordinate, thereby obtaining a stable interactive guidance video.

[0070] In this embodiment of the invention, the overlap area between the device area indicated by the starting position coordinates and the device area indicated by the latest position coordinates in the same coordinate system can be determined first. If the overlap area is less than a preset area threshold (for example, the overlap area between S1 and S2 is less than 90%), it indicates that the current interactive guidance video may be unstable and needs to be processed. The expert's guidance gestures can be drawn according to the coordinates from a to b1. That is, the guidance gestures from a to b are uniformly fed back to the interval from a to b1 through interpolation (that is, when the overlap area is less than the preset area threshold, the uniform interpolation algorithm is used to draw the guidance gestures from the guidance gesture entry coordinate to the guidance gesture end coordinate on the interval from the guidance gesture entry coordinate to the latest guidance gesture end coordinate, thus obtaining a stable interactive guidance video). In this way, through uniform interpolation, the guidance gestures on the previous interactive guidance video can be stably transitioned to the next interactive guidance video, thereby achieving the stability of the interactive guidance video.

[0071] Optionally, after determining the overlapping surface of the device area indicated by the starting position coordinates and the device area indicated by the latest position coordinates in the same coordinate system, the method further includes: if the overlapping area is greater than or equal to a preset area threshold, determining that the initial interactive guidance video is in a stable state; and characterizing the initial interactive guidance video as a stable interactive guidance video.

[0072] In this embodiment of the invention, if the overlapping area is greater than or equal to a preset area threshold (for example, the overlapping area of ​​S1 and S2 is greater than or equal to 90%), it can be determined that the initial interactive guidance video is in a stable state, and the initial interactive guidance video can be directly transmitted to the AR glasses as a stable interactive guidance video for display on the display module of the AR glasses.

[0073] In this embodiment of the invention, the stabilized interactive guidance video is transmitted to AR glasses for display on the display module of the AR glasses, providing maintenance guidance for maintenance personnel wearing AR glasses.

[0074] The following describes in detail another optional implementation method.

[0075] Figure 3 This is a schematic diagram of an optional relay-protected AR glasses remote-guided image stabilization system according to an embodiment of the present invention, such as... Figure 3 As shown, it includes: an AR glasses system, an expert working terminal, and a video dynamic stabilization edge computing server. The AR glasses system includes: a camera module, a data transmission module, and a display module; the expert working terminal includes: a touch screen module and a motion capture module.

[0076] In this embodiment of the invention, the camera module can acquire real-time video data (i.e., on-site video data) of the relay protection maintenance site. The data transmission module can send the on-site video data to the video dynamic stabilization edge computing server and the expert operation terminal via 5G or wireless network. In this way, the expert operation terminal can display this on-site video data and generate guidance gesture data based on this on-site video data, and the video dynamic stabilization edge computing server can overlay the guidance gesture data onto the on-site video data and stably generate interactive guidance video.

[0077] In this embodiment, the display module can display a stable interactive guidance video, which can reduce the discomfort caused by shaking for maintenance personnel wearing AR glasses.

[0078] In this embodiment of the invention, a touchscreen is used to display on-site video data, enabling back-end experts to provide interactive guidance (including gesture guidance) based on the displayed real-time on-site video. The motion capture module can extract the expert's guidance gestures from the background to obtain guidance gesture data. This guidance gesture data is generated by the expert providing maintenance guidance to the maintenance equipment displayed in the real-time on-site video data. Optionally, to ensure extraction effectiveness, a green table mat can be used as the background.

[0079] In this embodiment of the invention, the video dynamic stabilization edge computing server can overlay the guidance gestures in the guidance gesture data onto the on-site video data collected by the AR glasses camera in the corresponding frame, and save the overlaid interactive guidance video according to a preset duration, and redetermine the latest position coordinates of the device area after each preset duration. Then, it determines the overlap area of ​​the device area indicated by the start position coordinates and the device area indicated by the latest position coordinates in the same coordinate system. If the overlap area is less than a preset area threshold, it indicates that the current interactive guidance video may be unstable and needs to be processed. An interpolation uniformity algorithm can be used to draw the guidance gestures from the guidance gesture entry coordinate to the guidance gesture end coordinate onto the interval from the guidance gesture entry coordinate to the latest guidance gesture end coordinate, thus obtaining a stable interactive guidance video. In this way, through interpolation uniformity, the guidance gestures on the previous interactive guidance video can be stably transitioned to the next interactive guidance video, thereby achieving the stabilization of the interactive guidance video.

[0080] In this embodiment of the invention, when an expert's gesture enters an area requiring interactive guidance, the coordinates of the expert's gesture and the coordinates of the area where the equipment to be inspected is located, as seen by the maintenance personnel through AR glasses, can be recorded. Then, by superimposing the expert's gesture onto the on-site video captured by the AR glasses camera in the corresponding frame, an interactive guidance video of a preset duration can be obtained. Furthermore, an interpolation uniformity algorithm is used to stabilize the interactive guidance video, thereby reducing the discomfort caused by shaking for maintenance personnel wearing AR glasses.

[0081] The following is a detailed description with reference to another embodiment.

[0082] Example 2

[0083] The image stabilization device based on AR glasses remote guidance provided in this embodiment includes multiple implementation units, each of which corresponds to a specific implementation step in Embodiment 1 above.

[0084] Figure 4 This is a schematic diagram of an optional image stabilization device based on remote guidance from AR glasses according to an embodiment of the present invention, such as... Figure 4 As shown, the image stabilization device may include: a first determining unit 40, a generating unit 41, a second determining unit 42, and a processing unit 43.

[0085] The first determining unit 40 is used to determine the coordinates of the guiding gesture when in interactive guidance mode;

[0086] The generation unit 41 is used to acquire guidance gesture data and on-site video data, and generate an initial interactive guidance video of a preset duration based on the guidance gesture data, on-site video data and guidance gesture entry coordinates. The start time of the initial interactive guidance video is the start time of the interactive guidance mode or the start time of each preset duration. Each frame of interactive guidance image in the initial interactive guidance video includes the equipment area where the target maintenance equipment is located and the guidance gesture.

[0087] The second determining unit 42 is used to determine the end coordinates of the guidance gesture and the latest position coordinates of the device area when the duration of each interactive guidance reaches the preset duration.

[0088] The processing unit 43 is used to process the initial interactive guidance video based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates of the device region on the first frame of the initial interactive guidance video to obtain a stable interactive guidance video.

[0089] The aforementioned image stabilization device can determine the entry coordinates of the guidance gesture by the first determining unit 40 when in interactive guidance mode, acquire guidance gesture data and on-site video data by the generating unit 41, and generate an initial interactive guidance video of a preset duration based on the guidance gesture data, on-site video data, and guidance gesture entry coordinates. The second determining unit 42 determines the end coordinates of the guidance gesture and the latest position coordinates of the device area each time the interactive guidance duration reaches the preset duration. The processing unit 43 processes the initial interactive guidance video based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates of the device area on the first frame of the interactive guidance image in the initial interactive guidance video to obtain a stable interactive guidance video. In this embodiment of the invention, if it is determined that the current state is in interactive guidance mode, the coordinates of the guidance gesture entry are determined, and based on the guidance gesture entry coordinates, the guidance gesture data is superimposed on the on-site video data to generate an initial interactive guidance video of a preset duration. After each interactive guidance duration reaches the preset duration, the coordinates of the guidance gesture end and the latest position coordinates of the equipment area where the target maintenance equipment is located are determined. Then, based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates of the equipment area on the first frame of the interactive guidance image in the initial interactive guidance video, the initial interactive guidance video is processed to obtain a stable interactive guidance video. By processing the interactive guidance video of the preset duration, not only can the amount of computation be reduced, but the continuity of video frames can also be improved, effectively achieving the stability of the interactive guidance video. This solves the technical problem in related technologies where the interactive guidance video cannot maintain stability when providing remote guidance through AR glasses, and achieves the technical effect of maintaining the stability of the interactive guidance video when providing remote guidance through AR glasses.

[0090] Optionally, the image stabilization device further includes: a first determining module for determining an interactive guidance area; a second determining module for determining to enter an interactive guidance mode when the guiding gesture of the guided object is detected to enter the interactive guidance area; and a third determining module for determining to exit the interactive guidance mode when the guiding gesture of the guided object is detected to leave the interactive guidance area.

[0091] Optionally, the image stabilization device further includes: a fourth determining module, used to determine the moment of entry of the guiding gesture before generating an initial interactive guiding video of a preset duration based on the guiding gesture data, on-site video data, and the guiding gesture entry coordinates; and a fifth determining module, used to determine the starting position coordinates of the equipment area where the target maintenance equipment is located at the guiding gesture entry moment based on the on-site video data.

[0092] Optionally, the generation unit includes: a first extraction module, used to extract the guidance gesture image at each moment within a preset duration from the guidance gesture data, wherein each moment corresponds to a frame image in the on-site video data, and each frame image in the on-site video data contains a device area; a first overlay module, used to overlay the guidance gesture image at each moment onto the image in the corresponding frame of the on-site video data based on the guidance gesture entry coordinates and start position coordinates, to obtain multiple frames of interactive guidance images; and a first generation module, used to generate an initial interactive guidance video based on the multiple frames of interactive guidance images.

[0093] Optionally, the image stabilization device further includes: a sixth determining module, configured to determine the latest guiding gesture entry coordinates in the device area indicated by the latest position coordinates based on the position of the guiding gesture entry coordinates in the device area indicated by the start position coordinates after determining the latest position coordinates of the device area; and a seventh determining module, configured to determine the latest guiding gesture end coordinates in the device area indicated by the latest position coordinates based on the position of the guiding gesture end coordinates in the device area indicated by the start position coordinates.

[0094] Optionally, the processing unit includes: an eighth determining module, used to determine the overlapping area of ​​the device area indicated by the start position coordinates and the device area indicated by the latest position coordinates in the same coordinate system; and a first drawing module, used to, when the overlapping area is less than a preset area threshold, use an interpolation uniform algorithm to draw the guidance gesture from the guide gesture entry coordinate to the guide gesture end coordinate onto the interval from the guide gesture entry coordinate to the latest guide gesture end coordinate, so as to obtain a stable interactive guidance video.

[0095] Optionally, the image stabilization device further includes: a ninth determining module, used to determine that the initial interactive guidance video is in a stable state after determining the overlapping surface of the device area indicated by the start position coordinates and the device area indicated by the latest position coordinates in the same coordinate system, provided that the overlapping area is greater than or equal to a preset area threshold; and a first characterization module, used to characterize the initial interactive guidance video as a stable interactive guidance video.

[0096] The image stabilization device described above may also include a processor and a memory. The first determining unit 40, the generating unit 41, the second determining unit 42, the processing unit 43, etc., are all stored in the memory as program units, and the processor executes the program units stored in the memory to realize the corresponding functions.

[0097] The aforementioned processor contains a kernel, which retrieves the corresponding program units from memory. One or more kernels can be configured. By adjusting kernel parameters, the initial interactive guidance video is processed based on the device region's start coordinates, latest coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates on the first frame of the initial interactive guidance video to obtain a stable interactive guidance video.

[0098] The aforementioned memory may include non-permanent memory in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash memory, and the memory includes at least one memory chip.

[0099] This invention also provides a computer program product, which, when executed on a data processing device, is suitable for executing an initialization program with the following method steps: In an interactive guidance mode, determining the coordinates of the guidance gesture entry, acquiring guidance gesture data and on-site video data, and generating an initial interactive guidance video of a preset duration based on the guidance gesture data, on-site video data, and guidance gesture entry coordinates; determining the guidance gesture end coordinates and the latest position coordinates of the device area each time the interactive guidance duration reaches the preset duration; and processing the initial interactive guidance video based on the start position coordinates, latest position coordinates, guidance gesture entry coordinates, and guidance gesture end coordinates of the device area on the first frame of the interactive guidance image in the initial interactive guidance video to obtain a stable interactive guidance video.

[0100] According to another aspect of the present invention, a computer-readable storage medium is also provided, the computer-readable storage medium including a stored computer program, wherein, when the computer program is executed, it controls the device where the computer-readable storage medium is located to perform the above-described image stabilization method based on remote guidance from AR glasses.

[0101] According to another aspect of the present invention, an electronic device is also provided, including one or more processors and a memory, the memory being used to store one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors cause the one or more processors to implement the above-described image stabilization method based on remote guidance from AR glasses.

[0102] Figure 5 This is a hardware structure block diagram of an electronic device (or mobile device) for an image stabilization method based on remote guidance for AR glasses, according to an embodiment of the present invention. Figure 5 As shown, an electronic device may include one or more processors (e.g., Figure 5The processors 502a, 502b, ..., 502n, etc., may include, but are not limited to, processing devices such as microprocessors (MCUs) or programmable logic devices (FPGAs), and a memory 505 for storing data. In addition, it may include: a display, an input / output interface (I / O interface), a universal serial bus (USB) port (which may be included as one of the ports of the I / O interface), a network interface, a keyboard, a power supply, and / or a camera. Those skilled in the art will understand that... Figure 5 The structure shown is for illustrative purposes only and does not limit the structure of the electronic device described above. For example, the electronic device may also include components that are more... Figure 5 The more or fewer components shown, or having the same Figure 5 The different configurations shown.

[0103] The sequence numbers of the above embodiments of the present invention are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0104] The embodiments or examples disclosed herein are not exhaustive, but merely illustrative of some embodiments or examples, and are not intended to limit the scope of protection of this disclosure. Unless otherwise specified, each step in a particular embodiment or example can be implemented as an independent embodiment, and the steps can be arbitrarily combined. For example, a solution after removing some steps in a particular embodiment or example can also be implemented as an independent embodiment, and the order of the steps in a particular embodiment or example can be arbitrarily interchanged. Furthermore, optional methods or examples in a particular embodiment or example can be arbitrarily combined; moreover, embodiments or examples can be arbitrarily combined. For example, some or all steps of different embodiments or examples can be arbitrarily combined, and a particular embodiment or example can be arbitrarily combined with optional methods or examples of other embodiments or examples.

[0105] In the above embodiments of the present invention, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions of other embodiments.

[0106] In the several embodiments provided by this invention, it should be understood that the disclosed technical content can be implemented in other ways. The device embodiments described above are merely illustrative; for example, the division of units can be a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed can be through some interfaces; the indirect coupling or communication connection of units or modules can be electrical or other forms.

[0107] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0108] Furthermore, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0109] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of the present invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0110] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. An image stabilization method based on AR glasses remote guidance, characterized in that, Comprise: In the case of being in the interactive guidance mode, the entering coordinates of the guidance gesture are determined; The guidance gesture data and the live video data are acquired, and based on the guidance gesture data, the live video data and the entering coordinates of the guidance gesture, an initial interactive guidance video with a preset time length is generated, wherein the starting time point of the initial interactive guidance video is the starting time point of the interactive guidance mode or the starting time point of each preset time length, and each frame of interactive guidance image in the initial interactive guidance video contains a device area where the target maintenance device is located and a guidance gesture; In the case that each interactive guidance time length reaches the preset time length, the ending coordinates of the guidance gesture are determined, and the latest position coordinates of the device area are determined; Based on the starting position coordinates of the device area on the first frame of interactive guidance image in the initial interactive guidance video, the latest position coordinates, the entering coordinates of the guidance gesture and the ending coordinates of the guidance gesture, the initial interactive guidance video is processed to obtain a stable interactive guidance video; Wherein, before generating the initial interactive guidance video with a preset time length based on the guidance gesture data, the live video data and the entering coordinates of the guidance gesture, it further comprises: determining the entering time of the guidance gesture; based on the live video data, the starting position coordinates of the device area where the target maintenance device is located at the entering time of the guidance gesture are determined; Wherein, after determining the latest position coordinates of the device area, it further comprises: based on the position of the device area indicated by the starting position coordinates at the entering coordinates of the guidance gesture, the latest entering coordinates of the guidance gesture on the device area indicated by the latest position coordinates are determined; based on the position of the device area indicated by the starting position coordinates at the ending coordinates of the guidance gesture, the latest ending coordinates of the guidance gesture on the device area indicated by the latest position coordinates are determined; Wherein, the step of obtaining the stable interactive guidance video comprises: determining the overlapping area of the device area indicated by the starting position coordinates and the device area indicated by the latest position coordinates in the same coordinate system; in the case that the overlapping area is less than a preset area threshold, the guidance gesture from the entering coordinates of the guidance gesture to the ending coordinates of the guidance gesture is drawn to the interval from the entering coordinates of the guidance gesture to the latest ending coordinates of the guidance gesture by using an interpolation uniform algorithm, to obtain the stable interactive guidance video.

2. The image stabilizing method according to claim 1, characterized by, The image stabilization method further comprises: Determining an interactive guidance area; In the case that the entering guidance gesture of the guidance object into the interactive guidance area is detected, the interactive guidance mode is determined to be entered; In the case that the guidance gesture of the guidance object is detected to leave the interactive guidance area, the interactive guidance mode is determined to be exited.

3. The image stabilizing method according to claim 1, characterized by, The step of generating the initial interactive guidance video with a preset time length based on the guidance gesture data, the live video data and the entering coordinates of the guidance gesture comprises: extracting a guide gesture image of each moment within the preset time length from the guide gesture data, wherein each moment corresponds to a frame of the live video data, and each frame of the live video data contains the device region; superimposing the guide gesture image of each moment into an image in the corresponding frame of the live video data based on the guide gesture entering coordinates and the start position coordinates, to obtain a plurality of interactive guidance images; generating the initial interactive guidance video based on the plurality of interactive guidance images.

4. The image stabilizing method according to claim 1, characterized by, After determining that the overlapping area of the device region indicated by the start position coordinates and the device region indicated by the latest position coordinates is the same in the coordinate system, the method further comprises: determining that the initial interactive guidance video is in a stable state when the overlapping area is greater than or equal to a preset area threshold; representing the initial interactive guidance video as the stable interactive guidance video.

5. An image stabilizing apparatus employing the image stabilizing method using AR glasses remote guidance according to claim 1, characterized by The method comprises: a first determination unit configured to determine guide gesture entering coordinates when in an interactive guidance mode; a generation unit configured to obtain guide gesture data and live video data, and generate an initial interactive guidance video of a preset time length based on the guide gesture data, the live video data, and the guide gesture entering coordinates, wherein a start moment of the initial interactive guidance video is an opening moment of the interactive guidance mode or a starting moment of each preset time length, and each frame of interactive guidance image in the initial interactive guidance video contains a device region of a target maintenance device and a guide gesture; a second determination unit configured to determine guide gesture ending coordinates and a latest position coordinates of the device region when each interactive guidance time length reaches the preset time length; a processing unit configured to process the initial interactive guidance video based on a start position coordinates of the device region on a first frame of interactive guidance image in the initial interactive guidance video, the latest position coordinates, the guide gesture entering coordinates, and the guide gesture ending coordinates, to obtain a stable interactive guidance video.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium comprises a stored computer program, wherein the computer program controls a device in which the computer-readable storage medium is located to execute the image stabilization method based on AR glasses remote guidance according to any one of claims 1 to 4 when the computer program is running.

7. An electronic device, comprising: The device comprises one or more processors and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to implement the image stabilization method based on AR glasses remote guidance according to any one of claims 1 to 4.

Images