A method for generating and interacting with remote AR information on and off-line

Abstract

The present invention is a method for generating and interacting with online / offline remote AR information, which relates to the field of remote maintenance technology. The method comprises collecting real-time video through a camera, transmitting the collected video information to a server, and transmitting the collected video information to a display after being processed by the server. After the camera captures the video information on site, the camera transmits it to a graphics correction module on the server in combination with the real-time orientation information of a posture collector, identifies the collected image, calls the corresponding accessory information from a database, combines the identified image information with the real-scene video on site, combines the virtual accessories called from the database with the real-scene on site, and sends it to the display. The method can solve the problem in the prior art that ordinary engineering personnel are unable to perform complex equipment maintenance alone, and can automatically provide technical support and teaching videos to engineering personnel, thereby lowering the entry threshold for engineers and reducing training costs.

Description

Technical Field

[0001] The present invention relates to the technical field of remote maintenance, and in particular to an information generation and interaction method, and in particular to an online / offline remote AR information generation and interaction method. Background Art

[0002] In the field of complex equipment maintenance, many large-scale equipment are installed in the field or remote factories. When engineers repair complex and large-scale equipment, it is difficult to ensure that all aspects are covered. There are few people who are proficient in the equipment, and it is difficult to meet actual needs. Therefore, equipment auxiliary support is needed during maintenance.

[0003] In the existing technology, on-site maintenance personnel generally consult manuals or make inquiries remotely through telephone and video, which has low communication efficiency and unintuitive understanding of on-site problems and maintenance methods. It may even cause secondary damage to the equipment, making it difficult to meet the needs of simplicity and efficiency. Summary of the Invention

[0004] The present invention proposes a method for online / offline remote AR information generation and interaction, which can solve the problem in the existing technology that ordinary engineering personnel are unable to perform complex equipment maintenance alone. It can automatically provide technical support and teaching videos to engineers, lower the entry threshold for engineers, and reduce training costs.

[0005] The technical solutions of the present invention are as follows:

[0006] A method for generating and interacting with online / offline remote AR information includes collecting real-time video through a camera, transmitting the collected video information to a server, and transmitting it to a display after being processed by the server. The method is characterized in that after the camera captures the video information on site, it is combined with the real-time orientation information of the attitude collector and transmitted to the graphics correction module of the server, the collected image is recognized, the corresponding accessory information is called from the database, and the recognized image information is combined with the real-scene video on site, and the virtual accessories called from the database are combined with the real-scene on site and sent to the display.

[0007] As a further optimization of this solution, the implementation of this method also requires the application of a labeling tool. The server places the collected image information in the form of videos and pictures in a memory, and the server calls the labeling tool to label the collected image information. The labeled images and labeling information are used as input for neural network training data, and the server calls the git tool to synchronize the labels with the database.

[0008] As a further optimization of this solution, the annotation tool supports allocation function, which can assign annotation tasks to multiple hosts.

[0009] As a further optimization of this solution, the server monitors the progress of annotation and the time point of annotation in real time. When new annotation data is input or existing annotation data is updated, the server will monitor and update the annotation sample database in real time, and synchronize the annotations in the database with the annotation tool in real time based on the new and old data modification timestamps.

[0010] As a further optimization of this solution, for real-time input remote video image data, the server uses the object detection model to detect objects in the image and uses the image semantic model to generate semantic information for the image;

[0011] The obtained image semantic information is compared with the resource annotations in the database, and the recommendation algorithm is used to obtain a set of resource files most relevant to the image for push. When long text appears, the QA model is used to extract the relevant parts for push.

[0012] As a further optimization of this solution, when the network speed is lower than the preset threshold or the network is disconnected, the AR head-mounted terminal interface uses Qt to implement the interface display framework, and transmits the camera image data to the server in real time through the P2P transmission protocol within the display framework.

[0013] As a further optimization of this solution, the camera's rotation angle is transmitted using a standard serial port RS-422 protocol, using a bidirectional full-duplex asynchronous working mode.

[0014] As a further optimization of this solution, the conversion method between the image information collected by the camera and the real-world image information includes:

[0015] First, the operator's position P in the world coordinate system can be obtained by measuring o , and then obtain the azimuth angle θ and elevation angle data φ through the helmet data, the Z axis of the virtual camera v=(cosθcosφ,sinθcosφ,sinφ) T , the Y axis of the camera Where n represents the Z axis of the world coordinate system, the X axis coordinate of the camera is w = u × v, and the coordinates of the three-dimensional point in the camera coordinate system are P c and the coordinate P in the world coordinate system w The relationship is: P w =[w T ,u T ,v T ]P c +P0, transform the model in the world coordinate system to the virtual camera coordinate system, and generate the final panorama based on the intrinsic parameter projection of the virtual camera.

[0016] As a further optimization of this solution, H.265 audio and video compression codec is adopted during video data transmission.

[0017] The working principle and beneficial effects of the present invention are:

[0018] In this application, on-site maintenance personnel wear corresponding equipment and use cameras to collect image information of the on-site environment and equipment, and transmit it to the graphics system of the server in combination with the azimuth angle information of the attitude collector. The graphics correction module of the server labels, extracts and analyzes the equipment based on the azimuth information of the attitude collector and the image information, and obtains a complete spatial image of the equipment based on a large amount of image information. The same method is used to present the on-site environmental information, combining the equipment with the environment, and identifying the equipment's accessories, functions and other information based on the analyzed image information. After confirming the corresponding accessories, the accessory information is marked in the output video and presented to the operator in the virtual video based on the actual position of the equipment and environment.

[0019] The application of this method can meet the needs of remote maintenance assistance. AR technology is used to generate a virtual reality scene by combining the captured real-life video images and related multimedia files (text, voice, pictures, videos and 3D models) in the database. The attributes of objects in the video of the head-end terminal are identified, and then the attribute identifiers are integrated with the real video scene to achieve the generation of virtual-reality interactive scenes and complete the enhancement of the real scene on site. The information interaction between the front-end and the server is completed through the H.265 compression encoding method, so that on-site maintenance personnel can obtain maintenance suggestions and guidance pushed by the back-end server expert system in real time. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0021] Figure 1 This is a structural diagram of this application;

[0022] Figure 2 is the virtual camera coordinate diagram in this application;

[0023] Figure 3 The following is a comparison chart of the effects of various compression modes;

[0024] Figure 4 It is the H.265 encoding framework. DETAILED DESCRIPTION

[0025] The following will be combined with the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the embodiments described are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of the present invention.

[0026] As the instruction manual Figure 1 As shown, a method for generating and interacting with remote AR information online / offline includes collecting real-time video through a camera, transmitting the collected video information to a server, and transmitting it to a display after being processed by the server. After the camera captures the video information on site, it is combined with the real-time orientation information of the attitude collector and transmitted to the graphics correction module of the server. The collected image is recognized, the corresponding accessory information is called from the database, and the recognized image information is combined with the real-scene video on site. The virtual accessories called from the database are combined with the real-scene on site and sent to the display.

[0027] This system consists of four modules: annotation tools, database, back-end services, and front-end interface.

[0028] The annotation tool facilitates user annotation of images and videos, supporting a combination of manual and automatic annotation. The database module is responsible for storing and managing various media data and their annotation information. The backend service module is responsible for data exchange with the other modules, detecting and recognizing input images, analyzing image recognition results, and automatically searching the database to integrate and push relevant information to the front-end interface. The front-end interface module generates virtual scenes based on the data pushed by the backend service and pushes them to the AR headset interface.

[0029] Virtual scene generation technology is a key technology for overlaying virtual and real scenes, enhancing real-world video images. It has long been a hot topic in the field of vision research. It utilizes computer graphics to achieve the simultaneous display of real and virtual scenes. On the embedded side, this technology is implemented through a multi-layer overlay. Attributes associated with objects identified in the real scene are displayed on a layer overlaid with the real scene. Because scene data includes text, images, video, audio, and other related data, the project processes each type of information separately to achieve the fusion of virtual information and real scenes. Because the generated virtual and real scenes meet the viewing needs of the human eye, the user experience is positive.

[0030] To enhance the sense of reality, given the increased computing power of embedded systems, in addition to placing virtual objects at target locations in the real environment, 3D modeling can also be used to integrate virtual objects into the real world. This allows for consistent feedback based on real-world lighting conditions, rendering the desired lighting effects on the virtual objects' surfaces. On the server side, differentiated stereoscopic display implementations and helmet-mounted binocular displays enable real-time, real-world 3D effects.

[0031] Media VR information primarily consists of real-time video captured by the camera and buttons, menus, and text displays required for user interaction. The system creates a spatial imaging stereo model based on the camera's real-time imaging model and displays it on the helmet-mounted display.

[0032] This method also requires the use of a labeling tool. The server stores the collected image information in the form of videos and pictures in a memory. The server calls the labeling tool to label the collected image information, and uses the labeled images and labeling information as input for neural network training data. The server calls the git tool to synchronize the labeling with the database. The labeling tool supports an allocation function, which can allocate labeling tasks to multiple hosts.

[0033] The server monitors the progress of annotation and the time point of annotation in real time. When new annotation data and updated data are input, the server monitors the changes in the database in real time and synchronizes the annotations in the database with the annotation tool in real time based on the new and old data modification timestamps.

[0034] The image data used in the research needs to be labeled. The labeled images and annotation information are used as input for neural network training data. Use labelme annotation to create customized annotation tasks and perform image annotation to label the images.

[0035] After running Labelme in pycharm, Labelme is an open source image annotation tool that includes the graphics development tool QT Designer. It is a graphics development tool that can be integrated into pycharm. It generates ui files, which can be converted into py files. Combined with pycharm, it can be used for graphical interface development.

[0036] The annotation tool supports a variety of annotation methods, including boxes, polygons, polylines, points, cuboids, and labels (whole-frame annotation). Annotation entries support adding attributes, and the attribute types support four types: multiple selection, single selection (true / false), text, and numbers.

[0037] After annotation is complete, the annotation tool synchronizes the annotations with the database through backend processing using Git. The tool also supports task assignment, allowing annotation tasks to be assigned to multiple people and monitoring annotation progress and speed. For large amounts of data, the tool offers automatic annotation, leveraging pre-trained models to automatically annotate images, videos, and more, allowing for manual fine-tuning of the annotations.

[0038] The AR headset itself relies on hardware head-mounted devices. The AR headset terminal module consists of the headset itself, a core control board, and a display terminal. It features video transmission, local object detection, voice playback, and local display. It can render and layout text, images, and videos based on different data file formats, and combine the input image's object detection results to generate a virtual scene.

[0039] In standalone mode, the AR headset terminal interface uses Qt to implement the interface display framework, and within the display framework, camera image data is transmitted to the server in real time via the P2P transmission protocol. During the research process, the target detection and classification functions were implemented in the core processing board of the helmet through model quantization and embedded board optimization technology. The local detection process can improve the availability of the detection and classification process under low-speed networks, and cooperate with the interface display framework to realize the rendering and image overlay functions of the virtual scene;

[0040] In the middleware mode, the virtual scene is pushed to the head-mounted display system in the actual head-mounted system data format for display.

[0041] For real-time input remote video image data, the server uses the object detection model to detect objects in the image and uses the image semantic model to generate semantic information for the image;

[0042] The obtained image semantic information is compared with the resource annotations in the database, and the recommendation algorithm is used to obtain a set of resource files most relevant to the image for push. When long text appears, the QA model is used to extract the relevant parts for push.

[0043] When the network speed is lower than the preset threshold or the network is disconnected, the AR head-mounted terminal interface uses Qt to implement the interface display framework, and transmits the camera image data to the server in real time through the P2P transmission protocol within the display framework.

[0044] In standalone mode, the AR headset terminal interface uses Qt to implement the interface display framework, and within the display framework, camera image data is transmitted to the server in real time via the P2P transmission protocol. During the research process, the target detection and classification functions were implemented in the core processing board of the helmet through model quantization and embedded board optimization technology. The local detection process can improve the availability of the detection and classification process under low-speed networks, and cooperate with the interface display framework to realize the rendering and image overlay functions of the virtual scene;

[0045] In the middleware mode, the virtual scene is pushed to the head-mounted display system in the actual head-mounted system data format for display.

[0046] As the instruction manual Figure 2 As shown, the rotation angle of the camera is transmitted using a standard serial port RS-422 protocol, using a bidirectional full-duplex asynchronous working mode.

[0047] When linked with the helmet, a serial port standard protocol is used to transmit the rotation angle of the helmet, and a virtual camera is used to realize interaction with VR multimedia information based on the scene projection model.

[0048] The communication protocol uses a fixed serial port. It uses the standard RS-422 protocol, bidirectional, full-duplex, asynchronous operation, a 40ms communication cycle, a baud rate of 115200 bit / s, 1 start bit, 8 data bits, 1 odd parity bit, and 2 stop bits. The specific data format can be defined separately.

[0049] The conversion method between the image information collected by the camera and the real-world image information includes:

[0050] First, the operator's position P in the world coordinate system can be obtained by measuring o , and then obtain the azimuth angle θ and elevation angle data φ through the helmet data, the Z axis of the virtual camera v=(cosθcosφ,sinθcosφ,sinφ) T , the Y axis of the camera Where n represents the Z axis of the world coordinate system, the X axis coordinate of the camera is w = u × v, and the coordinates of the three-dimensional point in the camera coordinate system are P c and the coordinate P in the world coordinate system w The relationship is: P w =[w T ,u T ,v T ]P c +P0, transform the model in the world coordinate system to the virtual camera coordinate system, and generate the final panorama based on the intrinsic parameter projection of the virtual camera.

[0051] In OpenGL, we place the surface model in the world coordinate system. We can then create a virtual camera to observe the surface model in the world coordinate system. Using the helmet wearer's eyes as a virtual camera, we can read the data from the helmet to obtain the operator's head rotation data and modify the definition of the virtual camera coordinate system accordingly.

[0052] We use the following method to estimate the transformation from the world coordinate system to the camera coordinate system. First, we can obtain the position P of the operator in the world coordinate system by measuring o , Secondly, we obtain the azimuth angle θ and elevation angle data φ through the helmet data. Figure 2 As shown, the axis pointing to the star symbol is the Z axis of the virtual camera v = (cosθcosφ, sinθcosφ, sinφ) T The camera's Y axis Where n represents the Z axis of the world coordinate system. The X axis coordinate of the camera is w = u × v. The coordinates P of the three-dimensional point in the camera coordinate system are c and the coordinate P in the world coordinate system w The relationship is: P w =[w T ,u T ,vT ]P c +P0

[0053] We transform the model in the world coordinate system to the virtual camera coordinate system and generate the final panorama based on the intrinsic projection of the virtual camera.

[0054] During the video data transmission process, H.265 audio and video compression codec is used.

[0055] H.265, also known as HEVC, is a video compression standard designed for the next generation of high-resolution video. The H.265 standard revolves around the existing video coding standard H.264, retaining some of the original technologies while improving some related technologies. The new technology uses advanced technology to improve the relationship between bitrate, encoding quality, latency and algorithm complexity to achieve optimal settings. Specific research contents include: improving compression efficiency, improving robustness and error recovery capabilities, reducing real-time latency, reducing channel acquisition time and random access latency, and reducing complexity. H.265 can achieve 720P (resolution 1280×720) ordinary high-definition audio and video transmission using a transmission speed of 1-2Mbps.

[0056] H.265 shifts the task of image segmentation from manual user settings to the encoder, allowing it to split the image into coding tree units (CTUs) in sizes such as 16×16, 32×32, or 64×64. Generally speaking, larger block sizes improve compression efficiency. Through a series of optimizations, H.265's video encoding and compression technology can halve the file size and download time of movies with the same image quality, providing a more comprehensive video compression solution for network bandwidth.

[0057] As the instruction manual Figure 3 and 4 As shown in the figure, by comparing various compression modes, it can be seen that the H.265 compression mode has good compression characteristics.

[0058] As shown in the accompanying figure, H.265 reuses many concepts defined in H.264. Both are block-based video coding technologies, so they have the same roots and similar coding methods, including:

[0059] Subdivide the picture into macroblocks and eventually into blocks;

[0060] Use intra-frame compression technology to reduce spatial redundancy;

[0061] Use inter-frame compression techniques to reduce temporal redundancy (motion estimation and compensation);

[0062] Use transformation and quantization to perform residual data compression;

[0063] • Use entropy coding to reduce eventual redundancy in residual and motion vector transmission and signaling.

[0064] The overall coding framework of H.265 can be divided into four parts: (1) prediction part, (2) transformation and quantization, (3) a complete decoder, and (4) entropy coding.

[0065] As the H.265 encoding method continues to improve, its application is becoming more and more extensive.

[0066] The above are only preferred embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A method for generating and interacting with AR information remotely online or offline, comprising collecting real-time video through a camera, transmitting the collected video information to a server, and transmitting the collected video information to a display after being processed by the server, characterized in that: After the camera captures the video information on site, it combines the real-time orientation information of the attitude collector and transmits it to the graphics correction module on the server side, recognizes the captured image, calls the corresponding accessory information from the database, and combines the recognized image information with the real scene video on site. The virtual accessory information called from the database is combined with the real scene on site and sent to the display. The conversion method between the image information captured by the camera and the real-world image information includes: first, the position P of the operator in the world coordinate system can be obtained by measuring o , and then obtain the azimuth angle θ and elevation angle data through the helmet data The Z axis of the virtual camera The camera's Y axis Where n represents the Z axis of the world coordinate system, the X axis coordinate of the camera is w = u × v, and the coordinates of the three-dimensional point in the camera coordinate system are P c and the coordinate P in the world coordinate system w The relationship is: P w =[w T ,u T ,v T ]P c +P0, transforms the model in the world coordinate system to the virtual camera coordinate system, and generates the final panoramic image based on the internal parameter projection of the virtual camera; for the real-time input remote video image data, the server uses the target detection model to detect the target in the image, and uses the image semantic model to generate semantic information for the image; the obtained image semantic information is compared with the resource annotations in the database, and the recommendation algorithm is used to obtain a set of resource files most relevant to the image for push. When long text appears, the QA model is used to extract the relevant part for push.

2. The method for generating and interacting with online / offline remote AR information according to claim 1, characterized in that: This method also requires the use of a labeling tool. The server stores the collected image information in the form of videos and pictures in a memory. The server calls the labeling tool to label the collected image information, and uses the labeled images and labeling information as input for neural network training data. The server calls the git tool to synchronize the labels with the database.

3. The method for generating and interacting with online / offline remote AR information according to claim 2, characterized in that: The annotation tool supports the allocation function, which can assign annotation tasks to multiple hosts.

4. The method for online / offline remote AR information generation and interaction according to claim 3, characterized in that: The server monitors the progress of annotation and the time point of annotation in real time. When new annotation data and updated data are input, the server monitors the changes in the database in real time and synchronizes the annotations in the database with the annotation tool in real time based on the new and old data modification timestamps.

5. The method for online / offline remote AR information generation and interaction according to claim 1, characterized in that: When the network speed is lower than the preset threshold or the network is disconnected, the AR head-mounted terminal interface uses Qt to implement the interface display framework, and transmits the camera image data to the server in real time through the P2P transmission protocol within the display framework.

6. The method for generating and interacting with online / offline remote AR information according to claim 1, characterized in that: The camera's rotation angle is transmitted using a standard serial port RS-422 protocol, using a bidirectional full-duplex asynchronous working mode.

7. The method for generating and interacting with online / offline remote AR information according to claim 1, characterized in that: During the video data transmission process, H.265 audio and video compression codec is used.

Images