Rail transit train remote maintenance interaction method based on AR glasses

Abstract

The invention belongs to the technical field of AR equipment, and particularly relates to a rail transit train remote overhaul interaction method based on AR glasses, which comprises the following steps: capturing overhaul video data and operation and maintenance process video data of an on-site rail transit train by using the AR glasses, providing a remote VPN channel by using a computer, and after the AR glasses are connected to the Internet, sending the overhaul video data and the operation and maintenance process video data to the remote VPN channel; transmitting the captured video data from an operator end of the remote control software to an expert end of the remote control software; and the background monitors the rail transit train maintenance site in real time according to the site maintenance video data received by the remote control software expert end, and establishes real-time interaction between the site and background experts through the AR glasses and the remote control software. An existing maintenance method based on AR equipment lacks real-time performance, and experts cannot provide real-time guidance, and according to the rail transit train remote maintenance interaction method based on the AR glasses, AR and remote control are combined, the real-time performance of interaction is greatly improved, and the equipment maintenance and operation efficiency is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of AR equipment, and in particular relates to an interactive method for remote maintenance of rail transit trains based on AR glasses. Background Art

[0002] Expert guidance is often indispensable during equipment inspection and maintenance. However, equipment manufacturers and equipment users are often not in the same region, and it is time-consuming, labor-intensive, and costly for equipment manufacturers’ experts to rush to the site for guidance. When the site is geographically limited or encounters emergency troubleshooting, it is impossible to obtain expert on-site guidance in a timely manner.

[0003] Augmented Reality (AR) technology can provide an immersive guidance experience by superimposing virtual information on the real world. As an advanced display device, AR glasses can directly display virtual content in the user's field of view, thereby enhancing the interactivity and effectiveness of inspection and maintenance. However, the existing remote guidance solutions based on AR glasses still have certain shortcomings, especially in terms of real-time remote expert guidance.

[0004] The patent document with publication number "CN110324560A" discloses "a remote video guidance maintenance system", which uses a camera to capture on-site scenes, realizes communication between the front and rear, and helps on-site personnel complete complex operations. However, it relies on video conferencing software such as Zoom, Microsoft Teams or dedicated traditional video call software, and lacks the immersive experience and interactivity of augmented reality.

[0005] The patent document with publication number "CN110349467A" discloses "an augmented reality-based general aviation maintenance training system and method", which uses AR technology for immersive maintenance training. Trainees understand the location and function of each component by seeing the virtual model of the equipment. However, these trainings are usually preset training content and cannot respond to trainees' needs in real time. Summary of the invention

[0006] In view of the shortcomings of the above-mentioned existing methods, the present invention provides an interactive method for remote maintenance of rail transit trains based on AR glasses, aiming to solve the core problem that traditional technical solutions lack real-time interaction and immersive experience.

[0007] To achieve the above-mentioned purpose, the present invention is implemented by the following technical solutions: A rail transit train remote maintenance interactive method based on AR glasses, the method comprising the following steps: Step S10: using AR glasses to capture on-site rail transit train inspection video data and operation and maintenance process video data, using a computer to provide a remote VPN channel, so that the AR glasses are connected to the Internet, and the captured on-site inspection video data and operation and maintenance process video data are transmitted from the operator end of the remote control software to the expert end of the remote control software; Step S20: The backend monitors the rail transit train maintenance site in real time based on the on-site maintenance video data received by the remote control software expert terminal, and establishes real-time interaction between the AR glasses site and the backend experts through the remote control software; Among them, the on-site maintenance video data of rail transit trains and the video data of the operation and maintenance process are recorded by the remote control software operator end and saved by the remote operation software expert end.

[0008] Furthermore, the AR glasses in step S10 include: Flash light: used to increase the brightness of the rail transit train maintenance site; Camera: used to capture video data at the rail transit train maintenance site; Microphone and speaker: used to capture voice data at the rail transit train maintenance site and realize voice intercom; Hardware system: used to transmit the captured video data and voice data to the remote control software via a computer connected to the Internet.

[0009] Furthermore, in step S10, the operation of the AR glasses is realized by the AR glasses APP, and the use process of the AR glasses is as follows: Get the wireless network IP address by configuring the wireless connection; Connect the AR glasses to the computer according to the obtained wireless network IP address; After the connection is successful, download the AR glasses APP in the AR glasses; After the download is complete, the APP starts and generates a UID account based on the current AR glasses device and joins the network; After successfully joining the network, when the remote device initiates a video call, choose to accept or reject; After receiving the video initiated by the remote device, the AR glasses display the camera shooting screen and display tool buttons on the screen, including voice call, flashlight, zoom in, zoom out and close. After the remote video ends, the APP interface returns to the state of successfully joining the network.

[0010] Furthermore, in step S10, the steps of establishing a connection between the expert end and the operator end of the remote control software are: When the expert end initiates a remote control request command through the IP address, the operator end accepts the request command and returns the connection address and authorization code command. The expert end establishes a remote connection with the operator end through the connection address and authorization code. The operator end returns a success callback command, and the remote control connection is successfully established; When the expert side initiates a voice call request command through the IP address, the operator side accepts the request command and returns the connection address and authorization code command. The expert side establishes a voice call connection with the operator side through the connection address and authorization code. The operator side returns a success callback, and the two-way voice call is successfully established.

[0011] Furthermore, in step S20, the background expert instructs the on-site operator by superimposing virtual guidance information in real time on the AR glasses screen through the expert end of the remote control software. The interactive operation steps between the expert end and the operator end of the remote control software are as follows: The expert side or the operator side initiates a whiteboard operation command to pause the AR glasses video screen and display a whiteboard in the AR glasses video screen. The background experts and on-site operators mark virtual guidance information on the rail transit train maintenance operation steps on the whiteboard, and the expert side and the operator side synchronize the whiteboard annotation content in real time; the expert side initiates a focus adjustment command, the operator side accepts the focus adjustment command, responds to the focus change, and the rail transit train maintenance on-site video data changes synchronously.

[0012] The virtual guidance information for the rail transit train maintenance operation steps marked on the whiteboard by the back-end experts and on-site operators includes virtual information such as instruction arrows, text boxes, text descriptions and digital labels.

[0013] In addition, during the equipment inspection and maintenance process, the data analysis module in the remote control software records and saves the video, and analyzes and generates operation reports to improve the guidance methods and content. The specific operation steps are as follows: After the connection between the operator side and the expert side is established, the operator side automatically runs the video recording command to record the video screen of the rail transit train maintenance site. After the recording is completed, the video data of the rail transit train maintenance site is synchronized to the expert side for storage.

[0014] In addition, to achieve the above-mentioned purpose, the present invention further proposes a rail transit train remote maintenance interactive device based on AR glasses, and the rail transit train remote maintenance interactive device based on AR glasses includes: AR glasses module: used to capture on-site video data and transmit the captured on-site video data to the remote control software via a computer connected to the Internet; Backstage guidance module: used to monitor the site in real time and guide personnel to repair equipment; Interaction module: used to establish real-time interaction between the site and the backstage; Data analysis module: used to record and record the equipment inspection and operation and maintenance process, generate operation reports and feedback, and help the backend improve guidance content and methods.

[0015] In addition, to achieve the above-mentioned purpose, the present invention also proposes a rail transit train remote maintenance interaction device based on AR glasses, and the rail transit train remote maintenance interaction device based on AR glasses includes: a memory, a processor, and a rail transit train remote maintenance interaction program based on AR glasses stored on the memory and runnable on the processor. When the rail transit train remote maintenance interaction program based on AR glasses is executed by the processor, the steps of the rail transit train remote maintenance interaction method based on AR glasses as described above are implemented.

[0016] The advantages and effects of the present invention are: The method of the present invention improves the traditional remote maintenance interaction method, and uses AR glasses technology to enhance the immersive experience of the traditional rail train remote maintenance interaction method. While retaining a good experience, it also greatly increases the convenience of interaction. The rear experts can provide immersive operation guidance to the operators at the maintenance site in real time, thereby improving the efficiency of equipment maintenance and the effect of operation and maintenance. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 The present invention is a flowchart of an interactive method for remote maintenance of rail transit trains based on AR glasses.

[0019] Figure 2 It is a schematic diagram of the operation flow of the AR glasses APP of the present invention.

[0020] Figure 3 The figure is a remote interaction logic diagram of the remote control software of the present invention.

[0021] Figure 4 It is a schematic diagram of the structure of the rail transit train remote maintenance interactive device based on AR glasses of the present invention.

[0022] Figure 5 This is a schematic block diagram of the structure of the interactive electronic equipment for remote maintenance of rail transit trains based on AR glasses of the present invention. DETAILED DESCRIPTION

[0023] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0024] The present invention provides a rail transit train remote maintenance interactive method based on AR glasses, the method is composed as follows Figure 1 As shown, the following steps are included: Step S10: using AR glasses to capture on-site rail transit train inspection video data and operation and maintenance process video data, using a computer to provide a remote VPN channel, so that the AR glasses are connected to the Internet, and the captured on-site inspection video data and operation and maintenance process video data are transmitted from the operator end of the remote control software to the expert end of the remote control software; Step S20: The backend monitors the rail transit train maintenance site in real time based on the on-site maintenance video data received by the remote control software expert terminal, and establishes real-time interaction between the AR glasses site and the backend experts through the remote control software; Among them, the on-site maintenance video data of rail transit trains and the video data of the operation and maintenance process are recorded by the remote control software operator end and saved by the remote operation software expert end.

[0025] The AR glasses in step S10 include: Flashlight: used to increase the brightness of the rail transit train maintenance site and can maintain lighting for a long time; Camera: used to capture video data of rail transit train maintenance sites, with anti-shake recording function, shooting resolution of 1080p or 2k and above, with multiple focal lengths, and also with night vision function; Microphone and speaker: used to capture voice data at the rail transit train maintenance site and realize voice calls. The microphone has a noise reduction function, which can shield some noise at the site, and has good sound collection, recording, dust and moisture resistance functions, etc. The speaker has a good sound release function and has a certain ability to prevent electromagnetic interference; Hardware system: It consists of an embedded computer or mobile device, which is used to transmit the captured video data and voice data of the rail transit train maintenance site to the remote control software through a computer connected to the Internet, and is also used to support the normal use and operation of AR glasses.

[0026] In addition, the operation of AR glasses is realized by the AR glasses pilot APP, and its usage process is as follows: Figure 2 As shown, including: Get the wireless network IP address by configuring the wireless connection; Connect the AR glasses to the computer according to the obtained wireless network IP address; After the connection is successful, download the AR glasses APP in the AR glasses; After the download is complete, the APP starts and generates a UID account based on the current AR glasses device. At the same time, it determines whether this UID account has been added to the network. If it has not been added to the network, the AR glasses interface will prompt: the remote control project has not been added yet, and it will be closed after the UID of the AR device is added to the network; if it has been added to the network, a pop-up window will be displayed when entering this interface for the first time, and a VPN connection request needs to be authorized. After clicking to confirm and accept the authorization, the interface will display the identification code information of the current AR device (consisting of IP address + port), and it is in a state of waiting for a connection request; After successfully joining the network, when the remote device initiates a video call, a pop-up window will pop up on the interface to remind you whether to accept the remote video call request. Select Accept or Reject. When you click Reject, the interface will return to the ID code display interface and resume the state of waiting for the remote video access request. When you click Accept, the remote video connection state will be entered, and the voice call function will be activated by default. After receiving the video initiated by the remote device, the AR glasses display the camera image and tool buttons on the screen, including: Voice call: Enabled by default, click to close to disconnect the voice call; Flashlight: activate the flashlight of the glasses to enhance the brightness of the sight; Zoom in and out: Equivalent to the focusing effect, you can zoom in and out the camera image; Close: Clicking Close is equivalent to actively disconnecting the current remote video connection; After the remote video ends, the APP interface returns to the state of waiting for connection request.

[0027] In addition, the step of establishing a connection between the expert end and the operator end of the remote control software in step S10 is as follows: Figure 3 As shown, specifically including: When the expert end initiates a remote control request command through the IP address, the operator end accepts the request command and returns the connection address and authorization code command. The expert end establishes a remote connection with the operator end through the connection address and authorization code. The operator end returns a success callback command, and the remote control connection is successfully established; When the expert side initiates a voice call request command through the IP address, the operator side accepts the request command and returns the connection address and authorization code command. The expert side establishes a voice call connection with the operator side through the connection address and authorization code. The operator side returns a success callback, and the two-way voice call is successfully established.

[0028] In addition, in step S20, the background expert instructs the on-site operator by superimposing virtual guidance information on the AR glasses screen in real time through the remote control software expert end. The interactive operation steps of the remote control software expert end and the operator end are as follows: Figure 3 As shown, specifically including: The expert side or the operator side initiates a whiteboard operation command to pause the AR glasses video screen and display a whiteboard in the AR glasses video screen. The background experts and on-site operators mark virtual guidance information on the rail transit train maintenance operation steps on the whiteboard, and the expert side and the operator side synchronize the whiteboard annotation content in real time; the expert side initiates a focus adjustment command, the operator side accepts the focus adjustment command, responds to the focus change, and the rail transit train maintenance on-site video data changes synchronously.

[0029] The virtual guidance information for the rail transit train maintenance operation steps marked on the whiteboard by the back-end experts and on-site operators includes virtual information such as instruction arrows, text boxes, text descriptions and digital labels.

[0030] In addition, during the equipment inspection and maintenance process, the data analysis module in the remote control software records and saves the video, and analyzes and generates operation reports to improve the guidance methods and content. The specific operation steps are as follows: After the connection between the operator side and the expert side is established, the operator side automatically runs the video recording command to record the video screen of the rail transit train maintenance site. After the recording is completed, the video data of the rail transit train maintenance site is synchronized to the expert side for storage.

[0031] In addition, the present invention also proposes a rail transit train remote maintenance interactive device based on AR glasses, please refer to Figure 4 , the rail transit train remote maintenance interactive device based on AR glasses includes: AR glasses module: used to capture on-site video data of rail transit train maintenance, and transmit the captured on-site video data to the remote control software through a computer connected to the Internet; Backstage guidance module: used by backstage experts to monitor the rail transit train maintenance site in real time and guide operators to repair rail transit train equipment; Interaction module: used to establish real-time interaction between rail transit train maintenance site and backstage experts; Data analysis module: used to record and record the inspection and maintenance process of rail transit train equipment, generate operation reports and feedback, and help back-end experts improve guidance content and methods.

[0032] The rail transit train remote maintenance interactive device based on AR glasses provided in the present application adopts the rail transit train remote maintenance interactive method based on AR glasses in the above-mentioned embodiment, which can solve the technical problems of the current remote maintenance interactive method lacking interactivity, lacking real-time and insufficient immersion. Compared with the prior art, the beneficial effects of the rail transit train remote maintenance interactive device based on AR glasses provided in the present application are the same as the beneficial effects of the rail transit train remote maintenance interactive method based on AR glasses provided in the above-mentioned embodiment, and the other technical features in the rail transit train remote maintenance interactive device based on AR glasses are the same as the features disclosed in the above-mentioned embodiment method, which will not be repeated here.

[0033] The present application provides an AR glasses-based rail transit train remote maintenance interaction device, and the AR glasses-based rail transit train remote maintenance interaction device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the AR glasses-based rail transit train remote maintenance interaction method in the above-mentioned embodiment one.

[0034] Reference below Figure 5 , which shows a schematic diagram of the structure of the rail transit train remote maintenance interactive device based on AR glasses suitable for implementing the embodiment of the present application. The rail transit train remote maintenance interactive device based on AR glasses in the embodiment of the present application may include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Pesonal Digital Assistants: Personal Digital Assistants), PADs (Portable Application Descriptions: Tablet Computers), PMPs (Portable Media Players: Portable Multimedia Players), etc., and fixed terminals such as digital TVs, desktop computers, etc. Figure 5 The rail transit train remote maintenance interactive device based on AR glasses shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0035] Figure 5The rail transit train remote maintenance interactive device based on AR glasses shown may include a processing device 1001 (such as a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM: Read Only Memory) 1002 or the program loaded from the storage device 1003 to the random access memory (RAM: Random Access Memory) 1004. Various programs and data required for the operation of the rail transit train remote maintenance interactive device based on AR glasses are also stored in RAM1004. The processing device 1001, ROM1002 and RAM1004 are connected to each other through a bus 1005. The input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touch screen, a touchpad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 1003 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 1009. The communication device 1009 can allow the rail transit train remote maintenance interactive device based on AR glasses to communicate wirelessly or wired with other devices to exchange data. Although the figure shows a rail transit train remote maintenance interactive device based on AR glasses with various systems, it should be understood that it is not required to implement or have all the systems shown. More or fewer systems may be implemented or provided alternatively.

[0036] In particular, according to the embodiments disclosed in the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, the embodiments disclosed in the present application include a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device, or installed from a storage device 1003, or installed from a ROM 1002. When the computer program is executed by the processing device 1001, the above-mentioned functions defined in the method of the embodiment disclosed in the present application are executed.

[0037] The rail transit train remote maintenance interactive device based on AR glasses provided in the present application adopts the rail transit train remote maintenance interactive method based on AR glasses in the above-mentioned embodiment, which can solve the technical problems of the current remote maintenance interactive method lacking interactivity, lacking real-time and insufficient immersion. Compared with the prior art, the beneficial effects of the rail transit train remote maintenance interactive device based on AR glasses provided in the present application are the same as the beneficial effects of the rail transit train remote maintenance interactive method based on AR glasses provided in the above-mentioned embodiment, and the other technical features in the rail transit train remote maintenance interactive device based on AR glasses are the same as the features disclosed in the method of the previous embodiment, which will not be repeated here.

[0038] The various parts disclosed in this application can be implemented by hardware, software, firmware or a combination thereof. In the description of the above embodiments, specific features, structures, materials or characteristics can be combined in any one or more embodiments or examples in a suitable manner.

[0039] The above descriptions are only some embodiments of the present application, and are not intended to limit the patent scope of the present application. All equivalent structural changes made using the contents of the present application specification and drawings under the technical concept of the present application, or direct / indirect applications in other related technical fields are included in the patent protection scope of the present application.

Claims

1. A rail transit train remote maintenance interactive method based on AR glasses, characterized in that: The method comprises the following steps: Step S10: using AR glasses to capture on-site rail transit train inspection video data and operation and maintenance process video data, using a computer to provide a remote VPN channel, so that the AR glasses are connected to the Internet, and the captured on-site inspection video data and operation and maintenance process video data are transmitted from the operator end of the remote control software to the expert end of the remote control software; Step S20: The backend monitors the rail transit train maintenance site in real time based on the on-site maintenance video data received by the remote control software expert terminal, and establishes real-time interaction between the AR glasses site and the backend experts through the remote control software; Among them, the on-site maintenance video data of rail transit trains and the video data of the operation and maintenance process are recorded by the remote control software operator end and saved by the remote operation software expert end.

2. According to claim 1, a rail transit train remote maintenance interactive method based on AR glasses is characterized in that: The AR glasses in step S10 include: Flash light: used to increase the brightness of the rail transit train maintenance site; Camera: used to capture video data at the rail transit train maintenance site; Microphone and speaker: used to capture voice data at the rail transit train maintenance site and realize voice intercom; Hardware system: used to transmit the captured video data and voice data to the remote control software via a computer connected to the Internet.

3. According to claim 1, a rail transit train remote maintenance interactive method based on AR glasses is characterized in that: The operation of the AR glasses in step S10 is implemented by the AR glasses APP, and the use process of the AR glasses is as follows: Get the wireless network IP address by configuring the wireless connection; Connect the AR glasses to the computer according to the obtained wireless network IP address; After the connection is successful, download the AR glasses APP in the AR glasses; After the download is complete, the APP starts and generates a UID account based on the current AR glasses device and joins the network; After successfully joining the network, when the remote device initiates a video call, choose to accept or reject; After receiving the video initiated by the remote device, the AR glasses display the camera shooting screen and display tool buttons on the screen, including voice call, flashlight, zoom in, zoom out and close. After the remote video ends, the APP interface returns to the state of successfully joining the network.

4. According to claim 1, a rail transit train remote maintenance interactive method based on AR glasses is characterized in that: The steps of establishing a connection between the expert end and the operator end of the remote control software in step S10 are: When the expert end initiates a remote control request command through the IP address, the operator end accepts the request command and returns the connection address and authorization code command. The expert end establishes a remote connection with the operator end through the connection address and authorization code. The operator end returns a success callback command, and the remote control connection is successfully established; When the expert side initiates a voice call request command through the IP address, the operator side accepts the request command and returns the connection address and authorization code command. The expert side establishes a voice call connection with the operator side through the connection address and authorization code. The operator side returns a success callback, and the two-way voice call is successfully established.

5. According to claim 1, a rail transit train remote maintenance interactive method based on AR glasses is characterized in that: In step S20, the background expert instructs the on-site operator by superimposing virtual guidance information on the AR glasses screen in real time through the remote control software expert terminal. The interactive operation steps between the remote control software expert terminal and the operator terminal are as follows: The expert side or the operator side initiates a whiteboard operation command to pause the AR glasses video screen and display a whiteboard in the AR glasses video screen. The background experts and on-site operators mark virtual guidance information on the rail transit train maintenance operation steps on the whiteboard, and the expert side and the operator side synchronize the whiteboard annotation content in real time; the expert side initiates a focus adjustment command, the operator side accepts the focus adjustment command, responds to the focus change, and the rail transit train maintenance on-site video data changes synchronously.

6. The method for remote maintenance of rail transit trains based on AR glasses according to claim 5 is characterized in that: The virtual guidance information includes an indication arrow, a text box, a text description and a numerical label.

7. The method for remote maintenance of rail transit trains based on AR glasses according to claim 1 is characterized in that: During the equipment inspection and maintenance process, the data analysis module in the remote control software records and records the video, and generates an operation report for improving the guidance method and content. The specific operation steps are as follows: After the connection between the operator side and the expert side is established, the operator side automatically runs the video recording command to record the video screen of the rail transit train maintenance site. After the recording is completed, the video data of the rail transit train maintenance site is synchronized to the expert side for storage.

8. A rail transit train remote maintenance interactive device based on AR glasses, characterized by: AR glasses module: used to capture on-site video data and transmit the captured on-site video data to the remote control software via a computer connected to the Internet; Backstage guidance module: used to monitor the site in real time and guide personnel to repair equipment; Interaction module: used to establish real-time interaction between the site and the backstage; Data analysis module: used to record and record the equipment inspection and operation and maintenance process, generate operation reports and feedback, and help the backend improve guidance content and methods.

9. A rail transit train remote maintenance interactive device based on AR glasses, characterized in that: The rail transit train remote maintenance interactive device based on AR glasses includes: A memory, a processor, and a remote guidance teaching program based on AR glasses stored in the memory and executable on the processor, wherein when the remote guidance teaching program based on AR glasses is executed by the processor, an AR glasses-based remote guidance teaching method as described in any one of claims 1 to 7 is implemented.

Images