Virtual reality simulation system and follow-up method

Abstract

The invention discloses a virtual reality simulation system and a follow-up method, and belongs to the technical field of virtual reality, the virtual reality simulation system comprises a holder and a VR head display device, the VR head display device comprises a head-mounted display, a first handle and a second handle, the head-mounted display is connected with the first handle and the second handle, and the first handle is connected with the holder. The motion states of the first handle and the second handle are mapped to the holder to control the holder to move along with the VR head display device, the holder comprises a base, a rotary table, an overturning frame, a first visual module, a second visual module and a radar module, the rotary table is rotationally connected to the base, the overturning frame is rotationally connected to the rotary table, the overturning frame is perpendicular to the rotating axis of the rotary table, and the first visual module is perpendicular to the second visual module. The first visual module is arranged close to one end of the overturning frame, the second visual module is arranged close to the other end of the overturning frame, the radar module is arranged between the first visual module and the second visual module, and the radar module can accurately sense objects in a scene in a large range, make up for a view blind area and achieve the purposes of monitoring and flexible operation.

Description

Technical Field

[0001] This invention relates to the field of virtual reality technology, and more specifically to a virtual reality simulation system and a follow-up method. Background Technology

[0002] Head-mounted display technology utilizes computer vision to track the user's head movements and postures. After learning through machine learning algorithms, it can recognize the user's head movements. Combined with virtual reality and augmented reality technologies, it can simulate head movements in a virtual environment, achieving the goal of satisfying human visual perception habits when remotely operating and controlling devices. This requires the use of sensor components such as cameras and accelerometers. Existing head-mounted devices are limited by the accuracy of sensors and the efficiency of computer processing. They are large in size and easily affected by external factors, with high latency and low imaging accuracy. At the same time, the computer needs to process a large amount of algorithm data, making it difficult to meet the accuracy and response speed requirements of real-time operation of the device. When used in space, they are easily affected by factors such as ambient light and electromagnetic interference, resulting in a narrow application range. In addition, the movement of head-mounted devices relies on the robot's own control system and sensor control. Errors or loss of control during operation may cause harm to the surrounding environment. There is a need for a simulation system for remotely controlling the movement of robotic equipment to meet the requirements of computing speed and safety.

[0003] A patent with publication number CN207817308U discloses a binocular AR head-mounted display device, including an AR housing, an AR buckle, an AR lens, and an AR display. The AR buckle is connected to one side of the AR housing, the AR lens is snapped into the AR housing, and the AR display is fixed to the other side of the AR housing. This invention has the advantages of small size, low cost, low technical requirements, and the ability to realize rich AR interactive applications. It acquires real-time information about the real environment through left and right cameras and displays it on the left and right screens accordingly, thereby better virtualizing the virtual environment and realizing binocular visual display that truly simulates human eyes. It also allows for quick switching between the AR virtual environment and the real environment through a switch button. However, this head-mounted display device can only display images and cannot transmit the posture of the human head outwards, lacking the function of interacting with external devices. Summary of the Invention

[0004] One of the objectives of this invention is to provide a virtual reality simulation system that solves the problem that operators cannot operate the equipment in real time for observation and flexible work.

[0005] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0006] A virtual reality simulation system includes a gimbal and a VR headset. The VR headset includes a head-mounted display, a first controller, and a second controller. The head-mounted display is connected to both the first and second controllers, mapping their motion states onto the gimbal. The head-mounted display controls the gimbal to follow the VR headset's movements. The gimbal includes a base, a turntable, a tilting frame, a first visual module, a second visual module, and a radar module. The turntable is rotatably connected to the base, and the tilting frame is rotatably connected to the turntable. The tilting frame is perpendicular to the turntable's rotation axis. The first visual module is positioned near one end of the tilting frame, and the second visual module is positioned near the other end. The radar module is positioned between the first and second visual modules and can perform composite imaging of objects within the scene, accurately collecting information about objects in the scene. The radar module can accurately perceive objects within a large area of ​​the scene, compensating for blind spots.

[0007] Furthermore, the first visual module includes a visible light camera and a first camera compartment. The first camera compartment is mounted on the flip frame, and the visible light camera is mounted inside the first camera compartment for visible light imaging. The image is then displayed to the operator through the head-mounted display, facilitating the operator's observation and operation.

[0008] Furthermore, the second visual module includes an infrared camera, a second camera compartment, and a supplementary light ring. The second camera compartment is mounted on the flip frame, the infrared camera is mounted inside the second camera compartment, and the supplementary light ring is mounted inside the second camera compartment. The supplementary light ring is coaxial with the infrared camera and serves to supplement the light of the object, enabling clear imaging even in low-light conditions.

[0009] Furthermore, it also includes a depth camera, which is mounted on the flip frame and faces the same direction as the radar module, for detecting the distance to objects and obtaining accurate data.

[0010] Preferably, the radar module includes a panoramic radar and a main control board. The panoramic radar is mounted on the flip frame, and the main control board is located on one side of the panoramic radar, which can transmit image data to the VR headset.

[0011] Preferably, the base includes a first driving device and a rotary support. The first driving device is disposed inside the base, the rotary support is rotatably connected to the base, and the turntable is connected to the rotary support. The first driving device drives the turntable to rotate, which can drive the turntable to rotate in the horizontal plane for adjusting the angle of the imaging element.

[0012] More preferably, the turntable includes a first fork and a second fork, which are respectively disposed near the two sides of the flipping frame. The radar module is disposed between the first fork and the second fork. The flipping frame penetrates the first fork and the second fork, thereby improving the load-bearing capacity of the turntable. The rotation of the flipping frame is stable and reliable, preventing the imaging from shaking during movement.

[0013] More preferably, the tilting frame includes a second drive device disposed on the turntable, the second drive device driving the tilting frame to rotate, thereby providing power for the pitch rotation of the imaging element.

[0014] More preferably, the VR headset and the gimbal communicate wirelessly, and the wireless communication is one of Bluetooth, 2.4G wireless network or Zigbee, which can conduct data communication at different distances and in different environments.

[0015] The second objective of this invention is to provide a follow-up method that solves the problem that the gimbal movements of the working equipment cannot keep up with the changes in the operator's head posture.

[0016] To achieve the above-mentioned objectives, the technical solution adopted by the present invention is as follows:

[0017] A gimbal and head-mounted display tracking method, implemented by the aforementioned virtual reality simulation system, specifically includes the following steps:

[0018] S1. Deployment: The gimbal is deployed to the area to be detected, and the operator wears the VR headset. The VR headset is wirelessly connected to the gimbal.

[0019] S2. Calibration: The gimbal is reset, the VR headset is aligned, and the positions of the two coincide to complete the position calibration.

[0020] S3. Movement: Operate the first handle and the second handle to control the turntable and the rotation respectively, thereby driving the first visual module, the second visual module and the radar module to rotate;

[0021] S4. Attitude control: The gimbal tracks the attitude of the head-mounted display and adjusts its angle accordingly; the head-mounted display receives the image data transmitted back by the gimbal.

[0022] S5. Data Processing: The image data collected by the gimbal is processed by image rendering and presented to the operator through the head-mounted display. The operator can remotely operate the gimbal to rotate it to complete detection, rescue and operation. It has high safety and a large observation range.

[0023] The beneficial effects of this invention are as follows:

[0024] (1) The virtual reality simulation system includes a VR head-mounted display and a gimbal. The VR head-mounted display can detect the posture and movement of the human head and hands, and transmit the human head and hand movement information to the gimbal so that the gimbal can move with the human's movements. It can meet the observation needs of the site under remote operation. The two visual modules have visible light imaging and infrared light imaging functions, which can identify living organisms under poor lighting conditions. The radar module can scan the site in real time, which improves the accuracy and range of object perception, and makes up for the shortcomings of visual imaging and prevents blind spots in observation.

[0025] (2) The turntable and flip frame on the virtual reality simulation system support the imaging element to rotate in the horizontal and vertical planes. It can follow the posture of the human head at high speed, expand the operator's observation range, reduce the difficulty of operation, and the VR headset and the gimbal are connected by multiple communication methods to ensure continuous communication in harsh environments.

[0026] (3) The gimbal and head-mounted display follow method detects the movement and posture of the head and hands through the VR head-mounted display device, and transmits the movement status information of the head and hands to the gimbal. The gimbal moves according to the transmitted information, so as to achieve the purpose of real-time observation of the situation on site, improve the safety of the on-site equipment operation, and can work in harsh environments such as high temperature, low temperature, and strong wind. In harsh environments, it can detect micro-motion signals to achieve the purpose of detection and rescue. Through the cooperation of infrared, visible light, and depth cameras, it can meet the purpose of searching for and rescuing living beings. Attached Figure Description

[0027] Figure 1 A front view of the virtual reality simulation system provided by the present invention;

[0028] Figure 2 A side view of the virtual reality simulation system provided by the present invention;

[0029] Figure 3 A top view of the virtual reality simulation system provided by the present invention;

[0030] Figure 4 This is a diagram illustrating the composition of the VR headset device provided by the present invention.

[0031] Figure label:

[0032] 1. Gimbal; 11. First drive unit; 12. Rotary support; 13. Base; 2. Turntable; 21. Tilting frame; 22. Second drive unit; 23. First fork arm; 24. Second fork arm; 3. First visual module; 31. Visible light camera; 32. First camera compartment; 4. Radar module; 41. Panoramic radar; 42. Pivot; 43. Main control board; 44. Depth camera; 5. Second visual module; 51. Infrared camera; 52. Second camera compartment; 6. Light ring; 7. VR headset; 71. First handle; 72. Second handle; 73. Head-mounted display. Detailed Implementation

[0033] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0034] Example 1

[0035] like Figures 1-4 As shown, this embodiment discloses a virtual reality simulation system, including a gimbal 1 and a VR headset 7. The VR headset 7 includes a head-mounted display 73, a first controller 71, and a second controller 72. The head-mounted display 73 is connected to the first controller 71 and the second controller 72, respectively, and maps the movement states of the first controller 71 and the second controller 72 onto the gimbal 1. The head-mounted display 73 controls the gimbal 1 to follow the movements of the VR headset 7. The gimbal 1 includes a base 13, a turntable 2, a tilting frame 21, a first visual module 3, a second visual module 5, and a radar module 4. The turntable 2 is rotatably connected to the base 13, and the tilting frame 21 is rotatably connected to the turntable 2. The tilting frame 21 is perpendicular to the rotation axis of the turntable 2. The first visual module 3 is located near one end of the tilting frame 21, and the second visual module 5 is located near the other end of the tilting frame 21. The radar module 4 is located between the first visual module 3 and the second visual module 5. The radar module 4 can detect objects in a large area, track moving objects, and supplement the blind spots of the first visual module 3 and the second visual module 5.

[0036] Among them, radar module 4 can accurately sense objects in the area. It has high sensitivity and high anti-interference ability and can work in harsh environments such as high temperature, low temperature and strong wind. It can detect micro-movements and even breathing signals in harsh environments and realize the detection of human movement to achieve the purpose of detection and rescue.

[0037] Furthermore, the first visual module 3 includes a visible light camera 31 and a first camera compartment 32. The first camera compartment 32 is mounted on the flip frame 21, and the visible light camera 31 is mounted inside the first camera compartment 32. The visible light camera 31 can directly image objects in the field and transmit the received images to the head-mounted display 73 via the main control board 43 for display. The first camera compartment 32 serves to protect the first visual module 3. The visible light camera 31 is an autofocus SLR camera.

[0038] Furthermore, the second visual module 5 includes an infrared camera 51, a second camera compartment 52, and a supplementary lighting ring 6. The second camera compartment 52 is mounted on the flip frame 21, the infrared camera 51 is mounted inside the second camera compartment 52, and the supplementary lighting ring 6 is mounted inside the second camera compartment 52. The supplementary lighting ring 6 is coaxial with the infrared camera 51. The infrared camera 51 is a binocular camera that can perform thermal imaging of objects in the field. After the thermal imaging image is transmitted to the head-mounted display 73, organisms in the field can be identified, and observation can be performed when the light is insufficient. In addition, the supplementary lighting ring 6 can be illuminated by infrared light and visible light, so that clear imaging can be achieved even in poor lighting conditions.

[0039] Furthermore, it also includes a depth camera 44, which is mounted on the flip frame 21 and faces the same direction as the radar module 4. The depth camera 44 can detect the depth of field of the shooting space, obtain the distance of each point in the image from the camera through the depth camera 44, measure the distance of the object, obtain the three-dimensional spatial coordinates of each point in the image, and play a role in accurate positioning.

[0040] Preferably, the radar module 4 includes a panoramic radar 41 and a main control board 43. The panoramic radar 41 is mounted on the flip frame 21, and the main control board 43 is mounted on one side of the panoramic radar 41. The panoramic radar 41 can perform periodic scanning of a 360-degree range. The main control board 43 is used to control the drive device to rotate the gimbal 1 to adjust its posture and transmit the collected data to the VR head-mounted display device 7.

[0041] Preferably, the base 13 includes a first drive device 11 and a rotary support 12. The first drive device 11 is disposed inside the base 13, and the rotary support 12 is rotatably connected to the base 13. The turntable 2 is connected to the rotary support 12. The first drive device 11 drives the turntable 2 to rotate. The first drive device 11 is connected to the rotary support 12 through a synchronous belt, driving the rotary support 12 to rotate. The rotary support 12 consists of a fixed plate and a rotating part. The fixed plate is installed on the top wall of the base 13, and the rotating part is rotatably connected to the fixed plate. It has the characteristics of large load-bearing capacity and stable and smooth rotation.

[0042] Preferably, the turntable 2 includes a first fork arm 23 and a second fork arm 24. The first fork arm 23 and the second fork arm 24 are respectively arranged close to both sides of the flipping frame 21. The radar module 4 is arranged between the first fork arm 23 and the second fork arm 24. The flipping frame 21 penetrates the first fork arm 23 and the second fork arm 24. The first fork arm 23 and the second fork arm 24 are rotatably connected to the flipping frame 21. The pivot 42 installed in the first fork arm 23 and the second fork arm 24 forms a two-point support, which can balance the load on the flipping frame 21 and make the rotation of the flipping frame 21 more flexible.

[0043] Preferably, the flip frame 21 includes a second drive device 22, which is mounted on the turntable 2. The second drive device 22 drives the flip frame 21 to rotate. Two second drive devices 22 are provided inside the flip frame 21. The two second drive devices 22 are respectively installed in the first fork arm 23 and the second fork arm 24, and simultaneously drive the flip frame 21 to rotate, so that the imaging element on the flip frame 21 can form a stable image without shaking.

[0044] Among them, the VR headset 7 and the gimbal 1 use wireless communication, which is one of Bluetooth, 2.4G wireless network or Zigbee. Through the hybrid communication method, data can be transmitted under harsh conditions, ensuring the real-time performance of data transmission.

[0045] Example 2

[0046] This embodiment discloses a gimbal and head-mounted display tracking method, implemented by a virtual reality simulation system, which specifically includes the following steps:

[0047] S1. Deployment: The gimbal 1 is deployed to the area to be detected, and the operator wears the VR headset 7. The VR headset 7 and the gimbal 1 are connected wirelessly.

[0048] S2. Calibration: Reset gimbal 1, straighten VR headset 7, and complete position calibration when the two positions coincide.

[0049] S3, Movement: Operate the first handle 71 and the second handle 72 to control the turntable 2 and the tilting frame 21 to rotate, thereby driving the first visual module 3, the second visual module 5 and the radar module 4 to rotate;

[0050] S4. Attitude control: The gimbal 1 tracks the attitude of the head-mounted display 73 and adjusts the angle accordingly. The head-mounted display 73 receives the image data transmitted back by the gimbal 1.

[0051] In step S4, a head tracking algorithm is used to calculate the posture and position of the head-mounted display 73 so that the image can be updated in real time according to the movement of the head. The head tracking algorithm detects the movement of the head by collecting accelerometer and gyroscope sensors installed on the head-mounted display 73. The gesture recognition algorithm converts the paths of the first handle 71 and the second handle 72 into interactive information in the virtual environment, so as to achieve the purpose of operating virtual objects by selecting and moving them, and to control the gimbal 1.

[0052] S5. Data processing: The image data collected by the gimbal 1 is processed by image rendering and presented to the operator through the head-mounted display 73.

[0053] In step S5, the VR head-mounted display device 7 uses an image rendering algorithm to render the image onto the head-mounted display 73, reducing canvas latency and improving image quality and frame rate. The image also uses an optical algorithm to coordinate the head-mounted display 73 with optical imaging, by correcting lens distortion and adjusting interpupillary distance and field of view, in order to improve image clarity and ensure the comfort of human eye observation.

[0054] Based on the disclosure and teachings of the foregoing specification, those skilled in the art can make changes and modifications to the above embodiments. Therefore, the present invention is not limited to the specific embodiments disclosed and described above, and any modifications and changes to the present invention should also fall within the protection scope of the claims of the present invention. Furthermore, although some specific terms are used in this specification, these terms are only for convenience of explanation and do not constitute any limitation on the present invention.

Claims

1. A virtual reality simulation system, comprising a gimbal (1) and a VR headset (7), wherein the VR headset (7) comprises a head-mounted display (73), a first handle (71), and a second handle (72), the head-mounted display (73) being connected to the first handle (71) and the second handle (72) respectively, mapping the motion state of the first handle (71) and the second handle (72) onto the gimbal (1), and the head-mounted display (73) controlling the gimbal (1) to follow the movement of the VR headset (7), characterized in that: The gimbal (1) includes a base (13), a turntable (2), a tilting frame (21), a first viewing module (3), a second viewing module (5), and a radar module (4). The turntable (2) is rotatably connected to the base (13), and the tilting frame (21) is rotatably connected to the turntable (2). The tilting frame (21) is perpendicular to the rotation axis of the turntable (2). The first viewing module (3) is located near one end of the tilting frame (21), and the second viewing module (5) is located near the other end of the tilting frame (21). The radar module (4) is located between the first viewing module (3) and the second viewing module (5).

2. The virtual reality simulation system according to claim 1, characterized in that: The first visual module (3) includes a visible light camera (31) and a first camera compartment (32). The first camera compartment (32) is mounted on the flip frame (21), and the visible light camera (31) is mounted inside the first camera compartment (32).

3. The virtual reality simulation system according to claim 2, characterized in that: The second visual module (5) includes an infrared camera (51), a second camera compartment (52), and a supplementary light ring (6). The second camera compartment (52) is mounted on the flip frame (21). The infrared camera (51) is mounted inside the second camera compartment (52). The supplementary light ring (6) is mounted inside the second camera compartment (52) and is coaxial with the infrared camera (51).

4. The virtual reality simulation system according to claim 1, characterized in that: It also includes a depth camera (44), which is mounted on the flip frame (21) and is oriented in the same direction as the radar module (4).

5. The virtual reality simulation system according to claim 4, characterized in that: The radar module (4) includes a panoramic radar (41) and a main control board (43). The panoramic radar (41) is mounted on the flip frame (21), and the main control board (43) is mounted on one side of the panoramic radar (41).

6. The virtual reality simulation system according to claim 1, characterized in that: The base (13) includes a first driving device (11) and a rotary support (12). The first driving device (11) is disposed inside the base (13). The rotary support (12) is rotatably connected to the base (13). The turntable (2) is connected to the rotary support (12). The first driving device (11) drives the turntable (2) to rotate.

7. The virtual reality simulation system according to claim 6, characterized in that: The turntable (2) includes a first fork arm (23) and a second fork arm (24). The first fork arm (23) and the second fork arm (24) are respectively arranged close to both sides of the flipping frame (21). The radar module (4) is arranged between the first fork arm (23) and the second fork arm (24). The flipping frame (21) penetrates the first fork arm (23) and the second fork arm (24).

8. The virtual reality simulation system according to claim 7, characterized in that: The tilting frame (21) includes a second drive device (22), which is mounted on the turntable (2) and drives the tilting frame (21) to rotate.

9. The virtual reality simulation system according to claim 1, characterized in that: The VR headset (7) and the gimbal (1) communicate wirelessly, and the wireless communication is one of Bluetooth, 2.4G wireless network or Zigbee.

10. A method for tracking a gimbal and head-mounted display, characterized in that: The virtual reality simulation system according to any one of claims 1-9 is implemented by specifically including the following steps: S1. Deployment: The gimbal (1) is deployed to the area to be detected, and the operator wears the VR headset (7). The VR headset (7) and the gimbal (1) are connected wirelessly. S2, Calibration: The gimbal (1) is reset, the VR headset (7) is aligned, and the positions of the two coincide to complete the position calibration; S3, Movement: Operate the first handle (71) and the second handle (72) to control the turntable (2) and the tilting frame (21) to rotate, thereby driving the first visual module (3), the second visual module (5) and the radar module (4) to rotate; S4, Attitude control: The gimbal (1) tracks the attitude of the head-mounted display (73) and adjusts the angle accordingly. The head-mounted display (73) receives the image data transmitted back by the gimbal (1). S5. Data processing: The image data collected by the gimbal (1) is processed by image rendering and presented to the operator through the head-mounted display (73).

Images