Search and rescue guiding device and method based on augmented reality

Abstract

The invention discloses a search and rescue guiding device and method based on augmented reality. The augmented reality device (3) is used for mixing data needing to be displayed in an augmented mode in the search and rescue guide control service data with a real environment to construct an augmented reality scene; the calculation and interface unit (1) is used for realizing protocol conversion between equipment networking and data links (11) in the search and rescue guiding device; the radio station (4) is used for realizing communication between the search and rescue guiding device and the air detection equipment (8) as well as between the search and rescue airplane (7) and the search and rescue command post (6); and the target detection equipment is used for acquiring information of the search and rescue guider, the target and the search and rescue scene. The search and rescue aircraft (7) can be guided to arrive at a search and rescue site to carry out target search and rescue work, and user decision-making efficiency is improved.

Description

Technical Field

[0001] The present invention relates to a search and rescue guidance device, in particular to an augmented reality-based search and rescue guidance device and method for the field of aviation search and rescue. Background Art

[0002] At present, aviation search and rescue guidance devices are mainly used to guide search and rescue aircraft to the search and rescue site to carry out target search and rescue work. They are composed of a tablet, a computing and interface unit, a radio station, a target detection equipment, etc. The search and rescue guidance control function is implemented by the tablet based on the map screen. It cannot directly present the real picture of the search and rescue site situation and the search and rescue target, and the search and rescue decision-making efficiency is low. Summary of the invention

[0003] The purpose of the present invention is to provide a search and rescue guidance device and method based on augmented reality. The present invention can guide a search and rescue aircraft to the search and rescue site to carry out target search and rescue work, thereby improving the decision-making efficiency of users.

[0004] Technical solution: A search and rescue guidance device based on augmented reality, comprising: a computing and interface unit, a target detection device, an augmented reality device, and a radio station; the target detection device, the augmented reality device, and the radio station are all connected to the computing and interface unit via a network cable to achieve data transmission;

[0005] Augmented reality equipment, used to mix the data that needs to be enhanced in the search and rescue guidance control business data with the real environment to build an augmented reality scene;

[0006] The computing and interface unit is used to realize the equipment networking and data link protocol conversion within the search and rescue guidance device;

[0007] Radio station, used to realize communication between search and rescue guidance equipment and air detection equipment, search and rescue aircraft and search and rescue command post;

[0008] Target detection equipment is used to obtain information about search and rescue guides, targets and search and rescue sites.

[0009] The aforementioned augmented reality-based search and rescue guidance device also includes a power supply for powering the augmented reality device, the computing and interface unit, the radio station, and the target detection device.

[0010] In the aforementioned search and rescue guidance device based on augmented reality, the augmented reality device includes: a data receiving and processing module, an enhanced display function module, and an immersive operation control module;

[0011] The data receiving and processing module receives the search and rescue scene data information output by the calculation and interface unit, and sends the search and rescue scene data information to the enhanced display function module; on the other hand, it exchanges the search and rescue guidance control business data with the immersive operation control module, sends the processed search and rescue guidance control business data to the calculation and interface unit, and sends the data that needs to be enhanced displayed in the search and rescue guidance control business data to the enhanced display function module;

[0012] The enhanced display function module displays the received on-site data information and the data that needs to be enhanced in the search and rescue guidance control business data in the field of view, and mixes it with the real environment to construct an augmented reality scene;

[0013] The immersive operation control module conducts search and rescue guidance and control operations based on the constructed augmented reality scene.

[0014] In the aforementioned augmented reality-based search and rescue guidance device, the data receiving and processing module, the enhanced display function module, and the immersive operation control module are integrated into an augmented reality wearable helmet, and the data that needs to be enhanced and displayed in the on-site data information and the search and rescue guidance control business data are displayed in the helmet field of view.

[0015] In the aforementioned search and rescue guidance device based on augmented reality, the operation of the immersive operation control module is achieved through gesture recognition and language recognition.

[0016] In the aforementioned augmented reality-based search and rescue guidance device, the computing and interface unit is used, on the one hand, to form a local area network of augmented reality equipment, radio stations, and target detection equipment through Ethernet; on the other hand, it is used to realize protocol conversion between Ethernet and data link.

[0017] In the aforementioned search and rescue guidance device based on augmented reality, the target detection equipment includes aerial detection equipment and ground detection equipment; when the ground is blocked, the aerial detection equipment is selected; when the ground is not blocked, the ground detection equipment is selected.

[0018] In the aforementioned search and rescue guidance device based on augmented reality, the radio station interacts with the aerial detection equipment through a data link and interacts with the ground detection equipment through Ethernet.

[0019] In the aforementioned search and rescue guidance device based on augmented reality, the radio station interacts with the search and rescue aircraft through data link and voice, and interacts with the search and rescue command post through satellite communication.

[0020] The aforementioned search and rescue method based on the augmented reality search and rescue guidance device:

[0021] Step 1. Detect the search and rescue guide's own location information, target information, and search and rescue scene information through the target detection equipment;

[0022] Step 2. Fill out the task application form based on the immersive operation control module to apply for the search and rescue mission. After receiving the task approval from the search and rescue command post, carry out the search and rescue aircraft task planning and fill out the task brief. When the search and rescue aircraft arrives at the search and rescue site, send the search and rescue aircraft task planning and task brief to the search and rescue aircraft to carry out the search and rescue work, and finally evaluate the search and rescue effect.

[0023] Step 3. The data receiving and processing module receives the search and rescue scene data information output by the calculation and interface unit, including the search and rescue guide's own location information, target information, equipment status information, and search and rescue aircraft status information, and sends the search and rescue scene data information to the enhanced display function module; on the other hand, it exchanges the search and rescue guidance control business data with the immersive operation control module, sends the processed search and rescue guidance control business data to the calculation and interface unit, and sends the task planning and effect evaluation data in the search and rescue guidance control business data to the enhanced display function module;

[0024] Step 4. The enhanced display function module displays the received search and rescue scene data information and the mission planning and effect evaluation data in the search and rescue guidance and control business data in the helmet field of view, and mixes them with the real environment to build an augmented reality scene;

[0025] Step 5. The immersive operation control module carries out search and rescue guidance and control operations based on the constructed augmented reality scene, including mission application, mission planning, mission briefing, target association and effect evaluation.

[0026] Beneficial effects: The search and rescue guidance device based on the augmented reality wearable helmet designed by the present invention can mix the search and rescue site data information and the search and rescue guidance control business data information that needs to be enhanced and displayed with the real environment to a certain extent in the helmet field of view, so as to realize the enhanced display of the search and rescue site data and the search and rescue guidance control business data. In the enhanced display screen, the search and rescue guide's own information can be displayed, and the target, search and rescue aircraft's movement speed, movement direction, longitude and latitude and other attribute information can be superimposed and displayed next to the enhanced display target and search and rescue aircraft icons. The search and rescue aircraft route, search and rescue effect evaluation results and other information can also be enhanced and displayed on the situation screen, so that these virtual information and the real search and rescue environment are integrated, providing a smarter and more intuitive search and rescue experience, giving the search and rescue guide a sense of being in the scene. Based on the enhanced display screen, the search and rescue guidance control business immersive operation control is carried out, which can reduce the cognitive burden of the search and rescue guide and improve the decision-making efficiency of the search and rescue guide when carrying out the search and rescue guidance control business. BRIEF DESCRIPTION OF THE DRAWINGS

[0027] Figure 1 Diagram of the composition of the search and rescue guidance device and its connection with external equipment;

[0028] Figure 2This is a diagram of information interaction between internal and external modules of the search and rescue guidance device;

[0029] Figure 3 This is a flow chart for the operation of the search and rescue guidance device. DETAILED DESCRIPTION

[0030] Embodiment 1. A search and rescue guidance device based on augmented reality, see Figure 1-Figure 3 , which can guide the search and rescue aircraft 7 to arrive at the search and rescue site to carry out target search and rescue work. It is composed of an augmented reality device 3, a computing and interface unit 1, a radio station 4, a target detection device, and a power supply 5. It directly presents the real picture of the search and rescue site situation and the search and rescue target based on the augmented reality device 3 and the search and rescue guidance control software, realizes the search and rescue site data reception and processing, enhanced display, immersive operation control and other search and rescue guidance control core business functions, and improves the user's decision-making efficiency.

[0031] The augmented reality device 3 is composed of a hardware system and a software system. The hardware system is an augmented reality wearable helmet, and the software system is a search and rescue guidance control software, which is composed of three business modules: a data receiving and processing module 12, an enhanced display function module 13, and an immersive operation control module 14. On the one hand, the data receiving and processing module 12 receives the search and rescue scene data information output by the calculation and interface unit 1, and sends the search and rescue scene data information to the enhanced display function module 13; on the other hand, it interacts with the immersive operation control module 14 to exchange search and rescue guidance control business data, sends the processed search and rescue guidance control business data to the calculation and interface unit 1, and sends the data in the search and rescue guidance control business data that needs to be enhanced to display to the enhanced display function module 13. The enhanced display function module 13 displays the received search and rescue scene information and the search and rescue guidance control business data that needs to be enhanced to display in the helmet field of view, and mixes them with the real environment. The immersive operation control module 14 carries out the search and rescue guidance control business based on the constructed augmented reality scene. The immersive operation is realized through gesture recognition and language recognition.

[0032] The computing and interface unit 1 realizes the networking of the equipment in the search and rescue guidance device. On the one hand, the augmented reality device 3, the radio station 4, and the target detection device are formed into a local area network through Ethernet; on the other hand, the protocol conversion between Ethernet and the data link 11 is realized.

[0033] Radio station 4 is used to realize communication between the search and rescue guidance device and the aerial detection equipment 8, the search and rescue aircraft 7 and the search and rescue command post 6, and can realize communication by satellite, data link and voice 10. The search and rescue guidance device interacts with the aerial detection equipment 8 through the data link 11, the search and rescue guidance device interacts with the search and rescue aircraft 7 through the data link and voice 10, and the search and rescue guidance device interacts with the search and rescue command post 6 through satellite communication 9.

[0034] The target detection equipment is divided into an aerial detection equipment 8 and a ground detection equipment 2, which can realize the acquisition of information of the search and rescue guide, the target and the search and rescue scene. Different target detection equipment can be selected according to different search and rescue mission scenarios. When the ground is blocked, the aerial detection equipment 8 can be selected, and when the ground is not blocked, the ground detection equipment 2 can be selected.

[0035] The power supply 5 is used to supply power to the augmented reality device 3, the computing and interface unit 1, the radio station 4, and the target detection device.

[0036] Embodiment 2. A search and rescue guidance device based on augmented reality, see Figure 1-Figure 3 , which can guide the search and rescue aircraft 7 to arrive at the search and rescue site to carry out target search and rescue work, is composed of an augmented reality device 3, a computing and interface unit 1, a radio 4, a target detection device, and a power supply 5. The augmented reality device 3, the radio 4, and the target detection device are all connected to the computing and interface unit 1 via a network cable to realize data transmission. The power supply 5 supplies power to the augmented reality device 3, the computing and interface unit 1, the radio 4, and the target detection device through a cable. The search and rescue guidance device interacts with the aerial detection equipment 8 through a data link 11, the search and rescue guidance device interacts with the search and rescue aircraft 7 through a data link and voice 10, and the search and rescue guidance device interacts with the search and rescue command post 6 through satellite communication 9. The composition of the device and its connection relationship with external equipment are shown in the figure below. Figure 1 shown.

[0037] For the search and rescue process based on the search and rescue guidance device, see Figure 3 , where A is the start, B is the acquisition of search and rescue site data, C is the search and rescue site data processing, D is the creation of augmented reality scene, E is the search and rescue site data display, F is the immersive operation control, G is the immersive operation data update, and H is the end; specifically as follows: the search and rescue guide first detects the search and rescue guide's own position information, target information, and search and rescue site information through the target detection equipment, and then fills in the task application form based on the augmented reality device 3 to carry out the search and rescue task application. After receiving the task approval from the search and rescue command post 6, the search and rescue aircraft 7 carries out task planning and fills in the task brief. When the search and rescue aircraft 7 arrives at the search and rescue site, the task plan and task brief are sent to the search and rescue aircraft 7 to carry out the search and rescue work, and finally the search and rescue effect is evaluated.

[0038] The augmented reality device 3 is divided into a hardware system and a software system. The hardware system is an augmented reality wearable helmet, which is customized based on the RK3588 processor and provides one Ethernet port, which can achieve a horizontal 43° / vertical 29° field of view, a 1620*1080 screen resolution, and a 60HZ data refresh rate. The software system is a search and rescue guidance control software, which consists of three business modules: a data receiving and processing module 12, an enhanced display function module 13, and an immersive operation control module 14. The information interaction relationship between the three modules is as follows: Figure 2As shown, the information represented by a includes: mission application, mission planning, mission briefing, target association, and effect evaluation; the information represented by b includes: search and rescue guide's own location information, target information, search and rescue site information, equipment status information, and search and rescue aircraft status information; the information represented by c includes: mission planning and effect evaluation.

[0039] On the one hand, the data receiving and processing module 12 receives the search and rescue site data information output by the computing and interface unit 1, including the search and rescue guide's own location information, target information, equipment status information, search and rescue aircraft status information, etc., and sends the search and rescue site data information to the enhanced display function module 13; on the other hand, it interacts with the immersive operation control module 14 for search and rescue guidance control business data, sends the processed search and rescue guidance control business data to the computing and interface unit 1, and sends the task planning and effect evaluation data in the search and rescue guidance control business data to the enhanced display function module 13.

[0040] The enhanced display function module 13 displays the collected search and rescue scene data information and the task planning and effect evaluation data in the search and rescue guidance and control business data in the helmet field of view, and mixes it with the real environment to help the search and rescue guide better understand the search and rescue environment and search and rescue business information.

[0041] The immersive operation control module 14 carries out search and rescue guidance control services based on the constructed augmented reality scene, including mission application, mission planning, mission briefing, target association and effect evaluation. Immersive operation control is achieved through gesture recognition and language recognition. The mission application function is realized by filling out the mission application form and sending it to the search and rescue command post 6; the mission planning function is realized by marking the route of the search and rescue aircraft 7 in the enhanced display scene of the helmet and sending it to the search and rescue aircraft 7; the mission briefing function is realized by editing the mission briefing table and sending it to the search and rescue aircraft 7. The mission briefing includes the coordinates of the search and rescue target, the rescue method and other contents; the air-to-ground target association function is realized by controlling the aircraft pod and obtaining the aircraft pod video; the effect evaluation function is realized based on the image recognition algorithm through the aircraft pod video. The operation flow chart of the search and rescue guidance control software is shown below. Figure 3 shown.

[0042] The computing and interface unit 1 deploys the computing and interface unit 1 software to realize the networking of equipment in the search and rescue guidance device and the protocol conversion between Ethernet and data link 11. It is based on the FT-2000 / 4 processor and supports 16GB of memory and a 512GB hard disk.

[0043] The radio station 4 is used to realize the communication between the search and rescue guidance device and the air detection equipment 8, the search and rescue aircraft 7 and the search and rescue command post 6, and can realize the communication of satellite, data link and voice 10. The shelf products with satellite, data link and voice 10 communication functions are selected to realize it.

[0044] The target detection equipment is divided into air detection equipment 8 and ground detection equipment 2. The air detection equipment 8 is implemented by a quad-rotor drone. Based on the quad-rotor drone, a radio station 4, a visible light camera and a laser rangefinder are added to realize visible light sighting and laser ranging functions. The ground detection equipment 2 is implemented by sighting and can realize visible light sighting, infrared sighting and laser ranging functions.

[0045] The power source 5 is implemented by a lithium battery and a power source 5 manager. The power source 5 output by the lithium battery is distributed to the augmented reality device 3, the computing and interface unit 1, the radio station 4, and the target detection device through the power source 5 manager.

Claims

1. A search and rescue guidance device based on augmented reality, characterized in that: include: An augmented reality device (3) is used to mix the data that needs to be enhanced and displayed in the search and rescue guidance control business data with the real environment to construct an augmented reality scene; A computing and interface unit (1) is used to realize the device networking in the search and rescue guidance device and the protocol conversion between the data links (11); A radio station (4) for realizing communication between the search and rescue guidance device and the air detection equipment (8), the search and rescue aircraft (7) and the search and rescue command post (6); Target detection equipment is used to obtain information about search and rescue guides, targets and search and rescue sites.

2. The search and rescue guidance device based on augmented reality according to claim 1, characterized in that: It also includes a power supply (5) for supplying power to the augmented reality device (3), the computing and interface unit (1), the radio station (4), and the target detection device.

3. The search and rescue guidance device based on augmented reality according to claim 1, characterized in that: The augmented reality device (3) comprises: a data receiving and processing module (12), an enhanced display function module (13), and an immersive operation control module (14); The data receiving and processing module (12) receives the search and rescue scene data information output by the calculation and interface unit (1) and sends the search and rescue scene data information to the enhanced display function module (13); on the other hand, it exchanges the search and rescue guidance control business data with the immersive operation control module (14), sends the processed search and rescue guidance control business data to the calculation and interface unit (1), and sends the data in the search and rescue guidance control business data that needs to be enhanced displayed to the enhanced display function module (13); The enhanced display function module (13) displays the received on-site data information and the data that needs to be enhanced in the search and rescue guidance control business data in the field of view, and mixes them with the real environment to construct an enhanced reality scene; An immersive operation control module (14) performs search and rescue guidance control operations based on the constructed augmented reality scene.

4. The search and rescue guidance device based on augmented reality according to claim 3, characterized in that: The data receiving and processing module (12), the enhanced display function module (13), and the immersive operation control module (14) are integrated into an augmented reality wearable helmet, and data that needs to be enhanced in the on-site data information and the search and rescue guidance control business data are displayed in the helmet field of view.

5. The search and rescue guidance device based on augmented reality according to claim 3, characterized in that: The operation of the immersive operation control module (14) is achieved through gesture recognition and language recognition.

6. The search and rescue guidance device based on augmented reality according to claim 1, characterized in that: The computing and interface unit (1) is used, on the one hand, to form a local area network with the augmented reality device (3), the radio station (4), and the target detection device through Ethernet; and on the other hand, to realize the protocol conversion between Ethernet and the data link (11).

7. The search and rescue guidance device based on augmented reality according to claim 1, characterized in that: The target detection equipment comprises an aerial detection equipment (8) and a ground detection equipment (2); when the ground is blocked, the aerial detection equipment (8) is selected; when the ground is not blocked, the ground detection equipment (2) is selected.

8. The search and rescue guidance device based on augmented reality according to claim 7, characterized in that: The radio station (4) interacts with the aerial detection equipment (8) via a data link (11) and interacts with the ground detection equipment (2) via Ethernet.

9. The search and rescue guidance device based on augmented reality according to claim 1, characterized in that: The radio station (4) interacts with the search and rescue aircraft (7) via data link and voice (10), and interacts with the search and rescue command post (6) via satellite communication (9).

10. The search and rescue method based on the search and rescue guidance device based on augmented reality according to any one of claims 1 to 9, characterized in that: Step 1. Detect the search and rescue guide's own location information, target information, and search and rescue scene information through the target detection equipment; Step 2. Fill out the task application form based on the immersive operation control module (14) to carry out the search and rescue task application. After receiving the task approval from the search and rescue command post (6), carry out the task planning of the search and rescue aircraft (7) and fill out the task brief. When the search and rescue aircraft (7) arrives at the search and rescue site, send the search and rescue aircraft (7) task planning and task brief to the search and rescue aircraft (7) to carry out the search and rescue work, and finally evaluate the search and rescue effect. Step 3. The data receiving and processing module (12) receives the search and rescue scene data information output by the calculation and interface unit (1), including the search and rescue guide's own position information, target information, equipment status information, and search and rescue aircraft (7) status information, and sends the search and rescue scene data information to the enhanced display function module (13); on the other hand, it exchanges the search and rescue guidance control business data with the immersive operation control module (14), sends the processed search and rescue guidance control business data to the calculation and interface unit (1), and sends the task planning and effect evaluation data in the search and rescue guidance control business data to the enhanced display function module (13); Step 4. The enhanced display function module (13) displays the collected search and rescue scene data information and the task planning and effect evaluation data in the search and rescue guidance and control business data in the helmet field of view, and mixes them with the real environment to construct an augmented reality scene; Step 5. The immersive operation control module (14) carries out search and rescue guidance control operations based on the constructed augmented reality scene, including mission application, mission planning, mission briefing, target association and effect evaluation.

Images