Individual soldier situation awareness system based on visual enhanced awareness technology

Abstract

The invention discloses an individual soldier situation awareness system based on a visual enhanced awareness technology, and the system comprises a man-machine cooperative target discovery enhancement and environment awareness enhancement display part, obtains the surrounding environment information and target information of a remote unmanned vehicle in real time through sensor equipment, carries out the three-dimensional reconstruction of an environment situation through an SLAM algorithm, and carries out the real-time detection of the environment situation. When the unmanned vehicle recognizes a dangerous target by using an image recognition technology, the category and position information of the dangerous target is sent to the head display equipment worn by the individual combat personnel for enhanced display; 360-degree video image splicing is carried out on the environment information, and an image matched with the rotating angle visual angle of the head of the soldier is intercepted from the spliced image and sent to head display equipment worn by the individual combat personnel to be displayed; the combat personnel can intuitively perceive the position and related information of the dangerous target in the remote environment, and can more accurately perceive the surrounding environment of the vehicle when controlling the unmanned vehicle, thereby improving the perceiving ability and cooperative combat effect of the remote combat.

Description

Technical Field

[0001] The present invention belongs to the technical field of human-machine collaboration, and in particular relates to a single-soldier situation awareness system based on visual enhancement perception technology. Background Art

[0002] The term "situational awareness" first originated from the field of military aviation. It is people's understanding of "what is happening" in the face of complex dynamic systems. In a sense, situational awareness is a comprehensive reflection of people's cognitive activities in a specific environment in order to complete a certain theme task. In modern warfare, the importance of battlefield situation awareness is self-evident. Therefore, improving battlefield situation awareness and shortening the OODA cycle, so as to achieve enemy judgment, decision-making and action before the enemy, plays a vital role in seizing the initiative on the battlefield and achieving the final victory of the war. The individual situational awareness system based on human-machine collaboration can greatly improve the soldiers' battlefield situation awareness ability, thereby accelerating the OODA cycle. The advantage of human-machine collaboration is that it can give full play to human intelligence and creativity, combined with the computing power and execution efficiency of the machine, thereby enhancing combat capabilities, improving efficiency and reducing risks.

[0003] Various combat operations in recent years have shown that the urgency of individual situational awareness in urban combat has increased significantly. In this context, individual combat equipment plays a very important role in improving situational awareness. In future combat, individual soldiers will not be a simple combination of people and weapons, but will be successful equipment of individual combat systems and individual information systems with independent combat capabilities. The battlefield will be transparent to soldiers, and the environment, information, and targets will be transparent and simple. The individual situational awareness system plays an important role in the future battlefield, which is conducive to improving the situational awareness and reconnaissance capabilities of soldiers.

[0004] The individual situational awareness system is a system that integrates advanced sensor technology, data processing and analysis algorithms, and is designed to enhance the perception and intelligence acquisition capabilities of individual combatants on the battlefield. The system collects, processes and analyzes various information in the environment, including images, videos, sounds, geographic locations, etc., to understand the dynamic changes of the surrounding environment and the enemy and friendly situation in real time. The individual situational awareness system can provide combatants with high-quality, multi-source intelligence data, and present it to combatants in an intuitive and convenient way, thereby helping them make accurate and rapid decisions and actions. The system is often carried on individual equipment, weapons or combat clothing to provide combatants with individualized and customized situational awareness capabilities, improving their survivability, combat effectiveness and collaborative combat capabilities in complex battlefield environments. As an important component of the individual situational awareness system, unmanned platforms such as drones / unmanned vehicles have the characteristics of light size, flexible action, easy control, and good concealment. By carrying scene perception equipment, they are widely used in individual coordinated operations and combat reconnaissance. The future informationized battlefield will inevitably be a comprehensive, high-efficiency, and high-dynamic collaborative operation between individual soldiers and other combat units in combination with various sensors.

[0005] Human-machine collaboration has broad application prospects in the military field, but it also faces many problems. First, there are still certain challenges in information sharing and transmission in human-machine collaboration. In a battlefield environment, soldiers and machines such as unmanned vehicles may be in different positions and need to exchange information in real time. However, due to the limited field of view of the human eye, soldiers cannot directly observe the environment around the unmanned vehicle, which limits the acquisition and exchange of information. Secondly, target recognition is another important issue in human-machine collaboration. Accurately identifying enemy targets is crucial for battlefield command and decision-making. However, in a complex battlefield environment, enemy targets may be affected by factors such as occlusion (visual blind spots, building blind spots), interference, etc., making target recognition challenging. In addition, there are limitations on remote operation in human-machine collaboration. Due to the vastness and complexity of the battlefield environment, soldiers usually cannot directly access certain areas or targets, so they need to rely on machines such as unmanned vehicles to perform tasks. However, soldiers cannot effectively observe and control the actions of machines at a long distance, which may lead to inaccurate operations or inability to adapt to battlefield changes in a timely manner. Finally, there are limitations in information interaction means and information fusion display in human-machine collaboration. At present, human-machine interaction in battlefield environments mainly relies on handheld terminals for information interaction. However, this method cannot help personnel quickly grasp the situation and make decisions in dangerous or emergency situations. The usual practice is to install a camera in front of the reconnaissance vehicle and transmit its video data back to the screen of a handheld terminal or a fixed terminal for display. The disadvantage of this method is that it cannot effectively observe the environment on the left and right sides and behind the reconnaissance vehicle, which is not conducive to the overall perception of the battlefield environment.

[0006] In the face of complex battlefield environments and the need for individual soldier situational awareness, the development of an individual soldier situational awareness system based on visual enhancement perception technology is of great significance. In order to meet the urgent needs of combatants for all-round situational awareness in complex battlefield environments, this invention focuses on human-machine collaborative visual enhancement perception technology and constructs an individual soldier situational awareness system to further enhance combatants' battlefield situational awareness capabilities and the effectiveness of human-machine collaborative combat. Summary of the invention

[0007] In view of the above technical problems, the present invention provides a single-soldier situation awareness system based on visual enhancement perception technology.

[0008] The technical solution adopted by the present invention to solve the technical problem is:

[0009] A single-soldier situation awareness system based on visual enhanced perception technology includes an environmental information acquisition module, an unmanned vehicle processing center, a communication module, a head posture measurement module, a single-soldier processing center and an enhanced display module.

[0010] The environment information acquisition module is used to obtain the surrounding environment information and target information of the remote unmanned vehicle in real time through sensor equipment and send it to the unmanned vehicle processing center;

[0011] After receiving the surrounding environment information and target information of the remote unmanned vehicle, the unmanned vehicle processing center uses the SLAM algorithm to reconstruct the environmental situation in three dimensions. When the unmanned vehicle uses image recognition technology to identify a dangerous target, the category information and location information of the dangerous target are sent to the individual soldier processing center through the communication module; and the environmental information is stitched together in 360-degree video images. When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image and sent to the individual soldier processing center through the communication module;

[0012] The head posture measurement module estimates the current head posture information and measures the soldier's head rotation angle, and sends it to the individual soldier processing center;

[0013] The individual soldier processing center receives the soldier's head rotation angle and sends it to the drone processing center through the communication module; and receives the category information and location information of the dangerous target sent by the unmanned vehicle processing center and the image of the corresponding perspective of the screenshot, and then converts and enhances it using augmented reality technology and sends it to the enhanced display module for display.

[0014] Preferably, the unmanned vehicle processing center includes a three-dimensional map construction module, a target recognition and positioning module, and an image processing module.

[0015] The three-dimensional map construction module is used to reconstruct the three-dimensional map of the environment situation using the SLAM algorithm based on the received environmental information of the remote unmanned vehicle;

[0016] The target recognition and positioning module is used to perform target recognition on the received image containing target information according to the preset target recognition algorithm, and judge whether it is a dangerous target. If it is a dangerous target, the category information and location information of the dangerous target are determined in combination with the reconstructed three-dimensional map, and the category information and location information of the dangerous target are sent to the individual processing center through the communication module;

[0017] The image processing module is used to stitch 360-degree video images of environmental information. When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image and sent to the individual soldier processing center through the communication module.

[0018] Preferably, the target recognition and positioning module is also used to track dangerous targets in continuous frames using a target tracking algorithm in response to real-time requirements for image target recognition.

[0019] Preferably, the environmental information acquisition module includes multiple cameras and radars. The multiple cameras are installed at different positions on the unmanned vehicle to cover a panoramic view around the unmanned vehicle. The cameras are fisheye cameras with a large field of view and RGB-D depth cameras.

[0020] Preferably, the unmanned vehicle processing center is further used to predict the trajectory of the unmanned vehicle based on the received environmental information combined with the unmanned vehicle's own information to obtain the predicted trajectory and send it to the individual soldier computing center through a communication device; and receive control instructions sent by the individual soldier computing center to control the movement of the unmanned vehicle and update the predicted trajectory;

[0021] The individual computing center receives the real-time predicted trajectory and sends it to the enhanced display module so that the predicted trajectory is displayed in the center of the screen; and sends control instructions to the unmanned vehicle processing center.

[0022] Preferably, the individual soldier processing center receives the category information and location information of the dangerous target and the screenshot image under the corresponding viewing angle sent by the unmanned vehicle processing center, and then converts and enhances them using augmented reality technology and sends them to the enhanced display module for display, including:

[0023] After receiving the category information and location information of the dangerous target and the screenshot image from the corresponding perspective sent by the unmanned vehicle processing center, the individual processing center fuses the operator's field of view information with the category information and location information of the dangerous target and displays it, marking the location and direction of the dangerous target by means of icons, arrows or virtual models and converting them for display on the head-mounted display.

[0024] Preferably, the image processing module is used to stitch 360-degree video images of the surrounding environment information of the unmanned vehicle, including:

[0025] Collect videos shot by all cameras and pre-process the video frame images in batches; use the ORB algorithm optimized by particle swarm optimization to extract and align feature points of video images; stitch them through the Poisson fusion algorithm; combine the video frame processing to realize the display of panoramic video.

[0026] Preferably, when the soldier's head rotation angle is received, an image matching the soldier's head rotation angle is intercepted from the spliced ​​image and sent to the individual soldier processing center through the communication module, including:

[0027] When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image, and the captured field of view image is subjected to real-time graphics processing such as denoising, enhancement, and target detection. The processed image is compressed and encoded and sent to the individual soldier processing center through the communication module.

[0028] Preferably, the head posture measurement module includes a gyroscope and a visual positioning device.

[0029] Preferably, the communication module uses a high-speed data transmission protocol or a network protocol for transmission.

[0030] The above-mentioned individual situation awareness system based on visual enhancement perception technology can obtain the surrounding environment information and target information of the remote unmanned vehicle in real time through the sensor equipment of the unmanned vehicle in the case of no GPS signal in the outdoor battlefield environment, and use the SLAM algorithm to realize three-dimensional reconstruction of the environmental situation. The unmanned vehicle uses image recognition technology to identify dangerous targets, and sends its category information and location information to the AR head display device worn by the individual combat personnel, and displays it in a consistent fusion with the visual information of the combat personnel. In this way, the combat personnel can intuitively perceive the location and related information of dangerous targets in the remote environment, thereby improving the perception ability and coordinated combat effect of remote combat. The 360-degree image real-time splicing technology of the on-board visual sensor is used to achieve all-round perception of the environment around the unmanned vehicle. Specifically, the combat personnel send the current head posture information to the unmanned vehicle through head posture measurement (gyroscope and visual positioning device). After receiving the information, the unmanned vehicle intercepts the video stream corresponding to the perspective of the combat personnel in the 360-degree image, and sends it back to the head display AR device worn by the combat personnel for display. This method enables combat personnel to more accurately perceive the vehicle's surrounding environment when controlling the unmanned vehicle, thereby improving the ability to perceive the vehicle's surrounding conditions during human-machine collaborative operations. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 This is a structural block diagram of a single-soldier situation awareness system based on visual enhancement perception technology in one embodiment of the present invention;

[0032] Figure 2This is a structural block diagram of a single-soldier situation awareness system based on visual enhancement perception technology in another embodiment of the present invention;

[0033] Figure 3 This is a motion trajectory diagram of an unmanned vehicle in one embodiment of the present invention. DETAILED DESCRIPTION

[0034] In order to enable those skilled in the art to better understand the technical solution of the present invention, the present invention is further described in detail below in conjunction with the accompanying drawings.

[0035] Human-machine collaborative visual enhancement perception technology refers to improving human perception and decision-making effects in specific tasks by combining the capabilities of humans and robots. Human-machine collaborative visual enhancement perception technology provides strong support and impetus for the development of individual situational awareness systems, and injects new vitality into human perception and decision-making effects in complex tasks. Traditional individual situational awareness systems mainly rely on personnel's own perception capabilities, such as vision and hearing, and then make decisions based on personal experience and judgment. However, due to the limitations of human perception and the complex and changing real environment, human perception alone often cannot meet the needs of complex tasks. Human-machine collaborative visual enhancement perception technology makes up for the shortcomings of human perception by combining human cognition and machine computing power. Specifically, it uses advanced technologies such as computer vision, sensor technology, and artificial intelligence algorithms to efficiently and accurately perceive and analyze information in the environment. At the same time, these technologies can also present the perceived information to human operators in an intuitive and easy-to-understand way. Through visual enhancement perception technology, individual situational awareness systems can obtain key information in the scene in real time, such as target recognition, terrain analysis, obstacle detection, etc. This information can help combatants better understand the battlefield situation, provide decision support and command and dispatch. Human-machine collaborative visual enhancement perception technology has important application value in the fields of military, security, rescue, etc. It can not only enhance the perception ability of individuals, but also promote collaborative operations among multiple individuals. By coordinating with robots, individual soldiers can perceive and respond to challenges in complex scenarios more quickly and accurately, improving the efficiency and safety of mission execution.

[0036] In the face of complex battlefield environments and the need for individual soldier situational awareness, the development of an individual soldier situational awareness system based on visual enhancement perception technology is of great significance. In order to meet the urgent needs of combatants for all-round situational awareness in complex battlefield environments, this invention focuses on human-machine collaborative visual enhancement perception technology and constructs an individual soldier situational awareness system to further enhance combatants' battlefield situational awareness capabilities and the effectiveness of human-machine collaborative combat.

[0037] In one embodiment, Figure 1As shown, a single-soldier situation awareness system based on visual enhancement perception technology includes an environmental information acquisition module, an unmanned vehicle processing center, a communication module, a head posture measurement module, a single-soldier processing center and an enhanced display module.

[0038] The environment information acquisition module is used to obtain the surrounding environment information and target information of the remote unmanned vehicle in real time through sensor equipment and send it to the unmanned vehicle processing center;

[0039] After receiving the surrounding environment information and target information of the remote unmanned vehicle, the unmanned vehicle processing center uses the SLAM algorithm to reconstruct the environmental situation in three dimensions. When the unmanned vehicle uses image recognition technology to identify a dangerous target, the category information and location information of the dangerous target are sent to the individual soldier processing center through the communication module; and the environmental information is stitched together in 360-degree video images. When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image and sent to the individual soldier processing center through the communication module;

[0040] The head posture measurement module estimates the current head posture information and measures the soldier's head rotation angle, and sends it to the individual soldier processing center;

[0041] The individual soldier processing center receives the soldier's head rotation angle and sends it to the drone processing center through the communication module; and receives the category information and location information of the dangerous target sent by the unmanned vehicle processing center and the image of the corresponding perspective of the screenshot, and then converts and enhances it using augmented reality technology and sends it to the enhanced display module for display.

[0042] Specifically, in an outdoor battlefield environment, wireless communications and GPS signals may be affected or even interrupted due to enemy interference or geographical restrictions. Therefore, the present invention needs to solve the technical problems of outdoor positioning and scene intelligent perception of unmanned vehicles in a GPS-free environment. By carrying scene intelligent perception devices such as cameras and radars, the unmanned vehicle can obtain the surrounding environment information and target information of the remote unmanned vehicle in real time, and use the SLAM algorithm to perform three-dimensional reconstruction of the environmental situation. In this way, the unmanned vehicle can accurately locate its own position in the absence of GPS signals and perceive changes in the surrounding environment.

[0043] In order to achieve omnidirectional perception of the environment around the unmanned vehicle, the present invention needs to solve the technical problem of real-time stitching of multiple images captured by the on-board sensor. The images acquired by the sensor cover all directions around the vehicle, but these images need to be stitched to form a seamless 360-degree panoramic image. Therefore, it is necessary to develop an efficient and accurate image processing algorithm to stitch multiple images to achieve omnidirectional perception.

[0044] In order to match the perspective of the combatant with the surrounding environment of the unmanned vehicle, this paper needs to solve the problem of the coordinated use of head posture measurement and head-mounted AR devices. Through head posture measurement technology, the combatant's head posture information, such as rotation and tilt angles, can be obtained in real time. This information can be used to determine the perspective of the combatant and display the corresponding video stream on the head-mounted AR device, so that the combatant can accurately perceive the environment around the vehicle.

[0045] In order to enable combatants to intuitively perceive dangerous target information in a remote environment, the present invention needs to solve the problem of consistent fusion and display of visual information and position information. This involves fusing the category information and position information of dangerous targets identified by the unmanned vehicle with the visual information of combatants and displaying them on the AR device. Through consistent fusion and display, combatants can more intuitively understand the location and related information of dangerous targets, thereby improving combat effectiveness.

[0046] By solving the above key scientific and technical problems, the all-round perception ability and coordinated combat effect of combatants in complex battlefield environments can be effectively enhanced. Combatants can obtain various information about the surrounding environment in real time through sensor equipment, head posture measurement modules, unmanned vehicles and other equipment, and display it intuitively on the head-mounted AR device, so that combatants can accurately perceive and understand the battlefield environment, make more informed decisions and actions, and improve combat efficiency and safety.

[0047] In one embodiment, Figure 2 As shown in the figure, the unmanned vehicle processing center includes a three-dimensional map construction module, a target recognition and positioning module, and an image processing module.

[0048] The three-dimensional map construction module is used to reconstruct the three-dimensional map of the environment situation using the SLAM algorithm based on the received environmental information of the remote unmanned vehicle;

[0049] The target recognition and positioning module is used to perform target recognition on the received image containing target information according to the preset target recognition algorithm, and judge whether it is a dangerous target. If it is a dangerous target, the category information and location information of the dangerous target are determined in combination with the reconstructed three-dimensional map, and the category information and location information of the dangerous target are sent to the individual processing center through the communication module;

[0050] The image processing module is used to stitch 360-degree video images of environmental information. When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image and sent to the individual soldier processing center through the communication module.

[0051] In one embodiment, the environmental information acquisition module includes multiple cameras and radars. The multiple cameras are installed at different positions on the unmanned vehicle to cover a panoramic view around the unmanned vehicle. The cameras are fisheye cameras with a large field of view and RGB-D depth cameras.

[0052] In one embodiment, the target recognition and positioning module is also used to track dangerous targets in continuous frames using a target tracking algorithm in response to real-time requirements for image target recognition.

[0053] Specifically, unmanned vehicles are equipped with image sensors for target detection: high-resolution image sensors, such as fisheye cameras with large field of view and RGB-D depth cameras, are installed on unmanned vehicles. These sensors can obtain image information of the surrounding environment and use image processing algorithms to preprocess the acquired images, such as denoising and image enhancement, to improve the accuracy and robustness of subsequent target detection. Target detection and recognition: target detection algorithms, such as convolutional neural networks (CNNs) or object detectors (such as YOLO, Faster R-CNN, etc.) based on deep learning, are applied to perform target detection and recognition on the preprocessed images. These algorithms can identify different objects in the image and extract their location and category information. (3) Target tracking and feature extraction: To meet the real-time needs of image target recognition, target tracking algorithms, such as target trackers based on Kalman filtering, correlation filters or deep learning (such as MOT, DeepSORT, etc.), can be applied to track dangerous targets in continuous frames.

[0054] The unmanned vehicle is equipped with cameras and radars, which use visual information for positioning and mapping. Using VSLAM algorithms, such as feature point-based SLAM (ORB-SLAM) or direct method-based SLAM (DSO), the unmanned vehicle can be positioned and mapped in three-dimensional space by extracting features, matching, and reconstructing three-dimensional images in real time. The unmanned vehicle transmits the location, direction, and category information of dangerous targets to the remote individual head-mounted display device through wireless communication technology radio. Data transmission requires real-time and stability, and a high-speed data transmission protocol can be used to ensure the reliability and security of the communication link.

[0055] In one embodiment, the unmanned vehicle processing center is further used to predict the trajectory of the unmanned vehicle based on the received environmental information combined with the unmanned vehicle's own information to obtain a predicted trajectory and send it to the individual soldier computing center through a communication device; and receive a control command sent by the individual soldier computing center to control the movement of the unmanned vehicle and update the predicted trajectory;

[0056] The individual computing center receives the real-time predicted trajectory and sends it to the enhanced display module so that the predicted trajectory is displayed in the center of the screen; and sends control instructions to the unmanned vehicle processing center.

[0057] Specifically, the trajectory of the unmanned vehicle is as follows: Figure 3 As shown, O is the rotation center, R is the turning radius, w is the width of the mobile robot, and D is the distance from the center of the mobile robot to the front end.

[0058] Taking the rotation center as the reference point and assuming the turning curvature is k, the turning radius R is:

[0059]

[0060] Assume that R1 is the turning radius of the right front wheel and R2 is the turning radius of the left front wheel. The calculation is as follows. The trajectory that the left and right front wheels will travel is the predicted trajectory of the mobile robot.

[0061]

[0062]

[0063] In one embodiment, the individual processing center receives the category information and location information of the dangerous target and the screenshot image under the corresponding viewing angle sent by the unmanned vehicle processing center, converts and enhances them using augmented reality technology, and then sends them to the enhanced display module for display, including:

[0064] After receiving the category information and location information of the dangerous target and the screenshot image from the corresponding perspective sent by the unmanned vehicle processing center, the individual processing center fuses the operator's field of view information with the category information and location information of the dangerous target and displays it, marking the location and direction of the dangerous target by means of icons, arrows or virtual models and converting them for display on the head-mounted display.

[0065] Specifically, after the processing center of the AR headset receives the category information and location information of the dangerous target transmitted by the unmanned vehicle, it uses augmented reality technology to display the target's direction, distance and category information in real time on the headset's display screen, and frames it with a red box for enhanced display. The headset device can adjust the viewing angle according to the operator's head posture, and fuse the operator's field of view information with the target information for display, so that the operator can intuitively perceive the position and direction of the dangerous target in the real environment. The category, location and direction information of the dangerous target can also be marked on the headset display screen by icons, arrows or virtual models, thereby improving the operator's ability to perceive the dangerous target.

[0066] Furthermore, the individual processing center is also used to perform head tracking based on the received head posture data, and update the image perspective in real time, so that the operator can observe the environment around the remote unmanned vehicle and improve perception and operational efficiency.

[0067] In one embodiment, the image processing module is used to stitch 360-degree video images of the surrounding environment information of the unmanned vehicle, including:

[0068] Collect videos shot by all cameras and pre-process the video frame images in batches; use the ORB algorithm optimized by particle swarm optimization to extract and align feature points of video images; stitch them through the Poisson fusion algorithm; combine the video frame processing to realize the display of panoramic video.

[0069] Specifically, multiple high-resolution cameras or fisheye cameras are selected and installed at different positions on the vehicle to cover a panoramic view around the vehicle. The images of each camera are distorted and calibrated so that they are accurately positioned in the physical space and can be accurately matched when stitched.

[0070] In the initial stage of video acquisition, we must first ensure that the four video capture devices are started at the same time and have the same frame rate. This is the basis for ensuring that each frame is correctly matched. Due to the gaps in human vision, only when the video playback frame rate exceeds 24 frames per second will the human eye present a smooth video picture.

[0071] The process steps for implementing the video stitching method are as follows:

[0072] The first step is to collect videos shot by 6 fisheye cameras and pre-process the video frame images in batches; the second step is to use the ORB algorithm optimized by particle swarm optimization to extract and align the feature points of the video images to improve the accuracy of the registration; the third step is to use the Poisson fusion algorithm for stitching; the fourth step is to combine the video frame processing to realize the display of panoramic video.

[0073] First, the six cameras are subjected to the same-frequency image information collection, image correction, change projection, image matching, perspective transformation and image fusion. In the process of video registration, the present invention directly extracts features of the original video using the ORB algorithm, and optimizes it using the particle swarm algorithm in the search strategy stage. Such a combination is conducive to improving the speed and accuracy of registration. In the process of video fusion, the present invention performs video image splicing based on the Poisson fusion algorithm, and the video splicing is completed to obtain the final panoramic video. The panoramic image after real-time splicing can be provided to the operator for environmental perception and remote monitoring.

[0074] In one embodiment, when the soldier's head rotation angle is received, an image matching the soldier's head rotation angle is intercepted from the spliced ​​image and sent to the individual soldier processing center through the communication module, including:

[0075] When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image, and the captured field of view image is subjected to real-time graphics processing such as denoising, enhancement, and target detection. The processed image is compressed and encoded and sent to the individual soldier processing center through the communication module.

[0076] In one embodiment, the head posture measurement module includes a gyroscope and a visual positioning device.

[0077] Specifically, the present invention uses sensors such as gyroscopes to estimate the posture of the head, and can also be combined with visual positioning technology, such as feature point tracking, pattern recognition or structured light, to accurately measure the position and posture of the head. The measurement results of the head posture can be used to calculate the operator's perspective, and the images of the six cameras around the unmanned vehicle are spliced ​​into a cylindrical 360-degree annular panorama. Only one parameter, the angle of rotation of the head, is needed to capture the image of the corresponding perspective from the panorama, thereby realizing remote real-time acquisition of the image of the surrounding environment of the unmanned vehicle at the corresponding perspective.

[0078] In one embodiment, the communication module uses a high-speed data transmission protocol or a network protocol for transmission.

[0079] Specifically, head posture data can be transmitted through network protocols to ensure the real-time and stability of transmission.

[0080] In summary, the present invention builds a single-soldier situational awareness system based on human-machine collaborative visual enhancement perception technology. The system mainly has two functional modules: human-machine collaborative target discovery enhancement display module and environmental perception enhancement display module. The system uses on-board visual sensors and other equipment to achieve all-round and accurate information perception and analysis, providing combatants with a richer and more intuitive way of displaying information, thereby improving the decision-making effect and task execution efficiency of combatants.

[0081] (1) Human-machine collaborative target discovery enhanced display module.

[0082] In the case of no GPS signal in the outdoor battlefield environment, the sensor equipment (fisheye camera, RGB-D depth camera, radar, etc.) of the unmanned vehicle can obtain remote environment and target information in real time, and use the SLAM algorithm to achieve three-dimensional reconstruction of the environmental situation. Soldiers cannot observe dangerous targets outside the field of vision or hidden behind obstacles due to limited vision. However, the unmanned reconnaissance vehicle in front can observe targets that soldiers cannot observe from other perspectives. The unmanned vehicle identifies the dangerous target and sends its category information and location information to the AR head display device worn by the individual combat personnel, which is displayed in a consistent manner with the visual information of the combat personnel. The scene observed by the combat personnel through the AR head display device includes both real visual information and the category information and location information of the fused dangerous target. In this way, combat personnel can intuitively perceive the location and related information of dangerous targets in the remote environment, thereby improving the perception ability and collaborative combat effect of remote combat.

[0083] (2) Human-machine collaborative environmental perception enhanced display module. The 360-degree image real-time stitching technology of the vehicle-mounted visual sensor is used to achieve all-round perception of the environment around the unmanned vehicle. Specifically, the combatant sends the current head posture information to the unmanned vehicle through head posture measurement (gyroscope and visual positioning device). After receiving the information, the unmanned vehicle intercepts the video stream corresponding to the combatant's perspective in the 360-degree image and sends it back to the head-mounted AR device worn by the combatant for display. Soldiers can remotely obtain the panoramic perspective image of the unmanned vehicle driver through the head-mounted display device, and select the corresponding field of view according to their own body posture to perceive and monitor the environment around the vehicle in real time. This human-machine collaborative control solution helps to improve the battlefield perception capabilities of combatants.

[0084] (3) In order to fully verify the individual situational awareness system developed in this application, a demonstration system of one person + unmanned vehicle is built. The system aims to provide powerful battlefield perception and command support capabilities through the coordinated operations of humans and unmanned vehicles.

[0085] First, soldiers will be equipped with AR headsets. These devices can obtain data from the unmanned vehicle's sensors in real time and project it into the soldier's field of view. Through AR technology, soldiers can perceive information from the unmanned vehicle without direct visual contact, including remote environmental images, target locations, and related information. This augmented reality perception method enables soldiers to better analyze and understand the remote environment.

[0086] Secondly, the unmanned vehicle is equipped with various sensor devices, such as fisheye cameras, RGB-D depth cameras, and radars. These sensors can obtain target information in the environment and perform three-dimensional reconstruction to achieve perception of the environmental situation. The unmanned vehicle can help soldiers find potential dangerous targets and instantly send their category information and location information to the soldiers' AR head display devices. Through this human-machine collaborative target discovery and enhanced display function, the soldiers' perception ability and collaborative combat effect in long-range combat will be effectively improved.

[0087] In addition, the unmanned vehicle can also achieve all-round environmental perception through its on-board visual sensors. Soldiers can send the current head posture information to the unmanned vehicle through head posture measurement technology. The unmanned vehicle captures the video stream in the 360-degree image at the same perspective as the soldier, and transmits it back to the AR head display device worn by the soldier for display. In this way, soldiers can more accurately perceive the environment around the vehicle when operating the unmanned vehicle, effectively improving the perception and response capabilities of the vehicle's surrounding conditions in human-machine collaborative operations.

[0088] By building a demonstration system of people + unmanned vehicles and conducting actual verification, the feasibility and effectiveness of the individual situational awareness system can be fully verified. At the same time, through the actual operation and operational testing of the system, possible problems and room for improvement can be discovered, and the performance of the system can be further improved and optimized. The construction of this demonstration system will promote the research results of this application to the field of practical application and make positive contributions to improving the level of my country's military modernization.

[0089] The above is a detailed introduction to a single-soldier situational awareness system based on visual enhancement perception technology provided by the present invention. This article uses specific examples to illustrate the principles and implementation methods of the present invention. The description of the above embodiments is only used to help understand the core idea of ​​the present invention. It should be pointed out that for ordinary technicians in this technical field, without departing from the principles of the present invention, the present invention can also be improved and modified, and these improvements and modifications also fall within the scope of protection of the claims of the present invention.

Claims

1. A single-soldier situation awareness system based on visual enhancement perception technology, characterized in that: It includes environment information acquisition module, unmanned vehicle processing center, communication module, head posture measurement module, individual soldier processing center and enhanced display module. The environment information acquisition module is used to obtain the surrounding environment information and target information of the remote unmanned vehicle in real time through sensor equipment and send it to the unmanned vehicle processing center; After receiving the surrounding environment information and target information of the remote unmanned vehicle, the unmanned vehicle processing center uses the SLAM algorithm to reconstruct the environmental situation in three dimensions. When the unmanned vehicle uses image recognition technology to identify a dangerous target, the category information and location information of the dangerous target are sent to the individual soldier processing center through the communication module; and the environmental information is stitched together in 360-degree video images. When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image and sent to the individual soldier processing center through the communication module; The head posture measurement module estimates the current head posture information and measures the soldier's head rotation angle, and sends it to the individual soldier processing center; The individual soldier processing center receives the soldier's head rotation angle and sends it to the drone processing center through the communication module; and receives the category information and location information of the dangerous target sent by the unmanned vehicle processing center and the image of the corresponding perspective of the screenshot, and then converts and enhances it using augmented reality technology and sends it to the enhanced display module for display.

2. The method according to claim 1, characterized in that The unmanned vehicle processing center includes a 3D map building module, a target recognition and positioning module, and an image processing module. The three-dimensional map construction module is used to reconstruct the three-dimensional map of the environment situation using the SLAM algorithm based on the received environmental information of the remote unmanned vehicle; The target recognition and positioning module is used to perform target recognition on the received image containing target information according to the preset target recognition algorithm, and judge whether it is a dangerous target. If it is a dangerous target, the category information and location information of the dangerous target are determined in combination with the reconstructed three-dimensional map, and the category information and location information of the dangerous target are sent to the individual processing center through the communication module; The image processing module is used to stitch 360-degree video images of environmental information. When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image and sent to the individual soldier processing center through the communication module.

3. The method according to claim 2, characterized in that The target recognition and positioning module is also used to track dangerous targets in continuous frames using a target tracking algorithm to meet the real-time requirements of image target recognition.

4. The method according to claim 1, characterized in that The environmental information acquisition module includes multiple cameras and radars. Multiple cameras are installed at different positions on the unmanned vehicle to cover a panoramic view around the unmanned vehicle. The cameras are fisheye cameras with a large field of view and RGB-D depth cameras.

5. The method according to claim 1, characterized in that The unmanned vehicle processing center is also used to predict the trajectory of the unmanned vehicle based on the received environmental information combined with the unmanned vehicle's own information to obtain the predicted trajectory and send it to the individual computing center through the communication device; and receive the control command sent by the individual computing center to control the movement of the unmanned vehicle and update the predicted trajectory; The individual computing center receives the real-time predicted trajectory and sends it to the enhanced display module so that the predicted trajectory is displayed in the center of the screen; and sends control instructions to the unmanned vehicle processing center.

6. The method according to claim 1, characterized in that The individual processing center receives the category information and location information of the dangerous target sent by the unmanned vehicle processing center, as well as the screenshot image under the corresponding viewing angle, and then converts and enhances it using augmented reality technology and sends it to the enhanced display module for display, including: After receiving the category information and location information of the dangerous target and the screenshot image from the corresponding perspective sent by the unmanned vehicle processing center, the individual processing center fuses the operator's field of view information with the category information and location information of the dangerous target and displays it, marking the location and direction of the dangerous target by means of icons, arrows or virtual models and converting them for display on the head-mounted display.

7. The method according to claim 1, characterized in that The image processing module is used to stitch 360-degree video images of the surrounding environment information of the unmanned vehicle, including: Collect videos shot by all cameras and pre-process the video frame images in batches; use the ORB algorithm optimized by particle swarm optimization to extract and align feature points of video images; stitch them through the Poisson fusion algorithm; combine the video frame processing to realize the display of panoramic video.

8. The method according to claim 1, characterized in that When the soldier's head rotation angle is received, an image matching the soldier's head rotation angle is intercepted from the stitched image and sent to the individual soldier processing center through the communication module, including: When the soldier's head rotation angle is received, an image that matches the soldier's head rotation angle is captured from the stitched image, and the captured field of view image is subjected to real-time graphics processing such as denoising, enhancement, and target detection. The processed image is compressed and encoded and sent to the individual soldier processing center through the communication module.

9. The method according to claim 1, characterized in that: The head posture measurement module includes a gyroscope and a visual positioning device.

10. The method according to claim 1, characterized in that The communication module uses high-speed data transmission protocol or network protocol for transmission.

Images