Danger source active early warning system based on AR head-mounted display

Abstract

The invention belongs to the technical field of active early warning, and particularly discloses a dangerous source active early warning system based on an AR head-mounted display. In the application, an external information acquisition module acquires first image information around a user and on a preset path; the hazard source identification module performs hazard source identification based on the first image information and the second image information; the central micro-processing module performs 3D modeling on a scene according to the received first image information to obtain a scene model, and marks a dangerous source in the scene model by using the image information of the dangerous source; the AR head-mounted display obtains second image information under a user view angle; correcting the position difference between the scene model and the real scene by using the second image information; and if the dangerous source marked in the scene model is located in the visual angle of the AR head-mounted display, superposing a virtual element in the visual field of the AR head-mounted display to mark the dangerous source. According to the technical scheme, efficient identification and display of the hazard source are realized.

Description

Technical Field

[0001] The present application belongs to the field of active early warning technology, and more specifically, relates to an active early warning system for dangerous sources based on an AR head display. Background Art

[0002] In contemporary society, danger warning is an important means to ensure personnel safety. It can provide information before potential danger occurs, help personnel take defensive measures, and prevent or avoid harm.

[0003] With the improvement of science and technology, various reconnaissance equipment can be applied to danger warning, such as radar, micro unmanned reconnaissance vehicles, unmanned reconnaissance aircraft, infrared life detectors and other equipment, so that more detection techniques can be applied to danger warning. However, although various types of sensors are currently used to collect danger information, the information is not processed and extracted. Users need to manually process the huge amount of intelligence information obtained to obtain the required battlefield information. This process consumes a lot of time, resulting in low efficiency in information utilization and untimely danger warning. Summary of the invention

[0004] In view of the defects of the prior art, the purpose of this application is to provide an active warning system for dangerous sources based on AR head display, aiming to solve the current technical problem of low efficiency in hazard warning processing.

[0005] To achieve the above objectives, in a first aspect, the present application provides a hazard source active warning system based on an AR head display, the hazard source active warning system comprising the following parts: An external information acquisition module, used to obtain first image information around the user and on a preset path, and send the first image information to the hazard source identification module and the central micro-processing module; A hazard source identification module, used for identifying the hazard source based on the first image information and the second image information, and sending the image information of the identified hazard source to the central micro-processing module; The central microprocessor module is used to perform 3D modeling of the scene according to the received first image information to obtain a scene model, mark the danger source in the scene model using the image information of the danger source, and send the scene model to the AR head display; The AR head display is used to obtain second image information from the user's perspective and send the second image information to the hazard source identification module; use the second image information to correct the position difference between the scene model and the real scene; if the hazard source marked in the scene model is within the perspective of the AR head display, a virtual element is superimposed in the field of view of the AR head display to identify the hazard source.

[0006] Preferably, in the central microprocessing module, the image information of the danger source is used to mark the danger source in the scene model, specifically: Acquire an image of the hazard source from the image information of the hazard source; Find out the model formed when participating in 3D modeling based on the image; The model is marked as a hazard source in the scenario model.

[0007] Preferably, the central microprocessing module is also used to extract the outline of the danger source in the scene model; read the second image information from the AR headset, and obtain the position of the AR headset based on the second image information, obtain the position of the danger source based on the image information of the danger source, and obtain the distance and direction between the AR headset and the danger source from the positions of the AR headset and the danger source; and send the outline of the danger source and the distance and direction between the danger source and the AR headset to the AR headset for display.

[0008] Preferably, in the AR head display, the position difference between the scene model and the real scene is corrected using the second image information, specifically: Obtaining an image, a position of the AR headset, an image depth of field, and an image viewing angle from the second image information; Extracting an image contour from the image; matching the image contour in the scene model to obtain an approximate position of the AR head display relative to the scene model; Derived the position of the AR head display relative to the scene model from the image depth of field, the image viewing angle and the approximate orientation; The real position of the scene model is corrected by the position of the AR headset.

[0009] Preferably, in the AR head display, if the danger source marked in the scene model is located within the field of view of the AR head display, a virtual element is superimposed in the field of view of the AR head display to identify the danger source, specifically: The position between the danger source and the AR head display is obtained based on the position of the danger source and the position of the AR head display; It is determined whether the position is within the field of view of the AR head display. If so, a virtual element is superimposed in the field of view of the AR head display to identify the danger source.

[0010] Preferably, in the hazard source identification module, the hazard source is identified through a pre-trained target recognition model; the target recognition model is constructed based on the YOLOv5 model, and the output of the residual block in the backbone network is inserted into the ECA module, and the channel attention weight is introduced in the feature extraction process; after the target recognition model is trained, the model is lightweighted through model pruning, and finally the lightweight target recognition model is transplanted into the hazard source identification module.

[0011] Preferably, the external information collection module includes but is not limited to one or more combinations of drones, unmanned vehicles, and robots.

[0012] Preferably, the first image information and the second image information include: an image, an image depth of field, an image viewing angle, an image acquisition time, an image acquisition device, and a position of the image acquisition device.

[0013] Preferably, the external information acquisition module and the AR head display include a positioning unit, and the positioning unit is used to obtain the positions of the external information acquisition module and the AR head display in real time.

[0014] In a second aspect, the present application provides an early warning method for a dangerous source active early warning system applied to any one of the AR head displays in the first aspect, the early warning method comprising the following steps: The external information acquisition module acquires first image information around the user and on a preset path in real time, and sends the first image information to the hazard source identification module and the central micro-processing module; The AR head display obtains the second image information from the user's perspective in real time, and sends the second image information to the hazard source identification module; The hazard source identification module identifies the hazard source based on the first image information and the second image information, and sends the image information of the identified hazard source to the central micro-processing module; The central microprocessor module performs 3D modeling of the scene according to the received first image information to obtain a scene model, marks the danger source in the scene model using the image information of the danger source, and sends the scene model to the AR head display; The AR headset uses the second image information to correct the position difference between the scene model and the real scene; if the danger source marked in the scene model is within the field of view of the AR headset, a virtual element is superimposed in the field of view of the AR headset to identify the danger source.

[0015] In general, the above technical solutions conceived by this application have the following beneficial effects compared with the prior art: (1) This application models the intelligence information collected by the external information collection module and uses visual enhancement technology to accurately present the hazard source to the user through an AR head display, thereby enabling the user to obtain the hazard source information intuitively and quickly without affecting the user's perception of the actual situation.

[0016] (2) This application not only relies on the AR headset carried by the user to collect intelligence information for hazard identification, but also uses the intelligence information collected by the external information collection module for hazard identification, thereby expanding the scope and perspective of hazard identification. Even if there is a hazard source outside the user's perspective in a complex area environment, it can effectively identify and inform the user, thereby realizing a penetrating identification and labeling of the hazard source.

[0017] (3) This application uses a lightweight, channel-attention-enhanced target recognition model to identify dangerous sources. It does not reduce the model effect while reducing the model scale, effectively improves the information extraction ability of users in the area, and greatly reduces the time users spend analyzing intelligence information. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 It is a structural schematic diagram of a hazard source active warning system based on an AR head display provided in an embodiment of the present application.

[0019] Figure 2 It is a schematic diagram of the physical composition of a hazard source active warning system based on an AR head display provided in an embodiment of the present application.

[0020] Figure 3 It is the augmented reality picture displayed in the AR head display provided in the embodiment of the present application.

[0021] Figure 4 It is a flow chart of the active warning method for dangerous sources provided in the embodiment of the present application. DETAILED DESCRIPTION

[0022] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0023] The terms “first”, “second” and the like in the specification and claims herein are used to distinguish different objects rather than to describe a specific order of the objects.

[0024] In the embodiments of the present application, words such as "exemplary" or "for example" are used to indicate examples, illustrations or descriptions. Any embodiment or design described as "exemplary" or "for example" in the embodiments of the present application should not be interpreted as being more preferred or more advantageous than other embodiments or designs. Specifically, the use of words such as "exemplary" or "for example" is intended to present related concepts in a specific way.

[0025] In the description of the embodiments of the present application, unless otherwise specified, “plurality” means two or more than two.

[0026] First, the technical terms involved in the embodiments of the present application are introduced.

[0027] AR headset: A head-mounted display that overlays virtual elements on the real world, allowing users to see a combination of reality and virtuality.

[0028] The embodiments of the present application are described below in conjunction with the drawings in the embodiments of the present application.

[0029] Embodiment 1: like Figure 1 As shown, the active warning system for dangerous sources in Example 1 of the present application specifically includes the following parts: (1) An external information acquisition module, used to obtain first image information around the user and on a preset path, and send the first image information to the hazard source identification module and the central micro-processing module.

[0030] Among them, the external information collection module includes but is not limited to one or more combinations of drones, unmanned vehicles, and robots.

[0031] The first image information includes: image, image depth of field, image viewing angle, image acquisition time, image acquisition device and position of the image acquisition device.

[0032] (2) A hazard source identification module, configured to identify the hazard source based on the first image information and the second image information, and send the image information of the identified hazard source to the central microprocessing module.

[0033] Among them, the identification of hazardous sources is completed through a pre-trained target recognition model; the target recognition model is built based on the YOLOv5 model, and the output of the residual block in the backbone network is inserted into the ECA module, and the channel attention weight is introduced in the feature extraction process; after the target recognition model is trained, the model is lightweighted through model pruning, and finally the lightweight target recognition model is transplanted into the hazardous source identification module.

[0034] (3) A central microprocessor module, configured to perform 3D modeling of the scene based on the received first image information to obtain a scene model, mark the hazard source in the scene model using the image information of the hazard source, and send the scene model to the AR headset.

[0035] Among them, the image information of the hazard source is used to mark the hazard source in the scene model, specifically: Acquire an image of the hazard source from the image information of the hazard source; Find out the model formed when participating in 3D modeling based on the image; The model is marked as a hazard source in the scenario model.

[0036] The central microprocessor module is also used to extract the outline of the danger source in the scene model; read the second image information from the AR headset, and obtain the position of the AR headset based on the second image information, obtain the position of the danger source based on the image information of the danger source, and obtain the distance and direction between the AR headset and the danger source from the positions of the AR headset and the danger source; and send the outline of the danger source and the distance and direction between the danger source and the AR headset to the AR headset for display.

[0037] (4) an AR head display, used to obtain second image information from the user's perspective and send the second image information to the hazard source identification module; use the second image information to correct the position difference between the scene model and the real scene; if the hazard source marked in the scene model is within the perspective of the AR head display, superimpose a virtual element in the field of view of the AR head display to identify the hazard source.

[0038] The second image information includes: image, image depth of field, image viewing angle, image acquisition time, image acquisition device and position of the image acquisition device.

[0039] The second image information is used to correct the position difference between the scene model and the real scene, specifically: Obtaining an image, a position of the AR headset, an image depth of field, and an image viewing angle from the second image information; Extracting an image contour from the image; matching the image contour in the scene model to obtain an approximate position of the AR head display relative to the scene model; Derived the position of the AR head display relative to the scene model from the image depth of field, the image viewing angle and the approximate orientation; The real position of the scene model is corrected by the position of the AR headset.

[0040] If the danger source marked in the scene model is within the field of view of the AR head display, a virtual element is superimposed in the field of view of the AR head display to identify the danger source, specifically: The position between the danger source and the AR head display is obtained based on the position of the danger source and the position of the AR head display; It is determined whether the position is within the field of view of the AR head display. If so, a virtual element is superimposed in the field of view of the AR head display to identify the danger source.

[0041] (5) Positioning unit. Both the external information acquisition module and the AR head display include a positioning unit. The positioning unit is used to obtain the positions of the external information acquisition module and the AR head display in real time.

[0042] Embodiment 2: like Figure 2 As shown, the active warning system for dangerous sources in Example 2 of the present application specifically includes the following parts: AR head display module, namely AR head display, is a head-mounted augmented reality device that integrates visible light camera, infrared camera, depth camera, and communication module, realizing the user's first-person perspective intelligence collection, intelligence transmission, three-dimensional penetrating intelligence display and other functions; The external intelligence information collection device, i.e., the external information collection module, is a highly automated intelligent vehicle such as a drone, an all-terrain unmanned vehicle, a robot dog, etc. that integrates detection and communication sensors such as visible light cameras, infrared cameras, depth cameras, and communication modules to collect intelligence information around the user and on the predetermined road; The positioning and navigation module, also known as the positioning unit, is located inside the AR head display module and the external intelligence information collection device. It is composed of a high-precision positioning chip and a navigation system, and is responsible for navigation path planning, external module precise positioning, and viewing angle positioning. The hazard source active warning module, namely the hazard source identification module, is composed of a video intelligence real-time detection and identification model, which is responsible for identifying the hazard source using the collected intelligence information and transmitting the identified hazard source information to the micro central processing unit; The micro central processing unit, namely the central micro processing module, is composed of a single-soldier portable server. It is responsible for constructing a scene model based on the received intelligence information, identifying the danger sources therein, and determining the angle and distance of the danger source relative to the user based on the position and viewing angle positioning information when the danger source information appears, and transmitting this information to the head display for display.

[0043] The above parts constitute a set of active early warning system for dangerous sources. During combat, the system can collect intelligence information within the user's current field of view and on the planned route through external intelligence information collection devices and information collection devices on the AR head display, and transmit the intelligence information to the micro central processing unit for information processing. The dangerous sources in the intelligence information are identified through the pre-trained neural network system. When the system identifies the dangerous source information, it calibrates the dangerous source information, determines the position and distance of the dangerous source relative to the AR head display according to the high-precision positioning module, and displays it in the AR head display accordingly, so as to realize active prediction of dangerous source information, especially dangerous sources hidden behind obstacles. The dangerous sources can be marked by penetrating display, such as Figure 3 As shown in the figure, all the dark and obscured danger sources in the user's perspective are identified.

[0044] Embodiment 3, The application system is introduced through the entire workflow of the application system: Construction of the active warning module for dangerous sources: The main purpose of this module is to realize real-time detection and analysis of the collected intelligence and identify dangerous sources. The core content of the module is the detection algorithm. The current mainstream single-stage target detection algorithm is used to construct the detection algorithm, such as the YOLOv5 model. Since target detection consumes a lot of computing power, and users cannot carry large servers with high computing power when conducting single-soldier operations, the constructed model is modified to be lightweight and adaptive to obtain a lightweight model. The lightweight model is then accelerated and compressed by further quantization and pruning. For the YOLOv5 model, it has the characteristics of large volume and high computing power requirements. The ECA-Net (Effcient Channel Attention) method is introduced to the model. By enhancing the efficiency of the model channel attention mechanism, the channel attention weight is learned to emphasize the information-rich channels and suppress irrelevant channels to achieve the purpose of suppressing irrelevant background information, improve the efficiency of the model, and reduce the computing overhead. Then the model is pruned on the network structure to delete branches that have a relatively small impact on the calculation results, so as to achieve the purpose of reducing the volume. Finally, the quantization model is used to improve the computing efficiency and achieve the purpose of improving the computing performance of the model. Through the above steps, the detection algorithm is constructed. After the algorithm is constructed, the model is trained. Use relevant data sets, perform image enhancement on the data sets and then send them to the model for training. Select a smaller learning rate at the beginning of training to warm up the model. After the training is completed, the module is constructed.

[0045] Establishment of external intelligence collection system: The main function of the external intelligence collection system is to collect battlefield information from a global perspective and collect information on the combat path in advance. It is mainly constructed by drones and all-terrain unmanned vehicles (or robot dogs). Through the collection equipment such as visible light cameras, depth cameras and infrared cameras attached to the equipment, battlefield information is fully collected.

[0046] Establishment of scene model: The external intelligence collection system transmits the collected battlefield intelligence to the micro central processing unit. The central processing unit uses visible light images and depth images to perform rough modeling of the environment model to facilitate subsequent perspective correction and correct presentation of intelligence information.

[0047] Identification of dangerous sources: The battlefield intelligence information collected by the external intelligence collection system is transmitted to the dangerous source active warning module, the intelligence information is processed by the detection algorithm in the active warning module, and the identified dangerous source information is then transmitted to the micro central processing unit.

[0048] Establishment of high-precision positioning and navigation module: The main function of the navigation and positioning module is to achieve high-precision positioning of the head display and external equipment to achieve accurate positioning of the intelligence collection perspective. The construction of this module is mainly realized through high-precision positioning chips. By installing high-precision positioning chips on each device, the position and angle relationship between the devices is established. As the intelligence information is transmitted between devices, the positioning information is transmitted synchronously. At the same time, the high-precision positioning and navigation system provides route planning for external devices to ensure automatic cruising of external devices.

[0049] Positioning correction of hazard source information: After receiving the hazard source identification information and positioning information, the micro central processing unit completes the establishment of the rough model, and then corrects the precise angle position and distance of the hazard source relative to the head display according to the positioning information, and transmits the corrected hazard source position, distance, and contour information to the AR head display module.

[0050] Presentation of hazard source information: The AR headset acquires the user's first-person perspective through the camera integrated in it, and completes the three-dimensional registration of the AR headset (i.e., the correct positioning of the headset) by extracting the real-time contour model from the picture to match the established rough model. The micro central processing unit transmits the direction, distance, and contour information of the hazard source to the AR headset, and superimposes the contour of the hazard source in the correct position through augmented reality technology, presenting it to the user in a three-dimensional, penetrating and intuitive manner.

[0051] The following is a description of the early warning method provided by the present application. The early warning method described below and the active early warning system for dangerous sources described above can be referred to in correspondence with each other. Figure 4 As shown, the following steps are included: The external information acquisition module acquires first image information around the user and on a preset path in real time, and sends the first image information to the hazard source identification module and the central micro-processing module; The AR head display obtains the second image information from the user's perspective in real time, and sends the second image information to the hazard source identification module; The hazard source identification module identifies the hazard source based on the first image information and the second image information, and sends the image information of the identified hazard source to the central micro-processing module; The central microprocessor module performs 3D modeling of the scene according to the received first image information to obtain a scene model, marks the danger source in the scene model using the image information of the danger source, and sends the scene model to the AR head display; The AR headset uses the second image information to correct the position difference between the scene model and the real scene; if the danger source marked in the scene model is within the field of view of the AR headset, a virtual element is superimposed in the field of view of the AR headset to identify the danger source.

[0052] It can be understood that the detailed implementation of each of the above steps can be found in the introduction of the aforementioned system embodiment, and will not be repeated here.

[0053] It will be easily understood by those skilled in the art that the above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

[0054] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any person skilled in the art who is familiar with the present technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.

Claims

1. An active warning system for dangerous sources based on AR head display, characterized in that: The active early warning system for dangerous sources includes: An external information acquisition module, used to obtain first image information around the user and on a preset path, and send the first image information to the hazard source identification module and the central micro-processing module; A hazard source identification module, used for identifying the hazard source based on the first image information and the second image information, and sending the image information of the identified hazard source to the central micro-processing module; The central microprocessor module is used to perform 3D modeling of the scene according to the received first image information to obtain a scene model, mark the danger source in the scene model using the image information of the danger source, and send the scene model to the AR head display; The AR head display is used to obtain second image information from the user's perspective and send the second image information to the hazard source identification module; use the second image information to correct the position difference between the scene model and the real scene; if the hazard source marked in the scene model is within the perspective of the AR head display, a virtual element is superimposed in the field of view of the AR head display to identify the hazard source.

2. The active early warning system for dangerous sources according to claim 1 is characterized in that: In the central microprocessing module, the image information of the danger source is used to mark the danger source in the scene model, specifically: Acquire an image of the hazard source from the image information of the hazard source; Find out the model formed when participating in 3D modeling based on the image; The model is marked as a hazard source in the scenario model.

3. The active early warning system for dangerous sources according to claim 1 is characterized in that: The central microprocessing module is further used to extract the outline of the danger source in the scene model; read the second image information from the AR head display, and obtain the position of the AR head display based on the second image information, obtain the position of the danger source based on the image information of the danger source, and obtain the distance and direction between the AR head display and the danger source from the positions of the AR head display and the danger source; The outline of the hazard source, the distance and direction between the hazard source and the AR headset are sent to the AR headset for display.

4. The active early warning system for dangerous sources according to claim 1 is characterized in that: In the AR head display, the position difference between the scene model and the real scene is corrected using the second image information, specifically: Obtaining an image, a position of the AR headset, an image depth of field, and an image viewing angle from the second image information; Extracting an image contour from the image; matching the image contour in the scene model to obtain an approximate position of the AR head display relative to the scene model; Derived the position of the AR head display relative to the scene model from the image depth of field, the image viewing angle and the approximate orientation; The real position of the scene model is corrected by the position of the AR headset.

5. The active early warning system for dangerous sources according to claim 1 is characterized in that: In the AR head display, if the danger source marked in the scene model is within the field of view of the AR head display, a virtual element is superimposed in the field of view of the AR head display to identify the danger source, specifically: The position between the danger source and the AR head display is obtained based on the position of the danger source and the position of the AR head display; It is determined whether the position is within the field of view of the AR head display. If so, a virtual element is superimposed in the field of view of the AR head display to identify the danger source.

6. The active early warning system for dangerous sources according to claim 1 is characterized in that: In the hazard source identification module, the hazard source is identified through a pre-trained target recognition model; the target recognition model is built based on the YOLOv5 model, and the output of the residual block in the backbone network is inserted into the ECA module, and the channel attention weight is introduced in the feature extraction process; after the target recognition model is trained, the model is lightweighted through model pruning, and finally the lightweight target recognition model is transplanted into the hazard source identification module.

7. The active early warning system for dangerous sources according to claim 1 is characterized in that: The external information collection module includes but is not limited to one or more combinations of drones, unmanned vehicles, and robots.

8. The active early warning system for dangerous sources according to claim 1 is characterized in that: The first image information and the second image information include: an image, an image depth of field, an image viewing angle, an image acquisition time, an image acquisition device, and a position of the image acquisition device.

9. The active early warning system for dangerous sources according to claim 1 is characterized in that: The external information acquisition module and the AR head display include a positioning unit, and the positioning unit is used to obtain the positions of the external information acquisition module and the AR head display in real time.

10. An early warning method applied to the active early warning system for dangerous sources according to any one of claims 1 to 9, characterized in that: The early warning method comprises the following steps: The external information acquisition module acquires first image information around the user and on a preset path in real time, and sends the first image information to the hazard source identification module and the central micro-processing module; The AR head display obtains the second image information from the user's perspective in real time, and sends the second image information to the hazard source identification module; The hazard source identification module identifies the hazard source based on the first image information and the second image information, and sends the image information of the identified hazard source to the central micro-processing module; The central microprocessor module performs 3D modeling of the scene according to the received first image information to obtain a scene model, marks the danger source in the scene model using the image information of the danger source, and sends the scene model to the AR head display; The AR headset uses the second image information to correct the position difference between the scene model and the real scene; if the danger source marked in the scene model is within the field of view of the AR headset, a virtual element is superimposed in the field of view of the AR headset to identify the danger source.

Images