A method for rapid generation of semantic battlefield situation based on image parsing

Abstract

This invention discloses a rapid method for generating semantic battlefield situational awareness based on image analysis, belonging to the field of battlefield situational awareness generation technology. The method includes: dividing the battlefield area into multiple local battlefields; conducting reconnaissance of each local battlefield area; generating a multi-source heterogeneous image database; unifying the spatiotemporal relationships of the multi-source heterogeneous image database; constructing a battlefield situational awareness element composition system; constructing a battlefield situational awareness element extraction technology system; adaptively extracting battlefield situational awareness elements; generating local battlefield situational awareness; and generating global battlefield situational awareness based on local battlefield situational awareness. This invention proposes a deep learning-based battlefield situational awareness element extraction architecture, designs a deep learning-based battlefield situational awareness generation method, proposes an attention-based conflict situational awareness fusion method and a battlefield area division method, as well as a complete spatiotemporal relationship unification method, and constructs a complete set of adaptive battlefield situational awareness generation methods, which can provide decision-making assistance to direct commanders.

Description

Technical Field

[0001] This invention belongs to the field of battlefield situation construction technology, specifically referring to a method for rapid generation of semantic battlefield situation based on image analysis, which can be used for rapid generation of semantic battlefield situation to provide support for combat command. Background Technology

[0002] Unmanned warfare is a significant trend in intelligent future warfare. Unmanned / unmanned collaboration and manned / unmanned collaboration are the main modes of future combat, and the operational collaboration capability of unmanned platforms is key to winning future wars. In complex battlefield conditions, unmanned combat platforms face harsh electromagnetic warfare environments. Network communication between unmanned platforms and between manned / unmanned platforms is limited by non-real-time and narrow bandwidth, making it impossible to support the real-time exchange of large amounts of information. Battlefield situation analysis can comprehensively reflect the real-time status of the entire battlefield environment, the deployment of enemy and friendly forces, and combat operations. It can also rapidly respond to and infer the enemy's strategic deployments, and its information volume is relatively small, making it suitable for the harsh communication environment of the battlefield. Therefore, a correct understanding of the battlefield situation is a prerequisite for unmanned platforms to achieve operational collaboration.

[0003] In unmanned warfare, imagery data is crucial intelligence data, serving as an important supplement to electromagnetic intelligence data. However, battlefield imagery data is often multi-source and heterogeneous, with significant spatiotemporal mismatches. Therefore, based on multi-source heterogeneous battlefield imagery data, research is needed to develop a semantic integrated situational awareness construction technology for complex battlefields that unifies spatiotemporal relationships. This will enable a unified semantic representation of the battlefield situation, facilitating real-time battlefield command and control, improving decision-making efficiency, and providing support for unmanned warfare.

[0004] Previous studies have divided situation building into multiple modules, such as data fusion, system construction, estimation algorithms, expert knowledge base, and algorithm evaluation. These modules are isolated and not organically integrated. Some modules are based on manually designed rules, which can seriously affect the effectiveness of the entire system. Moreover, situation assessment results are mostly situation classifications, such as offensive, defensive, and retreat, and cannot reveal the logical relationships between core battlefield objectives. All of these factors have hindered the further application of situation building in modern warfare. Summary of the Invention

[0005] In view of this, the present invention proposes a rapid method for generating semantic battlefield situation based on image parsing. This method obtains the local battlefield situation based on image parsing and generates the global battlefield situation based on the local battlefield situation through an attention mechanism.

[0006] To achieve the above objectives, the technical solution adopted by the present invention is as follows:

[0007] A method for rapid generation of battlefield semantic situation based on image parsing includes the following steps:

[0008] Step 1: Divide the battlefield area into multiple local battlefields;

[0009] Step 2: Conduct reconnaissance of each local battlefield area;

[0010] Step 3: Generate a multi-source heterogeneous image database;

[0011] Step 4: Unify the spatiotemporal relationships of the multi-source heterogeneous image database;

[0012] Step 5: Construct a system of battlefield situational elements;

[0013] Step 6: Construct a technical system for extracting battlefield situational elements;

[0014] Step 7: Adaptively extract battlefield situational elements;

[0015] Step 8: Generate local battlefield situation;

[0016] Step 9: Generate the global battlefield situation based on the local battlefield situation.

[0017] Furthermore, in step 1, one or more of the following rules are used to divide the battlefield area:

[0018] 1a) If there are mountains or rivers in the battlefield area, then these shall be used as the local battlefield boundary.

[0019] 1b) If there are multiple administrative regions within the battlefield area, then the administrative regions shall be directly regarded as local battlefield areas;

[0020] 1c) If multiple local battlefields have already formed within the battlefield area, the current local battlefield will be directly used as the result of the area division.

[0021] 1d) Use geometric shapes to divide the battlefield area;

[0022] 1e) Based on existing intelligence, cluster the combat targets within the battlefield area and use the clustering results as the basis for area division;

[0023] 1f) Based on past battle examples, divide the local area.

[0024] Furthermore, in step 2, the methods for conducting reconnaissance of each local battlefield area include:

[0025] 2a) Use drones to conduct reconnaissance of local battlefields;

[0026] 2b) Use manned aircraft to conduct reconnaissance of local battlefields;

[0027] 2c) Use satellites to conduct reconnaissance of local battlefields.

[0028] Furthermore, the multi-source heterogeneous image database mentioned in step 3 includes light images, infrared images, SAR images, and optical remote sensing images.

[0029] Furthermore, step 4 is specifically implemented as follows:

[0030] 4a) Based on past battle examples and target statistics of important buildings or areas, the local battlefield is divided into multiple key areas;

[0031] 4b) Divide the time into multiple time periods by using either average time period division or hotspot time period extraction;

[0032] 4c) Based on the results of local battlefield area division and time period division, construct a multi-source heterogeneous data catalog;

[0033] 4d) Place images located in the same local area in the same directory;

[0034] 4f) Place images from the same time period into the same directory.

[0035] Furthermore, the battlefield situation element composition system described in step 5 includes:

[0036] 5a) Force deployment and combat capability, including: the strength, order of battle, troop deployment, and troop losses of the enemy, our side, and our allies;

[0037] 5b) Environmental factors, including: traffic conditions, climate and meteorology, and geographical environment, are described below:

[0038] ① Transportation conditions: highways, railways, aviation, and waterways;

[0039] ② Meteorology and Climate: Weather forecast, airport weather, ocean waves, sea breeze;

[0040] ③Geographical environment: topography, river system, important landmarks.

[0041] Furthermore, the battlefield situation element extraction technology system described in step 6 includes:

[0042] 6a) Intelligent target detection of images is achieved through target detection to extract combat targets, and important markers are intelligently identified to extract information about them;

[0043] 6b) Achieve intelligent classification of image scenes through scene classification;

[0044] 6c) Accurate image recognition is achieved through ground feature recognition, identifying the specific category of a certain area in the image.

[0045] Furthermore, step 7 is specifically implemented as follows:

[0046] 7a) Based on the actual scenario, select deep learning models for target detection, scene classification, and ground feature recognition to extract battlefield elements;

[0047] 7b) Using pre-reconnaissance battlefield intelligence, the extracted situational elements are classified and statistically analyzed to obtain the ownership of the combat target, troop deployment and losses, and battlefield environment.

[0048] Furthermore, step 8 is specifically implemented as follows:

[0049] By employing deep neural networks and training them on battlefield data, the system can automatically select battlefield situation elements and generate battlefield situations by following the methods used in classification problems.

[0050] Furthermore, step 9 is specifically implemented as follows:

[0051] By introducing an attention mechanism, corresponding weight values ​​are automatically assigned to different local battlefield situations to measure the contribution of local battlefield situations to the understanding of the global battlefield situation, thereby generating the global battlefield situation. The global battlefield situation generation method based on the attention mechanism is described as follows:

[0052] α=(α1,…,α N )

[0053] α=softmax{σ[W2ReLU(W1·evident)]}

[0054] α1+…+α N =1

[0055] Where α represents the weight vector obtained by the Attention module, σ represents the sigmoid function, ReLU represents the non-linear activation function, softmax represents the softmax function, W1 and W2 are the parameters of the Attention module, and evident represents the local battlefield situation vector.

[0056] Multiply the weight vector by the local battlefield situation vector to obtain the weighted local battlefield situation vector;

[0057] The global battlefield situation vector is obtained by taking the maximum value of the corresponding element of the weighted local battlefield situation vector.

[0058] Compared with the prior art, the present invention has the following advantages:

[0059] 1. Battlefield situations are complex and ever-changing, containing various image data types. Moreover, situational elements generally include environmental features, object types, object dynamics, and other elements, making extraction difficult. Traditional manual situational element extraction methods lack theoretical guidance and are time-consuming and labor-intensive. This invention proposes a deep learning-based battlefield situational element extraction architecture that eliminates the need for manual situational element extraction based on expert experience, enabling adaptive extraction of battlefield situational elements and improving the efficiency of battlefield situational element extraction.

[0060] 2. Most existing situation generation methods require many assumptions and are usually only applicable to a single scenario. This invention proposes a battlefield situation generation method based on deep learning, which fits the mapping relationship between battlefield situation elements and the current battlefield situation. It does not rely on fixed rules to generate battlefield situation, realizes adaptive generation of battlefield situation, and improves the efficiency of battlefield situation generation.

[0061] 3. Multi-source heterogeneous battlefield image data suffers from severe spatiotemporal discretization, making battlefield situation conflicts difficult to avoid. Considering that different local battlefield situations contribute differently to the global battlefield situation, a conflict situation fusion method based on an attention mechanism is proposed. This method can learn and update itself based on battlefield data, thereby improving the accuracy of the generated battlefield situation results.

[0062] 4. This invention proposes complete guiding principles for the division of battlefield areas, including topography, administrative division, geometric shapes, existing intelligence, and past battle examples. One or more of these principles can be used to divide the global battlefield into multiple local battlefields.

[0063] 5. To address the problem of discrete spatiotemporal relationships in multi-source heterogeneous battlefield data, this invention proposes a complete method for unifying spatiotemporal relationships, including methods for selecting key areas and dividing time periods.

[0064] 6. This invention constructs a complete set of battlefield semantic situation adaptive generation methods, which do not require human intervention in the intermediate process and can quickly generate battlefield semantic situation.

[0065] 7. The method of this invention can abstract the original image information into semantic information, achieve exponential compression of information, facilitate information transmission, and provide decision support for commanders at a higher semantic level. Attached Figure Description

[0066] Figure 1 This is the overall flowchart of the present invention.

[0067] Figure 2 Classify battlefield situation elements.

[0068] Figure 3 A diagram illustrating the extraction scheme for battlefield situational elements.

[0069] Figure 4 Generate a scenario map for the local battlefield situation.

[0070] Figure 5 This is a schematic diagram of global battlefield situation generation based on an attention mechanism. Detailed Implementation

[0071] The technical solution and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0072] Reference Figure 1 A method for rapid generation of battlefield semantic situation based on image parsing includes the following steps:

[0073] Step 1: Divide the battlefield area into multiple local battlefields, specifically as follows:

[0074] 1a) If there are typical landforms such as mountains and rivers in the battlefield area, they can be used as the local battlefield boundary;

[0075] 1b) If there are multiple administrative regions within the battlefield area, the administrative regions can be directly used as local battlefield areas;

[0076] 1c) If multiple local battlefields have already formed within the battlefield area, the current local battlefield can be directly used as the result of the area division;

[0077] 1d) Use geometric shapes such as rectangles and hexagons to directly divide the region;

[0078] 1e) Based on existing intelligence, cluster the combat targets in the area and use the clustering results as the basis for area division;

[0079] 1f) Based on past battle examples, this area is divided into local zones.

[0080] Step 2: Conduct reconnaissance of each local battlefield area, specifically as follows:

[0081] 2a) Use drones to conduct reconnaissance of local battlefields;

[0082] 2b) Use manned aircraft to conduct reconnaissance of local battlefields;

[0083] 2c) Use satellites to conduct reconnaissance of local battlefields.

[0084] Step 3: Generate a multi-source heterogeneous image database, including images of light, infrared, SAR, optical remote sensing, etc.

[0085] Step 4: Unify the spatiotemporal relationships of the multi-source heterogeneous image database. The specific method is as follows:

[0086] 4a) Divide the local battlefield into multiple key areas, with specific guiding principles including: past battle examples, important buildings, and target statistics within the area;

[0087] 4b) Divide the time into multiple time periods; specific guiding principles include: dividing time periods evenly, extracting hot time periods (based on target statistics within that time period), etc.

[0088] 4c) Construct a multi-source heterogeneous data catalog based on the results of local battlefield area division and time period division;

[0089] 4d) Place images located in the same local area in the same directory;

[0090] 4f) Place images from the same time period into the same directory.

[0091] Step 5, construct a system of battlefield situational elements, such as Figure 2 As shown, it includes:

[0092] 5a) Force deployment and combat capability, including: the strength, order of battle, troop deployment, and troop losses of the enemy, our side, and our allies;

[0093] 5b) Environmental factors, including: traffic conditions, climate and meteorology, and geographical environment, are described below:

[0094] ① Transportation conditions: highways, railways, aviation, and waterways;

[0095] ② Meteorology and Climate: Weather forecast, airport weather, ocean waves, sea breeze;

[0096] ③Geographical environment: topography, river system, important landmarks.

[0097] Step 6, construct a technical system for extracting battlefield situational elements, including:

[0098] 6a) Target detection enables intelligent target detection of images, extracts combat targets, and intelligently identifies important markers to extract information about them;

[0099] 6b) Scene classification enables intelligent classification of image scenes, which can be used for climate and meteorological classification, geographical environment classification, etc.

[0100] 6c) Land feature recognition enables accurate image recognition, identifying the specific category of a certain area in the image, such as bridges, railways, etc.

[0101] Step 7, adaptively extract battlefield situational elements, such as... Figure 3 As shown, the specific method is as follows:

[0102] 7a) Based on the actual scenario, select appropriate deep learning models such as target detection, scene classification, and ground feature recognition to extract battlefield elements;

[0103] 7b) Finally, using the battlefield intelligence obtained from prior reconnaissance, the extracted situational elements are classified and statistically analyzed, such as the ownership of the combat target (enemy, friendly, or ally), troop deployment and losses, battlefield environment, etc.

[0104] Step 8: Generate local battlefield situation, such as Figure 4 As shown, the specific method is as follows:

[0105] By employing deep neural networks and training them on battlefield data, the system can automatically select battlefield situation elements and generate battlefield situations by following the methods used in classification problems, thereby minimizing the introduction of human factors into the model.

[0106] Step 9: Generate the global battlefield situation based on the local battlefield situation, such as... Figure 5 As shown, the specific method is as follows:

[0107] By introducing an attention mechanism, corresponding weight values ​​are automatically assigned to different local battlefield situations to measure the contribution of local battlefield situations to the understanding of the global battlefield situation, thereby generating the global battlefield situation. The global battlefield situation generation method based on the attention mechanism is described as follows:

[0108] α=(α1,…,α N )

[0109] α=softmax{σ[W2ReLU(W1·evident)]}

[0110] α1+…+α N =1

[0111] Where α represents the weight vector obtained by the Attention module, σ represents the sigmoid function, ReLU represents the non-linear activation function, softmax represents the softmax function, W1 and W2 are the parameters of the Attention module (which can be updated iteratively), and evident represents the local battlefield situation vector. Finally, the weight vector is multiplied by the local battlefield situation vector to obtain the weighted local battlefield situation vector. The maximum value of the corresponding element in the weighted local battlefield situation vector is then taken to obtain the global battlefield situation vector.

[0112] The effectiveness of this method can be further illustrated by the following simulation experiments:

[0113] 1. Experimental conditions and methods

[0114] The hardware platform is: Intel Xeon E5-2678 v3 32GB;

[0115] The software platform is: Linux Ubuntu 18.04LTS;

[0116] Programming language: Python 3.6;

[0117] Deep learning framework: PyTorch 1.6.0.

[0118] Experimental method: This method.

[0119] 2. Simulation Content and Results

[0120] Construct a multi-source image heterogeneous database, based on Figure 3 The proposed method adaptively extracts battlefield elements based on Figure 4 The proposed method adaptively fits the battlefield situation, based on Figure 5 The proposed method generates local and global battlefield situation diagrams.

[0121] This invention is geared towards unmanned combat. It utilizes multi-source heterogeneous data acquired from various unmanned equipment and payloads at different times and locations on the battlefield. Based on a deep neural network-based situational element extraction framework, it achieves adaptive extraction of battlefield situational elements. Based on a deep learning-based situational generation method, it introduces an attention mechanism and achieves the generation of complex battlefield situations with unified spatiotemporal relationships through unified time nodes and unified spatial locations.

[0122] This invention proposes a deep learning-based battlefield situation element extraction architecture, designs a deep learning-based battlefield situation generation method, proposes an attention-based conflict situation fusion method, proposes a battlefield area division method, proposes a complete spatiotemporal relationship unification approach, and constructs a complete set of battlefield semantic situation adaptive generation methods. It then abstracts the original image information into semantic information, which can achieve exponential information compression, facilitate information transmission, and provide decision support for commanders at a high semantic level.

Claims

1. A method for rapid generation of battlefield semantic situation based on image parsing, characterized in that, Includes the following steps: Step 1: Divide the battlefield area into multiple local battlefields; Step 2: Conduct reconnaissance of each local battlefield area; Step 3: Generate a multi-source heterogeneous image database; Step 4: Unify the spatiotemporal relationships of the multi-source heterogeneous image database; Step 5: Construct a system of battlefield situational elements; Step 6: Construct a technical system for extracting battlefield situational elements; Step 7: Adaptively extract battlefield situational elements; the specific method is as follows: 7a) Based on the actual scenario, select deep learning models for target detection, scene classification, and ground feature recognition to extract battlefield elements; 7b) Using pre-reconnaissance battlefield intelligence, the extracted situational elements are classified and statistically analyzed to obtain the ownership of the combat target, troop deployment and losses, and battlefield environment; Step 8: Generate local battlefield situation; the specific method is as follows: By employing deep neural networks and training them on battlefield data, we can achieve automatic selection of battlefield situation elements on the one hand, and generate local battlefield situations by following the processing methods for classification problems on the other hand. Step 9: Generate the global battlefield situation based on the local battlefield situation; the specific method is as follows: By introducing an attention mechanism, corresponding weight values ​​are automatically assigned to different local battlefield situations to measure the contribution of local battlefield situations to the understanding of the global battlefield situation, thereby generating the global battlefield situation. The global battlefield situation generation method based on the attention mechanism is described as follows: in, This represents the weight vector obtained by the Attention module. ReLU represents the sigmoid function, ReLU represents the non-linear activation function, and softmax represents the softmax function. and These are the parameters of the Attention module, and evident represents the local battlefield situation vector; Multiply the weight vector by the local battlefield situation vector to obtain the weighted local battlefield situation vector; The global battlefield situation vector is obtained by taking the maximum value of the corresponding element of the weighted local battlefield situation vector.

2. The method for rapid generation of battlefield semantic situation based on image parsing according to claim 1, characterized in that, In step 1, use one or more of the following rules to divide the battlefield area: 1a) If there are mountains or rivers in the battlefield area, then these shall be used as the local battlefield boundary; 1b) If there are multiple administrative regions within the battlefield area, then the administrative regions shall be directly regarded as local battlefield areas; 1c) If multiple local battlefields have already formed within the battlefield area, the current local battlefield will be directly used as the result of the area division. 1d) Use geometric shapes to divide the battlefield area; 1e) Based on existing intelligence, cluster the combat targets within the battlefield area and use the clustering results as the basis for area division; 1f) Based on past battle examples, divide the local area.

3. The method for rapid generation of battlefield semantic situation based on image parsing according to claim 1, characterized in that, In step 2, the methods for conducting reconnaissance of each local battlefield area include: 2a) Using drones to conduct reconnaissance of local battlefields; 2b) Use manned aircraft to conduct reconnaissance of local battlefields; 2c) Use satellites to conduct reconnaissance of local battlefields.

4. The method for rapid generation of battlefield semantic situation based on image parsing according to claim 1, characterized in that, The multi-source heterogeneous image database mentioned in step 3 includes visible light images, infrared images, and SAR images.

5. The method for rapid generation of battlefield semantic situation based on image parsing according to claim 1, characterized in that, The specific method for step 4 is as follows: 4a) Based on past battle examples and target statistics of important buildings or areas, the local battlefield is divided into multiple key areas; 4b) Divide the time into multiple time periods by using either average time period division or hotspot time period extraction; 4c) Based on the results of local battlefield area division and time period division, construct a multi-source heterogeneous data catalog; 4d) Place images located in the same local area in the same directory; 4e) Place images from the same time period into the same directory.

6. The method for rapid generation of battlefield semantic situation based on image parsing according to claim 1, characterized in that, The battlefield situation element composition system described in step 5 includes: 5a) Force deployment and combat capability, including: the strength, order of battle, troop deployment, and troop losses of the enemy, our side, and our allies; 5b) Environmental factors, including: traffic conditions, climate and meteorology, and geographical environment, are described below: ① Transportation conditions: highways, railways, air transport, and waterways; ② Climate and meteorology: Airport weather, sea waves, sea breeze; ③ Geographical environment: topography, river system, important landmarks.

7. The method for rapid generation of battlefield semantic situation based on image parsing according to claim 1, characterized in that, The battlefield situation element extraction technology system described in step 6 includes: 6a) Intelligent target detection of images is achieved through target detection, extracting combat targets and intelligently identifying important markers to extract information about them; 6b) Achieve intelligent classification of image scenes through scene classification; 6c) Accurate image recognition is achieved through ground feature recognition, identifying the specific category of a certain area in the image.

Images