A Method for Extracting Semantic Battlefield Situation

Abstract

The present invention discloses a method for extracting battlefield semantic situation, belonging to the technical field of battlefield situation construction. This method first constructs a multi-source image database; then unifies the spatio-temporal relationship of the multi-source image database; then conducts semantic description on the multi-source image data; and finally extracts multi-level summaries from the results of the semantic description to generate a battlefield semantic situation. The present invention constructs a full-process system for image semantic description and multi-level summary extraction based on artificial intelligence means, can obtain the logical relationship between "target and target" and "target and environment" on the battlefield, realizes the layer-by-layer extraction of semantic situation information from local to global, is closer to the human language system, the situation generation is simple and efficient, and compresses battlefield information exponentially, and can quickly provide decision-making support for commanders.

Description

Technical Field

[0001] The present invention belongs to the technical field of battlefield situation construction, and particularly relates to a method for extracting semantic battlefield situations, which can be used for semantic extraction of battlefield situations to provide support for combat command. Background Art

[0002] The intelligent operation is an important development trend of future wars. The correct understanding of battlefield situations is a prerequisite for winning. Under the condition of network informatization, the current battlefield environment has gradually evolved from the previous mechanized operation mode into a high-tech informatized war integrating sea, land and air. The whole war presents characteristics such as diversified operation modes, diversified operation objects and complex and changeable operation environments. In modern informatized wars, situation assessment is in the core position of the battlefield command and decision-making stage. In the face of a large number of highly uncertain information and intelligence, how to extract effective situation information and conduct rapid and accurate battlefield situation assessment is a challenging topic. On the one hand, under complex battlefield conditions, intelligent operation platforms face a harsh electromagnetic operation environment, and the network communication between operation platforms shows restricted non-real-time and narrow bandwidth, which cannot support the real-time exchange of large-capacity information such as images and videos. At this time, the operation platforms need to have a certain autonomous processing ability; on the other hand, battlefield data has changed from information scarcity in the past to information redundancy at the present stage. The results provided by existing methods are still not intuitive enough (usually some features and indicators), and usually still require secondary processing by human commanders, with low efficiency, which will inevitably affect the decision-makers' real-time grasp of the current battlefield situation, making corresponding decisions and formulating reasonable plans.

[0003] Facing the actual needs and existing problems in the battlefield, researching the technology of automatic situation generation and construction that can utilize multi-source heterogeneous data obtained in the battlefield under complex battlefield conditions, which can accurately reflect the battlefield situation within a certain time and space range, and using the obtained multi-source heterogeneous data to realize the semantic unified representation of battlefield situations and generate semantic descriptions of battlefield situations will undoubtedly improve the situation awareness ability, information processing ability and battlefield adaptability of operation platforms, and at the same time can also improve the decision-making efficiency of human commanders and gain the initiative in the battlefield. Summary of the Invention

[0004] In view of this, the present invention proposes a method for extracting semantic battlefield situations. This method obtains the semantic situation of the local battlefield area based on the image semantic description method and obtains the global semantic situation of the battlefield based on the abstract extraction method.

[0005] In order to achieve the above object, the technical solution adopted by the present invention is as follows:

[0006] A method for extracting semantic battlefield situations includes the following steps:

[0007] Step 1, construct a multi-source image database;

[0008] Step 2, unify the spatio-temporal relationships of the multi-source image database;

[0009] Step 3, perform semantic description on the multi-source image data;

[0010] Step 4, perform multi-level summary extraction on the results of semantic description to generate a battlefield semantic situation.

[0011] Furthermore, the images in the multi-source image database described in Step 1 include: visible light images, infrared images, SAR images, and optical remote sensing images.

[0012] Furthermore, the specific method for unifying the spatio-temporal relationships of the multi-source image database described in Step 2 is as follows:

[0013] 2a) Rename the images, adding time information and positioning information;

[0014] 2b) Place the images located in the same local spatial area in the same directory;

[0015] 2c) Place the images located within the same time period in the same directory.

[0016] Furthermore, the semantic description method in Step 3 is as follows:

[0017] 3a) The semantic description needs to include 5 types of attributes, including: time, space, friend-or-foe attribute, threat range, and combat intention;

[0018] 3b) Implement image semantic description based on an encoder-decoder structure and an attention mechanism, where the attention mechanism contains 3 layers of attention structures: attention to different regions of the image, attention to the generated words, and attention to the information between the image and language; in the encoder-decoder structure, different depths of ResNet are used as the encoder for different image data, and the decoder uses a long short-term memory network and a model containing 3 layers of attention structures.

[0019] Furthermore, the multi-level summary extraction and semantic generation method described in Step 4 is as follows:

[0020] 4a) Determine the number of levels of multi-level summary extraction. If the scale of the multi-source image database is small, then the battlefield division relationship is "local - single point", and the number of levels is 1; if the scale of the multi-source image database is medium, then the battlefield division relationship is "global - local - single point", and the number of levels is 2; if the scale of the multi-source database is large, then the battlefield division relationship is "global - large local - small local - single point", and the number of levels is 3; and so on.

[0021] 4b) If the level of multi-level abstract extraction is 1, the method of multi-level abstract extraction is "image semantic description result - 1st level abstract"; if the level of multi-level abstract extraction is 2, the method of multi-level abstract extraction is "image semantic description result - 1st level abstract - 2nd level abstract"; if the level of multi-level abstract extraction is 3, the method of multi-level abstract extraction is "image semantic description result - 1st level abstract - 2nd level abstract - 3rd level abstract"; and so on.

[0022] 4c) The methods of "image semantic description result - 1st level abstract" and "from (n - 1)th level abstract to nth level abstract" are both abstract extractions. The input of the abstract extraction is "multiple sentences", that is, the semantic description result of each image or the abstract extraction result of each local part at the previous level, and the output is one sentence or one paragraph, that is, the abstract extraction result at this level. The specific method of the abstract extraction is as follows:

[0023] ① For information attributes such as time, space, friend-or-foe attribute, threat range, and operation intention that need to be generated, it is necessary to set slots for the information to be filled, and generate corresponding abstract attribute values through setting rules; for the text description statements of the abstract, it is necessary to pre-set templates containing multiple slots to be filled with object relationships and attribute tags. These object relationships and attribute tags form empty slots, and the empty slots are filled to form the text description sentences of the image.

[0024] ② For the space attribute, select the image data or abstract data within the same local space area.

[0025] ③ For the time attribute, select the image data or abstract data within the same time period.

[0026] ④ For the friend-or-foe attribute, consider the enemy and friendly combat units separately, that is, the enemy units and the friendly units are divided into groups respectively, and the abstract extractions of the other 4 types of attributes are carried out separately.

[0027] ⑤ For the threat range, if a region contains combat units (assumed to be chariots and airplanes) with threat ranges of 40 km and 80 km, then with the cooperation of the airplanes (such as transport airplanes), the combat radius of the chariots can be extended to the combat radius of the airplanes, that is, the threat range of the chariots can be extended to the threat range of the airplanes. Take the maximum value of the threat range as the abstract value of the threat range attribute.

[0028] ⑥ For the operation intention, for the combat units within a certain range, use the relative maximum voting method for the operation intentions of all combat units. The operation intention with the most relative votes is used as the abstract value of the operation intention, that is, consider the "main intention" within a certain range.

[0029] 4d) If the number of levels for multi-level abstract extraction is 1, the 1st-level abstract is the result of semantic situation extraction; if the number of levels is 2, the 2nd-level abstract is the result of semantic situation extraction; if the number of levels is 3, the 3rd-level abstract is the result of semantic situation extraction; and so on.

[0030] The present invention has the following advantages compared with the prior art:

[0031] 1. In the previous situation system, we only knew "what targets are in a certain area and what the environment is", but we couldn't know "the logical relationships between targets and between targets and the environment". However, through the semantic description system of this patent, we can know the "logical relationships between targets and between targets and the environment" on the battlefield, which is closer to the human language system and convenient for human commanders to take the next step.

[0032] 2. The previous situation construction methods were not end-to-end but divided into multiple steps, some of which were based on expert knowledge bases or required direct participation of experts, resulting in problems such as low situation construction efficiency and poor robustness. While this patent constructs a full-process system for image semantic description and abstract extraction based on artificial intelligence means, and the situation generation is simple and efficient, which can quickly provide decision support for commanders.

[0033] 3. This patent proposes a multi-level abstract extraction method, which can construct a multi-level semantic description system to achieve layer-by-layer extraction of semantic situation information from local to global.

[0034] 4. Through the method of this patent, the original image information can be abstracted into text information, realizing exponential abstract compression of battlefield information and reducing the communication bandwidth requirements. BRIEF DESCRIPTION OF THE DRAWINGS

[0035] Figure 1 is the overall flowchart of the method of the present invention.

[0036] Figure 2 is the architecture diagram of image semantic description.

[0037] Figure 3 is the schematic diagram of template-based abstract extraction.

[0038] Figure 4 is the result diagram of semantic result extraction. EMBODIMENTS

[0039] The technical solutions and effects of the present invention will be further described in detail below with reference to the accompanying drawings.

[0040] Referring to Figure 1 , a battlefield semantic situation extraction method includes the following steps:

[0041] Step 1, construct a multi-source image database, where the multi-source images include: visible light, infrared, SAR, optical remote sensing, etc.

[0042] Step 2, unify the spatio-temporal relationship of the multi-source image database. The specific method is as follows:

[0043] 2a) Rename the images and add time information and positioning information;

[0044] 2b) Place the images located in the same local spatial area in the same directory;

[0045] 2c) Place the images located within the same time period in the same directory.

[0046] Step 3, perform semantic description on the multi-source image data. The specific method is as follows:

[0047] 3a) The semantic description needs to include 5 types of attributes, including: time, space, friend-or-foe attribute, threat range, and combat intention;

[0048] 3b) Implement image semantic description based on an encoder-decoder structure and an attention mechanism. The attention mechanism includes 3 layers of attention structures: attention to different regions of the image, attention to the generated words, and attention to the information between the image and language; in the encoder-decoder structure, different depths of ResNet are used as encoders for different image data, and the decoder uses a long short-term memory network and a model containing 3 layers of attention structures.

[0049] Step 4, perform multi-level summary extraction on the semantic description results to generate a battlefield semantic situation. The specific method is as follows: 4a) Determine the number of levels of multi-level summary extraction. If the scale of the multi-source image database is small, then the battlefield division relationship is "local - single point", and the number of levels is 1; if the scale of the multi-source image database is medium, then the battlefield division relationship is "global - local - single point", and the number of levels is 2; if the scale of the multi-source database is large, then the battlefield division relationship is "global - large local - small local - single point", and the number of levels is 3; and so on.

[0050] 4b) If the number of levels of multi-level summary extraction is 1, the multi-level summary extraction method is "image semantic description result - 1st level summary"; if the number of levels of multi-level summary extraction is 2, the multi-level summary extraction method is "image semantic description result - 1st level summary - 2nd level summary"; if the number of levels of multi-level summary extraction is 3, the multi-level summary extraction method is "image semantic description result - 1st level summary - 2nd level summary - 3rd level summary"; and so on.

[0051] 4c) The methods of "image semantic description result - level 1 summary" and "summary from level n - 1 to level n" are both summary extraction. The input of summary extraction is "multiple sentences", that is, the semantic description result of each image or the summary extraction result of each local part at the previous level, and the output is one sentence or one paragraph, that is, the summary extraction result at this level. The specific way of summary extraction is as follows:

[0052] ① For information attributes such as time, space, friend - enemy attribute, threat range, and combat intention that need to be generated, slots for the information to be filled need to be set, and corresponding summary attribute values are generated through setting rules; for the text description statements of the summary, templates containing multiple empty slots to be filled with object relationships and attribute labels need to be set in advance. These object relationships and attribute labels form empty slots, and the empty slots are filled to form the text description sentences of the image.

[0053] ② For the space attribute, select the image data or summary data within the same local space area.

[0054] ③ For the time attribute, select the image data or summary data within the same time period.

[0055] ④ For the friend - enemy attribute, consider the enemy and friendly combat units separately, that is, the enemy units and friendly units are divided into groups respectively, and the summary extraction of the other 4 types of attributes is carried out separately.

[0056] ⑤ For the threat range, if a region contains combat units (assumed to be a chariot and an aircraft) with threat ranges of 40 km and 80 km, then with the cooperation of the aircraft (such as a transport aircraft), the combat radius of the chariot can be extended to the combat radius of the aircraft, that is, the threat range of the chariot can be extended to the threat range of the aircraft. Take the maximum value of the threat range as the summary value of the threat range attribute.

[0057] ⑥ For the combat intention, for combat units within a certain range, use the relative maximum voting method for the combat intentions of all combat units. The combat intention with the most relative votes is used as the summary value of the combat intention, that is, consider the "main intention" within a certain range.

[0058] 4d) If the number of levels of multi - level summary extraction is 1, the level 1 summary is the semanticized situation extraction result; if the number of levels of multi - level summary extraction is 2, the level 2 summary is the semanticized situation extraction result; if the number of levels of multi - level summary extraction is 3, the level 3 summary is the semanticized situation extraction result; and so on.

[0059] The effect of this method can be further illustrated by the following simulation experiments:

[0060] 1. Experimental Conditions and Methods

[0061] The hardware platform is: Intel Xeon E5-2678 v3 32GB;

[0062] The software platform is: Linux Ubuntu 18.04 LTS;

[0063] Programming language: Python 3.6;

[0064] Deep learning framework: Pytorch 1.6.0.

[0065] Experimental method: This method.

[0066] 2. Simulation Content and Results

[0067] Construct multi-source image data. Based on Figure 2 the given architecture diagram of image semantic description, obtain the image and semantic description results; based on Figure 3 the given schematic diagram of template-based summary extraction, obtain the global semantic description results of the battlefield. The final semantic description results are as Figure 4 shown.

[0068] For the template-based summary extraction method, first, it extracts the keywords of the semantic descriptions of several images (such as plane, airport, parked, etc.), and extracts the relationships between objects based on visual dependency representation, using a four-tuple summary description composed of nouns, verbs, scenes, and prepositions. The advantage of the template-based method is that the resulting language description is more likely to be grammatically correct.

[0069] In summary, the present invention constructs a full-process system for image semantic description and multi-level summary extraction based on artificial intelligence means, can obtain the logical relationships between "object and object" and "object and environment" on the battlefield, realizes the layer-by-layer extraction of semantic situation information from local to global, is closer to the human language system, generates the situation simply and efficiently, and compresses battlefield information exponentially, and can quickly provide decision support for commanders.

Claims

1. A method for extracting battlefield semantic situation, characterized in that, It includes the following steps: Step 1, construct a multi-source image database; Step 2, unify the spatio-temporal relationships of the multi-source image database; Step 3, semantically describe the multi-source image data; Step 4, extract multi-level summaries from the results of semantic description to generate a battlefield semantic situation; the specific method is as follows: 4a) Determine the number of levels of multi-level summary extraction. If the scale of the multi-source image database is small, the battlefield division relationship is "local - single point", and the number of levels is 1; if the scale of the multi-source image database is medium, the battlefield division relationship is "global - local - single point", and the number of levels is 2; if the scale of the multi-source database is large, the battlefield division relationship is "global - large local - small local - single point", and the number of levels is 3; 4b) If the number of levels of multi-level summary extraction is 1, the multi-level summary extraction method is "image semantic description result - 1st level summary"; if the number of levels of multi-level summary extraction is 2, the multi-level summary extraction method is "image semantic description result - 1st level summary - 2nd level summary"; if the number of levels of multi-level summary extraction is 3, the multi-level summary extraction method is "image semantic description result - 1st level summary - 2nd level summary - 3rd level summary"; 4c) The methods of "image semantic description result - 1st level summary" and "summary from (n - 1)th level to nth level" are both summary extraction. The input of summary extraction is "multiple sentences", that is, the semantic description result of each image or the summary extraction result of each local area at the previous level, and the output is one sentence or one paragraph, that is, the summary extraction result at this level. The specific method of summary extraction is as follows: ① For the information attributes of time, space, friend-or-foe attribute, threat range, and combat intention that need to be generated, slots for the information to be filled need to be set, and corresponding summary attribute values are generated through setting rules; for the text description statements of the summary, templates containing multiple empty slots to be filled with object relationships and attribute labels need to be set in advance. These object relationships and attribute labels form empty slots, and the empty slots are filled to form the text description sentences of the image; ② For the space attribute, select the image data or summary data within the same local space area; ③ For the time attribute, select the image data or summary data within the same time period; ④ For the friend-or-foe attribute, consider the enemy and friendly combat units separately, that is, the enemy units and friendly units are respectively divided into a group, and the summary extraction of the other 4 types of attributes is carried out separately; ⑤ For the threat range, take the maximum value of the threat range as the summary value of the threat range attribute; ⑥ For the combat intention, for the combat units within a certain range, use the relative maximum voting method for the combat intentions of all combat units, and the combat intention with the most relative votes is used as the summary value of the combat intention.

2. The method for extracting battlefield semantic situation according to claim 1, characterized in that The images in the multi-source image database described in Step 1 include: visible light images, infrared images, SAR images, and optical remote sensing images.

3. A method for extracting battlefield semantic situation according to claim 1, characterized in that The specific method of unifying the spatio-temporal relationships of the multi-source image database described in Step 2 is as follows: 2a) Rename the images and add time information and positioning information; 2b) Place the images located in the same local space area in the same directory; 2c) Place the images located within the same time period in the same directory.

4. A method for extracting battlefield semantic situation according to claim 1, characterized in that, The semantic description method in step 3 is as follows: 3a) The semantic description includes five types of attributes: time, space, friendly / hostile attribute, threat range, and operation intention; 3b) The image semantic description is realized based on the encoding / decoding structure and the attention mechanism. Among them, the attention mechanism includes three layers of attention structures: attention to different regions of the image, attention to the generated words, and attention to the information between the image and language. In the encoding / decoding structure, ResNets with different depths are used as encoders for different image data, and the decoder uses the long short-term memory network and a model containing three layers of attention structures.

5. A method for extracting battlefield semantic situation according to claim 1, characterized in that Step 4 also includes: 4d) If the number of levels of multi-level summary extraction is 1, the 1st-level summary is the result of semantic situation extraction; if the number of levels of multi-level summary extraction is 2, the 2nd-level summary is the result of semantic situation extraction; if the number of levels of multi-level summary extraction is 3, the 3rd-level summary is the result of semantic situation extraction.

Images