An optoelectronic target recognition method based on multi-modal knowledge graph reasoning

Abstract

The application provides an optoelectronic target identification method based on multi-modal knowledge graph reasoning. By constructing a multi-modal knowledge graph of optoelectronic targets, optoelectronic targets are connected by relationships, and the related information and the association relationship thereof are obtained. The input information is dynamically linked with the multi-modal knowledge graph and reasoning, and the optoelectronic target selects the matching entity with the maximum probability from multiple candidate entities of the linking and reasoning result, and outputs all attribute information related to the matching entity to provide support for subsequent military upper application such as combat command control, battlefield situation awareness, auxiliary decision and threat assessment.

Description

Technical Field

[0001] This invention belongs to the field of multimodal knowledge graph construction, representation learning and reasoning technology, and specifically relates to a photoelectric target recognition method based on multimodal knowledge graph reasoning. Background Technology

[0002] Optoelectronic target recognition technology plays a crucial role in military applications such as operational command and control, battlefield situational awareness, tactical intent recognition, decision support, and threat assessment in the field of optoelectronic warfare. Currently, optoelectronic target recognition technology is mainly based on methods such as multi-sensor data fusion or expert systems. For example, Wang Tiehong's "Research on Multi-Sensor Data Fusion Model of Optoelectronic Information System" presents a multi-level sensor information fusion model structure. However, fusion models based on multi-level processing (data level, feature level, decision level, etc.) result in long algorithm flows and complex designs. Expert systems, a branch of artificial intelligence, present challenges in the field of military target recognition, where recognition knowledge is diverse, time-sensitive, and geographically specific. Expert systems suffer from difficulties in knowledge acquisition during the development phase, short effective lifespan after system delivery, and inability to dynamically and autonomously iterate. However, from the perspective of data format and type, the commonly used data types for photoelectric target identification include structured, semi-structured and unstructured data such as intelligence text, photoelectric detection images, SQL relational databases, and GIS geographic data. These data have characteristics such as multiple types, large data volume, complex structure, sparse key values, and high data redundancy in actual combat or training environments. Conventional methods based on multi-sensor data fusion and inference are difficult to fully adapt to multiple data types, have complex processing methods, and lack a self-iterative update process.

[0003] With the development of artificial intelligence and the internet, knowledge graphs possess powerful semantic expression, knowledge extraction and fusion, and reasoning capabilities, providing an effective solution for the knowledge-based organization, storage, and upper-level applications of multimodal data. Multimodal knowledge graphs mainly include {E, R, A, V, T}. R ,T A}, where E is an entity, R is a relation, A is a multimodal data attribute, V is a multimodal data attribute value, and T is a multimodal data attribute value. R A multimodal relation triple describes the relationship between entities. For example, in a multimodal knowledge graph, the relationship between the Paveway series missiles and laser semi-active guided weapons belongs to T. A An attribute triple describes the inherent attributes of an entity, such as the country of origin, speed, and function of the Paveway series missiles. For example, MMKG uses a Wikipedia URI as its query string to crawl images from a search engine and treats them as an attribute of the entity.

[0004] Existing photoelectric target recognition methods mainly target the detection and recognition of target categories, but do not delve into the fine-grained entity association of targets, that is, they cannot conduct in-depth mining and connection of other attributes and additional information of photoelectric targets.

[0005] Considering the unique characteristics of the optoelectronic countermeasures field and the limitations of pre-trained language models in specific task scenarios, existing models cannot achieve satisfactory results in optoelectronic target recognition. Therefore, a new optoelectronic target recognition method based on a multimodal pre-trained language model is needed, specifically for optoelectronic countermeasures and the optoelectronic target recognition task. This method extracts joint entity representation features of optoelectronic targets under different modalities, uses efficient representations to describe optoelectronic target objects, fuses entity representations of optoelectronic targets from different modalities, and encodes the modal features of optoelectronic targets into the context representation within the model. This enables the model to support both optoelectronic target object recognition and entity query linking tasks across different modalities. Summary of the Invention

[0006] (a) Technical problems to be solved

[0007] This invention proposes a photoelectric target recognition method based on multimodal knowledge graph reasoning to address the technical problem that conventional photoelectric target recognition methods are difficult to adapt to photoelectric target data with characteristics such as multiple types, large data volume, complex structure, sparse key values, and high data repetition in actual combat or training environments, and lack effective management and utilization.

[0008] (II) Technical Solution

[0009] To address the aforementioned technical problems, this invention proposes a photoelectric target recognition method based on multimodal knowledge graph reasoning, which includes the following steps:

[0010] S1. Constructing a multimodal knowledge graph of optoelectronic targets based on the field of optoelectronic countermeasures.

[0011] According to the structure, the sources of optoelectronic target knowledge data are divided into three types: structured data, semi-structured data, and unstructured data. Data preprocessing, information extraction, and knowledge fusion are performed on different types of data. The processed entities, relationships, attributes, and empirical rules are input into the optoelectronic target multimodal knowledge graph to construct an optoelectronic target multimodal knowledge graph based on the optoelectronic countermeasures field.

[0012] S2. Image Feature Vector Extraction

[0013] The size of the photoelectric target image data obtained from the photoelectric target multimodal knowledge graph constructed in step S1 is adjusted to 224*224 pixels, and then input into the CNN baseline model for image feature extraction;

[0014] S3. Encode the text format into a numerical vector.

[0015] When the knowledge data of optoelectronic targets is in the form of text data, the text information is encoded into digital codes that can be calculated by computer language according to the dictionary;

[0016] S4. Numerical Vector Extraction

[0017] When the input of optoelectronic target knowledge data is expressed as a key-value pair ({key:value}), the value itself is directly embedded as a vector into V = {v0, v1, ..., v...}. q-1}, where V is a numerical vector, v0,…v q-1 For specific numerical values;

[0018] S5. Input the image feature vectors, text-based numerical vectors, and numerical vectors obtained in the above steps into the Transformer model independently to obtain the image modality, text modality, and numerical modality embedding representation vectors X corresponding to the photoelectric target multimodal knowledge graph constructed in step S1. M ;

[0019] S6. Independently embed the different modal representation vectors X obtained in step S5. M The decomposed vectors K-key vector and V-value vector are used for multimodal information exchange;

[0020] S7. Input decoders of different modalities to calculate probability prediction distribution results.

[0021] Let the feature vectors of the image modality, text modality, and numerical modality outputs after multimodal interaction be represented as follows:

[0022] The following formulas are used to obtain the entity probability prediction distribution results for different modalities and the final probability prediction distribution result:

[0023]

[0024]

[0025] PRO=(1-α-β)PRO T +αPRO P +βPRO V

[0026] ReLU(x) = max(x,0)

[0027] Among them, PRO T PRO P PRO VPRO represents the probability prediction distributions for the text modality, image modality, numerical modality, and the final entity probability prediction, respectively; α and β are hyperparameters that adjust the proportions of the three modalities, obtained during model training; MLP is the multilayer perceptron computation.

[0028] S8. Photoelectric target recognition based on a trained multimodal knowledge graph reasoning model.

[0029] The photoelectric target multimodal knowledge graph dataset constructed in step S1 is input into steps S2-S7 for Transformer model training to obtain the specific values ​​of the hidden layer parameters of the model. The values ​​of hyperparameters α and β in step S7 are set according to the distribution size of the actual dataset. At this point, the inference model based on the multimodal knowledge graph is completed.

[0030] The photoelectric target data that needs to be reasoned is calculated according to steps S2-S7 to obtain the probability distribution PRO on the final output candidate entities. The candidate entities are arranged in descending order according to the probability, and the top K entities with the highest probability are output as the entity results of photoelectric target recognition. The attribute information of the K entities is obtained by querying the multimodal knowledge graph, and the photoelectric target recognition work is completed.

[0031] Furthermore, in step S1, structured data includes SQL data and XML data, semi-structured data includes Excel spreadsheets with custom data types, and unstructured data includes photoelectric target image data acquired by photoelectric sensors and text-based numerical data.

[0032] Furthermore, in step S1, data preprocessing includes word segmentation, named entity recognition, part-of-speech tagging, removal of duplicate information, and correction of invalid and missing values ​​for different types of data.

[0033] Furthermore, in step S1, information extraction includes entity extraction, relation extraction, and attribute extraction.

[0034] Furthermore, in step S1, knowledge fusion includes deambiguation and entity alignment of the extracted entities, relationships, attributes, and empirical rules.

[0035] Furthermore, in step S2, the CNN baseline model includes VGG, ResNet, or DenseNet models.

[0036] Furthermore, in step S4, the photoelectric target knowledge data includes the photoelectric target's azimuth angle, pitch angle, and azimuth velocity data.

[0037] Further, in step S5, the embedding representation vectors X of the image modal encoder, text modal encoder, and numerical modal encoder are calculated respectively according to the following formula. M :

[0038] X = X0 + X position

[0039] X M =MHA(LN(X))+X

[0040] Where X0 is an image feature vector, text encoding vector, or numerical vector, X position The parameters are the position embedding parameters inherent in the Transformer model, MHA is the multi-head standard calculation formula for the Transformer model, and LN is the normalization calculation of the network layer parameters in the Transformer model using the standard normal distribution function.

[0041] Furthermore, in step S6, when performing multimodal information interaction, the K-key vector and V-key vector of the mode with fewer elements among the image mode, text mode, and numerical mode are input into the K-key vector and V-key vector of the other two modes.

[0042] Further, in step S6, the embedding representation vector X of the text modality is calculated using the following formula. M K-bond vector and V value vector The embedding representation vectors X are input to the image mode and the numerical mode, respectively. M In the K-bond vector and V-value vector:

[0043]

[0044] Where [] represents the splicing operation, Attention is the calculation formula of the Transformer's self-attention mechanism, and head is the attention calculation after modal information interaction; Text modality embedding representation vector X M The corresponding decomposition vector; Image modality embedding representation vector X M The corresponding decomposition vector; For numerical mode embedding representation vector X M The corresponding decomposition vector; the subscript parameter i is the value of a specific element in the decomposition vector.

[0045] (III) Beneficial Effects

[0046] This invention proposes a method for optoelectronic target recognition based on multimodal knowledge graph reasoning. By constructing a multimodal knowledge graph of optoelectronic targets, relationships are connected between them, facilitating the acquisition of relevant information and connections. Input information is dynamically linked and reasoned with the multimodal knowledge graph. The optoelectronic target then selects the matching entity with the highest probability from multiple candidate entities in the linking and reasoning results, outputting all attribute information related to the matching entity to support subsequent high-level military applications such as combat command and control, battlefield situational awareness, decision support, and threat assessment. In this invention, the multimodal knowledge graph integrates different types of data, using joint representation learning across multiple modalities such as images, text, and numerical values ​​to provide more comprehensive and multidimensional information, thus improving the accuracy and richness of optoelectronic target recognition tasks. Constructing a multimodal knowledge graph based on optoelectronic targets allows for structured processing of equipment knowledge data, improving knowledge retrieval efficiency, establishing connections between information, and providing intelligence analysis and information services to support high-level military applications such as threat assessment and decision support. The reasoning model of the multimodal knowledge graph is based on neural networks, and its intelligence allows for continuous expansion and updating of model parameters to adapt to new data and new requirements. Attached Figure Description

[0047] Figure 1 The process of constructing a multimodal knowledge graph for optoelectronic targets;

[0048] Figure 2 A rendering showing the effect of displaying part of the multimodal knowledge graph for optoelectronic targets;

[0049] Figure 3 Schematic diagram of obtaining feature vectors through feature encoding;

[0050] Figure 4 Schematic diagram for multimodal interaction implementation;

[0051] Figure 5 This is a graph of a two-layer multilayer perceptron (MLP) neural network with one hidden layer. Detailed Implementation

[0052] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to the accompanying drawings and examples.

[0053] This embodiment proposes a photoelectric target recognition method based on multimodal knowledge graph reasoning. The photoelectric target recognition method specifically includes the following steps:

[0054] S1. Constructing a multimodal knowledge graph of optoelectronic targets based on the field of optoelectronic countermeasures.

[0055] Optoelectronic target knowledge data sources are categorized by structure into structured data (such as SQL data, XML data, etc.), semi-structured data (such as custom data type Excel spreadsheets, etc.), and unstructured data (such as optoelectronic target image data acquired from various optoelectronic sensors, text-based numerical data, etc.). Figure 1 The illustrated process for constructing a multimodal knowledge graph of optoelectronic targets involves preprocessing different types of data, including word segmentation, named entity recognition, part-of-speech tagging, removal of duplicate information, and correction of invalid and missing values. This is followed by information extraction processes such as entity extraction, relation extraction, attribute extraction, and empirical rule extraction. The extracted entities, relations, attributes, and empirical rules are then subjected to knowledge fusion processes such as deambiguation and entity alignment. Finally, the processed entities, relations, attributes, and empirical rules are input into the multimodal knowledge graph of optoelectronic targets, thus constructing a multimodal knowledge graph of optoelectronic targets based on the field of optoelectronic countermeasures.

[0056] like Figure 2 As shown, the multimodal knowledge graph of photoelectric targets {E,R,A,V,T} is specifically displayed. R ,T A} Detailed content. The entity E represents content such as: Paveway series, Pyros small tactical munitions, Aasmhammer modular guided bombs, US AGM-114 Hellfire missiles, US AN / AAQ-13 Lantiln, US AN / AAQ-14 Lantiln, French Rubis, US Pathfinde, US AN / AAQ-38NITE HAWK, Israeli Litening, and other electro-optical targets. The relationship R represents content such as: Paveway series, Pyros small tactical munitions, Aasmhammer modular guided bombs, and other electro-optical targets belong to laser semi-active guided weapons, and the relationship is "belongs to". A represents multimodal data attributes, such as: the country of manufacture of the Paveway series missile entity itself, its accuracy, functions, and a series of multimodal data attributes. V represents multimodal data attribute values, such as: the country of manufacture of the Paveway series missile entity itself is the United States, its accuracy is 1-3 meters, and other specific data attribute values. T R It is a multimodal relation triple, such as {Paveway series missiles, which belong to laser semi-active guided weapons}. A It is an attribute triple, such as {Paveway series missiles, country of origin, United States}.

[0057] S2. Image Feature Vector Extraction

[0058] The size of the photoelectric target image data obtained from the photoelectric target multimodal knowledge graph constructed in step S1 is adjusted to 224*224 pixels, and then input into a CNN baseline model such as VGG, ResNet or DenseNet for image feature extraction.

[0059] The image modal data in the photoelectric target multimodal knowledge graph data constructed in step S1 is input into the VGG, ResNet or DenseNet model to evaluate the photoelectric target classification task. The model with the best comprehensive evaluation index of accuracy, precision and recall on the photoelectric target classification task is used as the CNN baseline model for image feature vector extraction.

[0060] S3. Encode the text format into a numerical vector.

[0061] When the knowledge data of optoelectronic targets is in the form of text data, the text information is encoded into digital codes that can be calculated by computer language according to the dictionary.

[0062] S4. Numerical Vector Extraction

[0063] When the input of optoelectronic target knowledge data is expressed as key-value pairs {key:value}, since the numerical value itself is an attribute of the entity feature, the value itself is directly embedded as a vector into V = {v0, v1, ..., v...}. q-1}, where V is a numerical vector, v0,…v q-1 These are specific numerical values, such as the azimuth angle, pitch angle, and azimuth velocity of an electro-optical target, among other specific data.

[0064] S5. Input the image feature vectors, text-based numerical vectors, and numerical vectors obtained in the above steps into the Transformer model independently to obtain the image modality, text modality, and numerical modality embedding representation vectors X corresponding to the photoelectric target multimodal knowledge graph constructed in step S1. M .

[0065] Calculate the embedding representation vector X for the image modal encoder (P-EnCoder), text modal encoder (T-EnCoder), and numerical modal encoder (V-EnCoder) respectively according to the following formulas. M :

[0066] X = X0 + X position

[0067] X M =MHA(LN(X))+X

[0068] Where X0 is an image feature vector, text encoding vector, or numerical vector, X positionThe parameters are the position embedding parameters inherent in the Transformer model, MHA is the multi-head standard calculation formula for the Transformer model, and LN is the normalization calculation of the network layer parameters in the Transformer model using the standard normal distribution function.

[0069] S6. Independently embed the different modal representation vectors X obtained in step S5. M The decomposed vectors K-key vector and V-value vector are used for multimodal information exchange.

[0070] Figure 3 As shown in the specific technical schematic diagram, the photoelectric target data is divided into image modality, text modality, and numerical modality according to modality. Image modality data is extracted using the baseline model of a convolutional neural network (CNN), and then the features are encoded by the P-Encoder to obtain feature vectors. Text modality data is encoded into numerical form using text encoding, and then the features are encoded by the T-Encoder to obtain feature vectors. Numerical modality data is directly encoded by the V-Encoder to obtain feature vectors. Then, the feature vectors of the three independent modalities are fused and interacted between different modalities, so that each modality contains vector information from other modalities. Finally, the probability prediction values ​​of the three feature vectors after multimodal interaction are decoded by the P-Dncoder, T-Dncoder, and V-Dncoder respectively, and the sum of them is obtained to obtain the final inference probability value P.

[0071] The specific technical principles of the multimodal interaction module in the implementation of this method are as follows: Figure 4 As shown, the image modality, text modality, and numerical modality are composed of three separate transformer structures. When the number of text modality samples is small, the multimodal information interaction process involves inputting the K-key and V-key vectors of the text modality into the K-key and V-key vectors of the image modality and the numerical modality, respectively. If the number of samples in other modalities is small, the K-key and V-key vectors of the less numerous modalities are similarly input into the K-key and V-key vectors of the other two modalities.

[0072] The embedding representation vector X of the text modality is obtained through the following formula. M K-bond vector and V value vector The embedding representation vectors X are input to the image mode and the numerical mode, respectively. M In the K-bond vector and V-value vector.

[0073]

[0074] Where [] represents the splicing operation, Attention is the calculation formula of the Transformer's self-attention mechanism, and head is the attention calculation after modal information interaction; Text modality embedding representation vector X M The corresponding decomposition vector; Image modality embedding representation vector X M The corresponding decomposition vector; For numerical mode embedding representation vector X M The corresponding decomposition vector; the subscript parameter i is the value of a specific element in the decomposition vector.

[0075] S7. Input decoders of different modalities to calculate probability prediction distribution results.

[0076] Let the feature vectors of the image modality, text modality, and numerical modality outputs after multimodal interaction be represented as follows:

[0077] The following formulas are used to obtain the entity probability prediction distribution results for different modalities and the final probability prediction distribution result:

[0078]

[0079] PRO=(1-α-β)PRO T +αPRO P +βPRO V

[0080] ReLU(x) = max(x,0)

[0081] Among them, PRO T PRO P PRO V PRO represents the text modality, image modality, numerical modality, and the final entity probability prediction distribution, respectively; α and β are hyperparameters that adjust the proportions of the three modalities, obtained during model training; MLP is a multilayer perceptron computation, and the specific technical principles of the DeCoder decoding process in this method are as follows: Figure 5 As shown in the figure, a two-layer structure diagram is given. x1, x2, ... xn are the input feature vectors to be decoded, h1, h2, ... hn are the hidden layer parameters of the network structure obtained from training, and o1, o2, ... on are the final entity probability prediction distribution results for different modalities.

[0082] S8. Photoelectric target recognition based on a trained multimodal knowledge graph reasoning model.

[0083] The photoelectric target multimodal knowledge graph dataset constructed in step S1 is input into steps S2-S7 for Transformer model training to obtain the specific values ​​of the hidden layer parameters. Furthermore, the values ​​of hyperparameters α and β in step S7 are set according to the actual dataset distribution. At this point, the inference model based on the multimodal knowledge graph is complete.

[0084] The photoelectric target data requiring inference is calculated according to steps S2-S7 to obtain the probability distribution PRO of the final output candidate entities. The candidate entities are arranged in descending order of probability, and the top K entities with the highest probabilities are output as the entity results of photoelectric target recognition. Then, the K entities are queried from the multimodal knowledge graph to obtain all relevant attribute information and sent to the relevant upper-layer applications. At this point, the photoelectric target recognition work is completed.

[0085] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A photoelectric target recognition method based on multimodal knowledge graph reasoning, characterized in that, The photoelectric target recognition method includes the following steps: S1. Constructing a multimodal knowledge graph of optoelectronic targets based on the field of optoelectronic countermeasures. According to the structure, the sources of optoelectronic target knowledge data are divided into three types: structured data, semi-structured data, and unstructured data. Data preprocessing, information extraction, and knowledge fusion are performed on different types of data. The processed entities, relationships, attributes, and empirical rules are input into the optoelectronic target multimodal knowledge graph to construct an optoelectronic target multimodal knowledge graph based on the optoelectronic countermeasures field. S2. Image Feature Vector Extraction The size of the photoelectric target image data obtained from the photoelectric target multimodal knowledge graph constructed in step S1 is adjusted to 224*224 pixels, and then input into the CNN baseline model for image feature extraction; S3. Encode the text format into a numerical vector. When the knowledge data of optoelectronic targets is in the form of text data, the text information is encoded into digital codes that can be calculated by computer language according to the dictionary; S4. Numerical Vector Extraction When the input of optoelectronic target knowledge data is expressed as a key-value pair ({key:value}), the value itself is directly embedded as a vector into V = {v0, v1, ..., v...}. q-1 }, where V is a numerical vector, v0,…v q-1 For specific numerical values; S5. Input the image feature vectors, text-based numerical vectors, and numerical vectors obtained in the above steps into the Transformer model independently to obtain the image modality, text modality, and numerical modality embedding representation vectors X corresponding to the photoelectric target multimodal knowledge graph constructed in step S1. M ; S6. Independently embed the different modal representation vectors X obtained in step S5. M The decomposed vectors K-key vector and V-value vector are used for multimodal information exchange; S7. Input decoders of different modalities to calculate probability prediction distribution results. Let the feature vectors of the image modality, text modality, and numerical modality outputs after multimodal interaction be represented as follows: The following formulas are used to obtain the entity probability prediction distribution results for different modalities and the final probability prediction distribution result: PRO=(1-α-β)RPO T +aPRO P +βPRO V ReLU(x) = max(x,0) Among them, PRO T PRO P PRO V PRO represents the text modality, image modality, numerical modality, and the final entity probability prediction distribution, respectively; α and β are hyperparameters that adjust the proportions of the three modalities, obtained during model training; MLP is a multilayer perceptron computation. S8. Photoelectric target recognition based on the trained multimodal knowledge graph inference model: Input the photoelectric target multimodal knowledge graph dataset constructed in step S1 into steps S2-S7 to train the Transformer model, obtain the specific values ​​of the hidden layer parameters of the model, and set the values ​​of hyperparameters α and β in step S7 according to the distribution size of the actual dataset; at this point, the inference model based on the multimodal knowledge graph is completed. The photoelectric target data that needs to be reasoned is calculated according to steps S2-S7 to obtain the probability distribution PRO on the final output candidate entities. The candidate entities are arranged in descending order according to the probability, and the top K entities with the highest probability are output as the entity results of photoelectric target recognition. The attribute information of the K entities is obtained by querying the multimodal knowledge graph, and the photoelectric target recognition work is completed.

2. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 1, characterized in that, In step S1, structured data includes SQL data and XML data, semi-structured data includes Excel spreadsheets with custom data types, and unstructured data includes photoelectric target image data acquired by photoelectric sensors and text-based numerical data.

3. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 1, characterized in that, In step S1, data preprocessing includes word segmentation, named entity recognition, part-of-speech tagging, removal of duplicate information, and correction of invalid and missing values ​​for different types of data.

4. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 1, characterized in that, In step S1, information extraction includes entity extraction, relation extraction, and attribute extraction.

5. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 1, characterized in that, In step S1, knowledge fusion includes deambiguation and entity alignment of the extracted entities, relationships, attributes, and empirical rules.

6. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 1, characterized in that, In step B2, the CNN baseline model includes VGG, ResNet, or DenseNet models.

7. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 1, characterized in that, In step S4, the photoelectric target knowledge data includes the photoelectric target's azimuth angle, pitch angle, and azimuth velocity data.

8. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 1, characterized in that, In step S5, the embedding representation vectors X of the image modal encoder, text modal encoder, and numerical modal encoder are calculated according to the following formulas. M : X=X0+X position X M =MHA(LN(X))+X Where X0 is an image feature vector, text encoding vector, or numerical vector, X position The parameters are the position embedding parameters inherent in the Transformer model, MHA is the multi-head standard calculation formula for the Transformer model, and LN is the normalization calculation of the network layer parameters in the Transformer model using the standard normal distribution function.

9. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 1, characterized in that, In step S6, when performing multimodal information interaction, the K-key vector and V-key vector of the mode with fewer elements among the image mode, text mode, and numerical mode are input into the K-key vector and V-key vector of the other two modes.

10. The photoelectric target recognition method based on multimodal knowledge graph reasoning as described in claim 9, characterized in that, In step S6, the embedding representation vector X of the text modality is calculated using the following formula. M K-bond vector and V value vector The embedding representation vectors X are input to the image mode and the numerical mode, respectively. M In the K-bond vector and V-value vector: Where [] represents the splicing operation, Attention is the calculation formula of the Transformer's self-attention mechanism, and head is the attention calculation after modal information interaction; Text modality embedding representation vector X M The corresponding decomposition vector; Image modality embedding representation vector X M The corresponding decomposition vector; For numerical mode embedding representation vector X M The corresponding decomposition vector; the subscript parameter i is the value of a specific element in the decomposition vector.

Images