Electromagnetic target recognition method based on multi-source data fusion and related equipment

Abstract

The application provides an electromagnetic target identification method based on multi-source data fusion and related equipment, which comprises the following steps: acquiring electromagnetic signal data and image data of a target electromagnetic device; encoding the electromagnetic signal data through a pre-constructed electromagnetic encoder to obtain an electromagnetic hidden layer vector; encoding the image data through a pre-constructed image encoder to obtain an image hidden layer vector; obtaining a superimposed hidden layer vector based on the electromagnetic hidden layer vector and the image hidden layer vector; inputting the superimposed hidden layer vector into a pre-constructed electromagnetic target classifier to classify the target electromagnetic device, and determining the electromagnetic signal category of the target electromagnetic device based on a pre-set category label. The application fully utilizes multi-information sources such as electromagnetism and images, reduces external influences under the condition of a single data source through data fusion, improves the adaptability of the algorithm in a complex environment, and improves the classification accuracy and the robustness of the model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of information fusion, and in particular to an electromagnetic target identification method based on multi-source data fusion and related equipment. BACKGROUND

[0002] In daily life scenarios, the main composition of the electromagnetic environment is high-density, high-intensity, multi-spectrum electromagnetic waves emitted by various electronic devices and electromagnetic waves generated by nature, etc. In civil spectrum management, civil electromagnetic devices generate a variety of full-spectrum, high-density electromagnetic radiation signals, as well as mutual influence and interference caused by the use of a large number of electronic devices, resulting in time domain mutations, spatial shifts, and crowded overlaps in the frequency domain, which seriously affect the efficiency of electromagnetic spectrum regulation. The complexity of spectrum management mainly reflects the increase in civil electromagnetic use activities and the dependence of daily life on electromagnetic spectrum. With the gradual increase of electromagnetic spectrum bands, almost covering the entire electromagnetic spectrum band, a large number of human misuse of electromagnetic spectrum resources is the biggest factor affecting spectrum monitoring and management. Generally, wireless sensing data is susceptible to electromagnetic channel environment interference when propagating in free space, and the received electromagnetic data often has a large interference background environment. Similarly, image data such as surveillance cameras are easily affected by light brightness, weather, and other factors. Therefore, electromagnetic target identification detection under single data source conditions is easily affected by various disturbances. SUMMARY

[0003] Therefore, the purpose of the present application is to provide an electromagnetic target identification method based on multi-source data fusion and related equipment.

[0004] In order to achieve the above purpose, the present application provides an electromagnetic target identification method based on multi-source data fusion, comprising:

[0005] obtaining electromagnetic signal data and image data of a target electromagnetic device;

[0006] encoding the electromagnetic signal data through a pre-constructed electromagnetic encoder to obtain an electromagnetic hidden layer vector;

[0007] encoding the image data through a pre-constructed image encoder to obtain an image hidden layer vector;

[0008] obtaining a superimposed hidden layer vector based on the electromagnetic hidden layer vector and the image hidden layer vector;

[0009] inputting the superimposed hidden layer vector into a pre-constructed electromagnetic target classifier to classify the target electromagnetic device, and determining an electromagnetic signal category of the target electromagnetic device based on a pre-set category label.

[0010] Further, the electromagnetic encoder comprises a convolutional neural network, a long short-term memory network LSTM, and a fully connected layer.

[0011] Further, the image encoder comprises a convolutional neural network and a fully connected layer.

[0012] Further, the superimposed hidden layer vector is obtained based on the electromagnetic hidden layer vector and the image hidden layer vector, comprising: adding the electromagnetic hidden layer vector and the image hidden layer vector to obtain the superimposed hidden layer vector.

[0013] Further, the electromagnetic encoder, the image encoder and the electromagnetic target classifier are pre-trained, and a pre-training process of the electromagnetic encoder, the image encoder and the electromagnetic target classifier is as follows,

[0014] A training sample set is constructed, and the training sample set comprises electromagnetic data samples and image data samples.

[0015] The electromagnetic encoder, the image encoder and the electromagnetic target classifier are iteratively trained based on the training sample set and a total loss function until the total loss function converges.

[0016] Further, the iteratively training the electromagnetic encoder, the image encoder and the electromagnetic target classifier based on the training sample set and the total loss function until the total loss function converges comprises:

[0017] The electromagnetic data sample is encoded by the electromagnetic encoder to obtain an electromagnetic sample hidden layer vector.

[0018] The image data sample is encoded by the image encoder to obtain an image sample hidden layer vector.

[0019] A superimposed sample hidden layer vector is obtained based on the electromagnetic sample hidden layer vector and the image sample hidden layer vector.

[0020] The superimposed sample hidden layer vector is decoded by the electromagnetic decoder to obtain reconstructed electromagnetic data.

[0021] The superimposed sample hidden layer vector is decoded by the image decoder to obtain reconstructed image data.

[0022] The superimposed sample hidden layer vector is input into the electromagnetic target classifier to classify the training sample in the training sample set, to obtain a pseudo label of the training sample, and a detection accuracy of the electromagnetic target classifier is determined based on a true label of the training sample and the pseudo label.

[0023] A first loss function is constructed based on the electromagnetic data sample and the reconstructed electromagnetic data.

[0024] A second loss function and a third loss function are constructed based on the image data sample and the reconstructed image data.

[0025] constructing a fourth loss function based on the true label of the training sample and the detection accuracy;

[0026] constructing the total loss function based on the first loss function, the second loss function, the third loss function and the fourth loss function, and performing minimization calculation on the total loss function to make the total loss function converge.

[0027] Further, the electromagnetic decoder includes a convolutional neural network, a long short-term memory network (LSTM) and a fully connected layer, and the image decoder includes a convolutional neural network and a fully connected layer.

[0028] Based on the same inventive concept, the present application further provides an electromagnetic target identification device based on multi-source data fusion, comprising:

[0029] An acquisition module is configured to acquire electromagnetic signal data and image data of a target electromagnetic device;

[0030] An electromagnetic encoding module is configured to encode the electromagnetic signal data through a pre-constructed electromagnetic encoder to obtain an electromagnetic hidden layer vector.

[0031] An image encoding module is configured to encode the image data through a pre-constructed image encoder to obtain an image hidden layer vector.

[0032] A superimposed encoding module is configured to obtain a superimposed hidden layer vector based on the electromagnetic hidden layer vector and the image hidden layer vector.

[0033] A classification module is configured to input the superimposed hidden layer vector into a pre-constructed electromagnetic target classifier to classify the target electromagnetic device, and determine an electromagnetic signal category of the target electromagnetic device based on a pre-set category label.

[0034] Based on the same inventive concept, the present application further provides an electronic device, comprising a memory, a processor and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any one of the above.

[0035] Based on the same inventive concept, the present application further provides a non-transitory computer readable storage medium, which stores computer instructions for causing a computer to execute the method of any one of the above.

[0036] As can be seen from the above, the electromagnetic target recognition method based on multi-source data fusion and the related device provided by the application encode wireless perception data and image data by using a self-encoder, fuse the encoded data, and finally complete the electromagnetic target detection task by a classifier. The electromagnetic target recognition method fully utilizes multi-information sources such as electromagnetism and images, reduces external influences under the condition of a single data source through data fusion, improves the adaptability of the algorithm in a complex environment, and improves the classification accuracy and the robustness of the model. BRIEF DESCRIPTION OF DRAWINGS

[0037] In order to more clearly illustrate the technical solutions in the application or the related art, the drawings needed to be used in the embodiments or the related art description will be briefly introduced. Obviously, the drawings in the following description are only embodiments of the application, and other drawings can be obtained by those skilled in the art without creative labor.

[0038] Figure 1 The flowchart of the electromagnetic target recognition method based on multi-source data fusion of the embodiments of the application;

[0039] Figure 2 The structural diagram of the electromagnetic target recognition device based on multi-source data fusion of the embodiments of the application;

[0040] Figure 3 The structural diagram of the electronic device of the embodiments of the application. DETAILED DESCRIPTION

[0041] In order to make the purpose, technical solutions and advantages of the application more clear, the application will be further described in detail below with reference to the embodiments and the drawings.

[0042] It should be noted that, unless otherwise defined, the technical terms or scientific terms used in the embodiments of the application should be understood as the usual meaning understood by those skilled in the art in the field to which the application belongs. The terms "first", "second" and the like used in the embodiments of the application do not represent any order, number or importance, but are only used to distinguish different components. The terms "include" or "contain" and the like mean that the elements or objects before the terms cover the elements or objects listed after the terms and their equivalents, without excluding other elements or objects. The terms "connect" or "connected" and the like do not mean physical or mechanical connection, but can include electrical connection, whether direct or indirect. The terms "up", "down", "left", "right" and the like are only used to represent relative positional relationship, and when the absolute position of the described object changes, the relative positional relationship may also change accordingly.

[0043] The embodiments of this application will be described in detail below with reference to the accompanying drawings.

[0044] This application provides a method for electromagnetic target identification through multi-source data fusion, referencing... Figure 1 This includes the following steps:

[0045] Step S101: Acquire electromagnetic signal data and image data of the target electromagnetic device. The electromagnetic signals of the target electromagnetic device include radar signals and communication signals. The collected electromagnetic signals of the target electromagnetic device are used as electromagnetic signal data, and the image data is the acquired image information of the target electromagnetic device.

[0046] Step S102: Encode the electromagnetic signal data using a pre-constructed electromagnetic encoder to obtain the electromagnetic hidden layer vector. Specifically, the data input to the electromagnetic encoder is an orthogonal electromagnetic data sequence.

[0047] S=[[I1,Q1],[I2,Q2],…,[I n Q n ]]

[0048] Among them, I n Q represents the data of the in-phase path. n This represents orthogonal path data, where n is the sampling length, typically 1024.

[0049] In some embodiments, the electromagnetic encoder includes: a convolutional neural network, a long short-term memory (LSTM) network, and a fully connected layer. Hidden layer vectors are extracted from the input orthogonal electromagnetic data sequence using a one-dimensional convolutional neural network and LSTM, and the vector output by the LSTM is [h1, h2, ..., h...]. n Then, the vector [h1,h2,…,h] is... n The input is processed in a fully connected layer, and the electromagnetic hidden layer vector V is output through the fully connected layer. S =Φ1(W fc1 R lstm ), where W fc1 R represents the weight of the fully connected layer in the electromagnetic encoder. lstm This represents the output of the LSTM, i.e., R. lstm =[h1,h2,…,h n ], φ1 represents the fully connected layer of the electromagnetic encoder, V s This represents the electromagnetic hidden layer vector.

[0050] Step S103: Encode the image data using a pre-constructed image encoder to obtain the image hidden layer vector. Specifically, the acquired image of the target electromagnetic device is input into the image encoder, and the image hidden layer vector is output through the processing of the image encoder.

[0051] In some embodiments, the image encoder comprises a convolutional neural network and a fully connected layer. The image data encoder is trained using the convolutional neural network and outputs an image hidden layer vector using the fully connected layer, and the dimension of the obtained image hidden layer vector is consistent with the dimension of the electromagnetic signal hidden layer vector. The image hidden layer vector is denoted as V im = Φ2(W fc2 R cnn ), where W fc2 represents the weight of the fully connected layer in the image encoder, R cnn represents the output of the convolutional neural network, and Φ2 represents the fully connected layer of the image encoder. We define the total encoder as V enc = ENC(V S , v im , θ), where ENC(*) represents the total encoder, including the electromagnetic encoder and the image encoder, and θ is the parameter of the total encoder.

[0052] Step S104, obtaining a superimposed hidden layer vector based on the electromagnetic hidden layer vector and the image hidden layer vector. The electromagnetic hidden layer vector and the image hidden layer vector obtained in steps S102 and S103 are used to calculate the superimposed hidden layer vector.

[0053] In some embodiments, the step of obtaining the superimposed hidden layer vector based on the electromagnetic hidden layer vector and the image hidden layer vector comprises adding the electromagnetic hidden layer vector and the image hidden layer vector to obtain the superimposed hidden layer vector. Because the electromagnetic hidden layer vector and the image hidden layer vector have the same dimension, the superimposed hidden layer vector can be obtained by simply adding the vectors.

[0054] Step S105, inputting the superimposed hidden layer vector into a pre-constructed electromagnetic target classifier to classify the target electromagnetic device, and determining the electromagnetic signal category of the target electromagnetic device based on a preset category label. The electromagnetic target classifier is composed of multiple fully connected layers and an activation function. In this embodiment, the activation function used is the ReLU (The Rectified Linear Unit) function. The ReLU function is linear and does not have the problem of gradient explosion, and is the most widely used activation function in deep learning. The superimposed hidden layer vector is input into the electromagnetic target classifier, and the target electromagnetic device corresponding to the superimposed hidden layer vector is classified based on the preset category label to determine the category label to which the target electromagnetic device belongs. Each category label represents an electromagnetic signal category, so the electromagnetic signal category of the target electromagnetic device can be obtained.

[0055] In some embodiments, the electromagnetic encoder, the image encoder, and the electromagnetic target classifier are pre-trained, and the pre-training process is as follows.

[0056] constructing a training sample set, the training sample set comprising electromagnetic data samples and image data samples; iteratively training the electromagnetic encoder, the image encoder and the electromagnetic target classifier based on the training sample set and a total loss function until the total loss function converges.

[0057] Specifically, the electromagnetic encoder, the image encoder and the electromagnetic target classifier are all pre-trained, and the learning ability of the electromagnetic encoder and the image encoder is trained to improve the expression ability of the electromagnetic hidden layer vector and the image hidden layer vector. The training sample set is composed of electromagnetic data samples and image data samples, and each electromagnetic data sample and the corresponding image data sample are combined to fully train the electromagnetic encoder and the image encoder. The training sample set is divided into a training set and a test set, and the electromagnetic encoder, the image encoder and the electromagnetic target classifier are iteratively trained through the training set until the loss function converges, and the classification result of the electromagnetic target classifier is verified for accuracy after training, and the detection accuracy of the electromagnetic target classifier is obtained.

[0058] In some embodiments, the iteratively training the electromagnetic encoder, the image encoder and the electromagnetic target classifier based on the training sample set and a total loss function until the total loss function converges comprises:

[0059] encoding the electromagnetic data sample by the electromagnetic encoder to obtain an electromagnetic sample hidden layer vector;

[0060] encoding the image data sample by the image encoder to obtain an image sample hidden layer vector;

[0061] obtaining a superimposed sample hidden layer vector based on the electromagnetic sample hidden layer vector and the image sample hidden layer vector;

[0062] decoding the superimposed sample hidden layer vector by the electromagnetic decoder to obtain reconstructed electromagnetic data;

[0063] decoding the superimposed sample hidden layer vector by the image decoder to obtain reconstructed image data;

[0064] The electromagnetic decoder comprises a convolutional neural network, a long short-term memory network LSTM and a fully connected layer, and the image decoder comprises a convolutional neural network and a fully connected layer. The electromagnetic decoder and the image decoder have similar structures, and the purpose of the decoder is to reconstruct data from multi-modal representation. The encoded electromagnetic data sample and the image data sample are reconstructed respectively;

[0065] The superimposed sample hidden layer vector is input into the electromagnetic target classifier to classify a training sample in the training sample set, to obtain a pseudo label of the training sample, and a detection accuracy of the electromagnetic target classifier is determined based on a true label of the training sample and the pseudo label.

[0066] A first loss function is constructed based on the electromagnetic data sample and the reconstructed electromagnetic data, the first loss function being a cross-entropy loss function, specifically

[0067]

[0068] where y i represents a true electromagnetic data sample distribution, and Φ i represents a reconstructed electromagnetic data distribution after decoding by an electromagnetic decoder, and N represents a total number of training samples.

[0069] A second loss function and a third loss function are constructed based on the image data sample and the reconstructed image data, the second loss function being a mean square error loss function, specifically

[0070]

[0071] where V im represents an image hidden layer vector, and V dec represents reconstructed image data after decoding by an image decoder, and N represents a total number of training samples, and the third loss function being a divergence loss KL (Kullback-Leibler divergence) function, specifically

[0072]

[0073] where p(x i ) represents a true image data distribution, and q(x i ) represents a reconstructed image data distribution after decoding by an image decoder, and N represents a total number of training samples, and 1≤i≤N. The KL divergence between two probability distributions measures the degree of deviation therebetween, and in this embodiment, the degree of deviation between the image data distribution and the reconstructed image data distribution is measured by the KL divergence.

[0074] A fourth loss function is constructed based on the true label of the training sample set and the detection accuracy, the fourth loss function being a cross-entropy loss function, specifically

[0075]

[0076] where z i represents a true label of an electromagnetic data sample, and p i represents a probability that the electromagnetic sample i is predicted to be a corresponding category label.

[0077] The preset category labels in the embodiment specifically include: linear frequency modulation (LFM), frequency modulated continuous wave (FMCW), binary phase shift keying (BPSK), quadrature phase shift keying (QPSK), and quadrature amplitude modulation (QAM), where LFM and FMCW belong to radar signals, and BPSK, QPSK, and QAM belong to communication signals.

[0078] The total loss function is constructed based on the first loss function, the second loss function, the third loss function, and the fourth loss function, and the total loss function is minimized to converge. Specifically, the losses of the above four loss functions are combined, the encoder, the decoder, and the electromagnetic target detection identifier are jointly trained, and the optimized total loss function is defined as

[0079] Loss = λ1Loss1 + λ2Loss2 + λ3Loss3 + λ4Loss4

[0080] Wherein, λ1, λ2, λ3 and λ4 are weights of the corresponding loss functions, and the iterative loss value Loss is continuously optimized by minimizing calculation, specifically

[0081] min | λ1Loss1 + λ2Loss2 + λ3Loss3 + λ4Loss4 |

[0082] After multiple iterations, the loss value Loss converges. After pre-training, the optimal electromagnetic encoder, image encoder, electromagnetic decoder, image decoder, and electromagnetic target classifier are saved.

[0083] It should be noted that the method of the embodiment of the application can be executed by a single device, such as a computer or a server. The method of the embodiment of the application can also be applied to a distributed scenario and completed by multiple devices cooperating with each other. In this distributed scenario, one of the multiple devices can only execute one or more steps in the method of the embodiment of the application, and the multiple devices can interact with each other to complete the method.

[0084] It is to be understood that the foregoing description is directed to embodiments of the application. Various embodiments are described herein, including the best mode embodiments. However, various modifications and changes can be made without departing from the scope and spirit of the application as set forth in the claims below. It is intended that the application be construed as including all such modifications and changes.

[0085] Based on the same inventive concept, the application also provides an electromagnetic target identification device based on multi-source data fusion corresponding to any of the above-mentioned embodiment methods.

[0086] Reference Figure 2 , the electromagnetic target identification device based on multi-source data fusion comprises:

[0087] The acquisition module 201 is configured to acquire electromagnetic signal data and image data of a target electromagnetic device;

[0088] The electromagnetic encoding module 202 is configured to encode the electromagnetic signal data through a pre-constructed electromagnetic encoder to obtain an electromagnetic hidden layer vector;

[0089] The image encoding module 203 is configured to encode the image data through a pre-constructed image encoder to obtain an image hidden layer vector;

[0090] The superimposed encoding module 204 is configured to obtain a superimposed hidden layer vector based on the electromagnetic hidden layer vector and the image hidden layer vector;

[0091] The classification module 205 is configured to input the superimposed hidden layer vector into a pre-constructed electromagnetic target classifier to classify the target electromagnetic device, and determine an electromagnetic signal category of the target electromagnetic device based on a pre-set category label.

[0092] For the convenience of description, the above device is described in various modules based on functions. Of course, in the implementation of the application, the functions of each module can be implemented in the same or multiple software and / or hardware.

[0093] The device of the above-mentioned embodiment is used to implement the corresponding electromagnetic target identification method based on multi-source data fusion in any of the above-mentioned embodiments, and has the beneficial effects of the corresponding method embodiments, which will not be described here.

[0094] Based on the same inventive concept, the application also provides an electronic device corresponding to the method of any of the above embodiments, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the method of any of the above embodiments.

[0095] Figure 3 A more specific hardware structure of an electronic device is shown, which can include a processor 1010, a memory 1020, an input / output interface 1030, a communication interface 1040, and a bus 1050. The processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040 are connected to each other through the bus 1050 for communication within the device.

[0096] The processor 1010 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an ASIC (Application Specific Integrated Circuit), or one or more integrated circuits, etc., for executing related programs to implement the technical solutions provided by the embodiments of the present specification.

[0097] The memory 1020 can be implemented by a ROM (Read Only Memory), a RAM (Random Access Memory), a static storage device, a dynamic storage device, etc. The memory 1020 can store an operating system and other application programs, and when the technical solutions provided by the embodiments of the present specification are implemented by software or firmware, the related program codes are stored in the memory 1020 and executed by the processor 1010.

[0098] The input / output interface 1030 is used to connect input / output modules to realize information input and output. The input / output modules can be configured as components in the device (not shown in the figure) or externally connected to the device to provide corresponding functions. The input device can include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device can include a display, a speaker, a vibrator, an indicator light, etc.

[0099] The communication interface 1040 is used to connect the communication module (not shown in the figure) to realize the communication interaction between the device and other devices. The communication module can realize communication through wired means (such as USB, network cable, etc.) or through wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0100] The bus 1050 includes a path for transferring information between the various components (for example, the processor 1010, the memory 1020, the input / output interface 1030, and the communication interface 1040) of the device.

[0101] It should be noted that although the above device only shows the processor 1010, the memory 1020, the input / output interface 1030, the communication interface 1040 and the bus 1050, in the specific implementation process, the device can also include other components necessary for normal operation. In addition, those skilled in the art can understand that the above device can also only contain the components necessary to implement the embodiments of the present application, and does not have to contain all the components shown in the figure.

[0102] The electronic device of the above embodiment is used to implement the corresponding electromagnetic target identification method based on multi-source data fusion in any of the preceding embodiments, and has the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0103] Based on the same inventive concept, corresponding to the method of any of the above embodiments, the present application also provides a non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the electromagnetic target identification method based on multi-source data fusion as described in any of the above embodiments.

[0104] The computer readable medium of the present embodiment includes permanent and non-permanent, removable and non-removable media, which can be implemented by any method or technology to store information. The information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage device, or any other non-transmission medium that can be used to store information accessible by a computing device.

[0105] The computer instructions stored in the storage medium of the above embodiment are used to cause the computer to perform the electromagnetic target identification method based on multi-source data fusion as described in any of the above embodiments, and have the beneficial effects of the corresponding method embodiments, which are not repeated here.

[0106] Those of ordinary skill in the art will realize that the foregoing discussion of any of the embodiments has been presented for the purpose of illustration and description and is not intended to be exhaustive or to limit the application to the precise forms described, and that various adaptations and modifications are possible within the scope and spirit of the application. For example, while the embodiments discussed above have been described in the context of a memory device, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0107] In addition, to simplify the description and discussion, and so as not to make the embodiments of the application difficult to understand, the well-known power / ground connections to integrated circuit (IC) chips and other components can or can not be shown in the provided drawings. Further, devices can be shown in block diagram form so as not to make the embodiments of the application difficult to understand, and this also takes into account the fact that the details regarding the implementation of these block diagram devices are highly dependent on the platform in which the embodiments of the application are to be implemented (i.e., these details should be well within the understanding of one of ordinary skill in the art). Where specific details (e.g., circuitry) are set forth in order to describe an illustrative embodiment of the application, it should be apparent to one of ordinary skill in the art that the embodiments of the application can be practiced without or with variations of these specific details. Thus, the description should not be viewed as limiting the application, but rather as merely describing a particular embodiment.

[0108] While the application has been described in connection with specific embodiments thereof, it will be understood that many modifications, variations and alternatives will be apparent to those skilled in the art as a result of the foregoing description. For example, other memory architectures (e.g., dynamic RAM (DRAM)) can use the embodiments discussed.

[0109] It is therefore intended that the embodiments of the application embrace all such alternatives, modifications and variations as falling within the broad scope of the appended claims. Accordingly, any and all departures from the above described embodiments are intended to be included within the scope of the application as defined by the following claims.

Claims

1. An electromagnetic target recognition method based on multi-source data fusion, characterized in that, include: Acquire electromagnetic signal data and image data of the target electromagnetic device; The electromagnetic signal data is encoded by a pre-built electromagnetic encoder to obtain an electromagnetic hidden layer vector; The image data is encoded using a pre-built image encoder to obtain the image hidden layer vector; The superimposed hidden layer vector is obtained based on the electromagnetic hidden layer vector and the image hidden layer vector; The superimposed hidden layer vector is input into a pre-constructed electromagnetic target classifier to classify the target electromagnetic device, and the electromagnetic signal category of the target electromagnetic device is determined based on the preset category label.

2. The electromagnetic target identification method according to claim 1, characterized in that, The electromagnetic encoder includes: a convolutional neural network, a long short-term memory network (LSTM), and a fully connected layer.

3. The electromagnetic target identification method according to claim 1, characterized in that, The image encoder includes a convolutional neural network and a fully connected layer.

4. The electromagnetic target identification method according to claim 1, characterized in that, The process of obtaining the superimposed hidden layer vector based on the electromagnetic hidden layer vector and the image hidden layer vector includes: The electromagnetic hidden layer vector and the image hidden layer vector are added together to obtain the superimposed hidden layer vector.

5. The electromagnetic target identification method according to claim 1, characterized in that, Also includes: The electromagnetic encoder, the image encoder, and the electromagnetic target classifier are pre-trained. The pre-training process is as follows: Construct a training sample set, which includes electromagnetic data samples and image data samples; The electromagnetic encoder, the image encoder, and the electromagnetic target classifier are iteratively trained based on the training sample set and the total loss function until the total loss function converges.

6. The electromagnetic target identification method according to claim 5, characterized in that, The iterative training of the electromagnetic encoder, the image encoder, and the electromagnetic target classifier based on the training sample set and the total loss function until the total loss function converges includes: The electromagnetic data sample is encoded by the electromagnetic encoder to obtain the electromagnetic sample hidden layer vector; The image data samples are encoded by the image encoder to obtain the hidden layer vector of the image samples; The superimposed sample hidden layer vector is obtained based on the electromagnetic sample hidden layer vector and the image sample hidden layer vector; The hidden layer vectors of the superimposed samples are decoded by an electromagnetic decoder to obtain reconstructed electromagnetic data; The overlay sample hidden layer vectors are decoded by an image decoder to obtain reconstructed image data; The superimposed sample hidden layer vector is input into the electromagnetic target classifier to classify the training samples in the training sample set, obtain the pseudo label of the training sample, and determine the detection accuracy of the electromagnetic target classifier based on the real label of the training sample and the pseudo label. A first loss function is constructed based on the electromagnetic data samples and the reconstructed electromagnetic data; A second loss function and a third loss function are constructed based on the image data samples and the reconstructed image data; A fourth loss function is constructed based on the true labels of the training samples and the detection accuracy. The total loss function is constructed based on the first loss function, the second loss function, the third loss function, and the fourth loss function. The total loss function is then minimized to make it converge.

7. The electromagnetic target identification method according to claim 6, characterized in that, The electromagnetic decoder includes a convolutional neural network, a long short-term memory network (LSTM), and a fully connected layer; the image decoder includes a convolutional neural network and a fully connected layer.

8. An electromagnetic target identification device based on multi-source data fusion, characterized in that, include: The acquisition module is configured to acquire electromagnetic signal data and image data of the target electromagnetic device; An electromagnetic encoding module is configured to encode the electromagnetic signal data using a pre-built electromagnetic encoder to obtain an electromagnetic hidden layer vector. The image encoding module is configured to encode the image data using a pre-built image encoder to obtain an image hidden layer vector; The overlay coding module is configured to obtain an overlay hidden layer vector based on the electromagnetic hidden layer vector and the image hidden layer vector; The classification module is configured to input the superimposed hidden layer vector into a pre-constructed electromagnetic target classifier to classify the target electromagnetic device and determine the electromagnetic signal category of the target electromagnetic device based on a preset category label.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium storing computer instructions, characterized in that, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 7.

Images