Three-dimensional monitoring and sensing method for rich media data

Abstract

The invention discloses a three-dimensional monitoring and sensing method for rich media data, which comprises the following steps: acquiring to-be-monitored data, inputting the to-be-monitored data into a trained three-dimensional monitoring and sensing model to obtain a sensing result comprising situation description and a target recognition result, the three-dimensional monitoring perception model comprises a feature extraction module, a feature fusion module, a feature alignment module, a large language model and a feature mapper which are sequentially connected from the data input direction to the data output direction. According to the three-dimensional monitoring and sensing method for rich media data, different types of data are monitored, high-dimensional feature fusion of data of different modalities and different source domains is effectively realized, and deep environment three-dimensional sensing is realized; in the process of extracting features through the feature extraction module, feature extraction is carried out on data and images, shallow-layer and deep-layer fusion is carried out through the feature fusion module, information loss is reduced to the maximum extent, and a more accurate perception result is obtained.

Description

Technical Field

[0001] The present invention belongs to the technical field of intelligent perception monitoring, and in particular relates to a stereoscopic monitoring perception method for rich media data. Background Art

[0002] Stereoscopic monitoring and perception of rich media data is aimed at three-dimensional monitoring in different fields such as sea and air. Rich media signal data in different bands such as ultraviolet light, visible light, infrared light and radio signals realizes deep environmental stereoscopic perception, aiming to comprehensively and accurately analyze space, target and situation information in complex environments. However, with the explosive growth of data, traditional data analysis methods face many challenges in processing large-scale, multi-dimensional and highly complex rich media signal data.

[0003] Most existing perception methods are based on simple statistical models, which have obvious limitations in processing capabilities, model generalization capabilities, and intelligent processing methods. They often rely on prior knowledge in a specific field to process a certain type of data alone, and are difficult to adapt to different rich media signal data, and cannot effectively mine the hidden information and complex internal correlations in the data. Summary of the invention

[0004] The purpose of the present invention is to solve the problems raised in the background technology and to propose a stereoscopic monitoring perception method for rich media data.

[0005] To achieve the above object, the technical solution adopted by the present invention is:

[0006] The present invention proposes a stereoscopic monitoring perception method for rich media data, comprising acquiring data to be monitored, inputting the data into a trained stereoscopic monitoring perception model, and obtaining a perception result including a situation description and a target recognition result;

[0007] Wherein, the stereo monitoring perception model includes a feature extraction module, a feature fusion module, a feature alignment module, a large language model and a feature mapper which are sequentially connected from the data input to the output direction;

[0008] Using the data to be monitored as the input of the stereo monitoring perception model, and obtaining the first signal feature and the second signal feature through the feature extraction module;

[0009] Inputting the first signal feature and the second signal feature into a feature fusion module to obtain a signal fusion feature;

[0010] The signal fusion features are input into the feature alignment module to obtain the alignment features. The alignment features and the preset command features are input into the large language model to obtain the situation description.

[0011] The situation description is input into a feature mapper to obtain a target recognition result, and the target recognition result includes a target category and a target position.

[0012] Preferably, the data to be monitored is at least one of ultraviolet light information, visible light information, infrared light information and radio signal data in the same time and the same area, and the ultraviolet light information includes ultraviolet light data and ultraviolet light images, the visible light information includes visible light data and visible light images, and the infrared light information includes infrared light data and infrared light images.

[0013] Preferably, the feature extraction module includes a preprocessor and a feature encoding module. For the radio signal data in the data to be monitored, it is first processed by the preprocessor and then input into the feature encoding module. For the ultraviolet light information, visible light information and infrared light information in the data to be monitored, the images in each information are directly input into the feature encoding module.

[0014] Preferably, in the preprocessor, the radio signal data is first subjected to channelization processing to obtain a preset number of independent low-rate baseband sub-channel signals, and then each low-rate baseband sub-channel signal is subjected to fast Fourier transform to obtain a two-dimensional time-frequency image.

[0015] Preferably, the feature encoding module includes an image feature encoder and an original signal encoder;

[0016] For each image and two-dimensional time-frequency image in each information, input it to the image feature encoder, and in the image feature encoder, for each image: divide the image into × grid areas, obtaining first image feature representatives and feature weights from all grid areas through a first feature extraction structure, wherein the first image feature representatives and feature weights are both n-dimensional vectors, and each element in the first image feature representative is an integer with a value range of [1, m], and the value of each element in the feature weight is between 0 and 1;

[0017] According to each element value represented by the first image feature, a corresponding row vector is searched in a preset dictionary library, and the row vectors corresponding to all element values ​​represented by the first image feature constitute a query matrix with a dimension of n×d, and the dictionary library is m 1×d vectors;

[0018] The first signal feature is calculated by weighted summing the query matrix through the feature weight, and then the first signal feature corresponding to each image is obtained. The calculation formula of the first signal feature is as follows:

[0019] ;

[0020] in,

[0021] ;

[0022] in, represents the first signal characteristic, represents the query matrix row vector, represents the query matrix The feature weights corresponding to the row vectors, The first element value;

[0023] The data in each message and the radio signal data are input into an original signal encoder to obtain a second signal feature corresponding to each data, and the original signal encoder has a multi-layer perception structure;

[0024] The first feature extraction structure is a convolutional neural network.

[0025] Preferably, the feature fusion module includes a shallow feature fusion device and a deep feature fusion device connected in sequence from data input to output direction;

[0026] In the shallow feature fuser, the first signal feature and the second signal feature corresponding to each information or radio signal data in the monitored data are spliced ​​according to the feature channel dimension to obtain a first shallow fusion feature, and the corresponding first signal feature and the second signal feature are spliced ​​according to the feature vector dimension to obtain a second shallow fusion feature;

[0027] Pass each first shallow fusion feature through the first mapping module to obtain the first feature, pass each second shallow fusion feature through the second mapping module to obtain the second feature, then add the corresponding first feature and second feature to obtain the third feature, concatenate all the third features, and then input them into the deep feature fuser to obtain the signal fusion feature;

[0028] The first mapping module and the second mapping module are both two fully connected layers connected in sequence.

[0029] Preferably, the deep feature fuser is a multi-layer self-attention mechanism structure, and each layer of the self-attention mechanism structure performs attention mechanism calculation on the input of the layer, and the calculation formula is as follows:

[0030] ;

[0031] in, Indicates The output of the layer self-attention mechanism structure, Indicates The output of the layer self-attention mechanism structure, Represents the dimension, express function, Indicates The query vector of the layer, Indicates The key vector of the layer, Indicates The value vector of the layer, Indicates transpose.

[0032] Preferably, the feature alignment module and the feature mapper are both multiple fully connected layers connected in sequence.

[0033] Preferably, when training the stereoscopic monitoring perception model, the loss function of the situation description and the loss function of the target recognition result are weighted to obtain the total loss, and the parameters of the stereoscopic monitoring perception model are back-propagated and updated based on the total loss.

[0034] Preferably, the training data set for training the stereo monitoring perception model includes ultraviolet light information, visible light information, infrared light information and radio signal data in the same time and the same area.

[0035] Compared with the prior art, the present invention has the following beneficial effects:

[0036] This stereoscopic monitoring and perception method for rich media data monitors different types of data, effectively realizes the high-dimensional feature fusion of data of different modalities and different source domains, and realizes the stereoscopic perception of deep environment;

[0037] In the process of extracting features through the feature extraction module, both data and images are extracted, and shallow and deep fusion are performed through the feature fusion module to minimize information loss and obtain more accurate perception results;

[0038] During the image feature extraction process, the image feature encoder searches for the corresponding row vector in the preset dictionary library for each element value represented by the first image feature, and then performs weighted sum calculation on the query matrix through the feature weight to obtain the first signal feature, so as to form a more effective fusion and obtain better results. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] Figure 1 A module block diagram of a stereoscopic monitoring perception model of a stereoscopic monitoring perception method for rich media data of the present invention;

[0040] Figure 2 It is a module block diagram of the preprocessor of the present invention;

[0041] Figure 3 It is a module block diagram of the feature encoding module of the present invention;

[0042] Figure 4 It is a module block diagram of the feature fusion module of the present invention;

[0043] Figure 5 This is a module block diagram of the shallow feature fuser of the present invention. DETAILED DESCRIPTION

[0044] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0045] In one embodiment, Figure 1-Figure 5 As shown, a stereoscopic monitoring perception method for rich media data is provided, including:

[0046] Step 1: Obtain the data to be monitored and input it into the trained stereo monitoring perception model to obtain the perception results including situation description and target recognition results;

[0047] It should be noted that the data to be monitored is at least one of ultraviolet light information, visible light information, infrared light information and radio signal data (such as electromagnetic signal data) in the same time and the same area, and the ultraviolet light information includes ultraviolet light data and ultraviolet light images, the visible light information includes visible light data and visible light images, and the infrared light information includes infrared light data and infrared light images. The process of obtaining the data to be detected can be obtained through corresponding equipment (such as cameras, sensors, etc.).

[0048] Step 2: The stereo monitoring perception model includes a feature extraction module, a feature fusion module, a feature alignment module, a large language model and a feature mapper which are sequentially connected from the data input to the output direction. The data to be monitored is used as the input of the stereo monitoring perception model, and the first signal feature and the second signal feature are obtained through the feature extraction module;

[0049] Step 2.1, the feature extraction module includes a preprocessor and a feature encoding module. For the radio signal data in the data to be monitored, it is first processed by the preprocessor and then input into the feature encoding module. For the ultraviolet light information, visible light information and infrared light information in the data to be monitored, the images in each information are directly input into the feature encoding module.

[0050] Step 2.1: Figure 2As shown, in the preprocessor, the radio signal data is first subjected to channelization processing to obtain a preset number of independent low-rate baseband sub-channel signals (wherein the channelization processing is processed using a preset number of filters, and each low-rate baseband sub-channel signal is obtained after being processed by a filter. The number of filters can be determined according to actual needs. For example, if there are two filters, two low-rate baseband sub-channel signals are obtained), and then each low-rate baseband sub-channel signal is subjected to fast Fourier transform to obtain a two-dimensional time-frequency image (each low-rate baseband sub-channel signal after fast Fourier transform is plotted in the same coordinate system to obtain a two-dimensional time-frequency image).

[0051] Step 2.2: Figure 3 As shown (where Figure 3 ), the feature encoding module includes an image feature encoder and an original signal encoder.

[0052] Step 2.3: For each image and two-dimensional time-frequency image in each information, input them into the image feature encoder (the image feature encoder is a Transformer-based ViT network model), and in the image feature encoder, for each image: divide the image into × grid area (i.e. OK Column Total grid area, such as The value of is 16, and the image with a resolution of 224x224 pixels is divided into 256 grid areas). All grid areas are passed through the first feature extraction structure (where the first feature extraction structure is a convolutional neural network, such as a Resnet18 network) to obtain the first image feature representative and feature weight, where the first image feature representative and feature weight are both n-dimensional vectors (column vectors), and each element in the first image feature representative is an integer with a value range of [1,m]. For example, m is 8192, n is 8, and the value of each element in the feature weight is between 0 and 1, and the sum of all element values ​​is 1;

[0053] According to each element value represented by the first image feature, the corresponding row vector is queried in the preset dictionary library, and the row vectors corresponding to all the element values ​​represented by the first image feature constitute a query matrix with a dimension of n×d (such as d is 768, that is, an 8×768 query matrix), and the dictionary library is m 1×d vectors; wherein the dictionary library is obtained by pre-training using a large amount of image data, such as using a convolutional neural network (ResNet50, VGG, etc.) or a ViT network, taking any image as input, and outputting a 1×d vector, clustering the vectors obtained for each image, and setting the number of clustering categories to 8192. Finally, the 8192 clusters obtained are averaged to obtain 8192 1×d vectors to form a dictionary library;

[0054] The first signal feature is obtained by weighted summing the query matrix through the feature weights, and then the first signal feature corresponding to each image (that is, the first signal feature corresponding to the ultraviolet image, visible light image, infrared image and two-dimensional time-frequency image) is obtained, and the calculation formula of the first signal feature is as follows:

[0055] ;

[0056] in,

[0057] ;

[0058] in, represents the first signal characteristic, represents the query matrix row vector, represents the query matrix The feature weights corresponding to the row vectors, The first element value;

[0059] Step 2.4: The data in each information and the radio signal data are input into the original signal encoder to obtain the second signal features corresponding to each data (i.e., the second signal features corresponding to the ultraviolet light data, visible light data, infrared light data, and radio signal data respectively), and the original signal encoder has a multi-layer perception structure (MLP).

[0060] Step 3: Input the first signal feature and the second signal feature into a feature fusion module to obtain a signal fusion feature.

[0061] Step 3.1: Figure 4 As shown (where Figure 4 The module block diagram of the feature fusion module is shown in FIG. 1 , which takes visible light information as an example. The feature fusion module includes a shallow feature fuser and a deep feature fuser connected in sequence from data input to output.

[0062] Step 3.2: Figure 5As shown, in the shallow feature fusion device, the first signal features and the second signal features corresponding to each information or radio signal data in the monitored data are spliced ​​according to the feature channel dimension to obtain the first shallow fusion feature, and the corresponding first signal features and the second signal features are spliced ​​according to the feature vector dimension to obtain the second shallow fusion feature (wherein the corresponding first signal features and the second signal features represent: the first signal features obtained by the ultraviolet image in the ultraviolet information and the second signal features obtained by the ultraviolet data are corresponding, the first signal features obtained by the visible light image in the visible light information and the second signal features obtained by the visible light data are corresponding, the first signal features obtained by the infrared image in the infrared information and the second signal features obtained by the infrared data are corresponding, the first signal features obtained by the two-dimensional time-frequency image and the second signal features obtained by the radio signal data are corresponding, wherein there are several specific corresponding first signal features and second signal features, depending on the types contained in the monitored data, such as Figure 5 As shown, a module block diagram of a shallow feature fusion device is shown by taking a corresponding first signal feature and a second signal feature as an example. Similarly, the corresponding first feature and second feature below are also the same);

[0063] Pass each first shallow fusion feature through the first mapping module to obtain the first feature, pass each second shallow fusion feature through the second mapping module to obtain the second feature, then add the corresponding first feature and the second feature (add the corresponding position features) to obtain the third feature, and then splice all the third features (when there are multiple third features, splicing is required, when there is only one third feature, splicing is not required, and it is directly input to the deep feature fuser), and then input to the deep feature fuser to obtain the signal fusion feature; the first mapping module and the second mapping module are two fully connected layers connected in sequence;

[0064] It should be noted that, since the data to be monitored is at least one of ultraviolet light information, visible light information, infrared light information and radio signal data at the same time and in the same area, when the data to be monitored is one of ultraviolet light information, visible light information, infrared light information and radio signal data at the same time and in the same area, such as the data to be monitored is visible light information, after steps 2-3.2, the third feature is obtained and directly input into the deep feature fuser; when the data to be monitored is multiple types of ultraviolet light information, visible light information, infrared light information and radio signal data at the same time and in the same area, such as ultraviolet light information and visible light information, after steps 2-3.2, the third feature corresponding to the ultraviolet light information and the third feature corresponding to the visible light information are obtained, and then the two third features are spliced ​​and input into the deep feature fuser (that is, when the data to be monitored contains one type, after passing through the shallow feature fuser, the obtained third feature is directly input into the deep feature fuser; when the data to be monitored contains multiple types, after passing through the shallow feature fuser, the multiple third features obtained are first spliced ​​and then input into the deep feature fuser).

[0065] Step 3.3: The deep feature fuser is a multi-layer self-attention mechanism structure. Each layer of the self-attention mechanism structure performs attention mechanism calculation on the input of the layer, and the calculation formula is as follows:

[0066] ;

[0067] in,

[0068] ;

[0069] ;

[0070] ;

[0071] in, Indicates The output of the layer self-attention mechanism structure, Indicates The output of the self-attention mechanism structure is When it is equal to 1, , is the input of the deep feature fusion device, Represents the dimension, express function, Indicates The query vector of the layer, Indicates The key vector of the layer, Indicates The value vector of the layer, represents transpose; The dimension of each vector is , Indicates The output of the layer self-attention mechanism structure is mapped to the query vector through the first feature map. Indicates The output of the layer self-attention mechanism structure is mapped to the key vector through the second feature map. Indicates The output of the self-attention mechanism structure is mapped to a value vector through the third feature map. Each feature map is mapped through a fully connected layer with a dimension of The vector of The fully connected layer (i.e., the dimension is ), calculate the parameter matrix The vector is calculated by matrix multiplication.

[0072] Step 4: Input the signal fusion features into the feature alignment module to obtain the alignment features, and input the alignment features and the preset instruction features into the large language model to obtain the situation description; wherein the feature alignment module is used to map the output of the deep feature fuser and the input of the large language model to the same high-dimensional space;

[0073] It should be noted that the feature alignment module is a plurality of fully connected layers connected in sequence, such as two fully connected layers connected in sequence; the preset instruction features are converted from the task instructions of the text prompt into tokens (a token can be a word, a subword (such as a letter, a syllable or a subword fragment), a character, etc.), and then converted into instruction features through the BPE (Byte Pair Encoding) method, wherein the task instructions of the text prompt in this embodiment can be "please evaluate the current situation", "give a description of the situation", etc.; the large language model can be Qwen2.4-7B, Baichuan2-13B, etc.

[0074] Step 5: Input the situation description into the feature mapper to obtain a target recognition result, and the target recognition result includes the target category and the target position, wherein the feature mappers are multiple fully connected layers connected in sequence, such as two fully connected layers connected in sequence, and the number of neurons in each fully connected layer is not less than 2048.

[0075] In another embodiment, during the training of the stereo monitoring perception model:

[0076] Step I: First, construct a training data set, which includes ultraviolet light information, visible light information, infrared light information, and radio signal data at the same time and in the same area, and each information contains corresponding data and images;

[0077] Step II, labeling the training data set: the radio signal data is processed through a preprocessor to obtain the corresponding two-dimensional time-frequency image, the target category and position in each image (i.e., ultraviolet light image, visible light image, infrared light image, and two-dimensional time-frequency image) are labeled, and the situation description of the scene in the corresponding area of ​​the data (i.e., ultraviolet light data, visible light data, infrared light data, and radio signal data, and the area corresponding to the four types of data is the same, so there is only one situation description of the scene in the area). Among them, when training the stereo monitoring perception model, the input of each model includes at least two or more of the ultraviolet light information, visible light information, infrared light information, and radio signal data;

[0078] Stereo monitoring perception model pre-training stage:

[0079] Step III: Initialize the large language model and feature encoding module based on the preset pre-trained model parameters (the preset pre-trained model adopts an existing model, such as the Tongyi Qianwen model), freeze the parameters of the large language model and feature encoding module, and initialize the feature fusion module, feature alignment module and feature mapper with random parameters.

[0080] Step IV: Input the labeled data set into the stereo monitoring perception model for forward reasoning to obtain the target category and position;

[0081] Step V: Calculate the loss function value by combining the inferred target category and position with the manually annotated label; perform back propagation to update the model parameters according to the loss function value, and repeat steps IV to V until the model converges.

[0082] Stereo monitoring perception model fine-tuning training phase:

[0083] Step VI: Initialize the stereo monitoring perception model parameters based on the pre-training stage, wherein the large language model and the feature encoding module are initialized with the same preset pre-training model parameters as those in the pre-training stage, and keep the large language model in a frozen state, and do not participate in parameter updating during the training process;

[0084] Step VII: Input the labeled data set into the stereo monitoring perception model for forward reasoning to obtain the target category and location, as well as the situation description;

[0085] Step VIII, calculating the loss function value of the target category and position obtained by inference and the manually annotated label, and calculating the loss function value of the situation description obtained by inference and the manually performed situation description, weighting the loss function of the situation description and the loss function of the target recognition result to obtain the total loss, and back-propagating the parameters of the stereo monitoring perception model based on the total loss until the model converges;

[0086] The loss function of the target recognition result is Indicates that the weight is Indicates that the loss function of the situation description is expressed as Indicates that the weight is If for: ,and , , ,like .

[0087] The stereoscopic monitoring perception method of rich media data monitors different types of data, effectively realizes high-dimensional feature fusion of data of different modes (referring to image and data modes) and different source domains (referring to ultraviolet light information, visible light information, infrared light information and radio signal data of different source domains), and realizes deep environment stereoscopic perception; in the process of extracting features through the feature extraction module, feature extraction is performed on both data and images, and shallow and deep fusion is performed through the feature fusion module to minimize information loss and obtain more accurate perception results; in the process of extracting image features through the image feature encoder, each element value represented by the first image feature is queried in the preset dictionary library for the corresponding row vector, and then the query matrix is ​​weighted and summed by the feature weight to obtain the first signal feature, so that a more effective fusion is formed to obtain a better result.

[0088] It should be understood that, unless otherwise explicitly stated herein, there is no strict order restriction for the execution of the steps; the steps may be executed in other orders, and the steps may be executed at the same time or at different times.

[0089] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A stereoscopic monitoring perception method for rich media data, characterized by: The stereoscopic monitoring perception method of rich media data includes: Obtain the data to be monitored and input it into the trained stereoscopic monitoring perception model to obtain the perception results including situation description and target recognition results; Wherein, the stereo monitoring perception model includes a feature extraction module, a feature fusion module, a feature alignment module, a large language model and a feature mapper which are sequentially connected from the data input to the output direction; Using the data to be monitored as the input of the stereo monitoring perception model, and obtaining the first signal feature and the second signal feature through the feature extraction module; Inputting the first signal feature and the second signal feature into a feature fusion module to obtain a signal fusion feature; The signal fusion features are input into the feature alignment module to obtain the alignment features. The alignment features and the preset command features are input into the large language model to obtain the situation description. The situation description is input into a feature mapper to obtain a target recognition result, and the target recognition result includes a target category and a target position.

2. The stereoscopic monitoring perception method of rich media data according to claim 1, characterized in that: The data to be monitored is at least one of ultraviolet light information, visible light information, infrared light information and radio signal data in the same time and the same area, and the ultraviolet light information includes ultraviolet light data and ultraviolet light images, the visible light information includes visible light data and visible light images, and the infrared light information includes infrared light data and infrared light images.

3. The stereoscopic monitoring perception method of rich media data according to claim 2, characterized in that: The feature extraction module includes a preprocessor and a feature encoding module. For the radio signal data in the data to be monitored, it is first processed by the preprocessor and then input into the feature encoding module. For the ultraviolet light information, visible light information and infrared light information in the data to be monitored, the images in each information are directly input into the feature encoding module.

4. The stereoscopic monitoring perception method of rich media data according to claim 3, characterized in that: In the preprocessor, the radio signal data is firstly channelized to obtain a preset number of independent low-rate baseband sub-channel signals, and then each low-rate baseband sub-channel signal is fast Fourier transformed to obtain a two-dimensional time-frequency image.

5. The stereoscopic monitoring perception method of rich media data according to claim 4, characterized in that: The feature encoding module includes an image feature encoder and an original signal encoder; For each image and two-dimensional time-frequency image in each information, input it to the image feature encoder, and in the image feature encoder, for each image: divide the image into × grid areas, obtaining first image feature representatives and feature weights from all grid areas through a first feature extraction structure, wherein the first image feature representatives and feature weights are both n-dimensional vectors, and each element in the first image feature representative is an integer with a value range of [1, m], and the value of each element in the feature weight is between 0 and 1; According to each element value represented by the first image feature, a corresponding row vector is searched in a preset dictionary library, and the row vectors corresponding to all element values ​​represented by the first image feature constitute a query matrix with a dimension of n×d, and the dictionary library is m 1×d vectors; The first signal feature is calculated by weighted summing the query matrix through the feature weight, and then the first signal feature corresponding to each image is obtained. The calculation formula of the first signal feature is as follows: ； in, ； in, represents the first signal characteristic, represents the query matrix row vector, represents the query matrix The feature weights corresponding to the row vectors, The first element value; The data in each message and the radio signal data are input into an original signal encoder to obtain a second signal feature corresponding to each data, and the original signal encoder has a multi-layer perception structure; The first feature extraction structure is a convolutional neural network.

6. The stereoscopic monitoring perception method of rich media data according to claim 2, characterized in that: The feature fusion module includes a shallow feature fuser and a deep feature fuser connected in sequence from data input to output direction; In the shallow feature fuser, the first signal feature and the second signal feature corresponding to each information or radio signal data in the monitored data are spliced ​​according to the feature channel dimension to obtain a first shallow fusion feature, and the corresponding first signal feature and the second signal feature are spliced ​​according to the feature vector dimension to obtain a second shallow fusion feature; Pass each first shallow fusion feature through the first mapping module to obtain the first feature, pass each second shallow fusion feature through the second mapping module to obtain the second feature, then add the corresponding first feature and second feature to obtain the third feature, concatenate all the third features, and then input them into the deep feature fuser to obtain the signal fusion feature; The first mapping module and the second mapping module are both two fully connected layers connected in sequence.

7. The stereoscopic monitoring perception method of rich media data according to claim 6, characterized in that: The deep feature fuser is a multi-layer self-attention mechanism structure. Each layer of the self-attention mechanism structure performs attention mechanism calculation on the input of the layer, and the calculation formula is as follows: ； in, Indicates The output of the layer self-attention mechanism structure, Indicates The output of the layer self-attention mechanism structure, Represents the dimension, express function, Indicates The query vector of the layer, Indicates The key vector of the layer, Indicates The value vector of the layer, Indicates transpose.

8. The stereoscopic monitoring perception method of rich media data according to claim 1, characterized in that: The feature alignment module and the feature mapper are both multiple fully connected layers connected in sequence.

9. The stereoscopic monitoring perception method of rich media data according to claim 1, characterized in that: When training the stereo monitoring perception model, the loss function of the situation description and the loss function of the target recognition result are weighted to obtain the total loss, and the parameters of the stereo monitoring perception model are back-propagated and updated based on the total loss.

10. The stereoscopic monitoring perception method of rich media data according to claim 2, characterized in that: The training data set for training the stereo monitoring perception model includes ultraviolet light information, visible light information, infrared light information and radio signal data at the same time and in the same area.

Images