Audio-visual event positioning-oriented multi-mode credible semantic communication construction method

Through the multimodal trusted semantic communication method for audio-visual event positioning, the multimodal data transmission stability problem of traditional communication systems in complex environments is solved, efficient and reliable information transmission and event positioning are achieved, and powerful The secure communication framework improves system performance.

CN119011087AActive Publication Date: 2024-11-22JIANGSU UNIV

Patent Information

Application Number
CN202411225534.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2024-09-03
Publication Date
2024-11-22
Estimated Expiration
2044-09-03

AI Technical Summary

Technical Problem

Traditional communication systems are difficult to maintain efficient and stable when handling multimodal data transmission in complex environments, especially in audio-visual event positioning tasks, which are susceptible to physical noise and channel interference, resulting in information semantic loss or error, affecting task execution and System performance.

Method used

A multimodal trusted semantic communication method for audio-visual event positioning is proposed. Data is obtained through audio-visual sensors to realize the visual attention mechanism of cross-modal audio guidance, using trusted channel encoding and decoding, real-to-complex conversion, and using chan_layer Solve the problem of semantic information distortion, and calculate signal recovery through complex domain transformation and channel estimation, and finally calculate cross-modal similarity for event location.

Benefits of technology

It realizes reliable and efficient information transmission under complex channel conditions, ensures absolute fidelity of output information, provides a powerful secure communication framework, can maintain the confidentiality and integrity of communication in potential hostile networks, and improves multi-mode overall performance of state tasks.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119011087A_ABST
    Figure CN119011087A_ABST
Patent Text Reader

Abstract

The invention relates to a communication technology, in particular to an audio-visual event positioning-oriented multi-mode credible semantic communication construction method, and aims to improve the safety and reliability of an audio-visual event (AVE) positioning task. According to the invention, through advanced semantic coding and channel coding technologies, the integrity and privacy of data in a transmission process are effectively protected. By simulating the channel condition of the real world and adopting Reed-Solomon coding and AES encryption technologies, the accuracy and reliability of data transmission are remarkably improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to communication technology, and in particular to a method for constructing multimodal trusted semantic communication for audio-visual event positioning. Background Art

[0002] With the rapid development of science and technology, mobile devices and smart applications have become increasingly popular, greatly enriching people's daily lives and work efficiency. The widespread use of these devices and applications has led to an explosive growth in wireless data traffic. According to data from the International Telecommunication Union (ITU), global mobile data traffic has grown at an exponential rate in the past few years, and this trend is expected to continue in the next few years. This rapid growth in data traffic poses unprecedented challenges to modern communication systems.

[0003] Traditional communication systems mainly rely on bit-level data transmission, that is, converting data into a bit stream at the transmitter and then accurately recovering these bits to reconstruct the original data at the receiver. This process requires the communication system to have high-quality channel conditions and a high signal-to-noise ratio (SNR) to ensure the accuracy and reliability of data transmission. However, as the amount of data increases, these requirements of traditional communication systems become increasingly difficult to meet. In addition, traditional communication systems often have difficulty maintaining efficiency and stability when dealing with data transmission in complex environments.

[0004] The limitations of traditional communication systems are even more evident in multimodal tasks, especially audio-visual event (AVE) localization tasks. Multimodal tasks usually involve processing data from different sources (such as audio and video) and integrating them to obtain richer semantic information. For example, in applications such as security monitoring, environmental perception of autonomous vehicles, and smart medical diagnosis, accurate identification and localization of audio and visual events are crucial. However, due to the presence of physical noise and channel interference, multimodal information is susceptible to loss and interference during transmission, resulting in semantic loss or errors in information, which in turn affects the execution of tasks and the performance of the system.

[0005] To address these issues, a new communication paradigm is needed, one that shifts from traditional bit accuracy to semantic fidelity. This paradigm emphasizes the direct transmission and retrieval of the semantics of the content rather than a pure bit stream. Semantic communication is better able to handle data transmission under complex channel conditions because it focuses on the semantics of the information rather than the specific bit pattern. This shift requires communication systems to be able to handle not only the accurate transmission of data, but also to understand and utilize the semantic content of the data, thereby achieving more efficient and reliable communication in a variety of environments.

[0006] In addition, cross-modal information complementarity and semantic richness in multimodal tasks are crucial to improving the overall performance of the system. For example, in audio-visual event localization, audio and video data can complement each other to provide a more comprehensive scene understanding. Audio data can provide temporal information about the event, while video data can provide spatial information. By integrating information from these two modalities, the system can more accurately locate the location and time of the event. However, to achieve this integration, a communication framework that can effectively process and utilize multimodal data is required. Summary of the invention

[0007] Modern communication systems are facing the challenges brought by the rapid growth of wireless data traffic, and new technologies and methods need to be developed to improve the security and reliability of multimodal semantic communication, especially in key tasks such as audio-visual event localization. This requires not only the improvement of channel coding and decoding technology, but also the development of new semantic coding and decoding methods, as well as signal processing technology that can adapt to complex communication environments.

[0008] In order to solve the problems existing in the prior art, the present invention proposes a method for constructing multimodal trusted semantic communication for audiovisual event localization, comprising the following steps:

[0009] S1: The audio-visual sensor obtains audio-visual perception data of the same time interval, one user holds video data and one user holds audio data;

[0010] S2: Split the video and audio time series S into T non-overlapping but continuous segments; denoted as video segment sequence and audio clip sequences The duration of each segment is one second, where t is the time segment, ranging from 1 to T, T is a constant, V t represents the video data in time segment t, a t Represents the audio data in time segment t;

[0011] S3: Implementing cross-modal audio-guided visual attention mechanism for semantic communication between two users, achieving audio-guided visual attention;

[0012] S4: performing trusted channel coding on the video segments and the audio segments of the corresponding time intervals in the continuous synchronized audiovisual segments held by the two users respectively;

[0013] S5: Real to complex conversion module, which converts the feature tensor into a complex representation to cope with the complexity of data transmission in the analog channel;

[0014] S6: Using chan_layer to solve the semantic information distortion problem in the communication ecosystem, the semantic information distortion problem includes additive white Gaussian noise, Rayleigh fading and Rician fading;

[0015] S7: Calculate the channel matrix with the help of complex domain transformation and channel estimation method to achieve signal recovery;

[0016] S8: The receiver model decodes and decrypts the signal propagated through the channel in preparation for the final classification or prediction task;

[0017] S9: Calculate cross-modal similarity and predict event categories for localization.

[0018] Furthermore, the visual attention mechanism in the above S3 includes:

[0019] S3.1 Set V in step S2 t and a t Use the VGG-19 network to extract the initial visual features V and initial audio features A as the initial vectors, and the features over the entire time series are represented as audio features. and visual features

[0020] S3.2 transmits the audio from the audio user to the video user through a physical channel and encodes it using an audio semantic encoder:

[0021]

[0022] Among them A single is the output result, i.e., a single audio clip obtained after enhancement processing; is the audio feature, Is a function that processes audio features and enhances the signal, Θ ,1 represents the trainable parameters, which consists of a one-dimensional convolution with C channels and a kernel size of 7, and there are B convolutional blocks;

[0023]

[0024] Each block consists of a residual unit and a downsampling layer with strided convolutions, where the kernel size K is twice the stride S. The residual unit has two convolutions with kernel size 3 and a skip connection;

[0025] S3.3 integrates all the processed audio clips to obtain The integration process is as follows;

[0026]

[0027] Among them, DownSample(K,S) is a function, which means downsampling. K is the kernel size, which refers to the size of the convolution kernel used in the downsampling process; S is the step size, which refers to the step size of moving the convolution kernel in the downsampling process. Block i It is the i-th block after a series of operations. ResUnit is a residual unit containing several convolutional layers and a skip connection to retain the original input information.

[0028] S3.4 Audio features received by the user holding the video The recovered features are accurately recovered in the unimodal task-oriented decoding stage; the recovered features are used in the cross-modal semantic encoder to enhance the visual features. The cross-modal visual semantic encoder is defined as:

[0029]

[0030] where Θ is v The trainable parameters of the encoder, is the input visual feature, It is a function that processes visual features and enhances the signal;

[0031] S3.5 Calculate the attention weight α t :

[0032]

[0033] Where σ is the sigmoid function, M v 、M a The projected features obtained by projecting the visual and audio features into a shared space, W f is a trainable parameter;

[0034] S3.6 Using attention weight α t Audio Features and visual features Weighted visual features

[0035] Furthermore, the trusted channel coding in the above S4 includes:

[0036] S4.1 Use public key cryptography to establish a shared session key between the sender and receiver;

[0037] S4.2 uses a shared session key to symmetrically encrypt the transmitted information;

[0038] The S4.3 semantic encoder holds the key pair (pk, sk) of the elliptic curve cryptography algorithm, and the calculation formula is as follows:

[0039] X a=CE a (A 1 ; Φ a )

[0040] A 1 Indicates that the input data is the weighted V obtained in S3 at and received audio characteristics CE a Used to extract attention-related features, Φ a This is a vector representing CE a The learnable parameters in ;

[0041] S4.4 sets the AES encryption key K from usekey, encrypts the key K using the ECC public key, and obtains the ciphertext C 1 , the audio message and the video message are processed according to the data block model respectively;

[0042] S4.5 uses the key K and CBC mode to perform AES encryption on the processed message M to obtain the ciphertext C 2 , calculate the hash value h of message m according to the formula h = SHA-3 (m), and get the final ciphertext c = (c 1 ,c 2 ,h);

[0043] S4.6 encodes the obtained ciphertext c using Reed-Solomon coding.

[0044] Furthermore, the real number to complex number conversion module in the above S5 includes:

[0045] S5.1 Convolutional neural network layer learns diverse local features from multimodal semantics, where M represents a multimodal set, including weighted visual features V at and initial audio features A, and are respectively the temporal encoding of enhanced visual and audio features using LSTM at time t, They are the hidden state and unit state of the visual LSTM at the previous time t-1; similarly, the auditory LSTM They are the hidden state and unit state of the auditory LSTM at the previous time t-1;

[0046] S5.2 reshapes the encoded and learned data into a three-dimensional tensor and converts it into a complex form, i.e., f complex =RealToComplex(f cnn ).

[0047] Furthermore, the above S6 solves the problem of semantic information distortion in the communication ecosystem by:

[0048] S6.1 performs channel simulation. AWGN channel: generate Gaussian noise for the real and imaginary parts of the input signal respectively, and add the Gaussian noise to the input signal; set the channel matrix I to the unit matrix, indicating a distortion-free channel; Rayleigh fading channel: generate a complex channel matrix H and apply it to the input signal to form the output signal Y; calculate the conjugate transpose H of the channel matrix H H and the outer product H H H, Rician fading channel: The channel matrix H is constructed based on the line-of-sight and non-line-of-sight components and applied to the signal to produce the output Y;

[0049] S6.2 Rayleigh fading channel simulation: Generate a complex channel matrix H and apply it to the input signal to form the output signal Y; calculate the conjugate transpose H of the channel matrix H H and the outer product H H H;

[0050] S6.3 Rician fading channel simulation: Construct the channel matrix H based on the line-of-sight and non-line-of-sight components and apply it to the signal to produce the output Y.

[0051] Furthermore, the above S7 includes:

[0052] S7.1 defines the received signal vector as Y, the transmitted signal vector as X, the channel matrix as H, and the additive white Gaussian noise vector as N; the received signal can be expressed as: Y = HX + N;

[0053] S7.2 Signal detection and signal recovery: A zero-forcing minimum mean square error detector is used to eliminate multipath interference and reduce noise amplification; the purpose of the ZF-LMMSE detector is to find a detection matrix W so that the output is close to the actual transmitted signal The detection of the ZF-LMMSE detector can be shown as:

[0054]

[0055] Among them, H H is the conjugate transpose of the channel matrix H, is the variance of the noise, is the variance of the transmitted signal; I is the identity matrix.

[0056] Furthermore, the above S8 includes the following contents:

[0057] S8.1 After signal recovery and signal detection, the estimated complex signal is RS decoded by a trusted decoder and the data is converted into audio and visual feature information;

[0058] S8.2 The semantic decoder recovers the AES encrypted message c,

[0059] c 1 =First 160 bits of c

[0060] The first 160 bits are taken as c 1 , extract the last 128 bits of the ciphertext c as h, and extract the remaining bits in the middle of the ciphertext c as c 2 ; Use the ECC private key to decrypt it and get the decrypted message m;

[0061] S8.3 uses the combined audio and visual semantic information and adopts the positive sample propagation network By filtering samples and removing negative samples, semantic decoding is completed and the decoded audio and visual features are obtained respectively as A e and V e .

[0062] Furthermore, the above S9 includes:

[0063] The audio features obtained after S9.1 decoding are processed through two fully connected layers to obtain A e ; The decoded visual features are processed by two Conv2D blocks to obtain V e ; Calculate cosine similarity at the same time

[0064] S9.2 uses a shared fully connected layer to perform feature fusion and complete event prediction

[0065] F = ReLU(W · [A e ; V e ]+b)

[0066] Among them, W is the weight matrix, b is the bias vector, and F is the predicted result;

[0067] S9.3 Locate the location where the event occurs and determine the location of the audio event in the video frame. The positioning formula is:

[0068]

[0069] Where L represents the location result of the event, P is the set of all possible locations, and the location with the highest cosine similarity score is taken as the location where the event occurred.

[0070] The beneficial technical effects of the present invention are as follows:

[0071] 1. A method for constructing multimodal trusted semantic communication for audiovisual event localization is designed, and reliable information transmission is achieved through reliability channel encoders and decoders.

[0072] 2. A two-layer coding system is designed. By adding a layer of error correction code outside the traditional channel encoder, the small weight errors that may be introduced by the traditional codec pair can be corrected twice to ensure the absolute fidelity of the output information. With the help of a reliable channel encoder and decoder, it can be ensured that the information input to the encoder is exactly the same as the information output by the decoder, laying the foundation for the sender to transmit the ciphertext and the receiver to decrypt it later.

[0073] 3. The concept of hybrid encryption is used to achieve secure transmission of semantic information. First, a shared secure session key is established between the sender and the receiver using the principles of public key cryptography. Then, the information to be transmitted is symmetrically encrypted using this session key. This approach combines the advantages of public key and symmetric key cryptography, the former facilitating key distribution and the latter providing efficiency and speed in data encryption. The result is a powerful secure communication framework that can maintain the confidentiality and integrity of communications in potentially hostile networks. BRIEF DESCRIPTION OF THE DRAWINGS

[0074] Figure 1 A method for constructing multimodal trusted semantic communication for audiovisual event localization. DETAILED DESCRIPTION

[0075] The present invention will be described in detail below with reference to the accompanying drawings and specific embodiments. Figure 1 As shown, the present invention is a method for constructing multimodal trusted semantic communication for audio-visual event positioning, comprising the following steps:

[0076] S1: The audio-visual sensor obtains audio-visual perception data of the same time interval, one user holds video data and one user holds audio data;

[0077] S2: Split the video and audio time series S (corresponding to the same application scenario but sent to the remote receiver through different sensors and transmission ends) into T non-overlapping but continuous segments; denoted as video segment sequence and audio clip sequences The duration of each segment is one second, where t is the time segment, ranging from 1 to T, T is a constant, V t represents the video data in time segment t, a t Represents the audio data in time segment t;

[0078] S3: Implementing cross-modal audio-guided visual attention mechanism for semantic communication between two users, achieving audio-guided visual attention;

[0079] S4: performing trusted channel coding on the video segments and the audio segments of the corresponding time intervals in the continuous synchronized audiovisual segments held by the two users respectively;

[0080] S5: Real to complex conversion module, which converts the feature tensor into a complex representation to cope with the complexity of data transmission in the analog channel;

[0081] S6: Using chan_layer to solve the semantic information distortion problem in the communication ecosystem, wherein the semantic information distortion problem includes additive white Gaussian noise (AWGN), Rayleigh fading, and Rician fading;

[0082] S7: Calculate the channel matrix with the help of complex domain transformation and channel estimation method to achieve signal recovery;

[0083] S8: The receiver model decodes and decrypts the signal propagated through the channel in preparation for the final classification or prediction task;

[0084] S9: Calculate cross-modal similarity and predict event categories for localization.

[0085] To enable cross-modal collaboration, we designed an attention mechanism that allows audio features to guide the model's attention to video content. This involves mapping audio features to the timeline of the video sequence and calculating the correlation or similarity between audio and video features. Based on these calculated similarities, the system dynamically assigns attention weights to video frames, emphasizing those visual parts that are synchronized with the audio and have high information content. In this way, the model can focus on important visual content more effectively and improve overall understanding capabilities.

[0086] As a preferred embodiment of the present invention, the visual attention mechanism in S3 includes:

[0087] S3.1 Set V in step S2 t and a t Use the VGG-19 network to extract the initial visual features V and initial audio features A as the initial vectors, and the features over the entire time series are represented as audio features. and visual features

[0088] S3.2 transmits the audio from the audio user to the video user through a physical channel and encodes it using an audio semantic encoder:

[0089]

[0090] Among them A single is the output result, i.e., a single audio clip obtained after enhancement processing; is the audio feature, Is a function that processes audio features and enhances the signal, Θ a,1 represents the trainable parameters, which consists of a one-dimensional convolution with C channels and a kernel size of 7, and there are B convolutional blocks;

[0091]

[0092] Each block consists of a residual unit and a downsampling layer with strided convolutions, where the kernel size K is twice the stride S. The residual unit has two convolutions with kernel size 3 and a skip connection;

[0093] S3.3 integrates all the processed audio clips to obtain The integration process is as follows;

[0094]

[0095] Among them, DownSample(K,S) is a function, which means downsampling. K is the kernel size, which refers to the size of the convolution kernel used in the downsampling process; S is the step size, which refers to the step size of moving the convolution kernel in the downsampling process. Block i It is the i-th block after a series of operations. ResUnit is a residual unit containing several convolutional layers and a skip connection to retain the original input information.

[0096] S3.4 Audio features received by the user holding the video The recovered features are accurately recovered in the unimodal task-oriented decoding stage; the recovered features are used in the cross-modal semantic encoder to enhance the visual features. The cross-modal visual semantic encoder is defined as:

[0097]

[0098] where Θ is v The trainable parameters of the encoder, is the input visual feature, It is a function that processes visual features and enhances the signal;

[0099] S3.5 Calculate the attention weight α t :

[0100]

[0101] Where σ is the sigmoid function, M v 、M a The projected features obtained by projecting the visual and audio features into a shared space, W f is a trainable parameter;

[0102] S3.6 Using attention weight α t Audio Features and visual features Weighted visual features

[0103] As a preferred embodiment of the present invention, the trusted channel coding in S4 includes:

[0104] S4.1 Use public key cryptography to establish a shared session key between the sender and receiver;

[0105] S4.2 uses a shared session key to symmetrically encrypt the transmitted information;

[0106] The S4.3 semantic encoder holds the key pair (pk, sk) of the elliptic curve cryptography (ECC) algorithm, and the calculation formula is as follows:

[0107]

[0108] A 1 Indicates that the input data is the weighted V obtained in S3 at and received audio characteristics CE a Used to extract attention-related features, Φ a This is a vector representing CE a The learnable parameters in ;

[0109] S4.4 sets the AES encryption key K from usekey, encrypts the key K using the ECC public key, and obtains the ciphertext C 1 , the audio message and the video message are processed according to the data block model respectively;

[0110] S4.5 uses the key K and CBC (cipher block chaining) mode to perform AES encryption on the processed message M to obtain the ciphertext C 2 , calculate the hash value h of message m according to the formula h = SHA-3 (m), and get the final ciphertext c = (c 1 ,c 2 ,h);

[0111] S4.6 encodes the obtained ciphertext c using Reed-Solomon coding, specifies the number of roots (nroots) of the generating polynomial to be 1280, and uses the extended field GF(2^18).

[0112] In order to make full use of the rich information in video data, we innovatively reshape and extract video features through 2D convolutional layers, treating each time point in the sequence as a "height" in image processing to enhance the ability to capture spatiotemporal features. Afterwards, the video features are converted into complex form, and the characteristics of the complex domain are used to enhance the model's expressiveness and noise resistance. Complex L2 norm normalization is used to ensure the normalization of all feature vectors and maintain the consistency and effectiveness of the operation.

[0113] As a preferred embodiment of the present invention, the real number to complex number conversion module in S5 includes:

[0114] S5.1 Convolutional Neural Network (CNN) layer learns diverse local features from multimodal semantics, where M represents a multimodal set, including weighted visual features V at and initial audio features A, and are respectively the temporal encoding of enhanced visual and audio features using LSTM at time t, They are the hidden state and unit state of the visual LSTM at the previous time t-1; similarly, the auditory LSTM They are the hidden state and unit state of the auditory LSTM at the previous time t-1;

[0115] S5.2 reshapes the encoded and learned data into a three-dimensional tensor and converts it into a complex form, i.e., f complex =RealToComplex(f cnn ).

[0116] Considering the challenges in the actual communication environment, we integrated a channel simulation and signal recovery mechanism, namely chan_layer. It can not only simulate various channel environments such as additive white Gaussian noise (AWGN), Rayleigh fading, and Rician fading, but also recover the damaged signal under harsh channel conditions based on ideal channel estimation theory, such as matrix solution algorithm. By controlling the signal-to-noise ratio (SNR) and channel type, the model can be trained in simulated complex channels and learn adaptive strategies for different conditions, which significantly improves the generalization performance of the model in unknown environments.

[0117] As a preferred embodiment of the present invention, S6 solves the problem of semantic information distortion in the communication ecosystem and includes:

[0118] S6.1 performs channel simulation, AWGN (Additive White Gaussian Noise) channel: generate Gaussian noise for the real and imaginary parts of the input signal respectively, and add the Gaussian noise to the input signal; set the channel matrix I to the unit matrix, indicating a distortion-free channel; Rayleigh fading channel: generate a complex channel matrix H (representing the fading coefficients) and apply it to the input signal to form the output signal Y; calculate the conjugate transpose H of the channel matrix H H and the outer product H H H, Rician fading channel: The channel matrix H is constructed based on the line-of-sight (LOS) and non-line-of-sight (NLOS) components and applied to the signal to produce the output Y;

[0119] S6.2 Rayleigh fading channel simulation: Generate a complex channel matrix H (representing fading coefficients) and apply it to the input signal to form the output signal Y; Calculate the conjugate transpose H of the channel matrix H H and the outer product H H H;

[0120] S6.3 Rician fading channel simulation: The channel matrix H is constructed based on the line-of-sight (LOS) and non-line-of-sight (NLOS) components and applied to the signal to produce the output Y.

[0121] As a preferred embodiment of the present invention, S7 includes:

[0122] S7.1 defines the received signal vector as Y, the transmitted signal vector as X, the channel matrix as H, and the additive white Gaussian noise vector as N; the received signal can be expressed as: Y = HX + N;

[0123] S7.2 Signal detection and signal recovery: A zero-forced minimum mean square error (ZF-LMMSE) detector is used to eliminate multipath interference and reduce noise amplification; the purpose of the ZF-LMMSE detector is to find a detection matrix W so that the output is close to the actual transmitted signal. The detection of the ZF-LMMSE detector can be shown as:

[0124]

[0125] Among them, H H is the conjugate transpose of the channel matrix H, is the variance of the noise, is the variance of the transmitted signal; I is the identity matrix.

[0126] As a preferred embodiment of the present invention, S8 includes the following contents:

[0127] S8.1 After signal recovery and signal detection, the estimated complex signal is RS decoded by a trusted decoder and the data is converted into audio and visual feature information;

[0128] S8.2 The semantic decoder recovers the AES encrypted message c,

[0129] c 1 =First 160 bits of c

[0130] The first 160 bits are taken as c 1 , extract the last 128 bits of the ciphertext c as h, and extract the remaining bits in the middle of the ciphertext c as c 2 ; Use the ECC private key to decrypt it and get the decrypted message m;

[0131] S8.3 uses the combined audio and visual semantic information and adopts the positive sample propagation network By filtering samples and removing negative samples, semantic decoding is completed and the decoded audio and visual features are obtained respectively as A e and V e .

[0132] As a preferred embodiment of the present invention, S9 includes:

[0133] The audio features obtained after S9.1 decoding are processed through two fully connected layers (number of units = 256, activation function = 'ReLU') to obtain A e ; The decoded visual features are processed by two Conv2D blocks (number of units = 512, activation function = 'ReLU') to obtain V e ; Calculate cosine similarity at the same time

[0134] S9.2 uses a shared fully connected layer to perform feature fusion and complete event prediction

[0135] F = ReLU(W · [A e ; V e ]+b)

[0136] Among them, W is the weight matrix, b is the bias vector, and F is the predicted result;

[0137] S9.3 Locate the location where the event occurs and determine the location of the audio event in the video frame. The positioning formula is:

[0138]

[0139] Where L represents the location result of the event, P is the set of all possible locations, and the location with the highest cosine similarity score is taken as the location where the event occurred.

Claims

1. A method for constructing multimodal trusted semantic communication for audio-visual event localization, characterized in that: The following steps are involved: S1: The audio-visual sensor obtains audio-visual perception data of the same time interval, one user holds video data and one user holds audio data; S2: Split the video and audio time series S into T non-overlapping but continuous segments; denoted as video segment sequence and audio clip sequences The duration of each segment is one second, where t is the time segment, ranging from 1 to T, T is a constant, V t represents the video data in time segment t, a t Represents the audio data in time segment t; S3: Implementing cross-modal audio-guided visual attention mechanism for semantic communication between two users, achieving audio-guided visual attention; S4: performing trusted channel coding on the video segments and the audio segments of the corresponding time intervals in the continuous synchronized audiovisual segments held by the two users respectively; S5: Real to complex conversion module, which converts the feature tensor into a complex representation to cope with the complexity of data transmission in the analog channel; S6: Using chan_layer to solve the semantic information distortion problem in the communication ecosystem, the semantic information distortion problem includes additive white Gaussian noise, Rayleigh fading and Rician fading; S7: Calculate the channel matrix with the help of complex domain transformation and channel estimation method to achieve signal recovery; S8: The receiver model decodes and decrypts the signal propagated through the channel in preparation for the final classification or prediction task; S9: Calculate cross-modal similarity and predict event categories for localization.

2. A method for constructing multimodal trusted semantic communication for audiovisual event localization as claimed in claim 1, characterized in that: The visual attention mechanism in S3 includes: S3.1 Set V in step S2 t and a t Use the VGG-19 network to extract the initial visual features V and initial audio features A as the initial vectors, and the features over the entire time series are represented as audio features S. a and visual features S3.2 transmits the audio from the audio user to the video user through a physical channel and encodes it using an audio semantic encoder: Among them A single is the output result, i.e., a single audio clip obtained after enhancement processing; is the audio feature, Is a function that processes audio features and enhances the signal, Θ a,1 represents the trainable parameters, which consists of a one-dimensional convolution with C channels and a kernel size of 7, and there are B convolutional blocks; Each block consists of a residual unit and a downsampling layer with strided convolutions, where the kernel size K is twice the stride S. The residual unit has two convolutions with kernel size 3 and a skip connection; S3.3 integrates all the processed audio clips to obtain The integration process is as follows; Block i =ResUnit°DownSample(K,S) Among them, DownSample(K,S) is a function, which means downsampling. K is the kernel size, which refers to the size of the convolution kernel used in the downsampling process; S is the step size, which refers to the step size of moving the convolution kernel in the downsampling process. Block i It is the i-th block after a series of operations. ResUnit is a residual unit containing several convolutional layers and a skip connection to retain the original input information. S3.4 Audio features received by the user holding the video The recovered features are accurately recovered in the unimodal task-oriented decoding stage; the recovered features are used in the cross-modal semantic encoder to enhance the visual features. The cross-modal visual semantic encoder is defined as: where Θ is v The trainable parameters of the encoder, is the input visual feature, It is a function that processes visual features and enhances the signal; S3.5 Calculate the attention weight α t : Where σ is the sigmoid function, M v 、M a The projected features obtained by projecting the visual and audio features into a shared space, W f is a trainable parameter; S3.6 Using attention weight α t Audio Features and visual features Weighted visual features 3. The method for constructing multimodal trusted semantic communication for audiovisual event localization according to claim 1, characterized in that: The trusted channel coding in S4 includes: S4.1 Use public key cryptography to establish a shared session key between the sender and receiver; S4.2 uses a shared session key to symmetrically encrypt the transmitted information; The S4.3 semantic encoder holds the key pair (pk, sk) of the elliptic curve cryptography algorithm, and the calculation formula is as follows: X a =EC a (A1;Φ a ) A1 indicates that the input data is the weighted V obtained in S3 at and received audio characteristics CE a Used to extract attention-related features, Φ a This is a vector representing CE a The learnable parameters in ; S4.4 sets the AES encryption key K from usekey, encrypts the key K using the ECC public key, obtains the ciphertext C1, and processes the audio message and the video message according to the data block model respectively; S4.5 uses the key K and CBC mode to perform AES encryption on the processed message M to obtain the ciphertext C2, and calculates the hash value h of the message m according to the formula h=SHA-3(m) to obtain the final ciphertext c=(c1,c2,h); S4.6 encodes the obtained ciphertext c using Reed-Solomon coding.

4. The method for constructing multimodal trusted semantic communication for audiovisual event localization according to claim 1, characterized in that: The real to complex conversion module in S5 includes: S5.1 Convolutional neural network layer learns diverse local features from multimodal semantics, where M represents a multimodal set, including weighted visual features V at and initial audio features A, and are respectively the temporal encoding of enhanced visual and audio features using LSTM at time t, They are the hidden state and unit state of the visual LSTM at the previous time t-1; similarly, the auditory LSTM They are the hidden state and unit state of the auditory LSTM at the previous time t-1; S5.2 reshapes the encoded and learned data into a three-dimensional tensor and converts it into a complex form, i.e., f complex =RealToComplex(f cnn ).

5. The method for constructing multimodal trusted semantic communication for audiovisual event localization according to claim 1, characterized in that: The S6 solves the problem of semantic information distortion in the communication ecosystem by: S6.1 performs channel simulation. AWGN channel: generate Gaussian noise for the real and imaginary parts of the input signal respectively, and add the Gaussian noise to the input signal; set the channel matrix I to the unit matrix, indicating a distortion-free channel; Rayleigh fading channel: generate a complex channel matrix H and apply it to the input signal to form the output signal Y; calculate the conjugate transpose H of the channel matrix H H and the outer product H H H, Rician fading channel: The channel matrix H is constructed based on the line-of-sight and non-line-of-sight components and applied to the signal to produce the output Y; S6.2 Rayleigh fading channel simulation: Generate a complex channel matrix H and apply it to the input signal to form the output signal Y; Calculate the conjugate transpose H of the channel matrix H H and the outer product H H H; S6.3 Rician fading channel simulation: Construct the channel matrix H based on the line-of-sight and non-line-of-sight components and apply it to the signal to produce the output Y.

6. The method for constructing multimodal trusted semantic communication for audiovisual event localization according to claim 1, characterized in that: The S7 includes: S7.1 defines the received signal vector as Y, the transmitted signal vector as X, the channel matrix as H, and the additive white Gaussian noise vector as N; the received signal can be expressed as: Y = HX + N; S7.2 Signal detection and signal recovery: A zero-forcing minimum mean square error detector is used to eliminate multipath interference and reduce noise amplification; the purpose of the ZF-LMMSE detector is to find a detection matrix W so that the output is close to the actual transmitted signal The detection of the ZF-LMMSE detector can be shown as: Among them, H H is the conjugate transpose of the channel matrix H, is the variance of the noise, is the variance of the transmitted signal; I is the identity matrix.

7. The method for constructing multimodal trusted semantic communication for audiovisual event localization according to claim 1, characterized in that: The S8 includes the following contents: S8.1 After signal recovery and signal detection, the estimated complex signal is RS decoded by a trusted decoder and the data is converted into audio and visual feature information; S8.2 The semantic decoder recovers the AES encrypted message c, c1=First 160 bits of c Take the first 160 bits as c1, extract the last 128 bits of the ciphertext c as h, and extract the remaining bits in the middle of the ciphertext c as c2; decrypt it using the ECC private key to obtain the decrypted message m; S8.3 uses the combined audio and visual semantic information and adopts the positive sample propagation network By filtering samples and removing negative samples, semantic decoding is completed and the decoded audio and visual features are obtained respectively as A e and V e .

8. The method for constructing multimodal trusted semantic communication for audiovisual event localization as claimed in claim 1, characterized in that: The S9 includes: The audio features obtained after S9.1 decoding are processed through two fully connected layers to obtain A e ; The decoded visual features are processed by two Conv2D blocks to obtain V e ; Calculate cosine similarity at the same time S9.2 uses a shared fully connected layer to perform feature fusion and complete event prediction F=ReLU(W·[A e ;V e ]+b) Among them, W is the weight matrix, b is the bias vector, and F is the predicted result; S9.3 Locate the location where the event occurs and determine the location of the audio event in the video frame. The positioning formula is: Where L represents the location result of the event, P is the set of all possible locations, and the location with the highest cosine similarity score is taken as the location where the event occurred.

Citation Information

Patent Citations

  • Audio-visual event positioning method and device based on cross-modal attention mechanism

    CN112989977A

  • Audio-visual video analysis device and method based on multi-scale semantic network

    CN114519809A

  • Audio-visual event detection method and device based on event consistency

    CN115861879A

  • Audio visual sound source separation with cross-modal META consistency learning

    US20240161761A1

Cited By

  • Semantic communication method, system and terminal based on improved comparative learning

    CN119476303A

  • A semantic communication method, system and terminal based on improved contrastive learning

    CN119476303B