Multi-modal secure data transmission method and system based on differentiated code rate allocation

Abstract

The application belongs to the field of data security, and particularly relates to a multi-modal secure data transmission method and system based on differentiated code rate allocation. The method decouples the synchronously collected multi-modal data stream into a semantic primitive sequence, evaluates the semantic influence factors of each mode through a proxy model mask aligned with the backend secure recognition large model task, and extracts a task attention coefficient from the attention weight of the large model to generate a transmission importance coefficient. When the network bandwidth is limited and the time delay is high, the first deep compression strategy is performed on the lowest importance mode according to the transmission importance coefficient, the semantic redundant area is identified and compressed through cross entropy comparison for the high importance mode, and finally, the compressed data packet is added with a semantic label and a cross-modal identifier for priority queue scheduling and retransmission control, so as to maximize the secure recognition semantic throughput.

Description

Technical Field

[0001] This invention belongs to the field of data security, and in particular relates to a multimodal secure data transmission method and system based on differentiated bitrate allocation. Background Technology

[0002] With the deep integration of mobile internet and IoT technologies, the generation and dissemination of online public opinion are characterized by multimodality, fragmentation, and real-time nature. In fields such as public safety and emergency management, front-end data collection devices (such as drones and smart cameras) need to upload multimodal data streams, including text, high-definition video, and environmental audio, to AI-powered big data models on the cloud or edge for in-depth analysis in real time. However, front-end data collection environments typically face complex and variable network channel conditions, such as highly dynamic bandwidth, unstable round-trip latency, and random packet loss, posing significant challenges to the real-time and reliable transmission of multimodal data.

[0003] In existing technologies, the transmission of multimodal data mainly relies on traditional audio and video coding and streaming media transmission protocols, such as fixed bitrate coding and Dynamic Adaptive Streaming (DASH) based on H.264 / H.265. The core design goal of such methods is to minimize signal distortion or maximize user experience quality. Their compression and scheduling strategies are driven by the statistical characteristics of the data itself or general perceptual quality indicators, completely unaware of the specific semantic requirements of the backend AI tasks. This inevitably leads to a disconnect at the "transmission-computation" level: the network transmission layer may distribute bandwidth evenly to all data streams to ensure smoothness, resulting in the loss of visual details that are crucial for sensitivity judgment due to compression, or the discarding of audio segments containing key dialogues to ensure video quality. Although some studies have attempted to introduce differentiated transmission based on saliency detection, it still remains at the level of general visual attention and cannot fundamentally measure and adapt to the precise requirements of complex backend sensitivity judgment models for cross-modal semantic integrity and specific entity features (such as faces, license plates, and specific sound events). Therefore, under limited network conditions, how to dynamically and adaptively organize and transmit multimodal data with the final semantic accuracy of the backend sensitivity assessment task as the optimization goal has become a technical problem that urgently needs to be solved in this field. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention proposes a multimodal secure data transmission method and system based on differentiated bitrate allocation. This method decouples synchronously acquired multimodal data streams into semantic primitive sequences. It evaluates the semantic influence factors of each modality using a proxy model mask aligned with the backend security identification large-scale model task, and generates a transmission importance coefficient by weighting the task attention coefficient extracted from the large-scale model's attention weights. When network bandwidth is limited and latency is high, a first-depth compression strategy is applied to the least important modal based on the transmission importance coefficient. For highly important modalities, semantic redundancy regions are identified and compressed through cross-entropy comparison. Finally, semantic tags and cross-modal identifiers are added to the compressed data packets, and priority queue scheduling and retransmission control are implemented to maximize the semantic throughput of secure identification.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] Multimodal secure data transmission methods based on differentiated bitrate allocation include:

[0007] Acquire synchronously collected multimodal data streams and decouple each modal data stream into a corresponding semantic primitive sequence; the multimodal data streams include a first modal data stream, a second modal data stream, and a third modal data stream;

[0008] Using a proxy model aligned with the backend security identification large model task, a mask is applied to the semantic primitive sequence of each modality. Based on the distribution deviation of the output features of the proxy model before and after masking, a semantic influence factor for each modality is generated.

[0009] From the multi-head cross-attention weights of the backend security identification big model, the task attention coefficients of each modality are obtained, and the semantic influence factor is multiplied by the corresponding task attention coefficients to obtain the transmission importance coefficients of each modality.

[0010] The network channel state is detected, and adaptive compression is performed based on the network channel state and the transmission importance coefficient to obtain compressed data packets; the network channel state includes available bandwidth, round-trip time, and packet loss rate.

[0011] Semantic importance tags and cross-modal association identifiers are added to the compressed data packets. Priority queue scheduling and retransmission control are performed based on the semantic importance tags and the current network window state to maximize the throughput of security identification semantics.

[0012] Specifically, adaptive compression is performed based on the network channel state and the transmission importance coefficient, including:

[0013] When the available bandwidth is lower than a preset bandwidth threshold and the round-trip delay is higher than a preset delay threshold, a first deep compression strategy is executed on the modal data stream with the lowest transmission importance coefficient.

[0014] For the two remaining modal data streams other than the modal data stream with the lowest transmission importance coefficient, the modal semantic primitive sequence with the highest transmission importance coefficient is determined as the first sequence, and the other remaining modal semantic primitive sequence is determined as the second sequence. Semantic redundancy regions are identified by comparing the difference between self-information entropy and conditional entropy, and a second deep compression strategy is applied to the semantic redundancy regions.

[0015] Specifically, the data streams of each modality are decoupled into corresponding sequences of semantic primitives, including:

[0016] The first mode data stream, the second mode data stream, and the third mode data stream are time-series segmented to obtain the first initial segment sequence, the second initial segment sequence, and the third initial segment sequence;

[0017] Each segment in the first initial segment sequence, and segments whose time windows overlap with those in the second and third initial segment sequences, are respectively denoted as cross-modal segment pairs; each cross-modal segment pair is input into a pre-trained cross-modal temporal alignment model, and the semantic alignment confidence and optimal time offset between the first and second modal segments and between the first and third modal segments in the cross-modal segment pair are calculated to generate a cross-modal alignment relationship matrix;

[0018] Based on the cross-modal alignment relationship matrix, cross-modal segments whose semantic alignment confidence exceeds a preset confidence threshold and whose optimal time offset is within a preset offset tolerance range are identified as aligned related segments.

[0019] Specifically, decoupling each modal data stream into a corresponding sequence of semantic primitives also includes:

[0020] All cross-modal segments within the same aligned and associated segment group are unified on the time axis to the smallest time interval covering all segments within the group, resulting in an aligned cross-modal segment group.

[0021] For each modality segment within each aligned cross-modal segment group, the variance of the semantic vector within a single modal segment is calculated. When the variance exceeds a preset variance threshold, the corresponding single modal segment is further segmented until the variance of the semantic vector within each segmented sub-segment is lower than the preset variance threshold. When the semantic vector similarity between two adjacent sub-segments of the same modality exceeds a preset merging threshold, the two adjacent sub-segments of the same modality are merged into a single semantic primitive.

[0022] The segments formed by further segmenting and merging each modality within the same aligned cross-modal segment group are respectively used as the first modal semantic primitive, the second modal semantic primitive, and the third modal semantic primitive, and arranged in timestamp order to form the first modal semantic primitive sequence, the second modal semantic primitive sequence, and the third modal semantic primitive sequence.

[0023] Specifically, semantic influence factors for each modality are generated, including:

[0024] The first modality semantic primitive sequence, the second modality semantic primitive sequence, and the third modality semantic primitive sequence are simultaneously input into the surrogate model, and the joint feature distribution output by the surrogate model is obtained as the baseline feature distribution.

[0025] A masking operation is performed on the first modal semantic primitive sequence. The masked first modal semantic primitive sequence, along with the unmasked second and third modal semantic primitive sequences, are input into the surrogate model to obtain the first mask feature distribution. The first KL divergence value between the first mask feature distribution and the baseline feature distribution is calculated. The first KL divergence value is mapped to a first semantic influence factor through a preset exponential function.

[0026] Similarly, the second and third semantic influence factors are obtained in the same way as the first semantic influence factor.

[0027] Specifically, the task attention coefficients for each modality are obtained, including:

[0028] During the inference process of the security identification task, obtain the cross-attention weight matrix generated by the multi-head cross-attention sub-layer in each Transformer decoder layer;

[0029] For each attention head in each Transformer decoder layer, the intermodal cross-attention submatrix and the intramodal self-attention submatrix are separated from the cross-attention weight matrix of the attention head;

[0030] For each attention head, calculate the intermodal cross-attention contribution values ​​of the first modality, the second modality, and the third modality in that attention head.

[0031] Specifically, obtaining the task attention coefficient for each modality also includes:

[0032] For each attention head, calculate the intramodal self-attention contribution values ​​of the first mode, the second mode, and the third mode in the attention head;

[0033] The average cross-attention contribution value and average self-attention contribution value of the first modality in all attention heads of all layers are weighted and summed according to the preset cross-attention weight coefficient and self-attention weight coefficient to obtain the original task attention value of the first modality.

[0034] The original task attention values ​​of the second and third modalities are obtained in the same way as the original task attention values ​​of the first modal.

[0035] The normalized results of the original task attention values ​​of the first modality, the second modality, and the third modality are used as the task attention coefficients of the first modality, the second modality, and the third modality, respectively.

[0036] Specifically, semantically redundant regions are identified by comparing the difference between self-information entropy and conditional entropy, including:

[0037] The modal semantic primitive sequence with the largest transmission importance coefficient is determined as the first sequence, and the modal semantic primitive sequence with the second largest transmission importance coefficient is determined as the second sequence.

[0038] Traverse each semantic primitive in the second sequence and calculate the independent information content of the semantic primitive. The independent information content is the self-information entropy of the semantic primitive under the condition that the information is provided only by the second sequence itself. Semantic primitives with independent information content greater than a preset independent information content threshold are determined as non-redundant semantic primitives. A set of non-redundant semantic primitives is formed by all non-redundant semantic primitives in the second sequence.

[0039] Specifically, identifying semantically redundant regions by comparing the difference between self-information entropy and conditional entropy also includes:

[0040] Traverse each semantic primitive in the first sequence, and calculate the conditional information content of the semantic primitive using the set of non-redundant semantic primitives as a condition. The conditional information content is the conditional entropy of the semantic primitive under the given set of non-redundant semantic primitives. Calculate the difference between the independent information content and the conditional information content of the semantic primitive. Semantic primitives whose difference is less than a preset redundancy difference threshold are identified as redundant semantic primitives, and a semantic redundancy region is formed by all redundant semantic primitives in the first sequence.

[0041] A multimodal secure data transmission system based on differentiated bitrate allocation includes:

[0042] The decoupling module is configured to acquire synchronously collected multimodal data streams and decouple each modal data stream into a corresponding semantic primitive sequence; the multimodal data streams include a first modal data stream, a second modal data stream, and a third modal data stream;

[0043] The evaluation module is configured to use a proxy model aligned with the backend security identification large model task to perform masking operations on the semantic primitive sequences of each modality, and generate semantic influence factors for each modality based on the distribution deviation of the output features of the proxy model before and after masking.

[0044] The attention weighting module is configured to extract the task attention coefficients corresponding to each modality from the multi-head cross-attention weights of the backend security identification big model, and multiply the semantic influence factor with the corresponding task attention coefficients to obtain the transmission importance coefficients of each modality.

[0045] The compression decision module is configured to detect the network channel state, perform adaptive compression based on the network channel state and the transmission importance coefficient, and obtain compressed data packets; the network channel state includes available bandwidth, round-trip time, and packet loss rate.

[0046] The transmission control module is configured to add semantic importance tags and cross-modal association identifiers to the compressed data packets, and perform priority queue scheduling and retransmission control based on the semantic importance tags and the current network window state to maximize the throughput of security identification semantics.

[0047] Compared with the prior art, the beneficial effects of the present invention are:

[0048] This invention fundamentally solves the problem of the disconnect between traditional transmission mechanisms and AI semantic analysis tasks by constructing a transmission architecture deeply coupled with the backend security identification task. Specifically, firstly, by decoupling multimodal data streams into semantic primitive sequences and using a proxy model aligned with the backend large model for mask evaluation, the independent semantic contribution of each modality of data to the security identification task can be accurately quantified. Secondly, by extracting the task attention coefficient from the multi-head cross-attention weights of the large model and weighting it with semantic influence factors to generate a transmission importance coefficient, a paradigm shift in transmission priority is achieved from "channel adaptation" to "task semantic awareness." When the network is constrained, deep compression is performed on low-importance modalities based on this coefficient, while high-importance modalities are identified and compressed through cross-comparison of self-information entropy and conditional entropy. This maximizes bandwidth utilization while ensuring the inference accuracy of the backend model. Finally, by adding semantic importance tags and cross-modal association identifiers to compressed data packets, and in conjunction with differentiated priority scheduling and retransmission control strategies, reliable delivery of high-value semantic primitives is prioritized under unreliable network conditions. Ultimately, this maximizes the throughput of security identification semantics and significantly improves the analysis accuracy and real-time response of complex security services in weak network environments. Attached Figure Description

[0049] Figure 1This is a flowchart of the multimodal secure data transmission method based on differentiated bitrate allocation in Embodiment 1 of the present invention;

[0050] Figure 2 This is a flowchart illustrating how the data streams of each modality are decoupled into corresponding semantic primitive sequences in Embodiment 2 of the present invention.

[0051] Figure 3 This is a structural diagram of a multimodal secure data transmission system based on differentiated bitrate allocation, as described in Embodiment 4 of the present invention. Detailed Implementation

[0052] Example 1

[0053] Please see Figure 1 The present invention provides an embodiment of a multimodal secure data transmission method based on differentiated code rate allocation, comprising:

[0054] S1. Acquire synchronously collected multimodal data streams and decouple each modal data stream into a corresponding semantic primitive sequence; the multimodal data streams include a first modal data stream, a second modal data stream, and a third modal data stream;

[0055] S2. Using a proxy model aligned with the backend security identification large model task, a mask is applied to the semantic primitive sequence of each modality. Based on the distribution deviation of the output features of the proxy model before and after masking, a semantic influence factor for each modality is generated.

[0056] S3. Obtain the task attention coefficient of each modality from the multi-head cross-attention weight of the back-end security identification big model, and multiply the semantic influence factor with the corresponding task attention coefficient to obtain the transmission importance coefficient of each modality.

[0057] S4. Detect the network channel state, and perform adaptive compression based on the network channel state and the transmission importance coefficient to obtain the compressed data packet, specifically as follows:

[0058] When the available bandwidth is lower than a preset bandwidth threshold and the round-trip delay is higher than a preset delay threshold, a first deep compression strategy is executed on the modal data stream with the lowest transmission importance coefficient. In this embodiment, the specific implementation logic of the first deep compression strategy is as follows: all vector values ​​of each semantic primitive in the semantic primitive sequence corresponding to the modal data stream are discarded, and only the modal identifier field, timestamp range field, and a structured summary text field generated by the proxy model are retained. The structured summary text field describes the basic event types and key entity information contained in the modal data stream, and does not contain any detailed semantic content. When the receiving end receives the data packet processed by the first deep compression strategy, it identifies the modal category to which the data packet belongs based on the modal identifier field, determines the time span of the modal data based on the timestamp range field, and obtains the basic information of the modal data stream based on the structured summary text field. In this semantic summary, when the backend security identification big data model performs multimodal joint inference, the structured summary text is used as the context information input for that modality, replacing the complete semantic primitive sequence in cross-modal semantic association calculation. In this embodiment, the design basis of the first deep compression strategy is that the data stream of the modality with the lowest transmission importance coefficient has almost no independent semantic contribution to the security determination task of the backend security identification big data model. Under extremely limited network conditions, transmitting only the structured summary instead of the complete semantic primitive sequence can compress the transmission bandwidth occupied by the modality data stream to an extremely low level while still preserving the basic context information of that modality for the backend security identification big data model, avoiding the breakage of cross-modal semantic association due to the complete absence of the modality. Simultaneously, the released bandwidth resources are dynamically allocated to the transmission of modality data streams with higher transmission importance coefficients to ensure the reliable transmission of key semantic information for security identification, thereby achieving the overall goal of maximizing the semantic throughput of security identification. In this embodiment, the preset bandwidth threshold and preset latency threshold serve as the joint judgment threshold for triggering the differentiated compression strategy, and their setting is closely coupled with the rigid requirements of the backend security identification big data model for the semantic integrity and timeliness of the input data. The bandwidth threshold is set based on the statistical analysis of the minimum transmission bitrate required to ensure the basic semantic fidelity of key modalities on the sample security event dataset. It is the sum of the basic bitrates required for each modality when the backend model recognition accuracy is not lower than the preset service availability benchmark (e.g., 95%), with a certain engineering margin added. In this embodiment, it is set to 2Mbps. The round-trip latency threshold is determined according to the end-to-end timeliness requirements from front-end acquisition to back-end decision-making. It is rigidly constrained by the maximum tolerable response latency required for the security identification task (e.g., for sudden public safety events, the requirement is that the time from acquisition to output of the sensitivity judgment result should not exceed 2 seconds). After deducting the average inference time of the large model and the data processing overhead, the maximum tolerable latency limit of network transmission is derived in reverse. In this embodiment, it is set to 150 milliseconds.The logic behind this dual-threshold joint determination is that only when the bandwidth conditions deteriorate to the point that they cannot support the transmission of a complete semantic primitive sequence, and the latency is so high that it will cause end-to-end response timeouts and render the security identification results ineffective, will the extreme compression and scheduling mechanism, which sacrifices low-importance modal details, be activated to ensure the rapid delivery of high-value semantic information. This maximizes the throughput of security identification semantics in extremely limited network scenarios.

[0059] For the two remaining modal data streams other than the modal data stream with the lowest transmission importance coefficient, the semantic primitive sequence of the modal with the highest transmission importance coefficient is determined as the first sequence, and the other remaining modal semantic primitive sequence is determined as the second sequence. Semantic redundancy regions are identified by comparing the difference between self-information entropy and conditional entropy, and a second deep compression strategy is applied to the semantic redundancy regions. The network channel state includes available bandwidth, round-trip time, and packet loss rate.

[0060] S5. Add semantic importance tags and cross-modal association identifiers to the compressed data packets, and perform priority queue scheduling and retransmission control according to the semantic importance tags and the current network window state to maximize the throughput of security identification semantics.

[0061] Example 2

[0062] Please see Figure 2 It should be further explained that in this embodiment, each modal data stream is decoupled into a corresponding semantic primitive sequence, including:

[0063] S101. Perform time-series segmentation on the first mode data stream, the second mode data stream, and the third mode data stream to obtain the first initial segment sequence, the second initial segment sequence, and the third initial segment sequence;

[0064] It should be further explained that one specific implementation method for time-series segmentation in this embodiment is as follows:

[0065] A1011. Acquire synchronously acquired multimodal data streams, including a first modal data stream, a second modal data stream, and a third modal data stream. In this example, the first modal data stream is a video stream, the second modal data stream is an audio stream, and the third modal data stream is a text stream, all three being synchronized in acquisition time.

[0066] A1012. For the first modal data stream, perform adaptive temporal segmentation based on scene switching detection to generate a first initial segment sequence, specifically including:

[0067] A1012-1. Extract continuous image frames from the first modal data stream and define candidate segmentation points on the time axis with a preset basic segmentation interval. In this example, the basic segmentation interval is set to 2 seconds, that is, the video stream generates candidate segmentation boundaries in intervals of 2 seconds. In this embodiment, the preset basic segmentation interval is set as follows: obtain a sample video dataset for security identification scenarios, the sample video dataset contains video samples labeled with security events, calculate the average time interval between two adjacent scene transitions in the video samples, and set the basic segmentation interval based on the average time interval; at the same time, when setting the basic segmentation interval, the granularity requirements of video semantic primitives are also taken into account, so that the duration of the segmented video segments can fully contain an independent security event semantic unit, and avoid the semantic content of multiple different events mixed in the same video segment due to the segmentation granularity being too coarse; in this example, the average time interval between adjacent scene transitions in the sample video dataset is approximately 2 seconds, so the basic segmentation interval is preset to 2 seconds.

[0068] A1012-2. For each candidate segmentation point, obtain the two adjacent image frames before and after it, calculate the image embedding vectors of the two image frames respectively, and calculate the cosine similarity between the two image embedding vectors to obtain the scene switching score.

[0069] In this embodiment, the two adjacent image frames before and after the candidate segmentation point are respectively input into the pre-trained VisionTransformer image encoder. The processing procedure of the image encoder is as follows:

[0070] First, the input image frame is scaled to a preset size of 224×224 pixels, and then the scaled image frame is divided into a sequence of fixed-size image blocks, each block being 16×16 pixels. The segmentation method is to start from the top left corner of the image, proceed from left to right and top to bottom, and perform non-overlapping sliding segmentation with a step size of 16 pixels, resulting in a total of (224 / 16)×(224 / 16)=196 image blocks. For each of the 196 image blocks, all pixel values ​​of the image block are unfolded into a 768-dimensional vector in channel order. The 8-dimensional vector is multiplied by a 768×768 linear projection weight matrix to obtain a 768-dimensional block embedding vector for the image patch. A learnable position encoding vector of dimension 768, uniquely corresponding to the spatial location of the image patch in the original image, is then superimposed on each of the 196 block embedding vectors, with corresponding elements added together, generating 196 position-aware block embedding vectors. A learnable category label embedding vector of dimension 768 is then added to the front of these 196 position-aware block embedding vectors, resulting in an input sequence consisting of 197 embedding vectors. In this embodiment, the dimension of the 768×768 linear projection weight matrix is ​​determined by the requirement for a linear mapping from the original pixel space of the image patch to the model embedding dimension space. Specifically, the VisionTransformer image encoder scales the input image frame to 224×224 pixels and then performs non-overlapping segmentation into 16×16 pixel image blocks. Each image block contains 16×16×3=768 pixel values ​​(3 represents the RGB three channels). After expanding all the pixel values ​​of each image block in channel order, a 768-dimensional original pixel vector is obtained, which constitutes the input dimension of the matrix. To map the semantic information of each image block from the original pixel space to the embedding space required by the subsequent Transformer encoder layers of the model, a linear projection operation is introduced. The dimension of this embedding space is set to 768 in the model design, therefore the output dimension of the weight matrix is ​​also 768. This weight matrix serves as a trainable parameter for the model. During the pre-training process of the image encoder, it takes a dataset of sample images from a security recognition scenario as input. By minimizing the loss function of pre-training tasks such as image classification or feature reconstruction, the backpropagation algorithm iteratively updates each weight value in the matrix, enabling the matrix to learn the mapping relationship from pixel-level features to high-dimensional semantic features. Finally, it is saved as a fixed parameter matrix after training convergence, which is used for the extraction of image embedding vectors in this embodiment.

[0071] Next, the input sequence consisting of the 197 embedded vectors is input into a Transformer encoder consisting of 12 stacked Transformer encoder layers. Each Transformer encoder layer sequentially performs multi-head self-attention calculation, residual connection and layer normalization, feedforward neural network transformation, residual connection and layer normalization operations. Among them, the number of heads for multi-head self-attention calculation is 12, and the query vector dimension, key vector dimension and value vector dimension of each attention head are all 64. The hidden layer dimension of the feedforward neural network is 3072, and the activation function is the GELU activation function.

[0072] Third, from the 197 output vectors output from the 12th Transformer encoder layer, the output vector corresponding to the category label embedding vector is extracted. The output vector has a dimension of 768 and is used as the image embedding vector for the image frame.

[0073] Fourth, perform cosine similarity calculation on the image embedding vectors of the two image frames. The cosine similarity is calculated as follows: denote the image embedding vector of the first image frame as vector... The image embedding vector of the second image frame is denoted as vector. Calculate the dot product of vectors A and B. The dot product equals Calculate the L2 norm of vector A, which is equal to the square root of the sum of the squares of each of the 768 elements in vector A; calculate the L2 norm of vector B, which is equal to the square root of the sum of the squares of each of the 768 elements in vector B; divide the dot product by the product of the L2 norms of vector A and vector B, and the resulting cosine similarity value ranges from -1 to 1. The closer the value is to 1, the more similar the semantic features of the two image frames are. The cosine similarity value is used as a scene switching score to characterize the degree of scene switching between the two image frames.

[0074] A1012-3. When the scene switching score is lower than a preset scene switching threshold, it is determined that there is a scene change at the candidate segmentation point, and the candidate segmentation point is marked as the actual segmentation point; when the scene switching score is higher than or equal to the scene switching threshold, it is determined that the scene at the candidate segmentation point is continuous, and no segmentation is performed. In this example, the scene switching threshold is set to 0.75. If the cosine similarity between the frames before and after a candidate segmentation point is 0.6, which is lower than the threshold, then segmentation is performed at this point; if the similarity is 0.9, then no segmentation is performed. The threshold is set based on the following: In a security identification scenario, a sample video dataset containing typical security events such as traffic accidents and public safety incidents is collected. The actual time points of scene transitions in each video are manually labeled as ground truth values ​​for scene transitions. For all candidate segmentation points in the sample video dataset, the cosine similarity between the image embedding vectors of the two adjacent image frames is calculated. The cosine similarity of each candidate segmentation point is compared statistically with the manually labeled ground truth values ​​for scene transitions. A precision-recall curve for scene transition detection is plotted, and the threshold that achieves the highest F1 score for scene transition detection is selected. The cosine similarity value at which the scene changes reaches its maximum value is used as the preset scene switching threshold. Statistically, the optimal cosine similarity cutoff point is 0.75. When the cosine similarity is below 0.75, the semantic features between the two image frames are significantly different, indicating that a significant scene change has occurred at the candidate segmentation point, and segmentation should be performed. When the cosine similarity is higher than or equal to 0.75, the semantic features between the two image frames maintain a high degree of consistency, indicating that the scene is continuous at the candidate segmentation point, and segmentation should not be performed. This ensures a high recall rate for scene switching detection while keeping the false segmentation rate within an acceptable range.

[0075] A1012-4. Starting from the start timestamp of the first modal data stream, sequentially find the nearest actual segmentation point along the time axis. The video data segment between two adjacent actual segmentation points is considered as a video fragment, and a fragment identifier containing a fragment number, start timestamp, and end timestamp is assigned to each video fragment. This results in a first initial fragment sequence composed of multiple video fragments arranged in chronological order. As a specific example, in this example, a 35-second video of a traffic accident scene is used as the input to the first modal data stream. The start timestamp of this video is 0.0 seconds, and the end timestamp is 35.0 seconds. Adaptive temporal segmentation based on scene transition detection is performed as follows: First, candidate segmentation points are defined sequentially on the time axis from 0.0 seconds to 35.0 seconds at a preset basic segmentation interval of 2 seconds. The timestamps of the candidate segmentation points are 2.0 seconds, 4.0 seconds, 6.0 seconds, 8.0 seconds, 10.0 seconds, 12.0 seconds, 14.0 seconds, 16.0 seconds, 18.0 seconds, 20.0 seconds, 22.0 seconds, 24.0 seconds, 26.0 seconds, 28.0 seconds, 30.0 seconds, 32.0 seconds, and 34.0 seconds, respectively. Then, for each candidate segmentation point, the two adjacent image frames are obtained and input into the pre-trained VisionTransformer image encoder to extract 768-dimensional image embedding vectors and calculate the cosine similarity as the scene transition score. The scene transition score is compared with the preset scene transition threshold of 0.75. The calculated cosine similarity between the frames before and after the candidate segmentation point at 5.2 seconds is... The cosine similarity of the frames before and after the candidate segmentation point at 12.8 seconds is 0.68, the cosine similarity of the frames before and after the candidate segmentation point at 21.5 seconds is 0.71, and the cosine similarity of the frames before and after the candidate segmentation point at 28.0 seconds is 0.65. All of these are below the threshold of 0.75, indicating that there is a scene change at the above four candidate segmentation points, and they are marked as actual segmentation points. The cosine similarity of the frames before and after the other candidate segmentation points is greater than or equal to 0.75, indicating that the scene is continuous and no segmentation is performed. Finally, starting from the starting timestamp of 0.0 seconds, the segments are sequentially segmented along the time axis according to the actual segmentation points to obtain the first initial segment sequence, which includes the time interval of segment 1 from 0.0 seconds to 5.2 seconds, segment 2 from 5.2 seconds to 12.8 seconds, segment 3 from 12.8 seconds to 21.5 seconds, segment 4 from 21.5 seconds to 28.0 seconds, and segment 5 from 28.0 seconds to 35.0 seconds. In the subsequent cross-modal temporal alignment, the video is paired with segments whose time windows overlap in the second and third initial segment sequences to form multiple aligned cross-modal segment groups. After semantic vector variance determination and re-segmentation or merging, each segment is finally used as a first-modal semantic primitive to participate in the calculation of subsequent semantic influence factors and adaptive compression transmission.

[0076] A1013. For the second modal data stream, perform adaptive temporal segmentation based on speech activity detection to generate a second initial segment sequence, specifically including:

[0077] A1013-1. The second modal data stream is processed by framing, with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. The short-time energy and zero-crossing rate features of each frame are extracted. In this embodiment, the specific process of framing the second modal data stream is as follows: Standardized raw audio data after audio stream separation and resampling is obtained. The sampling rate of the standardized raw audio data is 16000Hz, meaning it contains 16000 sampling points per second. The amplitude of each sampling point ranges from -32768 to 32767, representing a 16-bit signed integer. Sliding framing is performed on the standardized raw audio data with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. A frame length of 25 milliseconds corresponds to 16000 × 0.025 = 400 sampling points per frame, and a frame shift of 10 milliseconds corresponds to an offset of 16000 × 0.01 = 160 sampling points between the starting sampling points of adjacent frames. Starting from the initial sampling point index 0 of the standardized raw audio data, the data is sequentially truncated with a frame shift of 160 sampling points, generating sampling point indices 0 to 399 for the first frame, 160 to 559 for the second frame, 320 to 719 for the third frame, and so on, generating a total of 3498 frames of audio data, with each frame containing 400 sampling points. For each frame of audio signal, short-time energy and zero-crossing rate features are calculated using time-domain feature extraction. The short-time energy is calculated by squaring the amplitudes of the 400 sampling points in the frame and summing them to obtain the short-time energy value of the frame. The zero-crossing rate is calculated by iterating through all 399 pairs of adjacent sampling points in the frame, determining whether the amplitude signs of each pair of adjacent sampling points are different. If the amplitude of the previous sampling point is positive and the amplitude of the next sampling point is negative, or the amplitude of the previous sampling point is negative and the amplitude of the next sampling point is positive, then it is determined that the pair of adjacent sampling points has crossed zero once. The total number of zero crossings in the frame is counted, and the total number of zero crossings is divided by the total number of adjacent sampling point pairs in the frame, 399, to obtain the zero-crossing rate value of the frame. The short-time energy value and zero-crossing rate value calculated for each frame are combined into a two-dimensional vector. The first dimension of the two-dimensional vector is the short-time energy value, and the second dimension is the zero-crossing rate value. The vectors are arranged in chronological order of the frames to form a frame-level acoustic feature sequence with a dimension of 3498×2.

[0078] The frame length of 25 milliseconds and the frame shift of 10 milliseconds in this embodiment are set as follows: A frame length of 25 milliseconds is a commonly used parameter for short-time analysis of audio signals in security recognition scenarios. This duration can fully cover the pronunciation cycle of a single Chinese character or two-character word. Within this time window, the audio signal can be approximated as a stationary signal, satisfying the stationarity assumption for short-time feature extraction. At the same time, it avoids the time resolution from decreasing due to excessively long frame lengths, which would prevent the capture of rapid start and end changes in speech activity. It also avoids the frequency resolution from being insufficient due to excessively short frame lengths, which would prevent the effective differentiation of acoustic features between speech and background noise. A frame shift of 10 milliseconds corresponds to an overlap of 15 milliseconds between adjacent frames. This overlap ratio ensures that the acoustic features extracted between adjacent frames have sufficient temporal continuity, enabling the speech activity detection model to accurately determine the precise timestamps of the start and end points of speech with a time precision of 10 milliseconds. At the same time, it controls the total number of frames in the frame-level acoustic feature sequence within a reasonable range, avoiding the increase in the computational burden of the subsequent speech activity detection model due to excessively long feature sequences caused by excessively small frame shifts.

[0079] A1013-2. The short-time energy and zero-crossing rate features are input into a pre-trained speech activity detection model, which outputs the speech presence probability for each frame. When the speech presence probability of multiple consecutive frames is higher than a preset speech presence threshold, the starting point of the consecutive frame interval is marked as the speech start point. When the speech presence probability of multiple consecutive frames is lower than a preset silence threshold, the ending point of the consecutive frame interval is marked as the speech end point. The audio segment between each speech start point and the next speech end point is considered as an audio segment. In this example, the speech presence threshold is set to 0.8, and the silence threshold is set to 0.3. The threshold is set based on the following: In security identification scenarios, diverse environmental audio data, including recordings of traffic accident scenes and public safety incidents, are collected. Each frame of audio signal is manually labeled with either a speech frame label or a non-speech frame label as the training ground value. The speech presence probability value output by the pre-trained speech activity detection model for each frame of sample audio data is compared with the manually labeled ground value to plot the receiver operating characteristic (ROC) curve for speech activity detection. The speech presence probability cutoff point that ensures a speech frame detection recall rate of no less than 95% and a non-speech frame false positive rate of no more than 5% is selected as the speech presence threshold. Statistically, this optimal cutoff point is 0.80. At the same time, the speech presence probability cutoff point that ensures a non-speech frame detection recall rate of no less than 90% and a speech frame false positive rate of no more than 10% is selected as the silence threshold. Statistically, this optimal cutoff point is 0.30. When the probability of speech presence in multiple consecutive frames is higher than 0.80, it indicates that the speech signal is stably present in the frame interval and is marked as the speech start point; when the probability of speech presence in multiple consecutive frames is lower than 0.30, it indicates that the speech signal has reliably terminated in the frame interval and is marked as the speech end point. This ensures a high recall rate for speech activity detection while keeping the false positive rate of speech start and end points within an acceptable range. In this embodiment, the specific number of consecutive frames in voice activity detection is not a fixed value, but a dynamic criterion jointly determined by the minimum voice duration preset by the system, the silence confirmation duration, and the frame shift parameters of audio framing. This setting is based on the requirement to protect the integrity of valid voice segments in security recognition scenarios: on the one hand, to avoid misjudging short bursts of noise such as collision sounds and electromagnetic pulses as valid voice, a minimum voice duration is set as the starting point for the confirmation window. Only when the probability of each frame of voice within this duration window is higher than a preset voice presence threshold is the voice activity officially considered to have started. On the other hand, to prevent the natural pauses or brief hesitations between sentences from being mistakenly judged as the end of voice activity, an independent silence confirmation duration is set. Only when the probability of each frame of voice within this duration window is consistently lower than a preset silence threshold is the voice activity reliably terminated.In this example, the frame shift for audio framing is set to 10 milliseconds, and the minimum speech duration is set to 60 milliseconds. This means that the probability of speech presence in 6 consecutive frames must be higher than a threshold for a speech start point to be marked. Simultaneously, the silence confirmation duration is set to 120 milliseconds, meaning that the probability of speech presence in 12 consecutive frames must be lower than a threshold for a speech end point to be marked. This mechanism, which transforms absolute duration constraints into consecutive frame counts, effectively filters out transient noise interference while accommodating pauses and variations in normal speech flow. This ensures that the segmented audio fragments remain semantically complete and coherent, thus meeting the stringent input requirements of the backend security recognition model for semantic integrity.

[0080] In this embodiment, the pre-trained speech activity detection model is constructed using a feedforward neural network architecture. The feedforward neural network consists of a cascaded input layer, a first hidden layer, a second hidden layer, and an output layer. The input layer contains two nodes, which receive the short-time energy value and zero-crossing rate value of a single frame of audio signal, respectively. The first hidden layer contains 16 nodes, each of which performs a weighted summation of its inputs and applies the ReLU activation function. The ReLU activation function is calculated as f(x) = max(0,x), meaning that when the input x is greater than 0, the output is x, and when the input x is less than or equal to 0, the output is 0. The second hidden layer also contains 16 nodes, each of which similarly performs a weighted summation of its inputs and applies the ReLU activation function. The output layer contains one node, which performs a weighted summation of its inputs and applies the Sigmoid activation function. The Sigmoid activation function is calculated as follows: The output value is the probability value of speech presence in the audio signal frame that is determined to be a speech frame, with a value ranging from 0 to 1. The training method of the speech activity detection model is as follows: Diverse environmental audio data in security recognition scenarios are collected. The audio data is divided into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds. Short-time energy values ​​and zero-crossing rate values ​​are extracted frame by frame as input features. Speech frame labels or non-speech frame labels are manually labeled frame by frame as training targets, where the speech frame label value is 1 and the non-speech frame label value is 0. The labeled frame-level feature data is divided into a training set (80%) and a validation set (20%) according to a preset ratio. Binary cross-entropy is used as the loss function. The binary cross-entropy loss function is calculated as the binary cross-entropy loss for a single sample. , where y is the true label value of the frame and p is the predicted speech presence probability output by the model; the Adam optimizer is used to iteratively update the weight matrix and bias vector of each layer of the network with a learning rate of 0.001. After each training round, the average binary cross-entropy loss value on the validation set is calculated. Training is terminated when the validation set loss value does not decrease for 5 consecutive rounds. The model parameters when the validation set loss value is the lowest are saved to obtain the pre-trained speech activity detection model.

[0081] A1013-3. Assign a segment identifier to each audio segment, including the segment number, start timestamp, and end timestamp, to obtain a second initial segment sequence composed of multiple audio segments arranged in chronological order. For example, three segments with vocals can be segmented from the audio stream: segment 1 (2.3s-5.8s), segment 2 (12.1s-16.5s), and segment 3 (28.4s-32.7s).

[0082] A1014. For the third modality data stream, perform temporal segmentation based on semantic integrity analysis to generate a third initial segment sequence, specifically including:

[0083] A1014-1. If the third modality data stream is a continuous text stream, then based on punctuation and semantic dependency analysis, the text stream is segmented into sentence-level text units. For each sentence-level text unit, a pre-trained semantic encoder is used to extract the sentence semantic vector. In this embodiment, the pre-trained semantic encoder is constructed using a Transformer encoder architecture. The Transformer encoder contains 6 identical Transformer encoder layers. Each Transformer encoder layer consists of a multi-head self-attention sub-layer and a feedforward neural network sub-layer cascaded with a layer normalization operation via residual connections. The multi-head self-attention sub-layer is configured with 8 attention heads, and each attention head has a query vector dimension, key vector dimension, and value vector dimension of 64. In the self-attention calculation, the input matrix is ​​linearly projected to generate the query matrix, key matrix, and value matrix for each attention head, and the product of the query matrix and the transpose of the key matrix is ​​calculated. The product, divided by a scaling factor of 8 (i.e., √64), is normalized row by row using the Softmax function to obtain the attention weight matrix, which is then multiplied by the value matrix to obtain the output of the attention head. The feedforward neural network sublayer contains two fully connected layers. The first fully connected layer maps the input dimension of 512 to the hidden layer dimension of 2048 and uses the GELU activation function for nonlinear transformation. The GELU activation function is calculated as f(x) = x·Φ(x), where Φ(x) is the cumulative distribution function of the standard normal distribution. The second fully connected layer maps the hidden layer dimension of 2048 back to the output dimension of 512. The input embedding dimension of the Transformer encoder is 512, and the maximum sequence length is set to 512 tags.

[0084] The training process of the semantic encoder is as follows: A text dataset for security identification scenarios is collected, including security event description text, security identification annotation text, and security scenario classification text. The text dataset is segmented into sentences to obtain a set of sentence samples. A byte-pair encoding word segmenter is used to segment each sentence sample to obtain a tag sequence. A classification tag [CLS] is added at the beginning of the tag sequence, and a separator tag [SEP] is added at the end. For each tag sequence, 15% of the tags are randomly selected for masking. 80% of the selected tags are replaced with a mask tag [MASK], 10% are replaced with random tags uniformly sampled from the vocabulary, and 10% remain unchanged. The masked tag sequence is input into the Transformer encoder. The encoder output vector corresponding to the masked position is taken. A linear projection layer maps the 512-dimensional output vector to the vocabulary dimension V. The Softmax function is used to calculate the vocabulary probability distribution. For the i-th masked position, the probability of predicting the j-th tag in the vocabulary is... In this embodiment, score_j represents the unnormalized raw score output by the linear projection layer for the j-th tag in the vocabulary during the pre-training of the semantic encoder. This score, after exponential operation and normalization, is converted into the probability that the model predicts the masked position as tag j. During the pre-training process of the semantic encoder, score_k represents the unnormalized raw score output by the linear projection layer for the k-th tag in the vocabulary (i.e., all tags traversed when summing the denominator). It is of the same type as score_j and is used for the normalized summation of the denominator of the Softmax function to calculate the probability that the masked position is predicted as a specific tag j.

[0085] The cross-entropy loss function is used to calculate the loss value between the predicted probability distribution and the original label. For a single masked label, the loss is -ln(p_true), where p_true is the model's predicted probability of the true label. The total loss is the average loss of all masked labels. The AdamW optimizer is used to iteratively update the model parameters with a learning rate of 0.0001 and a batch size of 32. After each training round, the validation loss value is calculated on the validation set. Training is terminated when the validation loss value does not decrease for three consecutive rounds. The model parameters with the lowest validation loss value are saved to obtain the pre-trained semantic encoder.

[0086] It should be noted that this semantic encoder processes text data and does not involve parameter settings such as frame length and frame shift of audio signals.

[0087] A1014-2. Starting from the first sentence-level text unit, the cosine similarity of the semantic vector of the current sentence and the semantic vector of the next adjacent sentence is calculated sequentially to obtain the semantic similarity of the adjacent sentences. When the semantic similarity of the adjacent sentences is greater than the preset strong association threshold, the current sentence and the next sentence are merged into a semantic paragraph. This merging process is repeated until no adjacent sentences meet the merging condition. In this embodiment, the strong association threshold is set as follows: based on the text dataset collected in the security identification scenario, a pre-trained semantic encoder is used to extract the semantic vectors of all adjacent sentences and calculate their similarity. With the manually annotated sentence merging truth value as the benchmark, those skilled in the art select the semantic similarity boundary point that achieves the optimal balance between the accuracy and recall of sentence merging under different similarity values ​​as the preset threshold. In this embodiment, the strong association threshold is statistically set to 0.85. When the semantic similarity of adjacent sentences is higher than this value, it is determined that the two sentences are highly semantically connected and belong to the same complete semantic paragraph, and are merged. Otherwise, it indicates that there is a semantic shift or topic change, and they are not merged. This ensures the integrity of the text semantic units while effectively preventing the semantic primitive granularity from becoming too coarse due to excessive merging.

[0088] A1014-3. Each semantic segment obtained from the final merging is treated as a text fragment. Based on the timestamp information of the original data corresponding to that semantic segment, a fragment identifier containing a fragment number, a start timestamp, and an end timestamp is assigned to it, ultimately resulting in a third initial fragment sequence composed of multiple text fragments arranged in chronological order. If there is no explicit timestamp, the text fragments are mapped to corresponding time intervals based on the synchronization relationship between the text stream and the video and audio. For example, the text description corresponding to the video may be segmented into fragment 1 (0.0s-10.5s), fragment 2 (10.5s-25.0s), etc.

[0089] S102. Each segment in the first initial segment sequence, and segments whose time windows overlap with those in the second initial segment sequence and the third initial segment sequence, are respectively denoted as cross-modal segment pairs; each cross-modal segment pair is input into a pre-trained cross-modal temporal alignment model, and the semantic alignment confidence and optimal time offset between the first modal segment and the second modal segment, and between the first modal segment and the third modal segment in the cross-modal segment pair are calculated to generate a cross-modal alignment relationship matrix;

[0090] In this embodiment, the pre-trained cross-modal temporal alignment model is constructed from four parts: a visual encoding branch, a semantic encoding branch, a common spatial projection layer, and an alignment prediction network.

[0091] In this embodiment, the visual encoding branch uses a 12-layer VisionTransformer encoder. The input image size is 224×224 pixels, and the image patch size is 16×16 pixels. The input image is divided into (224 / 16)×(224 / 16)=196 image patches. Each image patch is linearly projected to obtain a 768-dimensional block embedding vector, which is then superimposed with a learnable position encoding vector. A learnable class label embedding vector is added to the front of the sequence, resulting in an input sequence consisting of 197 embedding vectors. The input sequence is processed by a 12-layer Transformer encoder layer, each layer containing a multi-head self-attention sub-layer and a feedforward neural network sub-layer. The number of multi-head self-attention heads is 12, and the query vector dimension, key vector dimension, and value vector dimension of each attention head are all 64. The hidden layer dimension of the feedforward neural network is 3072, and the activation function is GELU. A 768-dimensional feature vector is extracted from the corresponding position of the class label output by the 12th layer as the visual feature vector.

[0092] In this embodiment, the semantic encoding branch employs a 6-layer Transformer encoder. The input is a text tag sequence segmented by a byte-pair encoding tokenizer and supplemented with classification and delimiter tags, with an embedding dimension of 512 and a maximum sequence length of 512. After processing by the 6-layer Transformer encoder, each layer contains a multi-head self-attention sub-layer and a feedforward neural network sub-layer. The multi-head self-attention sub-layer has 8 heads, and the query vector dimension, key vector dimension, and value vector dimension of each attention head are all 64. The hidden layer dimension of the feedforward neural network is 2048, and the activation function is GELU. A 512-dimensional feature vector is extracted from the corresponding position of the classification tag output by the 6th layer as the semantic feature vector.

[0093] In this embodiment, the public space projection layer includes a first linear layer and a second linear layer. The weight matrix of the first linear layer has a dimension of 768×256, no bias term, and no activation function, mapping the 768-dimensional visual feature vector to a 256-dimensional visual public vector. The weight matrix of the second linear layer has a dimension of 512×256, no bias term, and no activation function, mapping the 512-dimensional semantic feature vector to a 256-dimensional semantic public vector.

[0094] In this embodiment, the alignment prediction network consists of a concatenation layer, a first hidden layer, a second hidden layer, and an output layer connected sequentially. The concatenation layer concatenates the 256-dimensional visual common vector and the 256-dimensional semantic common vector along the feature dimension to form a 512-dimensional joint vector. The first hidden layer contains 128 nodes, which apply a ReLU activation function after weighted summation of the input. The second hidden layer contains 64 nodes, which also apply a ReLU activation function after weighted summation of the input. The output layer contains two nodes. The first output node calculates the semantic alignment confidence value in the range (0,1) by applying a sigmoid activation function after weighted summation of the input; the second output node calculates the offset prediction value in the range of any real number without applying an activation function, and directly outputs the predicted offset value in seconds.

[0095] In this embodiment, the training process of the cross-modal temporal alignment model is as follows: Video and audio data recorded synchronously in a security identification scenario are collected; keyframes are extracted from the video at 2-second intervals to construct video segments; audio streams are segmented using speech activity detection to construct audio segments; start and end timestamps are labeled for each video and audio segment; video and audio segments with overlapping time windows are paired; each pair is manually labeled as a positive sample pair for semantic alignment, with a label value of 1 for positive sample pairs and 0 for negative sample pairs; for positive sample pairs, the precise time offset value of the audio segment relative to the video segment is further labeled in seconds; for negative sample pairs, the label value is 0. The offset value of each pair is marked as 0; each paired video segment is scaled to 224×224 pixels and input into the visual encoding branch to obtain a 768-dimensional visual feature vector, which is then mapped to a 256-dimensional visual common vector through the first linear layer; the text content obtained from the audio segment through speech recognition is input into the semantic encoding branch to obtain a 512-dimensional semantic feature vector, which is then mapped to a 256-dimensional semantic common vector through the second linear layer; the visual common vector and the semantic common vector are concatenated into a 512-dimensional joint vector and input into the alignment prediction network. The first output node outputs the alignment confidence prediction value, and the second output node outputs the offset prediction value; the alignment is calculated using the binary cross-entropy loss function. The alignment loss between the predicted confidence value and the positive / negative sample labels is calculated as follows: For a single sample, the alignment loss Loss_align = -[y1×ln(p1)+(1-y1)×ln(1-p1)], where y1 is the true label value and p1 is the predicted alignment confidence value. The offset loss between the predicted offset value and the manually labeled offset value is calculated using the smoothed L1 loss function. For a single sample, when the absolute difference between the predicted offset pred and the labeled offset truth is less than 1, the offset loss Loss_offset = 0.5×(pred-truth)²; when the absolute difference is greater than or equal to 1, the offset loss is... s_offset=|pred-truth|-0.5; The alignment loss and offset loss are weighted and summed by preset weight coefficients of 0.5 and 0.5 to obtain the total loss. The total loss Loss_total=0.5×Loss_align+0.5×Loss_offset; The Adam optimizer is used to iteratively update all parameters of the model with a learning rate of 0.0001 and a batch size of 32. After each training round, the validation total loss is calculated on the validation set. Training is terminated when the validation total loss does not decrease for 5 consecutive rounds. The model parameters with the lowest validation total loss are saved to obtain the pre-trained cross-modal temporal alignment model.

[0096] S103. Based on the cross-modal alignment relationship matrix, cross-modal segments whose semantic alignment confidence exceeds a preset confidence threshold and whose optimal time offset is within a preset offset tolerance range are identified as aligned related segments. All cross-modal segments within the same aligned related segment group are unified on the time axis to the minimum time interval covering all segments within the group, resulting in an aligned cross-modal segment group. In this embodiment, the preset confidence threshold and the preset offset tolerance range are based on a balance between the accuracy requirements of cross-modal temporal alignment in the security recognition scenario and the tolerance boundary of the backend security recognition large model for the semantic integrity of the input. Specifically, the preset confidence threshold is set to 0.80. This is based on receiver operating characteristic (ROC) curve analysis of the semantic alignment confidence value output by the cross-modal temporal alignment model and the manually labeled alignment ground truth value on a labeled verification dataset containing typical safety recognition scenarios such as traffic accidents and public safety incidents. The lowest confidence threshold point that ensures an alignment decision recall rate of no less than 90% and a precision rate of no less than 85% is selected as the threshold. This ensures that cross-modal segment pairings with semantic alignment confidence exceeding this threshold have highly reliable semantic relevance, avoiding the misclassification of semantically unrelated cross-modal segments as aligned segments, which would lead to cross-modal noise being mixed into the subsequent semantic primitive sequence. The preset offset tolerance range is set to [-1.0 seconds, 1.0 seconds], based on the statistical analysis of the physical temporal deviation between visual and auditory events in multimodal information content under safety recognition scenarios. The characteristics and the tolerance capability of the backend security recognition model for temporal alignment errors are jointly determined. Specifically, in security recognition scenarios, there is usually a natural time offset between the moment an event occurs in the video frame and the moment the corresponding speech description is uttered in the audio, due to factors such as sound propagation delay and processing delay of the acquisition equipment. Statistically, this offset does not exceed ±1.0 seconds in most cases. Furthermore, the backend security recognition model is robust to temporal deviations of the input modal semantic primitives within a range of ±1.0 seconds during cross-modal semantic joint encoding. Temporal misalignments exceeding this range will cause significant cross-modal semantic association errors in the model's security attribute determination. Therefore, the upper and lower limits of the offset tolerance range are set to 1.0 seconds and -1.0 seconds, respectively, to ensure that cross-modal segments within this range can be correctly grouped into the same aligned and associated segment group and have a unified time interval. Specifically, the offset is defined as the start timestamp of the audio segment minus the start timestamp of the video segment. A positive offset indicates that the audio event occurs later than the video event; a negative offset indicates that the audio event occurs earlier than the video event. In security identification scenarios, there is a common phenomenon of "hearing the sound before seeing the image." For example, in traffic accident videos, the sound of a collision or a cry for help may be captured by the acquisition device an instant before the corresponding image appears.Therefore, the preset offset tolerance range [-1.0 seconds, 1.0 seconds] means that the time offset of an audio segment no more than 1.0 seconds earlier than a video segment and no more than 1.0 seconds later is considered to be within the effective range for cross-modal timing alignment.

[0097] S104. For each modality segment within each aligned cross-modal segment group, calculate the variance of the semantic vector within each single-modal segment. When the variance exceeds a preset variance threshold, further segment the corresponding single-modal segment until the variance of the semantic vector within each segmented sub-segment is lower than the preset variance threshold. When the semantic vector similarity between two adjacent sub-segments of the same modality exceeds a preset merging threshold, merge the two adjacent sub-segments of the same modality into a single semantic primitive. In this embodiment, the preset variance threshold is set based on the following: In a security identification scenario, synchronous multimodal sample data is collected and decoupled into a semantic primitive sequence. For each aligned cross-modal segment group... The internal semantic vectors of single-modal segments within a modal segment group are extracted, and the variance distribution of the internal semantic vectors of each segment is statistically analyzed. Based on the manually annotated semantic primitive segmentation truth value, those skilled in the art select the variance boundary point that optimizes the homogeneity within semantic primitives and avoids over-segmentation as the preset variance threshold. In this embodiment, the variance threshold is set to 0.15. When the variance of the internal semantic vector of a single-modal segment exceeds 0.15, it is determined that there is a significant semantic mutation or multiple independent events mixed in the segment, and further segmentation is required until the variance of all sub-segments after segmentation is lower than this threshold, thereby ensuring that each semantic primitive is semantically homogeneous and complete.

[0098] In this embodiment, the preset merging threshold is set based on the following: Based on the set of homomodal sub-segments generated after re-segmentation, a pre-trained semantic encoder is used to calculate the semantic vector similarity of all adjacent homomodal sub-segments. Using manually annotated semantic primitive merging ground truth as a benchmark, those skilled in the art weigh the merging accuracy under different similarity values ​​and select the optimal similarity boundary point as the preset merging threshold, which effectively aggregates coherent semantics without confusing differing semantics. In this embodiment, the merging threshold is set to 0.90. When the semantic vector similarity of two adjacent homomodal sub-segments is higher than 0.90, they are determined to be highly semantically continuous and belong to the same complete security event description, and are merged into a single semantic primitive. Otherwise, they remain independent. This dynamically aggregates semantically coherent segments based on refined segmentation, avoiding fragmentation of semantic primitive sequences due to excessive segmentation.

[0099] S105. The segments of each modality within the same aligned cross-modal segment group, after being further segmented and merged, are respectively used as the first modal semantic primitive, the second modal semantic primitive, and the third modal semantic primitive, and arranged in the order of timestamps to form the first modal semantic primitive sequence, the second modal semantic primitive sequence, and the third modal semantic primitive sequence.

[0100] As a specific example, in this embodiment, a 35-second video clip of a traffic accident scene is used as the input to the multimodal data stream. This data consists of multimodal information captured by a front-end acquisition device and uploaded in real time to a back-end safety identification model for automated safety assessment. The data includes a first modal data stream (video stream), a second modal data stream (audio stream), and a third modal data stream (text stream), all collected simultaneously, with a start timestamp of 0.0 seconds for each. The video stream contains visual information such as vehicle collisions, onlookers, and driver arguments; the audio stream contains voice conversations of people at the scene; and the text stream contains text descriptions synchronized with the video.

[0101] In step S101, candidate segmentation points are defined on the timeline of the video stream at a preset basic segmentation interval of 2 seconds. For each candidate segmentation point, the two adjacent image frames before and after it are obtained. The two image frames are input into the pre-trained VisionTransformer image encoder to extract image embedding vectors and calculate cosine similarity as a scene switching score. Candidate segmentation points with scene switching scores lower than the preset scene switching threshold of 0.75 are marked as actual segmentation points. Finally, the first initial segment sequence is obtained, which includes segment V1 (0.0s-5.2s), segment V2 (5.2s-12.8s), segment V3 (12.8s-21.5s), segment V4 (21.5s-28.0s), and segment V5 (28.0s-35.0s). The audio stream is segmented into frames with a frame length of 25 milliseconds and a frame shift of 10 milliseconds, and short-time energy values ​​and zero-crossing rate values ​​are extracted. The frame-level acoustic feature sequence is then processed frame by frame. Inputting a pre-trained speech activity detection model, the speech start and end points are determined using a speech presence threshold of 0.8 and a silence threshold of 0.3, resulting in a second initial segment sequence containing segment A1 (2.3s-5.8s), segment A2 (12.1s-16.5s), and segment A3 (28.4s-32.7s). Segment A1 corresponds to the speech content "How do you drive?", segment A2 corresponds to the speech content "Hit someone and still want to run away", and segment A3 corresponds to the speech content "Call the police immediately". The text stream is then segmented into sentences based on punctuation marks. The semantic vectors of the sentences are extracted by a pre-trained semantic encoder, and the semantic similarity between adjacent sentences is calculated. Adjacent sentences with a similarity greater than the preset strong association threshold of 0.85 are merged, resulting in a third initial segment sequence containing segment T1 (0.0s-10.5s), segment T2 (10.5s-25.0s), and segment T3 (25.0s-35.0s).

[0102] Execute S102, traverse all segments in the three initial segment sequences, and determine the cross-modal segment pairs with overlapping time windows as (V1,A1), (V1,T1), (V3,A2), (V3,T2), (V5,A3), and (V5,T3). Input the above 6 pairs into the pre-trained cross-modal temporal alignment model in sequence to obtain the semantic alignment confidence value and the optimal time offset value of each pair. Specifically, the alignment confidence of pair (V1,A1) is 0. The following pairs of data are used to generate cross-modal alignment relation matrices: (V1, T1) with an alignment confidence of 0.88 and an offset of 0.1 seconds; (V3, A2) with an alignment confidence of 0.87 and an offset of -0.2 seconds; (V3, T2) with an alignment confidence of 0.85 and an offset of 0.0 seconds; (V5, A3) with an alignment confidence of 0.91 and an offset of 0.1 seconds; and (V5, T3) with an alignment confidence of 0.89 and an offset of 0.0 seconds.

[0103] Execute S103, set the preset confidence threshold to 0.80, set the preset offset tolerance range to [-1.0 seconds, 1.0 seconds], all 6 pairs are determined to be aligned related segments, and the pairs containing common segments are merged into the same aligned related segment group, resulting in the aligned cross-modal segment group Group1 containing segments V1, A1, and T1, Group2 containing segments V3, A2, and T2, and Group3 containing segments V5, A3, and T3.

[0104] Execute S104 to extract semantic vectors from each modal segment within each aligned cross-modal segment group and calculate the variance of the semantic vectors within a single modal segment. Set a preset variance threshold of 0.15 and a preset merging threshold of 0.90. After detection, the variance of the semantic vector within segment T2 in Group2 is 0.21, which exceeds the threshold and needs to be further segmented. Along the time axis, T2 is segmented into sub-segments T2-1 (10.5s-18.0s) and T2-2 (18.0s-25.0s) at timestamp 18.0 seconds. After the resegmentation, the variance of the semantic vectors within the sub-segments is all below 0.15, and the similarity of the semantic vectors of adjacent sub-segments is 0.82, which is below the merging threshold and they are not merged. The variances of the remaining segments are all below the threshold and are not further segmented.

[0105] Execute S105, assigning segments V1, A1, and T1 within Group 1 as the first modal semantic primitive U1-1, the second modal semantic primitive U2-1, and the third modal semantic primitive U3-1, respectively. Assigning segments V3, A2, sub-segments T2-1, and T2-2 within Group 2 as the first modal semantic primitive U1-2, the second modal semantic primitive U2-2, and the third modal semantic primitives U3-2 and U3-3, respectively. Then, assign segments V5, A3, and T3 within Group 3 as... The three semantic primitives, U1-3 (first modality), U2-3 (second modality), and U3-4 (third modality), are arranged in timestamp order to generate the following sequences: (U1-1, U1-2, U1-3), (U2-1, U2-2, U2-3), and (U3-1, U3-2, U3-3, U3-4). This completes the entire process of decoupling the three modal data streams into semantic primitive sequences. These semantic primitive sequences will then be transmitted to the backend security identification model. Based on the cross-modal semantic associations and security domain knowledge learned during pre-training, this model performs regional entity extraction and attribution determination, information security scenario classification, multi-dimensional security attribute determination, and security risk level generation. Finally, it outputs a structured security identification result containing regional identification results, scenario identification results, security risk levels, and handling suggestions.

[0106] It should be further explained that the semantic influence factors generated for each modality in this embodiment include:

[0107] S201. The first modal semantic primitive sequence, the second modal semantic primitive sequence, and the third modal semantic primitive sequence are simultaneously input into a proxy model aligned with the backend security identification large-scale model task. The joint feature distribution output by the proxy model is obtained as the baseline feature distribution. The network structure and parameter configuration of the proxy model are consistent with the feature extraction layer of the backend security identification large-scale model. In this embodiment, the backend security identification large-scale model is a pre-trained large language model deployed on the server by those skilled in the art for automated security identification of multimodal information content. Its network structure is completely consistent with the proxy model, containing 32 Transformer layers. The r decoder layer, each Transformer decoder layer consists of a multi-head self-attention sublayer and a feedforward neural network sublayer cascaded through residual connections and layer normalization operations. The multi-head self-attention sublayer has 40 attention heads, and each attention head has a query vector dimension, key vector dimension, and value vector dimension of 128. The first fully connected layer of the feedforward neural network sublayer maps the input dimension of 5120 to the hidden layer dimension of 20480 and uses the GELU activation function. The second fully connected layer maps the hidden layer dimension of 20480 back to the output dimension of 5120. The hidden layer dimension is 5120, and the vocabulary size is 128000. The training method of the backend security recognition large model is as follows: A multimodal sample dataset for security identification scenarios is collected. Each sample in the multimodal sample dataset contains synchronously collected first-modal data streams, second-modal data streams, and third-modal data streams, as well as manually labeled ground truth values ​​for regional identification results, scene identification results, security risk levels, and handling suggestions. For each sample, the first-modal semantic primitive sequence, the second-modal semantic primitive sequence, and the third-modal semantic primitive sequence are arranged in the same modal order and labeled in the same way as in S2011 to construct a multimodal joint input sequence, which is then input into the back-end security identification large model. The back-end security identification large model performs layer-by-layer Tr processing on the input multimodal joint input sequence. The Ansformer decoder layer processes the data, extracting the feature vector corresponding to the end marker of the joint sequence from the last layer output as a multimodal joint feature representation. This multimodal joint feature representation is then mapped through a task output layer to the output space corresponding to each security identification subtask, generating prediction results for region identification, scene identification, security risk level, and handling suggestions, respectively. A multi-task joint loss function is used to calculate the total loss between the prediction results of each subtask and the manually labeled ground truth. In this embodiment, the multi-task joint loss function is determined as follows: the region identification loss is denoted as L_geo, and calculated using the cross-entropy loss function, with the expression: Where y_geo_i is the true one-hot encoded label of the region category, and p_geo_i is the predicted probability of the region category output by the model; the scene recognition loss is denoted as L_scene, and is calculated using the cross-entropy loss function, the expression of which is: Where y_scene_j is the true one-hot encoded label of the scene category, and p_scene_j is the predicted probability of the scene category output by the model; the loss for determining the safety risk level is denoted as L_risk, and is calculated using the cross-entropy loss function, the expression of which is: Where y_risk_k is the true one-hot encoded label of the risk level, and p_risk_k is the predicted probability of the risk level output by the model; the loss from generating the action suggestion is denoted as L_action, and is calculated using the autoregressive cross-entropy loss function, the expression of which is: Where y_action_t,m represents the true label of the m-th marker in the vocabulary at position t in the action suggestion text sequence, and p_action_t,m is the model's predicted probability for that position. The multi-task joint loss function L_total is calculated as L_total=α×L_geo+β×L_scene+γ×L_risk+δ×L_action, where α, β, γ, and δ are the weighting coefficients of the region identification loss, scene identification loss, security risk level determination loss, and action suggestion generation loss, respectively. All four coefficients are preset non-negative hyperparameters that satisfy the normalization constraint α+β+γ+δ=1. The weighting coefficients are set based on the following criteria: In the security identification scenario, the influence of each sub-task on the final security decision is used as the benchmark. A combination of coefficients that achieves the optimal balance between the accuracy of security risk level determination and the availability of handling suggestions is selected on the validation set through grid search. In this embodiment, the weighting coefficients for the region identification loss (α) are set to 0.15, the scene identification loss (β) to 0.25, the security risk level determination loss (γ) to 0.35, and the handling suggestion generation loss (δ) to 0.25. This ensures that the optimization intensity of each sub-task during model training is consistent with the priority of the backend security identification business. The AdamW optimizer is used to iteratively update the model parameters with a learning rate of 0.00001 and a batch size of 16. After each round of training, the total validation loss is calculated on the validation set. Training is terminated when the total validation loss does not decrease for three consecutive rounds. The model parameters with the lowest total validation loss are saved to obtain the backend security identification large model.

[0108] In this embodiment, the specific task of the backend security identification big model is as follows: receiving the first modality semantic primitive sequence, the second modality semantic primitive sequence, and the third modality semantic primitive sequence delivered via the transmission link; constructing a multimodal joint input sequence according to the same modal arrangement order and marker addition method as in S2011; inputting the multimodal joint input sequence into the backend security identification big model; performing cross-modal self-attention calculation and feedforward neural network transformation layer by layer through the 32-layer Transformer decoder layer; extracting the 5120-dimensional feature vector corresponding to the end marker position of the joint sequence from the final output matrix output by the 32nd layer; and inputting the feature vector into the regional entity extraction and attribution determination subnetwork, the information security scenario classification subnetwork, and the multi-dimensional security attribute determination subnetwork, respectively. The system consists of a network and security risk level generation subnetwork, regional subnetworks outputting regional identification results, an information security scenario classification subnetwork outputting scenario identification results, a multi-dimensional security attribute judgment subnetwork combining scenario identification results to generate preliminary security attribute judgment results and calling multiple domain judgment agents to output domain judgment scores in parallel, then weighted summing to obtain a security score, a security risk level generation subnetwork generating a security risk level based on the comparison between the security score and the security level judgment benchmark, and a disposal suggestion generation subnetwork matching disposal suggestion text and push department identifiers based on the security risk level and scenario identification results, ultimately outputting a structured security identification result containing regional identification results, scenario identification results, security risk level, and disposal suggestions, in order to achieve automated and accurate identification of whether multimodal information content has security risks.

[0109] In this embodiment, the proxy model is constructed using the same network structure as the backend security identification model. This network structure includes 32 Transformer decoder layers. Each Transformer decoder layer consists of a multi-head self-attention sub-layer and a feedforward neural network sub-layer, cascaded with residual connections and layer normalization operations. The multi-head self-attention sub-layer has 40 attention heads, and each attention head has a query vector dimension, key vector dimension, and value vector dimension of 128. The first fully connected layer of the feedforward neural network sub-layer maps the input dimension of 5120 to the hidden layer dimension of 20480 and uses the GELU activation function. The second fully connected layer maps the hidden layer dimension of 20480 back to the output dimension of 5120. The hidden layer dimension is 5120, and the vocabulary size is 128000. After construction, the proxy model loads all model parameters saved during the training and convergence of the backend security identification model. These model parameters include all Transformer decoder layers. The weight matrix, bias vector, layer normalization parameters, and embedding layer parameters of the encoder layer ensure that the proxy model is completely consistent with the backend security identification model at the parameter level, possessing the same cross-modal semantic joint encoding and feature extraction capabilities. The training method of the proxy model is as follows: the proxy model is deployed on the network edge node, the joint feature distribution vector generated by the backend security identification model for the input multimodal sample data is obtained as the distillation target, the multimodal semantic primitive sequence is input into the proxy model to obtain the predicted joint feature distribution vector, the KL divergence value between the predicted joint feature distribution vector and the distillation target is calculated as the distillation loss, the AdamW optimizer is used to iteratively update the parameters of the proxy model with a learning rate of 0.00001 and a batch size of 16, the distillation loss on the validation set is calculated after each round of training, and training is terminated when the validation set distillation loss does not decrease for 3 consecutive rounds, the model parameters with the lowest validation set distillation loss are saved, and a trained proxy model aligned with the backend security identification model task is obtained.

[0110] It should be further explained that, in this embodiment, the joint feature distribution output by the proxy model is used as the baseline feature distribution, including:

[0111] S2011. Obtain the first modal semantic primitive sequence (U1-1, U1-2, ..., U1-P), the second modal semantic primitive sequence (U2-1, U2-2, ..., U2-Q), and the third modal semantic primitive sequence (U3-1, U3-2, ..., U3-R) generated in S105, where P is the total number of semantic primitives in the first modal semantic primitive sequence, Q is the total number of semantic primitives in the second modal semantic primitive sequence, and R is the total number of semantic primitives in the third modal semantic primitive sequence. Each semantic primitive is a vector with dimension 5120.

[0112] A preset first modality start marker embedding vector is added at the beginning position of the first modality semantic primitive sequence, and a preset first modality end marker embedding vector is added at the end position of the first modality semantic primitive sequence. Both the first modality start marker embedding vector and the first modality end marker embedding vector are learnable parameter vectors of dimension 5120, provided by the embedding layer of the proxy model.

[0113] A preset second modality start marker embedding vector is added at the beginning position of the second modality semantic primitive sequence, and a preset second modality end marker embedding vector is added at the end position of the second modality semantic primitive sequence. Both the second modality start marker embedding vector and the second modality end marker embedding vector are learnable parameter vectors of dimension 5120, provided by the embedding layer of the proxy model.

[0114] A preset third modality start marker embedding vector is added at the beginning position of the third modality semantic primitive sequence, and a preset third modality end marker embedding vector is added at the end position of the third modality semantic primitive sequence. Both the third modality start marker embedding vector and the third modality end marker embedding vector are learnable parameter vectors of dimension 5120, provided by the embedding layer of the surrogate model.

[0115] The first modality semantic primitive sequence, the second modality semantic primitive sequence, and the third modality semantic primitive sequence after adding the labeled embedding vectors are concatenated in the order of first modality, second modality, and third modality. A preset joint sequence start-label embedding vector is added to the beginning of the concatenated sequence, and a preset joint sequence end-label embedding vector is added to the end of the concatenated sequence. The joint sequence start-label embedding vector and the joint sequence end-label embedding vector are both learnable parameter vectors of dimension 5120, provided by the embedding layer of the surrogate model. Finally, a multimodal joint input sequence consisting of (P+Q+R+8) labeled embedding vectors is generated, and the dimension of the multimodal joint input sequence is (P+Q+R+8)×5120. As a specific example, the first modality semantic primitive sequence generated by S105 contains three semantic primitives: U1-1, U1-2, and U1-3; the second modality semantic primitive sequence contains three semantic primitives: U2-1, U2-2, and U2-3; and the third modality semantic primitive sequence contains four semantic primitives: U3-1, U3-2, U3-3, and U3-4. P=3, Q=3, R=4, and each semantic primitive is a 5120-dimensional vector. When constructing the multimodal joint input sequence, a 5120-dimensional first-mode start-label embedding vector is added before U1-1, and a 5120-dimensional first-mode end-label embedding vector is added after U1-3; a 5120-dimensional second-mode start-label embedding vector is added before U2-1, and a 5120-dimensional second-mode end-label embedding vector is added after U2-3; a 5120-dimensional third-mode start-label embedding vector is added before U3-1, and a 5120-dimensional third-mode end-label embedding vector is added after U3-4. The three modal sequences with added labels are concatenated sequentially in the order of first mode, second mode, and third mode. A 5120-dimensional joint sequence start-label embedding vector is added at the beginning of the concatenated sequence, and a 5120-dimensional joint sequence end-label embedding vector is added at the end. Finally, a multimodal joint input sequence consisting of 3+3+4+8=18 5120-dimensional label embedding vectors is generated, with a sequence dimension of 18×5120.

[0116] S2012. Input the multimodal joint input sequence into the surrogate model and extract the baseline feature distribution vector, specifically including:

[0117] S2012-1. The embedding layer of the surrogate model receives the multimodal joint input sequence and superimposes a position encoding vector uniquely corresponding to the position of the label in the multimodal joint input sequence onto each label embedding vector in the multimodal joint input sequence; the position encoding vector is generated as follows: for dimensions with odd position indices of 2k+1 in the sequence, its position encoding value is sin(position index / 10000).(2k / 5120) For a dimension with an even position index of 2k, its position encoding value is cos(position index / 10000). (2k / 5120) ), where k is an integer ranging from 0 to 2559; the dimension of the position encoding vector is 5120, and the superposition operation is to add corresponding elements together, and after superposition, an input embedding matrix with a dimension of (P+Q+R+8)×5120 is generated.

[0118] S2012-2. The input embedding matrix is ​​input to the Transformer decoder stack of the proxy model; the Transformer decoder stack consists of 32 identical Transformer decoder layers stacked sequentially, with the output matrix of the previous layer directly serving as the input matrix of the next layer, and the first layer receiving the input embedding matrix as its input matrix. The processing procedure for each Transformer decoder layer is as follows:

[0119] S2012-2-1. Receive an input matrix with a dimension of (P+Q+R+8)×5120. Multiply the input matrix by a query weight matrix with a dimension of 5120×(40×128), a key weight matrix with a dimension of 5120×(40×128), and a value weight matrix with a dimension of 5120×(40×128). Reshape the multiplication results into tensors of (P+Q+R+8)×40×128. Then, through a dimension permutation operation, obtain 40 independent query matrices, key matrices, and value matrices, each with a dimension of (P+Q+R+8)×128.

[0120] S2012-2-2. For each of the 40 attention heads, multiply the query matrix of the attention head by the transpose of the key matrix to obtain an original attention score matrix of dimension (P+Q+R+8)×(P+Q+R+8). Divide each element of the original attention score matrix by the scaling factor 8 to obtain a scaled attention score matrix. Perform Softmax function normalization on each row of the scaled attention score matrix so that the sum of the element values ​​of each row is 1, and obtain the attention weight matrix of the attention head.

[0121] S2012-2-3. For each of the 40 attention heads, multiply the attention weight matrix of that attention head by its value matrix to obtain the output matrix of that attention head, with a dimension of (P+Q+R+8)×128; concatenate the output matrices of all 40 attention heads along the last dimension to obtain a concatenated output matrix with a dimension of (P+Q+R+8)×5120; multiply the concatenated output matrix by an output projection weight matrix with a dimension of 5120×5120 to obtain the multi-head self-attention output matrix;

[0122] S2012-2-4. The multi-head self-attention output matrix and the input matrix received in S2012-2-1 are joined element-wise to obtain a residual concatenation, resulting in a first residual matrix. Layer normalization is then performed on the first residual matrix along its last dimension to obtain a first intermediate matrix with dimensions (P+Q+R+8)×5120. The layer normalization operation is calculated as follows: for each row of the input matrix, the mean and variance of the 5120 elements in that row are calculated. The mean is subtracted from each element in the row, and the result is divided by the square root of the variance and a small constant. This is then multiplied by a learnable scaling parameter and a learnable translation parameter is added. In this embodiment, the "small constant" refers to a fixed, minimal positive value set for calculating numerical stability; in this embodiment, it is set to 1×10⁻⁶. -5 Its function is to prevent division by zero errors caused by excessively small variance in the denominator; the "learnable scaling parameter" and "learnable translation parameter" are two trainable vectors defined in the layer normalization operation. The dimension of each vector is the same as the last dimension of the input feature, that is, 5120 dimensions. During model initialization, all element values ​​of the scaling parameter are initialized to 1, and all element values ​​of the translation parameter are initialized to 0. During the distillation training of the surrogate model, these two parameters are used as part of the model parameters and are iteratively updated by the optimizer according to the distillation loss gradient. Finally, they are stored together with other parameters of the surrogate model in the trained model weights to adaptively adjust the scale and position of the normalized feature distribution.

[0123] S2012-2-5. The first intermediate matrix is ​​input into the feedforward neural network sub-layer. The first fully connected weight matrix of the feedforward neural network sub-layer has a dimension of 5120×20480, and the first fully connected bias vector has a dimension of 20480. After matrix multiplication and bias addition, a first fully connected output matrix with a dimension of (P+Q+R+8)×20480 is obtained. The GELU activation function is applied to each element of the first fully connected output matrix to obtain the activation output matrix. The second fully connected weight matrix of the feedforward neural network sub-layer has a dimension of 20480×5120, and the second fully connected bias vector has a dimension of 5120. The activation output matrix is ​​then subjected to matrix multiplication and bias addition to obtain a feedforward output matrix with a dimension of (P+Q+R+8)×5120.

[0124] S2012-2-6. Perform element-wise addition of the feedforward output matrix and the first intermediate matrix to obtain a second residual matrix. Perform layer normalization operation on the second residual matrix along the last dimension in the same way as in S2012-2-4 to obtain the output matrix of the current Transformer decoder layer.

[0125] S2012-3. From the final output matrix with dimension (P+Q+R+8)×5120 output from the 32nd layer Transformer decoder layer, extract the row vector corresponding to the position index of the joint sequence end marker in the multimodal joint input sequence. The row vector has a dimension of 5120. Use the row vector as the baseline feature distribution vector to complete the acquisition of the baseline feature distribution.

[0126] As a concrete example, continuing the processing of the aforementioned 35-second traffic accident scene video, the multimodal joint input sequence constructed by S2011 contains 18 labeled embedding vectors, with a sequence dimension of 18×5120. According to S2012-1, the embedding layer of the surrogate model receives this 18×5120 multimodal joint input sequence. For each labeled embedding vector in the sequence at position indices 1 to 18, a 5120-dimensional position encoding vector calculated based on that position index using sine and cosine functions is superimposed. The superposition method is to add corresponding elements together, generating an input embedding matrix with a dimension of 18×5120. According to S2012-2, the input embedding matrix is ​​fed into the first layer of the surrogate model, the Transformer decoder layer. Each layer performs the following operations in sequence: multiply the 18×5120 input matrix with three 5120×5120 weight matrices and reshape it into 40 18×128 query matrices, key matrices, and value matrices. For each attention head, calculate the product of the query matrix and the transpose of the key matrix to obtain an 18×18 original attention score matrix. After dividing by a scaling factor of 8, it is normalized row by row by Softmax to obtain an 18×18 attention weight matrix. Then, it is multiplied with the value matrix to obtain an 18×128 output matrix. The output matrices of the 40 heads are concatenated and projected back to 18×5120. After residual connection with the input matrix, it is layer normalized. After transformation by a feedforward neural network with two fully connected layers, it is residually connected again and layer normalized. The output 18×5120 matrix is ​​used as the input of the next layer. After the same processing is performed on all 32 layers, the output of the 32nd layer is the final output matrix with a dimension of 18×5120. According to S2012-3, the final output matrix has 18 rows. Each row corresponds to a 5120-dimensional feature vector at a position in the multimodal joint input sequence. The position index of the joint sequence end marker in the sequence is 18. The 5120-dimensional row vector corresponding to the 18th row is extracted, and this row vector is used as the baseline feature distribution vector to complete the acquisition of the baseline feature distribution.

[0127] S202. Perform a masking operation on the first modal semantic primitive sequence, input the masked first modal semantic primitive sequence, the unmasked second modal semantic primitive sequence, and the third modal semantic primitive sequence into the proxy model, obtain the first mask feature distribution, calculate the first KL divergence value between the first mask feature distribution and the benchmark feature distribution, and map the first KL divergence value into a first semantic influence factor through a preset exponential function.

[0128] In this embodiment, a specific implementation of performing a masking operation on the first modal semantic primitive sequence and generating a first semantic influence factor is as follows: All vector values ​​of each semantic primitive in the first modal semantic primitive sequence are replaced with zero vectors, each zero vector having a dimension of 5120, to obtain a masked first modal semantic primitive sequence; the masked first modal semantic primitive sequence is then concatenated with the unmasked second modal semantic primitive sequence generated in S105 and the third modal semantic primitive sequence, following the same modal arrangement order and label addition method as in S2011, to form a multimodal mask joint input sequence; the multimodal mask joint input sequence is then input to the proxy model, and the final output from the 32nd layer Transformer decoder layer of the proxy model is obtained following the same layer-by-layer processing procedure as in S2012. In the matrix, the 5120-dimensional feature vector corresponding to the position of the joint sequence end marker is extracted as the first mask feature distribution vector; the first mask feature distribution vector is normalized by the Softmax function to obtain the normalized first mask feature distribution vector; the reference feature distribution vector obtained in S2012 is normalized by the Softmax function to obtain the normalized reference feature distribution vector. The Softmax function is calculated as follows: for an input vector of dimension 5120, the normalized value of its i-th element is equal to the power of e divided by the sum of the powers of e of all 5120 elements in the input vector, where e is a natural constant; the first KL divergence value between the normalized first mask feature distribution vector and the normalized reference feature distribution vector is calculated. The formula for calculating the first KL divergence value is: ,in, Represents the normalized baseline characteristic distribution vector. represents the normalized first mask feature distribution vector, where i is the dimension index of the element in the distribution vector, and the value ranges from 1 to 5120; and Let the normalized probability values ​​of the baseline feature distribution and the first mask feature distribution in the i-th dimension be respectively, satisfying the following conditions: and ln represents the natural logarithm function with the natural constant e as the base. In this embodiment, the KL divergence value quantifies the degree of difference between the output feature distribution of the surrogate model and the complete input baseline feature distribution after the first modality semantic primitive sequence is masked. The larger the divergence value, the more severe the semantic information loss caused by masking this modality, and the greater the independent semantic contribution of this modality to the security recognition task. The first KL divergence value is converted into a first semantic influence factor through a mapping function. The first semantic influence factor is calculated as 1 - e. -第一KL散度值 The first semantic influence factor ranges from 0 to 1. When the first KL divergence value is 0, the first semantic influence factor is 0, indicating that the feature distribution output by the surrogate model after the first modality semantic primitive sequence of the mask is completely consistent with the baseline feature distribution, and the first modality has no independent semantic contribution to the security recognition task. When the first KL divergence value approaches infinity, the first semantic influence factor approaches 1, indicating that the feature distribution output by the surrogate model after the first modality semantic primitive sequence of the mask is significantly different from the baseline feature distribution, and the first modality has an irreplaceable independent semantic contribution to the security recognition task.

[0129] S203. Similarly, the second and third semantic influence factors are obtained in the same way as the first semantic influence factor, specifically as follows:

[0130] S2031. Perform a masking operation on the second modal semantic primitive sequence, input the masked second modal semantic primitive sequence, the unmasked first modal semantic primitive sequence, and the third modal semantic primitive sequence into the surrogate model, obtain the second mask feature distribution, calculate the second KL divergence value between the second mask feature distribution and the baseline feature distribution, and map the second KL divergence value to a second semantic influence factor through a preset exponential function. The value range of the second semantic influence factor is 0 to 1.

[0131] S2032. Perform a masking operation on the third modality semantic primitive sequence. Input the masked third modality semantic primitive sequence, the unmasked first modality semantic primitive sequence, and the second modality semantic primitive sequence into the surrogate model to obtain the third mask feature distribution. Calculate the third KL divergence value between the third mask feature distribution and the baseline feature distribution. Map the third KL divergence value to a third semantic influence factor using a preset exponential function. The value range of the third semantic influence factor is 0 to 1. In this embodiment, the second and third semantic influence factors are obtained in the same way as the first semantic influence factor. Those skilled in the art can make specific settings according to the method for obtaining the first semantic influence factor, which will not be further elaborated here.

[0132] Example 3

[0133] It should be further explained that the task attention coefficients for each modality obtained in this embodiment include:

[0134] S301. Obtain the cross-attention weight matrix generated by the multi-head cross-attention sublayer in each Transformer decoder layer during the security recognition task inference process of the back-end security recognition big model on the first modality semantic primitive sequence, the second modality semantic primitive sequence, and the third modality semantic primitive sequence. In this embodiment, taking a certain generation time step of the back-end security recognition big model in performing security recognition task inference as an example, the acquisition form of the cross-attention weight matrix is ​​illustrated as follows: Suppose that at this time step, the multimodal joint input sequence contains a total of 18 tags, corresponding to P=3 first modality semantic primitive tags, Q=3 second modality semantic primitive tags, R=4 third modality semantic primitive tags, and 8 sequence structure tags, and the total sequence length (P+Q+R+8)=18; take the attention weight matrix of the first attention head in the 32nd layer of the Transformer decoder layer of the back-end security recognition big model as an example of the cross-attention weight matrix. The dimension of this matrix is ​​18×18, and the element value in the i-th row and j-th column of the matrix represents The attention weight value of the label at position i to the label at position j, the sum of the 18 elements in each row is 1; the row index i and column index j of this matrix correspond to the label positions in the multimodal joint input sequence, where positions 1 to 3 correspond to the first modality semantic primitive labels, positions 4 to 6 correspond to the second modality semantic primitive labels, positions 7 to 10 correspond to the third modality semantic primitive labels, and positions 11 to 18 correspond to sequence structure labels such as the start label of each modality, the end label of each modality, the start label of the joint sequence, and the end label of the joint sequence; in the cross-attention weight matrix, rows 1-3 correspond to the first modality semantic primitive labels, rows 4-6 correspond to the second modality semantic primitive labels, rows 7-10 correspond to the third modality semantic primitive labels, and rows 11-18 correspond to the sequence structure labels;

[0135] S302. For each attention head in each Transformer decoder layer, separate the inter-modal cross-attention sub-matrix and the intra-modal self-attention sub-matrix from the cross-attention weight matrix of the attention head. The separation method is as follows:

[0136] S3021. Based on the modality attribution information of each marker in the multimodal joint input sequence, determine the modality category to which each marker belongs. The modality categories include four categories: first mode, second mode, third mode, and sequence structure marker.

[0137] S3022. For the cross-attention weight matrix of this attention head, its dimension is (P+Q+R+8)×(P+Q+R+8). The element in the i-th row and j-th column of the matrix represents the attention weight value of the tag at the i-th position to the tag at the j-th position. Elements whose modal category of query position i is different from that of key position j are included in the inter-modal cross-attention submatrix, and elements whose modal category of query position i is the same as that of key position j are included in the intra-modal self-attention submatrix. As a concrete example, continuing the processing of the aforementioned 35-second traffic accident scene video, suppose the multimodal joint input sequence contains 18 tags, corresponding to P=3 first-modal semantic primitive tags, Q=3 second-modal semantic primitive tags, R=4 third-modal semantic primitive tags, and 8 sequence structure tags, with a total sequence length of 18. Take the cross-attention weight matrix of the first attention head in the 32nd layer of the Transformer decoder layer. This matrix has a dimension of 18×18, where rows 1 to 3 correspond to the first-modal semantic primitive tags, rows 4 to 6 correspond to the second-modal semantic primitive tags, rows 7 to 10 correspond to the third-modal semantic primitive tags, and rows 11 to 18 correspond to the sequence structure tags such as the modality start tag, the modality end tag, the joint sequence start tag, and the joint sequence end tag. The column index and row index tag positions are completely consistent. Perform a separation operation on this matrix. The process is performed element-wise, row by row: Attention weight values ​​for query position i belonging to the first modality category (rows 1 to 3) and key position j belonging to the second modality category (columns 4 to 6), the third modality category (columns 7 to 10), or the sequence structure marker category (columns 11 to 18) are all assigned to the intermodal cross-attention submatrix corresponding to the first modality. Attention weight values ​​for query position i belonging to the first modality category (rows 1 to 3) and key position j also belonging to the first modality category (columns 1 to 3) are assigned to the intramodal self-attention submatrix corresponding to the first modality. Similarly, the same separation operation is performed on the second and third modalities. Finally, the intermodal cross-attention submatrix and intramodal self-attention submatrix corresponding to the three modalities are separated from the 18×18 cross-attention weight matrix for subsequent calculation of the average cross-attention contribution and average self-attention contribution of each modality in this attention head.

[0138] S303. For each attention head, calculate the intermodal cross-attention contribution value of the first modality, the second modality, and the third modality in that attention head. The calculation method is as follows:

[0139] For the first modality, the attention weights of all the positions marked with the first modality as query positions and non-first modality as key positions in the inter-modal cross-attention submatrix are summed to obtain the total cross-attention of the first modality in the attention head. The total cross-attention is divided by the total number of rows in the attention head where the first modality is marked as query positions to obtain the average cross-attention contribution of the first modality in the attention head.

[0140] For the second modality, following the same acquisition method as the first modality, the attention weight values ​​of all those marked with the second modality as the query position and those marked with non-second modality as the key position in the inter-modal cross-attention submatrix are summed to obtain the total cross-attention of the second modality in the attention head. The total cross-attention is then divided by the total number of rows in the attention head marked with the second modality as the query position to obtain the average cross-attention contribution value of the second modality in the attention head.

[0141] For the third modality, following the same acquisition method as the first modality, all attention weight values ​​in the inter-modal cross-attention sub-matrix with the third modality marker as the query position and non-third modality marker as the key position are summed to obtain the total cross-attention value of the third modality in this attention head. The total cross-attention value is then divided by the total number of rows in this attention head where the third modality marker is the query position to obtain the average cross-attention contribution value of the third modality in this attention head. As a specific example, continuing the processing process of the aforementioned 35-second traffic accident scene video, the inter-modal cross-attention sub-matrix of the first attention head after separation in the 32nd layer Transformer decoder layer is used for calculation: the first modality semantic primitive marker corresponds to rows 1 to 3, and there are 3 rows of attention weight rows where the query position is the first modality. The 15 element values ​​from column 4 to column 18 in row 1 are summed to obtain 0.45, the 15 element values ​​from column 4 to column 18 in row 2 are summed to obtain 0.41, and the 15 element values ​​from column 4 to column 18 in row 3 are summed to obtain 0.41. The sum of these three values ​​is 0.43, and the total cross-attention of the first modality in this attention head is 1.29. Dividing this by the total number of rows where the first modality's tags are used as query positions (3), we get the average cross-attention contribution of the first modality in this attention head: 0.43. The second modality's semantic primitive tags correspond to rows 4 to 6. The query positions are the attention weight rows for the second modality (3 rows in total). The sum of the 15 element values ​​in columns 1 to 3 and 7 to 18 of row 4 is 0.42. The sum of the 15 element values ​​in columns 1 to 3 and 7 to 18 of row 5 is... To reach 0.38, the sum of the 15 element values ​​from columns 1 to 3 and columns 7 to 18 in row 6 is 0.35. The sum of these three is 1.15, which is the total cross-attention value of the second modality in this attention head. Dividing this by the total number of rows where the second modality is used as the query position (3), we get the average cross-attention contribution value of the second modality in this attention head, which is 0.38. The third modality semantic primitives correspond to rows 7 to 10. The query position is the attention weight row of the third modality (4 rows in total). The sum of the 14 element values ​​from columns 1 to 6 and columns 11 to 18 in row 7 is... The sum of the 14 element values ​​in rows 8 (columns 1-6 and 11-18) is 0.36. Similarly, the sum of the 14 element values ​​in rows 9 (columns 1-6 and 11-18) is 0.34. Finally, the sum of the 14 element values ​​in rows 10 (columns 1-6 and 11-18) is 0.33. The total sum of these four sums is 1.42, which is the total cross-attention value of the third modality in this attention head. Dividing this by the total number of rows (4) where the third modality is used as the query position, we obtain the average cross-attention contribution value of the third modality in this attention head, which is 0.36. By traversing all 32 layers (1280 attention heads) in this way, we can obtain the total average cross-attention contribution value of each modality across all attention heads.

[0142] S304. For each attention head, calculate the intramodal self-attention contribution values ​​of the first, second, and third modes within that attention head. The calculation method is as follows:

[0143] For the first mode, the attention weight values ​​of all the attention positions marked as query positions and key positions in the self-attention submatrix within the mode are summed after excluding the attention weight values ​​of each position to itself. This yields the total self-attention of the first mode in the attention head. The total self-attention is then divided by the total number of rows in the attention head marked as query positions to obtain the average self-attention contribution value of the first mode in the attention head.

[0144] For the second modality, following the same acquisition method as the first modality, the attention weight values ​​of all positions marked as query locations and key locations in the self-attention submatrix within the modality are summed after excluding the attention weight values ​​of each location to itself. This yields the total self-attention of the second modality in the attention head. The total self-attention is then divided by the total number of rows in the attention head marked as query locations to obtain the average self-attention contribution value of the second modality in the attention head.

[0145] For the third modality, following the same acquisition method as the first modality, the attention weight values ​​of all positions marked with the third modality as query positions and positions marked with the third modality as key positions in the self-attention sub-matrix within the modality are summed after excluding the attention weight values ​​of each position to itself. This yields the total self-attention of the third modality in the attention head. The total self-attention is then divided by the total number of rows in the attention head marked with the third modality as query positions to obtain the average self-attention contribution value of the third modality in the attention head.As a concrete example, continuing the processing of the aforementioned 35-second traffic accident scene video, following S303's calculation of the same attention head: the first modal semantic primitive tag corresponds to rows 1 to 3, and its intramodal self-attention sub-matrix contains 9 elements from columns 1 to 3 of rows 1 to 3. The diagonal elements are 0.12 of column 1 in row 1, 0.18 of column 2 in row 2, and 0.16 of column 3 in row 3. The non-diagonal elements, namely 0.28 of column 2 in row 1, 0.15 of column 3 in row 1, 0.20 of column 1 in row 2, 0.22 of column 3 in row 2, 0.14 of column 1 in row 3, and 0.21 of column 2 in row 3, are summed to obtain the first... The total self-attention of the modality in this attention head is 1.20. Dividing this by the total number of rows where the first modality is used as the query position (3), we get the average self-attention contribution of the first modality in this attention head as 0.40. The semantic primitives of the second modality correspond to rows 4 to 6. Its modal self-attention sub-matrix contains 9 elements from columns 4 to 6 of rows 4 to 6. The diagonal elements are 0.25 of column 4 of row 4, 0.30 of column 5 of row 5, and 0.24 of column 6 of row 6. The off-diagonal elements are 0.20 of column 5 of row 4, 0.15 of column 6 of row 4, 0.18 of column 4 of row 5, 0.16 of column 6 of row 5, and 0 of column 4 of row 6. 15. Summing the values ​​of 0.22 in row 6 and column 5 yields a total self-attention value of 1.06 for the second modality in this attention head. Dividing this by the total number of rows where the second modality is used as the query position (3), we get an average self-attention contribution value of 0.35 for the second modality in this attention head. The semantic primitives of the third modality correspond to rows 7 to 10. Its intra-modal self-attention sub-matrix contains 16 elements from columns 7 to 10 of rows 7 to 10. The diagonal elements are 0.18 in column 7 of row 7, 0.16 in column 8 of row 8, 0.22 in column 9 of row 9, and 0.20 in column 10 of row 10. The off-diagonal elements are 0.15 in column 8 of row 7. The values ​​of the following 12 off-diagonal elements are summed: 0.13 (row 7, column 9), 0.12 (row 7, column 10), 0.14 (row 8, column 7), 0.17 (row 8, column 9), 0.14 (row 8, column 10), 0.16 (row 9, column 7), 0.18 (row 9, column 8), 0.15 (row 9, column 10), 0.13 (row 10, column 7), 0.16 (row 10, column 8), and 0.21 (row 10, column 9). The total self-attention value of the third modality in this attention head is 1.84. Dividing this by the total number of rows (4) where the third modality is used as the query position, the average self-attention contribution value of the third modality in this attention head is 0.46. By traversing all 32 layers (1280 attention heads) in this way, the total average self-attention contribution value of each modality in all attention heads can be obtained.

[0146] S305. For the first modality, the average cross-attention contribution value and average self-attention contribution value in all attention heads of all layers are weighted and summed according to the preset cross-attention weight coefficient and self-attention weight coefficient to obtain the original task attention value of the first modality.

[0147] For the second modality, following the same acquisition method as the first modality, the average cross-attention contribution value and average self-attention contribution value in all attention heads of all layers are weighted and summed according to the cross-attention weight coefficient and the self-attention weight coefficient to obtain the original task attention value of the second modality;

[0148] For the third modality, following the same acquisition method as the first modality, its average cross-attention contribution value and average self-attention contribution value in all attention heads of all layers are weighted and summed according to the cross-attention weight coefficient and the self-attention weight coefficient to obtain the original task attention value of the third modality;

[0149] The cross-attention weight coefficient is used to characterize the importance of intermodal semantic association in the security recognition task, and the self-attention weight coefficient is used to characterize the importance of intramodal semantic integrity in the security recognition task. The sum of the cross-attention weight coefficient and the self-attention weight coefficient is 1. As a specific example, continuing the processing of the aforementioned 35-second traffic accident scene video, the entire 32 layers of the backend security recognition large model's Transformer decoder layers, each containing 40 attention heads, totaling 1280 attention heads, are traversed. For each attention head, the average cross-attention contribution value and average self-attention contribution value of the first, second, and third modalities under that attention head are obtained according to the aforementioned calculation rules for the cross-attention contribution value between modalities and the self-attention contribution value within a modality. The average cross-attention contribution value of the same modality corresponding to all 1280 attention heads is summed and divided by the total number of heads, 1280, to obtain the global average cross-attention contribution value of the corresponding modality. Similarly, the global average self-attention contribution value of each modality is calculated. The calculated global average cross-attention contribution value of the first modality is 0. 42. The global average self-attention contribution value is 0.38, the global average cross-attention contribution value of the second modality is 0.33, the global average self-attention contribution value is 0.285, and the global average cross-attention contribution value of the third modality is 0.51 and the global average self-attention contribution value is 0.45. Weighted summation is performed according to the preset cross-attention weight coefficient of 0.6 and self-attention weight coefficient of 0.4. The original task attention value of the first modality is calculated as 0.42×0.6+0.38×0.4=0.404, the original task attention value of the second modality is 0.33×0.6+0.285×0.4=0.312, and the original task attention value of the third modality is 0.51×0.6+0.45×0.4=0.486. This completes the calculation of the original task attention values ​​for the three modalities, which are used for subsequent dynamic maximum and minimum value normalization processing.

[0150] S306. Using the original task attention values ​​of the first modality, the second modality, and the third modality corresponding to the same set of multimodal data as the sole calculation benchmark, extract the maximum value among the three as the normalization upper limit and the minimum value as the normalization lower limit, and perform group-by-group dynamic maximum and minimum value normalization processing on the three original task attention values; the calculation formula for the single modality task attention coefficient is: task attention coefficient = (original task attention value of the modality - normalization lower limit) ÷ (normalization upper limit - normalization lower limit); after calculation, the three normalized coefficient values ​​are all mapped to the range of 0 to 1, and the normalized results are used as the task attention coefficients of the first modality, the second modality, and the third modality, respectively.

[0151] It should be further explained that this embodiment applies a second deep compression strategy to the semantically redundant region, including:

[0152] S401. Obtain the first transmission importance coefficient corresponding to the first modal semantic primitive sequence, the second transmission importance coefficient corresponding to the second modal semantic primitive sequence, and the third transmission importance coefficient corresponding to the third modal semantic primitive sequence. Sort the three transmission importance coefficients in descending order of value, determine the modal semantic primitive sequence with the smallest transmission importance coefficient, determine the modal semantic primitive sequence with the largest transmission importance coefficient among the remaining two modal semantic primitive sequences as the first sequence, and determine the modal semantic primitive sequence with the second largest transmission importance coefficient as the second sequence.

[0153] S402. For the first sequence and the second sequence, identify semantically redundant regions in the following manner:

[0154] S4021. The compression processing order is determined as follows: first, the second sequence is screened for non-redundant semantic primitives, and then the first sequence is identified for semantic redundancy based on the screening results. In this embodiment, the compression processing order is set based on the following: the transmission importance coefficient of the second sequence is lower than that of the first sequence. In the security identification task of the backend security identification large model, the independent semantic contribution of the second sequence is relatively small. Prioritizing the screening of non-redundant semantic primitives for the second sequence can lock in the core semantic information that cannot be compressed, providing a stable condition reference for subsequent cross-modal redundancy determination of the first sequence. If the first sequence is identified for redundancy first, the semantic information of the second sequence on which it depends may change before and after compression, causing the reference target of the cross-modal reference pointer to be removed in subsequent compression, resulting in reference failure.

[0155] S4022. Traverse each semantic primitive in the second sequence, input the semantic primitive into a pre-trained self-information entropy calculation model, and calculate the independent information content of the semantic primitive. The independent information content is the self-information entropy of the semantic primitive under the condition that the information is provided only by the second sequence itself. When the independent information content is greater than a preset independent information content threshold, the semantic primitive is determined as a non-redundant semantic primitive. When the independent information content is less than or equal to the preset independent information content threshold, the semantic primitive is marked as a candidate compressible semantic primitive. After traversal, a non-redundant semantic primitive set is formed by all non-redundant semantic primitives in the second sequence. In this embodiment, the preset independent information content threshold is set to 0.35. This threshold is determined based on the statistical distribution characteristics of semantic primitive information content in a security identification scenario. Specifically, the normalized independent information content values ​​of all semantic primitives in the semantic primitive sequence dataset generated by the proxy model are statistically analyzed, and the 30th percentile of the cumulative distribution function is used as the threshold. This ensures that approximately 70% of the semantic primitives are deemed to have sufficient independent semantic information content and are retained, while approximately 30% are marked as candidate compressible semantic primitives for subsequent compression. Differential compression processing is then performed on the candidate compressible semantic primitives according to their independent information content: when the independent information content of a candidate compressible semantic primitive is greater than or equal to... When the independent information content is in the preset low information content range of 0.2 and less than 0.35, the complete feature vector of the semantic primitive is replaced by an all-zero vector, while its timestamp field and modality identifier field are retained to maintain cross-modal temporal alignment. When the independent information content of the candidate compressible semantic primitive is in the preset extremely low information content range of less than 0.2, a corresponding single semantic primitive structured summary text is generated to replace the complete feature vector. The structured summary only records the core event types and key entity attributes covered by the semantic primitive and does not contain detailed semantic features. In this embodiment, the independent information content of the candidate compressible semantic primitive U1-2 is 0.28, which is in the preset low information content range, so its complete feature vector is replaced by a zero vector.

[0156] In this embodiment, the pre-trained self-information entropy calculation model is constructed as follows: a feedforward neural network consisting of an input layer, two hidden layers, and an output layer is used. The input layer has 5120 nodes and receives a single semantic primitive vector with a dimension of 5120. The first hidden layer contains 1024 nodes and uses the ReLU activation function, the second hidden layer contains 256 nodes and uses the ReLU activation function, and the output layer contains 1 node and uses the Sigmoid activation function, outputting the normalized independent information value of the semantic primitive, with a value range of [0,1]. The training method of the self-information entropy calculation model is as follows: a semantic primitive sequence dataset generated by the proxy model for multimodal sample data in a security identification scenario is obtained. This dataset contains N semantic primitive vectors, and the frequency of each semantic primitive vector appearing in the dataset is counted. The negative natural logarithm of that frequency As the true self-information entropy value of the semantic primitive, the minimum value min_val and the maximum value max_val of the true self-information entropy values ​​of all semantic primitives are calculated. Each true self-information entropy value is then normalized to the [0,1] interval using the formula (label-min_val) / (max_val-min_val) as the training label. In the training data statistics process of the self-information entropy calculation model in this embodiment, count represents the total number of times the specific semantic primitive vector to be calculated appears in all N semantic primitive sequence samples generated by the proxy model, i.e., the frequency of the same vector. In the training process of the self-information entropy calculation model in this embodiment, label represents the normalized true self-information entropy value used to supervise model training, i.e., the training label. Specifically, for each semantic primitive vector in the semantic primitive sequence dataset, its frequency of occurrence in the dataset is first counted. and with the negative natural logarithm of that frequency This serves as the true self-information entropy value of the semantic primitive; subsequently, the minimum value (min_val) and maximum value (max_val) of the true self-information entropy values ​​of all semantic primitives are calculated, and each true self-information entropy value is processed according to the formula... The vector is normalized to its maximum and minimum values, mapping it to the [0,1] interval. The mapped value is the label corresponding to the semantic primitive vector. During model training, this label serves as a supervision signal, and the mean squared error loss is calculated between it and the predicted normalized independent information value output by the model. This drives the model to learn the mapping relationship from the semantic primitive vector to its normalized self-information entropy value.

[0157] The semantic primitive vectors are used as input, and the corresponding normalized true self-information entropy values ​​are used as labels. Mean squared error is employed. As the loss function, where M is the batch size, pred_i is the model prediction value, and label_i is the normalized label value, the Adam optimizer is used to iteratively update the network parameters with a learning rate of 0.0001 and a batch size of 64. After each training round, the mean squared error value on the validation set is calculated. Training is terminated when the mean squared error value on the validation set does not decrease for 10 consecutive rounds. The model parameters with the lowest mean squared error value on the validation set are saved to obtain the pre-trained self-information entropy calculation model.

[0158] S4023. Traverse each semantic primitive in the first sequence, input the semantic primitive into a pre-trained conditional entropy calculation model, use the non-redundant semantic primitive set as a condition, and use the conditional entropy calculation model to calculate the conditional information content of the semantic primitive, where the conditional information content is the conditional entropy of the semantic primitive under the given non-redundant semantic primitive set; calculate the difference between the independent information content and the conditional information content of the semantic primitive; when the difference is less than a preset redundancy difference threshold, determine that the semantic primitive is a redundant semantic primitive and classify it into a semantic redundancy region; when the difference is greater than or equal to the preset redundancy difference threshold, determine that the semantic primitive is a non-redundant semantic primitive. In this embodiment, the preset redundancy difference threshold is set to 0.20. The basis for setting this threshold is as follows: on the labeled verification dataset of the security identification scenario, redundant labeling is performed on cross-modal semantic primitive pairings, the difference between the independent information content and the conditional information content of each pair of semantic primitives is calculated, and the lowest difference boundary point that makes the redundancy judgment recall rate not less than 90% and the precision rate not less than 85% is selected as the threshold; when the difference is less than 0.20, it indicates that the conditional information content and independent information content of the semantic primitive under the given non-redundant semantic primitive set are close, and its semantic information has been fully expressed by the non-redundant semantic primitives in the second sequence, and it can be safely replaced by cross-modal reference pointers without causing the loss of security identification semantic information.

[0159] In this embodiment, the pre-trained conditional entropy calculation model is constructed as follows: a dual-input feedforward neural network architecture is adopted, which consists of a conditional input branch, a target input branch, and a cascaded fusion output layer. The conditional input branch receives a conditional feature vector obtained by concatenating the vectors of all non-redundant semantic primitives in the non-redundant semantic primitive set. The dimension of the conditional feature vector is K×5120, where K is the number of non-redundant semantic primitives in the non-redundant semantic primitive set. The conditional input branch includes a global average pooling layer and two fully connected layers. The global average pooling layer compresses the K×5120 conditional feature vector along the semantic primitive dimension into a 1×5120 pooling vector. The first fully connected layer contains 1024 nodes and... The system employs the ReLU activation function. The second fully connected layer contains 256 nodes and uses the ReLU activation function, outputting a 256-dimensional conditional encoding vector. The target input branch receives a single semantic primitive vector of dimension 5120 and contains two fully connected layers: the first fully connected layer contains 1024 nodes and uses the ReLU activation function, and the second fully connected layer contains 256 nodes and uses the ReLU activation function, outputting a 256-dimensional target encoding vector. The fusion output layer concatenates the 256-dimensional conditional encoding vector with the 256-dimensional target encoding vector to obtain a 512-dimensional fusion vector. This 512-dimensional fusion vector is then input into a fully connected hidden layer containing 128 nodes and uses the ReLU activation function, followed by further input... An output layer containing one node and using the Sigmoid activation function outputs the normalized conditional information value of the semantic primitive, with a value range of [0,1]. The training method of the conditional entropy calculation model is as follows: obtain the semantic primitive sequence dataset generated by the proxy model for multimodal sample data in a security recognition scenario; for each semantic primitive vector v_i in the dataset, specify another semantic primitive sequence as the condition set C; and count the conditional occurrence frequency p(v_i|C)=count(v_i,C) / total(C) of the semantic primitive vector v_i under the condition set C is known, where count(v_i,C) represents the number of samples in which the semantic primitive vector v_i appears simultaneously in the condition set C. The total number of occurrences in the condition set C is represented by `total(C)`. The negative natural logarithm of the condition occurrence frequency, -ln(p(v_i|C)), is used as the true conditional information value for this semantic primitive. The minimum and maximum values ​​of all true conditional information values, `c_min` and `c_max`, are calculated. Each true conditional information value, `c_label`, is normalized to the [0,1] interval using the formula (c_label-c_min) / (c_max-c_min) as the training label. The concatenation of all semantic primitive vectors in the condition set C is used as the conditional input, the semantic primitive vector `v_i` is used as the target input, and the corresponding normalized true conditional information value is used as the label. Mean squared error is used. As the loss function, where D is the batch size, c_pred_j is the conditional information content value predicted by the model, and c_label_j is the normalized label value, the Adam optimizer is used to iteratively update the network parameters with a learning rate of 0.0001 and a batch size of 32. After each training round, the mean squared error value on the validation set is calculated. Training is terminated when the mean squared error value on the validation set does not decrease for 10 consecutive rounds. The model parameters with the lowest mean squared error value on the validation set are saved to obtain the pre-trained conditional entropy calculation model.

[0160] S403. For each redundant semantic primitive in the semantic redundancy region, perform a second deep compression strategy: do not transmit the complete vector representation of the redundant semantic primitive, generate a cross-modal reference pointer to replace the redundant semantic primitive, the cross-modal reference pointer includes the modality identifier of the referenced modality, the index number of the referenced semantic primitive in the set of non-redundant semantic primitives, and a reference confidence score; for the non-redundant semantic primitives in the first sequence, retain their complete vector transmission. In this embodiment, the reference confidence score is calculated as follows: the difference between the independent information and the conditional information corresponding to the redundant semantic primitive is taken, and the reference confidence score is calculated according to formula 1 - (difference / preset redundancy difference threshold). The reference confidence score ranges from (0,1). The larger the value, the higher the reliability of the cross-modal reference pointer. When the receiver performs semantic recovery, it performs weighted fusion of the vector value of the referenced semantic primitive and the locally reconstructed semantic vector according to the reference confidence score. The weights are the reference confidence score and 1 - reference confidence score, respectively, to achieve robust reconstruction of cross-modal semantics.

[0161] As a concrete example, continuing the processing of the aforementioned 35-second traffic accident scene video: The first transmission importance coefficient of the first modal semantic primitive sequence (U1-1, U1-2, U1-3) is calculated to be 0.33, the second transmission importance coefficient of the second modal semantic primitive sequence (U2-1, U2-2, U2-3) is 0.00, and the third transmission importance coefficient of the third modal semantic primitive sequence (U3-1, U3-2, U3-3, U3-4) is 0.68. The three transmission importance coefficients are sorted from largest to smallest as {0.68, 0.33, 0.00}. The second modal semantic primitive sequence with the smallest transmission importance coefficient is selected for the first deep compression strategy. The third modal semantic primitive sequence with the largest transmission importance coefficient is selected as the first sequence, and the first modal semantic primitive sequence with the second largest transmission importance coefficient is selected as the second sequence. First, the second sequence, i.e. the first modality semantic primitive sequence, is screened for non-redundant semantic primitives. U1-1, U1-2, and U1-3 are input into the pre-trained self-information entropy calculation model in sequence, and the normalized independent information values ​​are 0.52, 0.28, and 0.61, respectively. The preset independent information threshold is 0.35. U1-1 and U1-3 are determined to be non-redundant semantic primitives, and U1-2 is a candidate compressible semantic primitive. The set of non-redundant semantic primitives is composed of U1-1 and U1-3. Then, semantic redundancy region identification is performed on the first sequence, i.e., the third modality semantic primitive sequence. U3-1, U3-2, U3-3, and U3-4 are sequentially input into the pre-trained conditional entropy calculation model. Using the non-redundant semantic primitive set {U1-1, U1-3} as conditions, the normalized conditional information values ​​of each semantic primitive are 0.43, 0.68, 0.15, and 0.72, respectively, and the independent information values ​​are 0.45, 0.72, 0.18, and 0.75, respectively. The calculated difference values ​​are 0.02 for U3-1, 0.04 for U3-2, 0.03 for U3-3, and 0.03 for U3-4. The preset redundancy difference threshold is 0.20. If the difference between U3-1 and U3-3 is less than the redundancy difference threshold, it is classified as a semantic redundancy region. If the difference between U3-2 and U3-4 is greater than or equal to the redundancy difference threshold, it is classified as a non-redundant semantic primitive.The second deep compression strategy is executed on U3-1 and U3-3 in the semantically redundant region. U3-1 generates a cross-modal reference pointer {referenced modality identifier: first modality, referenced semantic primitive index number: 1, reference confidence score: 1-(0.02 / 0.20)=0.90}, and U3-3 generates a cross-modal reference pointer {referenced modality identifier: first modality, referenced semantic primitive index number: 2, reference confidence score: 1-(0.03 / 0.20)=0.85}. The complete vector representations of U3-1 and U3-3 are not transmitted. For the non-redundant semantic primitives U3-2 and U3-4 in the first sequence, their complete vectors are preserved and transmitted. For the non-redundant semantic primitives U1-1 and U1-3 in the second sequence, the complete vectors are preserved and transmitted. The candidate compressible semantic primitive U1-2 is replaced with a zero vector, thus completing the entire execution process of the second deep compression strategy.

[0162] It should be further explained that this embodiment performs priority queue scheduling and retransmission control to maximize the throughput of security identification semantics, including:

[0163] S501. Add a semantic importance tag to each data packet after adaptive compression processing. The semantic importance tag is generated by: obtaining the transmission importance coefficient corresponding to the semantic primitives contained in the data packet, mapping the transmission importance coefficient to a preset semantic importance level range, and obtaining the semantic importance level value of the data packet; the preset semantic importance level range includes three levels: high importance level, medium importance level, and low importance level, wherein the transmission importance coefficient corresponding to the high importance level is greater than or equal to 0.60, the transmission importance coefficient corresponding to the medium importance level is greater than or equal to 0.30 and less than 0.60, and the transmission importance coefficient corresponding to the low importance level is less than 0.30.

[0164] S502. Add a cross-modal association identifier to each data packet after adaptive compression. The cross-modal association identifier is generated as follows: obtain the group identifier of the aligned cross-modal fragment group to which the data packet belongs before compression, and add the group identifier as the cross-modal association identifier to the header field of the data packet, so that each modal data packet belonging to the same aligned cross-modal fragment group carries the same cross-modal association identifier. The receiving end performs cross-modal association and reorganization of data packets of different modalities according to the cross-modal association identifier. For data packets that only contain structured summary text fields after processing by the first deep compression strategy, the group identifier of the aligned cross-modal fragment group to which it belongs is also added as the cross-modal association identifier to maintain the integrity of the cross-modal association relationship.

[0165] S503. Based on the semantic importance label and the current network window status, the data packets to be sent are allocated to different priority levels of the priority queue in descending order of semantic importance level value. Among them, high-importance data packets are allocated to the high-priority queue, medium-importance data packets are allocated to the medium-priority queue, and low-importance data packets are allocated to the low-priority queue. Within the same priority level, the data packets are queued according to their timestamp order.

[0166] S504. Obtain the current network window status, which includes the current congestion window size and the current available sending window size. When the current available sending window size is sufficient to accommodate all data packets to be sent in the high-priority queue, send all data packets in the high-priority queue first, and allocate the remaining window space to the data packets in the medium-priority queue and the low-priority queue in sequence. When the current available sending window size is insufficient to accommodate all data packets to be sent in the high-priority queue, send the data packets in the high-priority queue in sequence according to the timestamp until the window is exhausted. Perform a temporary storage operation on the high-priority data packets, medium-priority data packets and low-priority data packets that could not be sent, and wait for the next sending cycle to re-schedule the priority queues.

[0167] S505. For data packets that have been sent and are awaiting acknowledgment, perform differentiated retransmission control based on their semantic importance tags: When a high-importance data packet does not receive an acknowledgment response from the receiver within a preset retransmission timeout period, a retransmission operation is immediately performed, with the maximum number of retransmissions set to the preset maximum number of retransmissions; when a medium-importance data packet does not receive an acknowledgment response within a preset retransmission timeout period, a retransmission operation is performed after a preset delayed retransmission time, with the maximum number of retransmissions set to the preset maximum number of retransmissions; when a low-importance data packet does not receive an acknowledgment response within a preset retransmission timeout period, no retransmission operation is performed, the data packet is marked as a transmission failure, and removed from the sending queue. In this embodiment, the entire process of priority queue scheduling and differentiated retransmission control aims to maximize the semantic throughput of security identification. The semantic throughput of security identification refers to the amount of effective semantic information successfully transmitted per unit time that can support the backend security identification model to achieve a preset recognition accuracy rate. It is quantified in "number of effective semantic primitives per second," and its statistical rules are as follows: a sliding time window with a duration of 1 second and a step size of 100 milliseconds is used as the statistical period. Within each statistical window, the following conditions are met simultaneously: the receiver returns an acknowledgment response; the transmission importance coefficient of the corresponding semantic primitive's modality is not lower than a preset effective judgment threshold of 0.30; and the semantic primitive is preserved. The total number of semantic primitives that meet the three conditions of retaining complete vector representation or lossless recovery of semantic information through cross-modal reference pointers is divided by the statistical window duration to obtain the current security identification semantic throughput value. The sending window resources are allocated according to the importance level from high to low through a priority queue scheduling mechanism to compress the queuing waiting delay of high-value semantic primitives. The differentiated retransmission control mechanism ensures reliable retransmission of high-importance data packets and abandons retransmission of low-importance data packets to release bandwidth resources. Under the condition of limited network, the number of effective semantic primitives delivered per unit time and semantic effectiveness are continuously improved, and the security identification semantic throughput is maximized.

[0168] As a specific example, continuing the processing of the aforementioned 35-second traffic accident scene video: After adaptive compression, the non-redundant semantic primitives U3-2 and U3-4 in the third modality semantic primitive sequence retain complete vector transmission, with a transmission importance coefficient of 0.68, mapping to a high importance level. A semantic importance label is added, indicating a high importance level, and the group identifier of the aligned cross-modal fragment group is added as a cross-modal association identifier. The non-redundant semantic primitives U1-1 and U1-3 in the first modality semantic primitive sequence retain complete vector transmission, with a transmission importance coefficient of 0.33, mapping to a medium importance level. A semantic importance label is added, indicating a medium importance level. The second modality semantic primitive sequence, after processing with the first deep compression strategy, retains only the structured summary text field, with a transmission importance coefficient of 0.00, mapping to a low importance level. A semantic importance label is added, indicating a low importance level. The transmission importance coefficient of the cross-modal reference pointer data packet is consistent with the referenced first sequence non-redundant semantic primitives, mapping to a high importance level. Assuming the current network window has an available sending window size that can accommodate four complete semantic primitive data packets, U3-2, U3-4, and two cross-modal reference pointer data packets in the high-priority queue are sent first. U1-1 and U1-3 in the medium-priority queue are temporarily stored for the next sending cycle. The structured digest text data packets in the low-priority queue are sent last. In retransmission control, if the high-priority data packet U3-2 does not receive an acknowledgment within the retransmission timeout period, a retransmission operation is performed immediately. If the medium-priority data packet U1-1 does not receive an acknowledgment within the retransmission timeout period, a retransmission operation is performed after a preset delay time. If the low-priority structured digest text data packet does not receive an acknowledgment, no retransmission operation is performed, and it is marked as a transmission failure. This maximizes the throughput of secure semantic recognition under constrained network conditions.

[0169] Example 4

[0170] Please see Figure 3 Another embodiment of the present invention provides a multimodal secure data transmission system based on differentiated bitrate allocation, comprising:

[0171] The decoupling module is configured to acquire synchronously collected multimodal data streams and decouple each modal data stream into a corresponding semantic primitive sequence; the multimodal data streams include a first modal data stream, a second modal data stream, and a third modal data stream;

[0172] The evaluation module is configured to use a proxy model aligned with the backend security identification large model task to perform masking operations on the semantic primitive sequences of each modality, and generate semantic influence factors for each modality based on the distribution deviation of the output features of the proxy model before and after masking.

[0173] The attention weighting module is configured to extract the task attention coefficients corresponding to each modality from the multi-head cross-attention weights of the backend security identification big model, and multiply the semantic influence factor with the corresponding task attention coefficients to obtain the transmission importance coefficients of each modality.

[0174] The compression decision module is configured to detect the network channel state, perform adaptive compression based on the network channel state and the transmission importance coefficient, and obtain compressed data packets; the network channel state includes available bandwidth, round-trip time, and packet loss rate.

[0175] The transmission control module is configured to add semantic importance tags and cross-modal association identifiers to the compressed data packets, and perform priority queue scheduling and retransmission control based on the semantic importance tags and the current network window state to maximize the throughput of security identification semantics.

[0176] Example 5

[0177] An electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement a multimodal secure data transmission method based on differentiated bitrate allocation.

[0178] A computer-readable storage medium having computer instructions stored thereon, which, when executed, perform a method for multimodal secure data transmission based on differentiated bitrate allocation.

[0179] The embodiments of the present invention have been described above with reference to the accompanying drawings. However, the present invention is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments under the guidance of the present invention without departing from the spirit and scope of the present invention. All of these variations are within the scope of protection of the present invention.

Claims

1. A multimodal secure data transmission method based on differentiated bit rate allocation, characterized in that, include: Acquire synchronously collected multimodal data streams and decouple each modal data stream into a corresponding semantic primitive sequence; the multimodal data streams include a first modal data stream, a second modal data stream, and a third modal data stream; Using a proxy model aligned with the backend security identification large model task, a mask is applied to the semantic primitive sequence of each modality. Based on the distribution deviation of the output features of the proxy model before and after masking, a semantic influence factor for each modality is generated. From the multi-head cross-attention weights of the backend security identification big model, the task attention coefficients of each modality are obtained, and the semantic influence factor is multiplied by the corresponding task attention coefficients to obtain the transmission importance coefficients of each modality. The network channel state is detected, and adaptive compression is performed based on the network channel state and the transmission importance coefficient to obtain compressed data packets; The network channel status includes available bandwidth, round-trip time, and packet loss rate; Add semantic importance tags and cross-modal association identifiers to the compressed data packets, and perform priority queue scheduling and retransmission control based on the semantic importance tags and the current network window state.

2. The multimodal secure data transmission method based on differentiated bitrate allocation as described in claim 1, characterized in that, The adaptive compression based on the network channel state and the transmission importance coefficient includes: When the available bandwidth is lower than a preset bandwidth threshold and the round-trip delay is higher than a preset delay threshold, a first deep compression strategy is executed on the modal data stream with the lowest transmission importance coefficient. For the two remaining modal data streams other than the modal data stream with the lowest transmission importance coefficient, the modal semantic primitive sequence with the highest transmission importance coefficient is determined as the first sequence, and the other remaining modal semantic primitive sequence is determined as the second sequence. Semantic redundancy regions are identified by comparing the difference between self-information entropy and conditional entropy, and a second deep compression strategy is applied to the semantic redundancy regions.

3. The multimodal secure data transmission method based on differentiated bitrate allocation as described in claim 2, characterized in that, The step of decoupling each modal data stream into a corresponding semantic primitive sequence includes: The first mode data stream, the second mode data stream, and the third mode data stream are time-series segmented to obtain the first initial segment sequence, the second initial segment sequence, and the third initial segment sequence; Each segment in the first initial segment sequence, and segments whose time windows overlap with those in the second and third initial segment sequences, are respectively denoted as cross-modal segment pairs; each cross-modal segment pair is input into a pre-trained cross-modal temporal alignment model, and the semantic alignment confidence and optimal time offset between the first and second modal segments and between the first and third modal segments in the cross-modal segment pair are calculated to generate a cross-modal alignment relationship matrix; Based on the cross-modal alignment relationship matrix, cross-modal segments whose semantic alignment confidence exceeds a preset confidence threshold and whose optimal time offset is within a preset offset tolerance range are identified as aligned related segments.

4. The multimodal secure data transmission method based on differentiated bitrate allocation as described in claim 3, characterized in that, The step of decoupling each modal data stream into a corresponding semantic primitive sequence further includes: All cross-modal segments within the same aligned and associated segment group are unified on the time axis to the smallest time interval covering all segments within the group, resulting in an aligned cross-modal segment group. For each modality segment within each aligned cross-modal segment group, the variance of the semantic vector within a single modal segment is calculated. When the variance exceeds a preset variance threshold, the corresponding single modal segment is further segmented until the variance of the semantic vector within each segmented sub-segment is lower than the preset variance threshold. When the semantic vector similarity between two adjacent sub-segments of the same modality exceeds a preset merging threshold, the two adjacent sub-segments of the same modality are merged into a single semantic primitive. The segments formed by further segmenting and merging each modality within the same aligned cross-modal segment group are respectively used as the first modal semantic primitive, the second modal semantic primitive, and the third modal semantic primitive, and arranged in timestamp order to form the first modal semantic primitive sequence, the second modal semantic primitive sequence, and the third modal semantic primitive sequence.

5. The multimodal secure data transmission method based on differentiated bitrate allocation as described in claim 4, characterized in that, The semantic influence factors for generating each modality include: The first modality semantic primitive sequence, the second modality semantic primitive sequence, and the third modality semantic primitive sequence are simultaneously input into the surrogate model, and the joint feature distribution output by the surrogate model is obtained as the baseline feature distribution. A masking operation is performed on the first modal semantic primitive sequence. The masked first modal semantic primitive sequence, along with the unmasked second and third modal semantic primitive sequences, are input into the surrogate model to obtain the first mask feature distribution. The first KL divergence value between the first mask feature distribution and the baseline feature distribution is calculated. The first KL divergence value is mapped to the first semantic influence factor through a preset exponential function.

6. The multimodal secure data transmission method based on differentiated bitrate allocation as described in claim 5, characterized in that, The process of obtaining the task attention coefficient for each modality includes: During the inference process of the security identification task, obtain the cross-attention weight matrix generated by the multi-head cross-attention sublayer in each Transformer decoder layer; For each attention head in each Transformer decoder layer, the intermodal cross-attention submatrix and the intramodal self-attention submatrix are separated from the cross-attention weight matrix of the attention head; For each attention head, calculate the intermodal cross-attention contribution values ​​of the first modality, the second modality, and the third modality in that attention head.

7. The multimodal secure data transmission method based on differentiated bitrate allocation as described in claim 6, characterized in that, The process of obtaining the task attention coefficients for each modality also includes: For each attention head, calculate the intramodal self-attention contribution values ​​of the first mode, the second mode, and the third mode in the attention head; The average cross-attention contribution value and average self-attention contribution value of the first modality in all attention heads of all layers are weighted and summed according to the preset cross-attention weight coefficient and self-attention weight coefficient to obtain the original task attention value of the first modality. The original task attention values ​​of the second and third modalities are obtained in the same way as the original task attention values ​​of the first modal. The normalized results of the original task attention values ​​of the first modality, the second modality, and the third modality are used as the task attention coefficients of the first modality, the second modality, and the third modality, respectively.

8. The multimodal secure data transmission method based on differentiated bitrate allocation as described in claim 7, characterized in that, The method of identifying semantically redundant regions by comparing the difference between self-information entropy and conditional entropy includes: The modal semantic primitive sequence with the largest transmission importance coefficient is determined as the first sequence, and the modal semantic primitive sequence with the second largest transmission importance coefficient is determined as the second sequence. Traverse each semantic primitive in the second sequence and calculate the independent information content of the semantic primitive. The independent information content is the self-information entropy of the semantic primitive under the condition that the information is provided only by the second sequence itself. Semantic primitives with independent information content greater than a preset independent information content threshold are determined as non-redundant semantic primitives. A set of non-redundant semantic primitives is formed by all non-redundant semantic primitives in the second sequence.

9. The multimodal secure data transmission method based on differentiated bitrate allocation as described in claim 8, characterized in that, The method of identifying semantically redundant regions by comparing the difference between self-information entropy and conditional entropy also includes: Traverse each semantic primitive in the first sequence, and calculate the conditional information content of the semantic primitive using the set of non-redundant semantic primitives as a condition. The conditional information content is the conditional entropy of the semantic primitive under the given set of non-redundant semantic primitives. Calculate the difference between the independent information content and the conditional information content of the semantic primitive. Semantic primitives whose difference is less than a preset redundancy difference threshold are identified as redundant semantic primitives, and a semantic redundancy region is formed by all redundant semantic primitives in the first sequence.

10. A multimodal secure data transmission system based on differentiated bitrate allocation, used to implement the multimodal secure data transmission method based on differentiated bitrate allocation as described in any one of claims 1-9, characterized in that, include: The decoupling module is configured to acquire synchronously collected multimodal data streams and decouple each modal data stream into a corresponding semantic primitive sequence; the multimodal data streams include a first modal data stream, a second modal data stream, and a third modal data stream; The evaluation module is configured to use a proxy model aligned with the backend security identification large model task to perform masking operations on the semantic primitive sequences of each modality, and generate semantic influence factors for each modality based on the distribution deviation of the output features of the proxy model before and after masking. The attention weighting module is configured to extract the task attention coefficients corresponding to each modality from the multi-head cross-attention weights of the backend security identification big model, and multiply the semantic influence factor with the corresponding task attention coefficients to obtain the transmission importance coefficients of each modality. The compression decision module is configured to detect the network channel state, perform adaptive compression based on the network channel state and the transmission importance coefficient, and obtain compressed data packets; The network channel status includes available bandwidth, round-trip time, and packet loss rate; The transmission control module is configured to add semantic importance tags and cross-modal association identifiers to the compressed data packets, and perform priority queue scheduling and retransmission control based on the semantic importance tags and the current network window state to maximize the throughput of security identification semantics.

Images