Multimodal data encryption transmission method and system

Abstract

The present invention discloses a multimodal data encryption transmission method and system, the method comprising: performing time-series synchronous slicing on multimodal data according to data types, extracting key frames of each slice after time-series synchronous slicing and associating them to the same timestamp, the data types of multimodal data including: video data, audio data and text data; generating an encryption key through cross-modal feature fusion based on the timestamp, signaling header and intra-frame data of the key frame, encrypting the header of the corresponding key frame to obtain an encrypted slice; transmitting the encrypted slice through the main transmission channel, generating an unencrypted virtual slice when the length of the slice does not match the minimum slice length of the corresponding data type, and transmitting the virtual slice through the auxiliary signaling channel to coordinate the transmission path. By binding the key frames of video, audio and text to the same timestamp through time-series slicing and generating a dynamic key based on cross-modal feature fusion, the problem of lack of coordination in multimodal data encryption is solved.

Description

Technical Field

[0001] The present invention relates to the field of data processing technology, and in particular to a multimodal data encryption transmission method and system. Background Art

[0002] With the rapid development of artificial intelligence and large-scale model technology, the need for collaborative processing and secure transmission of multimodal data such as video, audio, and text has become increasingly prominent. In scenarios such as remote interaction, intelligent monitoring, and virtual reality, multimodal data must maintain temporal synchronization and content integrity, while also facing security risks such as sensitive information leakage and transmission tampering. Traditional encryption schemes are usually designed for single-modal data, such as using stream encryption for video streams and end-to-end encryption for text. However, they lack cross-modal collaborative protection mechanisms, leading to the following key issues:

[0003] First, there's a lack of a multimodal collaborative encryption mechanism. Existing technologies struggle to uniformly handle the dynamic correlations between video keyframes, audio voiceprint features, and text semantic nodes. For example, video encryption fails to consider the temporal binding between keyframes, audio voiceprints, and text keywords, resulting in a loss of synchronization between cross-modal data after transmission. This disconnect reduces the identifiability and traceability of multimodal data when used for large-scale model training or generative applications.

[0004] Second, there's insufficient differentiated protection for keyframes. Traditional encryption methods (such as AES and RSA) uniformly encrypt the entire data, without distinguishing the security levels of keyframes from non-keyframes. Keyframes in videos, voiceprint feature frames in audio, and semantic nodes in text contain high-value information, yet they receive the same encryption strength as ordinary data. Hackers can infer the structure of key information by analyzing non-keyframes or launch targeted attacks against keyframes, weakening the overall security of multimodal data.

[0005] The third issue is the reliability of the transmission path. When multimodal data is transmitted across separate channels, transmission delays or path conflicts are prone to occur due to differences in the slice lengths of video, audio, and text. Existing solutions rely on retransmission mechanisms for error correction, but fail to coordinate slice scheduling at the transport layer, resulting in reassembly failures or synchronization misalignment at the receiving end. Especially in cross-network transmission scenarios, path instability can further amplify data loss and distortion, impacting the complete decryption of encrypted data and the reconstruction of multimodal associations.

[0006] Fourth, there's the tracing bottleneck in large-scale model scenarios. Generative large-scale models must identify the source and associated features of multimodal data to enable copyright tracking or content auditing. Traditional encrypted transmission only ensures data confidentiality and fails to embed cross-modal feature identifiers (such as timestamp binding and voiceprint-semantic associations) during the encryption process. This makes it difficult to trace the data's origin and the generation logic between modalities after decryption, failing to meet the multimodal data traceability requirements of large-scale models. Summary of the Invention

[0007] The purpose of the embodiments of the present invention is to provide a multimodal data encryption transmission method and system, which solves the coordination and synchronization problems in multimodal data encryption through time-series synchronization slicing and cross-modal key frame association technology.

[0008] To solve the above technical problems, a first aspect of an embodiment of the present invention provides a multimodal data encryption transmission method, comprising the following steps:

[0009] Performing time-series synchronous slicing on the multimodal data according to data types, extracting key frames of each slice after the time-series synchronous slicing and associating them with the same timestamp, wherein the data types of the multimodal data include: video data, audio data, and text data;

[0010] Based on the timestamp, signaling header and intra-frame data of the key frame, an encryption key is generated by cross-modal feature fusion, and the header of the corresponding key frame is encrypted to obtain an encrypted slice;

[0011] The encrypted slice is transmitted through the main transmission channel. When the length of the slice does not match the minimum slice length of the corresponding data type, an unencrypted virtual slice is generated. The virtual slice is transmitted through the auxiliary signaling channel to coordinate the transmission path. The virtual slice is discarded at the data receiving end. The virtual slice and the corresponding encrypted slice carry the same session identifier.

[0012] Furthermore, the step of slicing the multimodal data in time-series synchronization according to the data type, extracting the key frames of each slice and associating them with the same timestamp after the time-series synchronization slicing, includes:

[0013] Slice the video data using intra-frame compression and extract video key frames through inter-frame coding;

[0014] The audio data is intra-frame encoded using the address information of the video key frame as the reference point, and the audio voiceprint key frame is extracted through voiceprint quantization analysis;

[0015] Extract text information of key audio nodes from text data, perform semantic analysis through the natural language processing engine, and extract text semantic key frames;

[0016] Obtaining the timestamp, signaling header information, and address information of the video keyframes, synchronizing the timestamp, signaling header information, and quantization value of the audio voiceprint keyframes based on the address information, and aligning the timestamp, signaling header information, and audio quantization value of the text semantic keyframes;

[0017] The video key frame, the audio voiceprint key frame, and the text semantic key frame are bound to the same timestamp to build a cross-modal association relationship.

[0018] Furthermore, the video data is sliced ​​using an intra-frame compression method, and video key frames are extracted through inter-frame coding, further comprising:

[0019] During the video frame processing, the improved Sobel-Feldman operator is used to dynamically evaluate the video frame quality. Some pixels are randomly selected in the video frame, and the pixel gradients are weighted according to the spatial distribution characteristics to generate a quantitative evaluation value that represents the degree of frame quality defects.

[0020] When the quantitative evaluation value is lower than a preset quality threshold, the current video frame is marked as a key frame and associated with a timestamp, a signaling header, and an address information.

[0021] Furthermore, the generating of an encryption key by cross-modal feature fusion based on the timestamp, signaling header and intra-frame data of the key frame includes:

[0022] randomly generating a first character string, replacing a random segment of compressed data after the video key frame, and calculating a video key component based on the length of the replacement operation;

[0023] Randomly generate a second character string to replace a random compressed data segment after the audio voiceprint key frame, and calculate the audio key component based on the length of the replacement operation;

[0024] Randomly generate a third character string to replace a random segment of compressed data after the text semantic key frame, and calculate the text key component according to the length of the replacement operation;

[0025] The video key component, the audio key component and the text key component are sequentially spliced, and the splicing result is input into a hash function to obtain the encryption key.

[0026] Furthermore, the step of sequentially concatenating the video key component, the audio key component, and the text key component, and inputting the concatenation result into a hash function to obtain the encryption key comprises:

[0027] Converting the video key component into a first binary sequence of fixed length, converting the audio key component into a second binary sequence of the same fixed length, and converting the text key component into a third binary sequence of the same fixed length;

[0028] splicing the first binary sequence, the second binary sequence, and the third binary sequence in the order of video, audio, and text to obtain a complete sequence;

[0029] The complete sequence is input into the national secret SM3 hash function, a 256-bit hash value is output as the encryption key, and the encryption key is written into the encryption field of the key frame header.

[0030] Furthermore, after obtaining the encryption key, the method further includes:

[0031] Calculate the compression ratio of non-key fields in video slices, audio slices, and text slices respectively;

[0032] generating residual data encryption information based on the compression ratio, and inserting the residual data encryption information into the residual data header to obtain a non-key field encryption unit;

[0033] The key frame encrypted slices and the non-key field encrypted units are merged in the order of the original data stream to generate a complete encrypted file and back it up to the video source server, and a server identifier for tracing is written in the header of the complete encrypted file.

[0034] Furthermore, when the length of the slice does not match the minimum slice length of the corresponding data type, an unencrypted virtual slice is generated, and the virtual slice is transmitted through the auxiliary signaling channel, including:

[0035] Obtain the lengths of the video, audio, and text slices to be transmitted, and compare them using the minimum length of the three as a reference length. If the length of any slice type is not equal to the reference length, generate a virtual slice containing only the splitting instructions. The virtual slice marks the split position, reassembly order, data type, session identifier, and abnormal transmission identification field of the original slice.

[0036] The encrypted slice is sent through the main transmission channel, and the virtual slice is sent through an auxiliary signaling channel that is isolated from the main transmission channel data. The auxiliary signaling channel only transmits slice scheduling instructions. When the data sending end detects an instruction conflict, the conflict position code is written into the abnormal transmission identification field of the virtual slice;

[0037] Based on the data receiving end discarding all virtual slices, matching virtual slice instructions with encrypted slices according to session identifiers, and reassembling them into complete video, audio and text data streams in the original order.

[0038] Furthermore, after sending the virtual slice through the auxiliary signaling channel that is data-isolated from the main transmission channel, the method further includes:

[0039] monitoring the integrity of the slice scheduling instructions in the virtual slice in real time based on a data receiving end;

[0040] When the slice scheduling instruction is missing or conflicting, the abnormal transmission identification field in the virtual slice is parsed, and a retransmission request is initiated based on the abnormal transmission identification marked by the virtual slice to ensure that the encrypted slice transmitted by the main transmission channel is coordinated with the instructions of the virtual slice.

[0041] Furthermore, the abnormal transmission identifier based on the virtual slice mark initiates a retransmission request to ensure coordination between the encrypted slice transmitted by the main transmission channel and the instruction of the virtual slice, including:

[0042] Parsing the abnormal transmission identification field in the virtual slice at the data receiving end. The abnormal transmission identification field is written by the data sending end when an instruction conflict is detected. It is used to locate the slice position where the instruction is missing and the conflicting instruction sequence number.

[0043] When it is identified that the instruction of the virtual slice is missing or the instruction sequence conflicts, requesting the data sending end to retransmit the virtual slice corresponding to the abnormal transmission identifier;

[0044] The data sending end retrieves the original virtual slice associated with the abnormal identifier from the cache, and resends the virtual slice separately through the auxiliary signaling channel;

[0045] The data receiving end verifies the retransmitted virtual slices with the locally stored instructions, checks the continuity and logical consistency of the split instructions, and triggers encrypted slice reassembly based on the session identifier if the verification passes.

[0046] When the instruction verification fails three times in a row, the data receiving end clears all cache slices of the current multimodal data stream, initiates a full data reset request carrying the current session identifier to the data sending end, and terminates the current transmission session.

[0047] Accordingly, a second aspect of an embodiment of the present invention provides a multimodal data encryption transmission system, which encrypts and transmits multimodal data based on the above-mentioned multimodal data encryption transmission method, including:

[0048] A time-series slicing module is used to perform time-series synchronization slicing on multimodal data according to data type, extract key frames of each slice after time-series synchronization slicing and associate them with the same timestamp. The data types of the multimodal data include: video data, audio data, and text data.

[0049] A key frame encryption module is used to generate an encryption key by cross-modal feature fusion based on the timestamp, signaling header and intra-frame data of the key frame, and encrypt the header of the corresponding key frame to obtain an encrypted slice;

[0050] A slice transmission module is used to transmit the encrypted slice through the main transmission channel, generate an unencrypted virtual slice when the length of the slice does not match the minimum slice length of the corresponding data type, and transmit the virtual slice through the auxiliary signaling channel to coordinate the transmission path. The virtual slice is discarded at the data receiving end, and the virtual slice and the corresponding encrypted slice carry the same session identifier.

[0051] Accordingly, a third aspect of an embodiment of the present invention provides an electronic device comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned multimodal data encryption transmission method.

[0052] Accordingly, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned multimodal data encryption transmission method.

[0053] The above technical solutions of the embodiments of the present invention have the following beneficial technical effects:

[0054] 1. By binding video, audio, and text keyframes to the same timestamp through time slicing and generating dynamic keys based on cross-modal feature fusion, this solves the problem of lack of coordination in multimodal data encryption. Dynamic quality assessment of video keyframes and quantitative alignment of audio voiceprints with text semantics ensure that the encryption process preserves the temporal correlation between modalities. This effect directly improves the ability to synchronously reconstruct decrypted data and provides traceable cross-modal feature identification for generative large models.

[0055] 2. Encryption keys are generated using the timestamp, signaling header, and intra-frame data of key frames, and the remaining encrypted information is generated by combining the compression rate of non-critical fields, achieving differentiated protection for high-value key frames. Binary fusion of video / audio / text key components and the national secret SM3 hash enhance the key's irreversibility and cracking resistance. Furthermore, by replacing key frames with random strings and then compressing the data, the data distribution is dynamically disrupted, effectively defending against targeted attacks on key frames and significantly improving overall security.

[0056] 3. When slice lengths do not match, virtual slices are transmitted via an auxiliary signaling channel, working in conjunction with the encrypted slices of the primary transmission channel. The virtual slices carry splitting instructions, session identifiers, and exception transmission identification fields, enabling the receiving end to accurately reassemble the data stream. Combined with instruction integrity monitoring and conflict retransmission mechanisms, especially by locating the conflict location through exception identification, limiting the number of retransmissions, and implementing a full reset backup strategy, the risk of data loss or distortion caused by path instability is significantly reduced, ensuring the reliable transmission of multimodal data. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] Figure 1 This is a flow chart of a multimodal data encryption transmission method provided by an embodiment of the present invention;

[0058] Figure 2 This is a module block diagram of a multimodal data encryption transmission system provided by an embodiment of the present invention.

[0059] Reference numerals:

[0060] 1. Time-series slicing module, 2. Key frame encryption module, 3. Slice transmission module. DETAILED DESCRIPTION

[0061] To make the objectives, technical solutions, and advantages of the present invention more clearly understood, the present invention will be further described in detail below in conjunction with specific embodiments and with reference to the accompanying drawings. It should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present invention. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present invention.

[0062] Please refer to Figure 1 A first aspect of an embodiment of the present invention provides a multimodal data encryption transmission method, comprising the following steps:

[0063] Step 100 , time-synchronized slicing of the multimodal data is performed according to the data type. After time-synchronized slicing, the key frames of each slice are extracted and associated with the same timestamp. The data types of the multimodal data include: video data, audio data, and text data.

[0064] Multimodal data is sliced ​​and processed in a time-synchronized manner based on the three data types of video, audio, and text to ensure that data segments of different modalities are strictly aligned on the timeline. Specifically, video data is segmented using intra-frame compression, and video key frames are extracted using inter-frame coding. Audio data is intra-frame encoded based on the address information of video key frames, and voiceprint key frames are extracted through voiceprint quantization analysis. Text data extracts text information from audio key nodes, and semantic analysis is performed using a natural language processing engine to generate semantic key frames. Ultimately, the timestamps, signaling header information, and address information of the three types of key frames are bound to the same timestamp to establish a cross-modal association relationship (such as the address of the video key frame is synchronized with the audio voiceprint quantization value, and the text semantic key frame is aligned with the audio quantization value), providing a unified timing anchor for subsequent encryption.

[0065] Step 200: Based on the timestamp, signaling header and intra-frame data of the key frame, an encryption key is generated through cross-modal feature fusion, and the header of the corresponding key frame is encrypted to obtain an encrypted slice.

[0066] Based on cross-modal associated key frames, the dynamic encryption key is generated by integrating its timestamp, signaling header and intra-frame data. The specific operations include: randomly generating the compressed data segment after replacing the video key frame with the first string, and calculating the video key component based on the replacement length; similarly, generating the compressed data after the second and third strings replace the audio voiceprint key frame and text semantic key frame respectively, and calculating the corresponding key components. The three types of key components are converted into equal-length binary sequences and spliced ​​in the order of video-audio-text, and input into the national secret SM3 hash function to generate a 256-bit encryption key. The key is written into the key frame header encryption field to achieve differentiated encryption protection for high-value key frames. At the same time, random data replacement is used to disrupt the data distribution of non-key frames to improve anti-attack capabilities. The non-key fields generate the remaining encrypted information based on the compression rate and insert it into the data header. Finally, they are merged into a complete encrypted file and backed up to the server.

[0067] Step 300: Transmit the encrypted slice through the main transmission channel. When the length of the slice does not match the minimum slice length of the corresponding data type, an unencrypted virtual slice is generated. The virtual slice is transmitted through the auxiliary signaling channel to coordinate the transmission path. The virtual slice is discarded at the data receiving end. The virtual slice and the corresponding encrypted slice carry the same session identifier.

[0068] Encrypted keyframe slices are transmitted over the main transmission channel, while dynamically coordinating transmission conflicts caused by differences in slice lengths. The specific mechanism is as follows: the lengths of video, audio, and text slices are compared in real time, using the minimum length as the benchmark. If the lengths of certain slice types do not match, a virtual slice is generated. Virtual slices only contain the splitting instructions, reassembly order, data type, session identifier, and exception transmission identification field (such as the conflict position code written when an instruction conflict is detected). They are transmitted over an auxiliary signaling channel independent of the main channel. The receiving end discards all virtual slices but uses the session identifiers they carry to match the encrypted slices of the main channel and reassemble the data stream in the original order. If a missing or conflicting instruction is detected, the exception identification field of the virtual slice is parsed to initiate a retransmission request: the sending end retrieves the original virtual slice from the cache and retransmits it, and the receiving end verifies the continuity of the instruction; if three consecutive failures occur, the cache is cleared and a full data reset is triggered, ensuring dynamic coordination and robustness of the transmission path.

[0069] This encrypted transmission method addresses the interoperability issues of multimodal data encryption by binding time-series synchronized slicing to cross-modal keyframes. It also utilizes keyframe feature fusion to generate dynamic keys, combined with a random substitution scrambling mechanism to enhance security protection. An innovative dual-channel transmission strategy (encrypted data on the primary channel + virtual instructions on the secondary channel) dynamically coordinates slice length differences, significantly reducing the risk of transmission conflicts and packet loss. Ultimately, while ensuring data confidentiality, it achieves precise, synchronized cross-modal reconstruction after decryption and provides traceable cross-modal feature identification for generative large models.

[0070] Furthermore, in step 100, the multimodal data is sliced ​​in time-series synchronization according to the data type, and after the time-series synchronization slicing, the key frames of each slice are extracted and associated with the same timestamp, including:

[0071] Step 110: Slice the video data using an intra-frame compression method, and extract video key frames through inter-frame coding.

[0072] Video data is sliced ​​using intra-frame compression methods (such as SNSIF and AVS standards), and inter-frame coding techniques are used to analyze the dynamic correlation of video frame sequences. During the slicing process, a modified Sobel-Feldman operator is used to perform gradient weighted calculations on randomly sampled pixels to generate a quantitative evaluation value for video frame quality. When this value falls below a preset threshold (e.g., 0.35), the current frame is marked as a video keyframe, and its timestamp, signaling header information, and physical address information are simultaneously recorded, providing a reference anchor point for multimodal synchronization.

[0073] Step 120 , intra-frame encoding is performed on the audio data using the address information of the video key frame as a reference point, and audio voiceprint key frames are extracted through voiceprint quantization analysis.

[0074] Using the address information of video keyframes as spatiotemporal reference points, the audio data is intra-frame encoded and sliced. Dynamic voiceprint features (including first-order and second-order differences ΔC_i) are calculated using an improved MFCC algorithm to construct a 64-dimensional deep voiceprint vector. A spectral clustering algorithm is used to analyze the spatial distribution of voiceprint features. When the intra-class distance exceeds 1.2 times the inter-class distance, the audio voiceprint keyframe is marked. This process ensures strict physical alignment between the audio keyframe and the video keyframe, achieving cross-modal spatiotemporal correlation.

[0075] Step 130 , extracting text information of key audio nodes from the text data, performing semantic analysis through a natural language processing engine, and extracting text semantic key frames.

[0076] Original text information is extracted from key nodes corresponding to audio voiceprint keyframes, and semantic parsing is performed using a BERT-Transformer hybrid model. Word vector associations are calculated by designing an attention weight matrix, and text importance is quantitatively analyzed based on the semantic entropy value (H_s). When the semantic entropy exceeds 5.8 bits, it is identified as a text semantic keyframe. This step converts quantized audio features into semantic nodes, establishing an abstract-level association between audio and text.

[0077] Step 140: obtain the timestamp, signaling header information and address information of the video key frame, synchronize the timestamp, signaling header information and quantization value of the audio voiceprint key frame based on the address information, and align the timestamp, signaling header information and audio quantization value of the text semantic key frame.

[0078] The timestamps, signaling headers, and address information of the video keyframes are integrated to synchronize the timestamps and quantization values ​​(such as the MFCC coefficient differential results) of the audio voiceprint keyframes. The signaling header information of the text semantic keyframes is also aligned with the audio quantization values. For example, the timestamps of the audio voiceprint keyframes are offset-calibrated based on the video address information, while the timestamps of the text semantic keyframes are bound to the audio quantization peak point, forming a longitudinal time series link for the trimodal data.

[0079] Step 150: Bind the video keyframes, audio voiceprint keyframes, and text semantic keyframes to the same timestamp to build a cross-modal association relationship.

[0080] Bind video keyframes, audio voiceprint keyframes, and text semantic keyframes to a unified timestamp to establish a cross-modal association relationship. Specifically, the video address information is mapped to the audio quantized coordinates, and then associated with the text semantic entropy peak points to generate a "video address-voiceprint feature-semantic node" triple. This association is written into the slice metadata, providing a spatiotemporal framework for cross-modal feature fusion for subsequent encrypted transmission.

[0081] The above implementation method solves the asynchronous problem of multimodal data slicing through video-driven multi-level key frame extraction (video quality assessment → voiceprint clustering → semantic entropy analysis) and a timing binding mechanism; audio encoding based on the video address and cascade alignment of voiceprint quantization to semantic nodes ensure the spatiotemporal consistency of cross-modal key features; the final constructed triple association relationship provides a fusion basis for dynamic key generation, while ensuring the synchronization accuracy and traceability of data reconstruction after decryption.

[0082] Furthermore, the step 110 of slicing the video data using an intra-frame compression method and extracting video key frames through inter-frame coding further includes:

[0083] Step 111: During the video frame processing, the improved Sobel-Feldman operator is used to dynamically evaluate the video frame quality. Some pixels are randomly selected from the video frame, and pixel gradients are weighted and calculated based on spatial distribution characteristics to generate a quantitative evaluation value representing the degree of frame quality defects.

[0084] During the video frame slicing process, a modified Sobel-Feldman operator is used to dynamically assess frame quality. 10% of the pixel sampling points are randomly selected from a single frame (using a 1920×1080 resolution as an example). Gradient values ​​are calculated based on the spatial distribution characteristics of these pixels (based on the image center coordinates (960, 540) and weighted by the Gaussian distribution standard deviation of 200-500 pixels). The intensity of pixel mutations is quantified using the second-order derivative of the Laplace operator. Ultimately, a frame quality evaluation score is generated that combines the spatial weighting and gradient magnitude. Lower values ​​indicate greater frame defects (such as blur, noise, or distortion).

[0085] Specifically, the video processing uses the improved Sobel-Feldman operator to detect defective frames and construct a video quality evaluation function :

[0086] ;

[0087] Where: N is the number of pixel sampling points of a single frame image, which is 10% random sampling points at a resolution of 1920×1080; Pixel The second derivative of the Laplace operator at ; is the image center coordinate (960,540), is the standard deviation of the spatial Gaussian distribution. In one example, the value range is [200, 500] pixels. The quality threshold can be set according to the actual defect regression experimental data. For example, when When the value is less than 0.35, it is judged as a defective frame and the key frame marking mechanism is triggered.

[0088] Specifically, the voiceprint feature extraction of audio processing adopts the improved MFCC algorithm:

[0089] ;

[0090] ;

[0091] Among them, the improved MFCC algorithm middle, The time window radius for dynamic feature extraction; After the current frame MFCC coefficients of the frame; Before the current frame MFCC coefficients of the frame; k is the offset index within the window (- to+ ); is the MFCC coefficient of the kth offset position in the window.

[0092] Construct a 64-dimensional deep voiceprint feature vector and divide the voiceprint feature space by spectral clustering algorithm. >1.2 The time is marked as a key voiceprint frame.

[0093] Specifically, the BERT-Transformer hybrid model is used for text semantic analysis, and the attention weight matrix is ​​designed:

[0094] ;

[0095] in, is the query vector of the i-th word; is the key vector of the jth word; d is the vector dimension of the attention mechanism; For words exist The co-occurrence probability in the context; n is the total number of words in the input sequence.

[0096] By calculating the semantic entropy ,when When the value is greater than 5.8 bits, it is determined to be a key semantic node.

[0097] Step 112: When the quantitative evaluation value is lower than the preset quality threshold, the current video frame is marked as a key frame and associated with a timestamp, a signaling header, and address information.

[0098] When the frame quality evaluation value falls below a preset threshold (e.g., 0.35, based on actual defect regression experiments), the current video frame is marked as a keyframe. The frame's precise timestamp (time stamp), signaling header (transmission control information), and physical address information (such as memory or storage location coordinates) are synchronously associated to provide a temporal and spatial reference for subsequent cross-modal alignment. Ordinary frames that fall below the threshold undergo only standard slicing and are not included in the keyframe encryption system.

[0099] Through the dynamic video quality assessment mechanism, key frames with high information density (such as scene switching and motion mutation frames) can be accurately identified to avoid redundant encryption caused by traditional uniform slicing; the improved operator combined with the spatial weighting strategy can significantly improve the sensitivity of defect detection (such as identifying subtle distortions below the 0.35 threshold); the strong binding of key frame markers and spatiotemporal metadata lays the core anchor point foundation for the synchronous encryption of multimodal data and the coordination of transmission paths.

[0100] Furthermore, in step 200, the encryption key is generated by cross-modal feature fusion based on the timestamp, signaling header, and intra-frame data of the key frame, including:

[0101] Step 210: randomly generate a first character string, replace a random segment of compressed data after the video key frame, and calculate the video key component according to the length of the replacement operation.

[0102] A random first string (e.g., 16 random characters) is generated to replace a random segment of compressed data following the video keyframe header (e.g., replacing a continuous 32-byte compressed video stream). The starting offset and length of the replaced data are recorded and used as the video key component. This operation dynamically disrupts the data distribution after the keyframe, increasing the difficulty of cracking.

[0103] Step 220: randomly generate a second character string to replace a random segment of compressed data after the audio voiceprint key frame, and calculate the audio key component according to the length of the replacement operation.

[0104] A second string is randomly generated (with an independent length from the first string), and the compressed data segment after the audio voiceprint keyframe is located (for example, replacing 20 consecutive bytes of the MFCC coefficient differential value). The audio key component is generated based on the byte length of the replacement operation. The voiceprint keyframe is determined by the voiceprint clustering algorithm to ensure that the replacement operation targets high-value audio features.

[0105] Step 230: randomly generate a third character string to replace a random segment of compressed data after the text semantic key frame, and calculate the text key component based on the length of the replacement operation.

[0106] A third string is randomly generated and replaced with a random segment from the compressed text data (e.g., BERT word vector stream) following the semantic keyframe. The replacement length is used as the text key component. Semantic keyframes are filtered using a semantic entropy threshold, and the replacement position avoids key semantic nodes to maintain basic text readability.

[0107] In step 240 , the video key component, the audio key component, and the text key component are sequentially concatenated, and the concatenation result is input into a hash function to obtain an encryption key.

[0108] The video, audio, and text key components (i.e., the three replacement length values) are concatenated in a fixed order into a combined string and then fed into the SM3 hash function, a national encryption standard. SM3 performs padding, iterative compression, and bitwise operations on the input to generate a 256-bit irreversible hash value as the final encryption key, which is written into the encrypted field in the keyframe header. The key generation process incorporates cross-modal operation features to achieve dynamic key binding.

[0109] The above steps convert physical operation characteristics into key materials by randomly replacing key frames, compressing data, and extracting the operation length as the key component, thereby enhancing the unpredictability of the key; the sequential splicing of the three types of key components is combined with the national secret SM3 hash to ensure that the key is collision-resistant and one-way; the collaborative mechanism of key frame header encryption and random data replacement effectively defends against statistical analysis attacks on encryption modes while ensuring the security of high-value frames.

[0110] Furthermore, in step 240, the video key component, the audio key component, and the text key component are sequentially concatenated, and the concatenation result is input into a hash function to obtain an encryption key, including:

[0111] Step 241: convert the video key component into a first binary sequence of a fixed length, convert the audio key component into a second binary sequence of the same fixed length, and convert the text key component into a third binary sequence of the same fixed length.

[0112] The video key component (i.e., the replacement length value of the compressed data after the video key frame) is converted into a binary sequence with a fixed number of bits (e.g., 16 bits), and the insufficient bits are padded with zeros; the audio key component and the text key component are synchronously converted into independent binary sequences with the same number of bits (e.g., 16 bits each), ensuring that the lengths of the three types of modal key components are unified and eliminating dimensional deviations caused by differences in data types.

[0113] Step 242 : Concatenate the first binary sequence, the second binary sequence, and the third binary sequence in the order of video, audio, and text to obtain a complete sequence.

[0114] The three binary sequences are concatenated in the strict order of video, audio, and text: the video sequence is placed in the upper order, the audio sequence is centered, and the text sequence is placed in the lower order (for example, 16-bit video + 16-bit audio + 16-bit text = a complete 48-bit sequence). This order strictly corresponds to the binding hierarchy of cross-modal keyframes (video → audio → text), maintaining the logical consistency of feature fusion.

[0115] Step 243: Input the complete sequence into the national encryption SM3 hash function, output a 256-bit hash value as the encryption key, and write the encryption key into the encryption field of the key frame header.

[0116] The concatenated binary sequence is fed into the National Cryptography SM3 hash function. SM3 performs message padding, iterative compression functions, and nonlinear bitwise operations (such as Boolean functions and shift operations) to generate an irreversible 256-bit hash value, which serves as the final encryption key. This key is written into a pre-defined encryption field (such as a 32-byte storage area) in the keyframe header, simultaneously triggering the encryption unit generation process for non-critical fields.

[0117] The above steps solve the dimensional mismatch problem of multimodal key components through fixed-length binary conversion, and sequential splicing retains the cross-modal association logic; the strong diffusion and collision resistance of SM3 hash ensure that the key has one-way irreversible characteristics, and defends against the risk of key derivation caused by length value leakage; the key is written into the key frame header to form an encryption identification closed loop, providing a dynamic and tamper-proof security anchor for multimodal data.

[0118] Furthermore, after obtaining the encryption key in step 240, the following steps are further included:

[0119] Step 251, respectively calculate the compression ratio of non-key fields in the video slice, audio slice and text slice.

[0120] The compression ratios of non-critical fields in video, audio, and text slices (i.e., ordinary data segments not marked as keyframes) are calculated separately: the compression ratio of non-critical video fields is calculated based on inter-frame redundancy, the compression efficiency of non-critical audio fields is calculated based on the statistical characteristics of the sampling points, and the compression ratio of non-critical text fields is calculated based on the frequency of character repetition. This value reflects the redundancy characteristics of the data itself and the encryption strength requirements.

[0121] Step 252: Generate remaining data encryption information based on the compression ratio, and insert the remaining data encryption information into the remaining data header to obtain a non-key field encryption unit.

[0122] Based on the compression ratio obtained in step 251, encryption information for the remaining data is dynamically generated: video slices generate a variable-length checksum based on the compression ratio, audio slices generate a voiceprint mask based on the sampling rate, and text slices generate a semantic obfuscation sequence based on character frequency. This encrypted information is inserted into the header of the corresponding non-critical field to form an independent encryption unit (for example, a checksum + offset is written into the header of a non-critical video frame), achieving hierarchical and lightweight protection of non-critical data.

[0123] In step 253, the key frame encrypted slices and the non-key field encrypted units are merged in the order of the original data stream to generate a complete encrypted file and back it up to the video source server, and a server identifier for tracing is written in the header of the complete encrypted file.

[0124] The encrypted keyframe slices and non-key field encrypted units are combined in the original data stream's time sequence to generate a complete encrypted file. A unique identifier for the video source server (such as an IP hash or digital fingerprint) is written to the file header, and the encrypted file is backed up to that server. This identifier is associated with the keyframe timestamp and session ID to establish an end-to-end traceability link.

[0125] The above steps adaptively generate differentiated encryption units through the compression rate of non-critical fields, maintaining the overall security level while reducing computational overhead; the sequential merging of key frames and non-critical data ensures the integrity of the decrypted data stream; the strong binding of server identifiers and encrypted files provides an unalterable trust anchor for cross-modal data traceability, meeting the audit requirements of large models for data sources.

[0126] Furthermore, in step 300, when the length of the slice does not match the minimum slice length of the corresponding data type, an unencrypted virtual slice is generated, and the virtual slice is transmitted through the auxiliary signaling channel, including:

[0127] Step 310, obtain the length values ​​of the video slice, audio slice and text slice to be transmitted, and use the minimum length value of the three as the benchmark length for comparison. If the length of any type of slice is not equal to the benchmark length, generate a virtual slice containing only the splitting instruction. The virtual slice marks the split position, reorganization order, data type, session identifier and abnormal transmission identification field of the original slice.

[0128] The lengths of the video, audio, and text slices to be transmitted are obtained in real time, and the minimum length of the three is used as a benchmark for comparison. If the length of a particular slice type does not match the benchmark (for example, a video slice is too long), an unencrypted virtual slice is generated. This virtual slice contains only structured instructions: recording the original slice split position (for example, split from the 1024th byte), the reassembly order (for example, segment A→B), the data type identifier (for example, "video"), the session ID (a unique transmission session identifier), and the abnormal transmission identifier field (initially empty, used for subsequent conflict recording). It does not carry any actual data payload.

[0129] Step 320: Send the encrypted slice through the main transmission channel, and at the same time send the virtual slice through the auxiliary signaling channel that is isolated from the main transmission channel data. The auxiliary signaling channel only transmits the slice scheduling instruction. When the data sending end detects an instruction conflict, the conflict position code is written into the abnormal transmission identification field of the virtual slice.

[0130] A physically isolated dual-channel transmission mechanism: the primary transmission channel sends encrypted real slices (including encrypted keyframe data), while the auxiliary signaling channel independently transmits virtual slices. The auxiliary channel only transmits command metadata and is completely isolated from the primary channel data. When the sender detects a command conflict (for example, two virtual slices point to the same reassembly location), the conflicting physical offset address and sequence number are written to the virtual slice's abnormal transmission identification field (such as "0x3F5A:Seq15"), ensuring that the receiver can locate the source of the conflict.

[0131] Step 330 , based on discarding all virtual slices at the data receiving end, matching the virtual slice instructions with the encrypted slices according to the session identifier, and reassembling them into complete video, audio, and text data streams in the original order.

[0132] The data receiver discards all virtual slices but extracts the instruction information for them: it matches the encrypted slices of the primary channel using the session ID, and restores the original data stream based on the split locations and reassembly order (e.g., splicing split video segments A and B in sequence). If the exception identification field of the virtual slice is not empty, it resolves the conflicting location and initiates a retransmission request. If there is no conflict, it directly integrates the encrypted video, audio, and text slices, outputting a synchronized multimodal data stream.

[0133] Furthermore, after sending the virtual slice through the auxiliary signaling channel isolated from the main transmission channel data in step 320, the method further includes:

[0134] Step 321 : monitor the integrity of the slice scheduling instructions in the virtual slice in real time based on the data receiving end.

[0135] The integrity of the slice scheduling instructions carried by the virtual slices is monitored in real time at the data receiving end. This includes checking the continuity of the instruction sequence (e.g., whether the split position is increasing), logical consistency (e.g., whether there are conflicts in the reassembly order), and data type matching (e.g., whether the video instruction corresponds to the video slice). This monitoring process is performed in parallel with the reception of encrypted slices on the main channel. The instructions and data stream are associated through session identifiers to ensure that the scheduling instructions in the virtual slices can be fully mapped to the actual encrypted data.

[0136] Step 322: When the slice scheduling instruction is missing or conflicting, parse the abnormal transmission identification field in the virtual slice, initiate a retransmission request based on the abnormal transmission identification marked by the virtual slice, and ensure that the encrypted slice transmitted by the main transmission channel is coordinated with the instructions of the virtual slice.

[0137] When monitoring detects missing instructions (e.g., sequence number jumps) or conflicts (e.g., two instructions pointing to the same data offset), the abnormal transmission identification field in the virtual slice is parsed (this field is written by the sender when a conflict is detected and contains the conflict position code and the instruction sequence number). Based on this identification, a directed retransmission request is initiated to the sender: the request carries the conflict position and session ID, triggering the sender to retrieve the original virtual slice from the cache and retransmit it via the auxiliary signaling channel. The retransmitted slice overwrites the missing or conflicting instructions, ensuring the coordinated reassembly of the encrypted slice and the scheduling instructions on the main channel.

[0138] The above steps accurately locate transmission anomalies through the receiving end's real-time integrity monitoring and conflict resolution mechanism for virtual slice instructions; the targeted retransmission strategy (non-full retransmission) based on the anomaly identification field significantly reduces error correction overhead; and ultimately ensures strict coordination between the main channel encrypted data and the auxiliary channel scheduling instructions, achieving highly robust reconstruction and zero-error synchronization of multimodal data streams.

[0139] Furthermore, in step 322, a retransmission request is initiated based on the abnormal transmission identifier of the virtual slice mark to ensure that the encrypted slice transmitted by the main transmission channel is coordinated with the instruction of the virtual slice, including:

[0140] Step 322a: parse the abnormal transmission identification field in the virtual slice at the data receiving end. The abnormal transmission identification is written by the data sending end when an instruction conflict is detected, and is used to locate the slice position where the instruction is missing and the conflicting instruction sequence number.

[0141] The abnormal transmission identification field in the virtual slice is parsed at the data receiving end. This field is dynamically written by the sending end when an instruction conflict is detected (for example, multiple virtual slices point to the same reorganization location). It contains the conflicting physical offset address (such as "0x3F5A") and the conflicting instruction sequence number (such as "Seq15"), accurately locating the source of the transmission anomaly.

[0142] Step 322b: When it is identified that the instruction of the virtual slice is missing or the instruction sequence conflicts, a request is made to the data sending end to retransmit the virtual slice corresponding to the abnormal transmission identifier.

[0143] When an instruction is detected to be missing (such as discontinuous serial numbers) or conflicting (such as overlapping two instructions), a directed retransmission request is initiated to the sender: the request message carries the conflict position and session ID of the exception identification field, and only requires the retransmission of the specific virtual slice associated with the exception identification to avoid retransmission of the entire data.

[0144] Step 322c: The data sending end retrieves the original virtual slice associated with the abnormal identifier from the cache, and resends the virtual slice separately through the auxiliary signaling channel.

[0145] Based on the received request message, the data sender retrieves the original virtual slice (including splitting instructions, data type tags and other metadata) from the local cache and resends the slice through an independent auxiliary signaling channel to ensure that the encrypted data flow of the main channel is not disturbed.

[0146] Step 322d, based on the data receiving end, the retransmitted virtual slice is spliced ​​and verified with the locally stored instructions to check the continuity and logical consistency of the split instructions. If the verification passes, the encrypted slice reorganization is triggered based on the session identifier.

[0147] The data receiver concatenates the retransmitted virtual slices with the locally stored instructions according to the sequence number, verifying the continuity of the split positions (e.g., incremental offsets) and the logical consistency of the reassembly (e.g., no overlap). If the verification passes, the encrypted slice reassembly process based on the session ID is triggered; if the verification fails, it is marked as an abnormal event.

[0148] Step 322e: When the command verification fails three times in a row, the data receiving end clears all cache slices of the current multimodal data stream, initiates a full data reset request carrying the current session identifier to the data sending end, and terminates the current transmission session.

[0149] If the command verification fails three times in a row (indicating that the path is unrecoverable), the receiving end clears all cached slices associated with the current session ID and initiates a full data reset request (carrying the session ID) to the sending end. The sending end terminates the current transmission session and retrieves the fully encrypted file from the video source server to initiate a new round of transmission.

[0150] The above steps resolve instruction conflicts at minimal cost through precise positioning of exception markers and a targeted retransmission mechanism. The three-level fault-tolerance strategy (single retransmission → splicing verification → session reset) ensures reliability while avoiding resource waste. Ultimately, atomic-level coordination between main channel encrypted data and virtual slice instructions is achieved, ensuring efficient reconstruction and zero-error decryption of multimodal data streams.

[0151] Accordingly, please refer to Figure 2 A second aspect of an embodiment of the present invention provides a multimodal data encryption transmission system, which encrypts and transmits multimodal data based on the multimodal data encryption transmission method described above, including:

[0152] The time-series slicing module 1 is used to perform time-series synchronization slicing on multimodal data according to data type. After time-series synchronization slicing, the key frames of each slice are extracted and associated with the same timestamp. The data types of multimodal data include: video data, audio data and text data.

[0153] Key frame encryption module 2, which is used to generate an encryption key based on the timestamp, signaling header and intra-frame data of the key frame through cross-modal feature fusion, encrypt the header of the corresponding key frame, and obtain an encrypted slice;

[0154] Slice transmission module 3 is used to transmit encrypted slices through the main transmission channel. When the length of the slice does not match the minimum slice length of the corresponding data type, an unencrypted virtual slice is generated. The virtual slice is transmitted through the auxiliary signaling channel to coordinate the transmission path. The virtual slice is discarded at the data receiving end. The virtual slice and the corresponding encrypted slice carry the same session identifier.

[0155] Accordingly, a third aspect of an embodiment of the present invention provides an electronic device comprising: at least one processor; and a memory connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor executes the above-mentioned multimodal data encryption transmission method.

[0156] Accordingly, a fourth aspect of an embodiment of the present invention provides a computer-readable storage medium having computer instructions stored thereon, which, when executed by a processor, implement the above-mentioned multimodal data encryption transmission method.

[0157] The embodiments of the present invention are intended to protect a multimodal data encryption transmission method and system, which has the following effects:

[0158] 1. By binding video, audio, and text keyframes to the same timestamp through time slicing and generating dynamic keys based on cross-modal feature fusion, this solves the problem of lack of coordination in multimodal data encryption. Dynamic quality assessment of video keyframes and quantitative alignment of audio voiceprints with text semantics ensure that the encryption process preserves the temporal correlation between modalities. This effect directly improves the ability to synchronously reconstruct decrypted data and provides traceable cross-modal feature identification for generative large models.

[0159] 2. Encryption keys are generated using the timestamp, signaling header, and intra-frame data of key frames, and the remaining encrypted information is generated by combining the compression rate of non-critical fields, achieving differentiated protection for high-value key frames. Binary fusion of video / audio / text key components and the national secret SM3 hash enhance the key's irreversibility and cracking resistance. Furthermore, by replacing key frames with random strings and then compressing the data, the data distribution is dynamically disrupted, effectively defending against targeted attacks on key frames and significantly improving overall security.

[0160] 3. When slice lengths do not match, virtual slices are transmitted via an auxiliary signaling channel, working in conjunction with the encrypted slices of the primary transmission channel. The virtual slices carry splitting instructions, session identifiers, and exception transmission identification fields, enabling the receiving end to accurately reassemble the data stream. Combined with instruction integrity monitoring and conflict retransmission mechanisms, especially by locating the conflict location through exception identification, limiting the number of retransmissions, and implementing a full reset backup strategy, the risk of data loss or distortion caused by path instability is significantly reduced, ensuring the reliable transmission of multimodal data.

[0161] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware. Furthermore, the present application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to magnetic disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0162] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or block in the flowchart and / or block diagram, as well as the combination of processes and / or blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 a process or multiple processes and / or boxes Figure 1A device that provides the functions specified in a block or multiple blocks.

[0163] These computer program instructions may also be stored in a computer readable memory that can direct a computer or other programmable data processing device to work in a specific manner, so that the instructions stored in the computer readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 a process or multiple processes and / or boxes Figure 1 The function specified in one or more boxes.

[0164] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operational steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing the instructions executed on the computer or other programmable device for implementing the process. Figure 1 a process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0165] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, ordinary technicians in the field should understand that the specific implementation methods of the present invention can still be modified or replaced by equivalents. Any modification or equivalent replacement that does not depart from the spirit and scope of the present invention should be covered by the scope of protection of the claims of the present invention.

Claims

1. A multimodal data encryption transmission method, characterized in that: The steps include: Performing time-series synchronous slicing on the multimodal data according to data types, extracting key frames of each slice after the time-series synchronous slicing and associating them with the same timestamp, wherein the data types of the multimodal data include: video data, audio data, and text data; Based on the timestamp, signaling header and intra-frame data of the key frame, an encryption key is generated by cross-modal feature fusion, and the header of the corresponding key frame is encrypted to obtain an encrypted slice; The encryption key is generated by cross-modal feature fusion based on the timestamp, signaling header, and intra-frame data of the key frame, including: randomly generating a first string, a second string, and a third string, replacing a random segment of compressed data after the video key frame, the audio voiceprint key frame, and the text semantic key frame, respectively; calculating the video key component, the audio key component, and the text key component based on the length of the replacement operation; and after splicing them in sequence, inputting the splicing result into a hash function to obtain the encryption key; The encrypted slice is transmitted through a primary transmission channel. When the length of the slice does not match the minimum slice length of the corresponding data type, an unencrypted virtual slice is generated. The virtual slice is transmitted through an auxiliary signaling channel to coordinate the transmission path. The virtual slice is discarded at the data receiving end. The virtual slice and the corresponding encrypted slice carry the same session identifier. Among them, when the length of the slice does not match the minimum slice length of the corresponding data type, an unencrypted virtual slice is generated, and the virtual slice is transmitted through the auxiliary signaling channel, including: obtaining the length values ​​of the video slice, audio slice and text slice to be transmitted, and comparing them with the minimum length value of the three as the benchmark length. If the slice length of any type is not equal to the benchmark length, a virtual slice containing only a splitting instruction is generated, and the virtual slice marks the split position, reorganization order, data type, session identifier and abnormal transmission identification field of the original slice; the encrypted slice is sent through the main transmission channel, and the virtual slice is sent through the auxiliary signaling channel isolated from the data of the main transmission channel. The auxiliary signaling channel only transmits slice scheduling instructions. When the data sending end detects an instruction conflict, the conflict position code is written into the abnormal transmission identification field of the virtual slice; based on the data receiving end, all virtual slices are discarded, the virtual slice instructions and the encrypted slices are matched according to the session identifier, and reorganized into complete video, audio and text data streams in the original order.

2. The multimodal data encryption transmission method according to claim 1, characterized in that: The method of performing time-series synchronous slicing on the multimodal data according to the data type, extracting the key frames of each slice and associating them with the same timestamp after the time-series synchronous slicing, includes: Slice the video data using intra-frame compression and extract video key frames through inter-frame coding; The audio data is intra-frame encoded using the address information of the video key frame as the reference point, and the audio voiceprint key frame is extracted through voiceprint quantization analysis; Extract text information of key audio nodes from text data, perform semantic analysis through the natural language processing engine, and extract text semantic key frames; Obtaining the timestamp, signaling header information, and address information of the video keyframes, synchronizing the timestamp, signaling header information, and quantization value of the audio voiceprint keyframes based on the address information, and aligning the timestamp, signaling header information, and audio quantization value of the text semantic keyframes; The video key frame, the audio voiceprint key frame, and the text semantic key frame are bound to the same timestamp to build a cross-modal association relationship.

3. The multimodal data encryption transmission method according to claim 2, characterized in that: The method of slicing the video data using an intra-frame compression method and extracting video key frames through inter-frame coding also includes: During the video frame processing, the improved Sobel-Feldman operator is used to dynamically evaluate the video frame quality. Some pixels are randomly selected in the video frame, and the pixel gradients are weighted according to the spatial distribution characteristics to generate a quantitative evaluation value that represents the degree of frame quality defects. When the quantitative evaluation value is lower than a preset quality threshold, the current video frame is marked as a key frame and associated with a timestamp, a signaling header, and an address information.

4. The multimodal data encryption transmission method according to claim 3, characterized in that: The step of sequentially concatenating the video key component, the audio key component, and the text key component, and inputting the concatenation result into a hash function to obtain the encryption key comprises: Converting the video key component into a first binary sequence of fixed length, converting the audio key component into a second binary sequence of the same fixed length, and converting the text key component into a third binary sequence of the same fixed length; splicing the first binary sequence, the second binary sequence, and the third binary sequence in the order of video, audio, and text to obtain a complete sequence; The complete sequence is input into the national secret SM3 hash function, a 256-bit hash value is output as the encryption key, and the encryption key is written into the encryption field of the key frame header.

5. The multimodal data encryption transmission method according to claim 3, characterized in that: After obtaining the encryption key, the method further includes: Calculate the compression ratio of non-key fields in video slices, audio slices, and text slices respectively; generating residual data encryption information based on the compression ratio, and inserting the residual data encryption information into the residual data header to obtain a non-key field encryption unit; The key frame encrypted slices and the non-key field encrypted units are merged in the order of the original data stream to generate a complete encrypted file and back it up to the video source server, and a server identifier for tracing is written in the header of the complete encrypted file.

6. The multimodal data encryption transmission method according to claim 5, characterized in that: After sending the virtual slice through the auxiliary signaling channel isolated from the main transmission channel data, the method further includes: monitoring the integrity of the slice scheduling instructions in the virtual slice in real time based on a data receiving end; When the slice scheduling instruction is missing or conflicting, the abnormal transmission identification field in the virtual slice is parsed, and a retransmission request is initiated based on the abnormal transmission identification marked by the virtual slice to ensure that the encrypted slice transmitted by the main transmission channel is coordinated with the instructions of the virtual slice.

7. The multimodal data encryption transmission method according to claim 6, characterized in that: The abnormal transmission identifier based on the virtual slice mark initiates a retransmission request to ensure that the encrypted slice transmitted by the main transmission channel is coordinated with the instruction of the virtual slice, including: Parsing the abnormal transmission identification field in the virtual slice at the data receiving end. The abnormal transmission identification field is written by the data sending end when an instruction conflict is detected. It is used to locate the slice position where the instruction is missing and the conflicting instruction sequence number. When it is identified that the instruction of the virtual slice is missing or the instruction sequence conflicts, requesting the data sending end to retransmit the virtual slice corresponding to the abnormal transmission identifier; The data sending end retrieves the original virtual slice associated with the abnormal identifier from the cache, and resends the virtual slice separately through the auxiliary signaling channel; The data receiving end verifies the retransmitted virtual slices with the locally stored instructions, checks the continuity and logical consistency of the split instructions, and triggers encrypted slice reassembly based on the session identifier if the verification passes. When the instruction verification fails three times in a row, the data receiving end clears all cache slices of the current multimodal data stream, initiates a full data reset request carrying the current session identifier to the data sending end, and terminates the current transmission session.

8. A multimodal data encryption transmission system, characterized in that: Encrypting and transmitting multimodal data based on the multimodal data encryption transmission method according to any one of claims 1 to 7 includes: A time-series slicing module is used to perform time-series synchronization slicing on multimodal data according to data type, extract key frames of each slice after time-series synchronization slicing and associate them with the same timestamp. The data types of the multimodal data include: video data, audio data, and text data. A key frame encryption module is used to generate an encryption key by cross-modal feature fusion based on the timestamp, signaling header and intra-frame data of the key frame, and encrypt the header of the corresponding key frame to obtain an encrypted slice; A slice transmission module is used to transmit the encrypted slice through the main transmission channel, generate an unencrypted virtual slice when the length of the slice does not match the minimum slice length of the corresponding data type, and transmit the virtual slice through the auxiliary signaling channel to coordinate the transmission path. The virtual slice is discarded at the data receiving end, and the virtual slice and the corresponding encrypted slice carry the same session identifier.

Images