Lightweight multi-mode audio-visual conversion method, device and system
By using structured pruning and quantized student models to process visual and audio streams in parallel, and combining a lightweight fusion layer and consistency check, the high latency and memory peak issues of multimodal audiovisual tasks are solved, achieving low latency, low power consumption and stable accuracy in multimodal audiovisual conversion.
Patent Information
- Application Number
- CN202610149234.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-02-03
- Publication Date
- 2026-05-12
AI Technical Summary
Existing solutions for multimodal audiovisual tasks have large parameter scales, high inference latency, and large memory peaks. Furthermore, the serial execution of preprocessing and postprocessing leads to pipeline idleness and jitter. Long-term operation on the edge results in changes in data distribution, causing threshold instability and output drift.
The system employs a structured pruning and quantized student model, combined with knowledge distillation training, to achieve parallel processing of visual and audio streams. It performs feature alignment and fusion through a lightweight fusion layer, performs post-processing in parallel, and uses a consistency verification service for calibration to reduce memory spikes and latency. Only low-dimensional embeddings or summaries are transmitted for verification.
It achieves low-latency, low-power multimodal audio-visual conversion, maintains stable accuracy, reduces pipeline idle time, lowers peak memory and bandwidth requirements, and ensures stable operation of the edge model on general-purpose devices.
Smart Images
Figure CN122021909A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of multimodal perception and intelligent edge computing technology, specifically to a lightweight multimodal audiovisual conversion method, device, and system. Background Technology
[0002] Multimodal audiovisual tasks often process visual and audio information simultaneously. Existing solutions suffer from large parameter scale, high inference latency, and large peak memory usage. Furthermore, preprocessing / postprocessing and inference are often executed sequentially, leading to pipeline idleness and jitter. Long-term operation on the edge can also cause threshold instability and output drift due to data distribution changes and offsets from the high-precision teachers on the cloud side. There is a lack of integrated edge solutions that combine lightweight design, three-stage parallel pipeline, and consistency verification. Summary of the Invention
[0003] The purpose of this invention is to provide a lightweight multimodal audiovisual conversion method, apparatus and system to solve the problems mentioned in the background art.
[0004] To achieve the above objectives, the present invention provides the following technical solution: a lightweight multimodal audiovisual conversion edge-side inference method, comprising:
[0005] A) Obtain the student model, which is obtained by training the teacher model through structured pruning, quantization and knowledge distillation. The structured pruning is performed on the convolutional channels, attention heads or equivalent structures, and operator folding and computation graph optimization are performed before deployment.
[0006] B) Perform region processing and normalization on the visual stream, and extract time-frequency features from the audio stream in parallel. The two streams are entered into independent queues and organized into batch inputs through a shared buffer.
[0007] C) The batch input is fed into a lightweight multimodal network containing a visual subnetwork and an audio subnetwork. Feature alignment and fusion are performed in the lightweight fusion layer, and an early departure branch is set in the intermediate layer to output in advance when the confidence reaches the threshold. The lightweight fusion layer is one of cross attention, gated weighting, or other deployable lightweight fusion structures.
[0008] D) Perform post-processing in parallel with inference, including result decoding, trajectory or keypoint smoothing and formatting, and reduce peak memory usage through memory reuse;
[0009] E) When the network is available, submit low-dimensional embeddings or summaries to the consistency verification service for samples or periodically sampled samples in the confidence interval [L,H] for shadow inference. Based on the comparison results, obtain the consistency deviation and return the threshold or scale calibration amount to update the end-side judgment. The consistency deviation is calculated based on one of the probability distribution difference measure or vector distance measure, and the threshold or scale calibration amount is updated accordingly. The consistency verification only submits the dimension-limited embeddings or summaries and the minimum necessary context without uploading the original audiovisual data.
[0010] Furthermore, the knowledge distillation includes distillation of one or more of the teacher's soft target output, selected intermediate features, and attention distribution.
[0011] Furthermore, the visual stream includes one or more of in-screen rendering frames, sampling frames, composite frames, or acquisition frames, and is not limited to originating from any external acquisition device.
[0012] A method for training a student model, characterized by comprising:
[0013] 1) Determine the teacher model and construct the student model skeleton;
[0014] 2) Apply structured pruning to the student model to obtain a sparse structure;
[0015] 3) Perform quantization-aware training or post-training quantization on the sparse structure and perform dynamic range calibration;
[0016] 4) Using teacher soft target output and / or intermediate features and / or attention distribution as supervision, minimize the combined distillation loss to obtain the student model;
[0017] 5) Set up a distillation head for the early exit branch to ensure the accuracy of the output path in advance.
[0018] Furthermore, the pruning target is the importance ranking of convolutional channels, attention heads, or equivalent structures, and the pruning ratio does not exceed a preset upper limit;
[0019] The quantization uses an integer bit width b∈[2,8] and is compatible with the end-side integer matrix multiplication and addition instruction set.
[0020] Furthermore, the quantization uses an integer bit width b∈[2,8] and is compatible with the end-side integer matrix multiplication and addition instruction set;
[0021] The dynamic range calibration is based on the scaling factor and zero point of each layer obtained by statistical analysis of the calibration set.
[0022] A lightweight multimodal audiovisual conversion system includes: —a parallel preprocessing module; —a lightweight multimodal inference module (including early termination and lightweight fusion layer); —a parallel postprocessing module; —a consistency verification module; —a resource and memory scheduling module (circular buffer, token arbitration, memory pool reuse).
[0023] An electronic device includes a processor and a memory, wherein the memory stores a computer program executable on the processor, and the processor executes the program to implement the method described in any of the preceding claims.
[0024] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.
[0025] A computer program product that, when run on a computer, performs the method described in any of the preceding claims.
[0026] Compared with the prior art, the beneficial effects of the present invention are: the lightweight multimodal audiovisual conversion method, device and system (1) low latency / low power consumption: the edge P95 inference latency is significantly reduced, and the memory peak and power consumption are reduced;
[0027] (2) Throughput improvement: pre-processing and inference are carried out in parallel to reduce pipeline idle time;
[0028] (3) Precision maintenance: Consistency verification between distillation and end-cloud ensures long-term operational accuracy and threshold stability;
[0029] (4) Bandwidth friendly: Only low-dimensional embeddings / digests are transmitted for consistency verification, protecting privacy and reducing bandwidth;
[0030] (5) High deployability: The quantized / pruned student model can run stably on the general edge inference engine. Attached Figure Description
[0031] Figure 1 Flowchart of a lightweight multimodal audiovisual conversion edge-side inference method;
[0032] Figure 2 Flowchart of student model training method;
[0033] Figure 3 Block diagram of a lightweight multimodal audiovisual conversion system;
[0034] Figure 4 Block diagram of an electronic device;
[0035] Figure 5 Block diagram of computer-readable storage medium;
[0036] Figure 6 This is a block diagram of a computer program product. Detailed Implementation
[0037] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0038] Please see Figure 1-6 This invention provides a technical solution: a lightweight multimodal audiovisual conversion edge-side inference method, characterized in that it includes:
[0039] A) Obtain the student model, which is obtained by training the teacher model through structured pruning, quantization and knowledge distillation. The structured pruning is performed on the convolutional channels, attention heads or equivalent structures, and operator folding and computation graph optimization are performed before deployment.
[0040] B) Perform region processing and normalization on the visual stream, and extract time-frequency features from the audio stream in parallel. The two streams are entered into independent queues and organized into batch inputs through a shared buffer.
[0041] C) The batch input is fed into a lightweight multimodal network containing a visual subnetwork and an audio subnetwork. Feature alignment and fusion are performed in the lightweight fusion layer, and an early departure branch is set in the intermediate layer to output in advance when the confidence reaches the threshold. The lightweight fusion layer is one of cross attention, gated weighting, or other deployable lightweight fusion structures.
[0042] D) Perform post-processing in parallel with inference, including result decoding, trajectory or keypoint smoothing and formatting, and reduce peak memory usage through memory reuse;
[0043] E) When the network is available, submit low-dimensional embeddings or summaries to the consistency verification service for samples or periodically sampled samples in the confidence interval [L,H] for shadow inference. Based on the comparison results, obtain the consistency deviation and return the threshold or scale calibration amount to update the end-side judgment. The consistency deviation is calculated based on one of the probability distribution difference measure or vector distance measure, and the threshold or scale calibration amount is updated accordingly. The consistency verification only submits the dimension-limited embeddings or summaries and the minimum necessary context without uploading the original audiovisual data.
[0044] I. Parallel Preprocessing
[0045] Video: Perform ROI detection (lightweight object detection / edge-morphology / template matching, etc.) on the input frame, output the cropping box and affine matrix, and perform cropping, scaling and normalization;
[0046] Audio: Parallel execution of pre-emphasis, framing, windowing, STFT and Mel filtering to obtain acoustic features;
[0047] Queues and zero-copy: Enter Q_v and Q_a respectively, and use shared memory / zero-copy to concatenate the cropped ROI and Mel fragments into batch input.
[0048] II. Lightweight Multimodal Networks
[0049] Visual subnetwork: depthwise separable convolution (with Squeeze-Excitation) or lightweight Transformer (linear attention / low-rank projection);
[0050] Audio subnet: 1D convolution / TCN / small Conformer;
[0051] Fusion layer: Cross-attention / gated weighting aligns and fuses visual and audio features;
[0052] Early termination branch: Set an early termination head (classification / regression / sequence determination) in the middle layer. When the confidence level is greater than or equal to the threshold (preferably 0.9), output directly and skip the tail layer.
[0053] III. Compression and Training
[0054] Structured pruning: Based on channel importance / sparse regularization, structured pruning is performed on convolutional channels or attention heads to maintain deployment friendliness;
[0055] Quantization: INT8 quantization is preferred for weights and activations (INT4 is optional); Quantization-Aware Training (QAT) or Post-Training Quantization (PTQ) can be used with dynamic range calibration.
[0056] Knowledge distillation: a multi-objective loss method using logits distillation from cloud-based teacher output, intermediate feature alignment, and attention heatmap alignment;
[0057] Two-stage / joint: distillation followed by pruning and then quantization can be performed, or joint optimization can be used;
[0058] Early retreat alignment: Distillation loss is designed for the early retreat head to ensure the accuracy and usability of the early retreat path.
[0059] IV. Parallel Post-processing
[0060] Results decoding, trajectory / keypoint smoothing, and formatting are performed in parallel with inference; intermediate tensors are reused using a memory pool to reduce allocation and reclamation overhead.
[0061] V. End-to-Cloud Consistency Verification
[0062] Trigger: Perform verification on critical samples with confidence levels in the range [L,H] (e.g., L=0.7, H=0.9) or periodically sampled samples;
[0063] Upload: Only upload low-dimensional embeddings / summaries and minimal necessary context (do not upload original images / audio);
[0064] Shadow reasoning: The cloud-based teacher model provides a comparative output;
[0065] Calibration: Calculate the bias and return the threshold / scale calibration amount or class recalibration; trigger retraining or model hot update when the bias continues to be abnormal.
[0066] VI. Resource Scheduling and Memory Reuse
[0067] Scheduler: Three parallel processes (preprocessing / inference / postprocessing) using a circular buffer and token arbitration;
[0068] Memory pool: Tensor blocks are categorized and reused by size; cross-stage zero-copy reduces peak memory usage;
[0069] Rate limiting: When resources are scarce, priority is given to ROI / audio features and early termination paths.
[0070] Preferably, the knowledge distillation includes distillation of one or more of the teacher's soft target output, selected intermediate features, and attention distribution.
[0071] Preferably, the visual stream includes one or more of in-screen rendering frames, sampling frames, composite frames, or acquisition frames, and is not limited to originating from any external acquisition device.
[0072] Preferably, the early termination branch is output in advance when the confidence level of the intermediate layer is not lower than the threshold, and continues to execute the complete inference of the subsequent layers when it is lower than the threshold.
[0073] Preferably, the parallel processing employs a circular buffer queue and token-based task scheduling to achieve a three-stage parallel pipeline of preprocessing, inference, and postprocessing.
[0074] Preferably, the memory reuse is managed hierarchically by tensor size through memory pools to reduce allocation and reclamation jitter.
[0075] Preferably, the visual subnet adopts any lightweight structure, and the audio subnet adopts one-dimensional convolution, temporal convolution, or other lightweight structures.
[0076] A method for training a student model, characterized by comprising:
[0077] 1) Determine the teacher model and construct the student model skeleton;
[0078] 2) Apply structured pruning to the student model to obtain a sparse structure;
[0079] 3) Perform quantization-aware training or post-training quantization on the sparse structure and perform dynamic range calibration;
[0080] 4) Using teacher soft target output and / or intermediate features and / or attention distribution as supervision, minimize the combined distillation loss to obtain the student model;
[0081] 5) Set up a distillation head for the early exit branch to ensure the accuracy of the output path in advance.
[0082] Preferably, the pruning target is the importance ranking of convolution channels, attention heads, or equivalent structures, and the pruning ratio does not exceed a preset upper limit;
[0083] The quantization uses an integer bit width b∈[2,8] and is compatible with the end-side integer matrix multiplication and addition instruction set.
[0084] Preferably, the quantization uses an integer bit width b∈[2,8] and is compatible with the end-side integer matrix multiplication and addition instruction set;
[0085] The dynamic range calibration is based on the scaling factor and zero point of each layer obtained by statistical analysis of the calibration set.
[0086] A lightweight multimodal audiovisual conversion system for implementing the method described in any one of claims 1–6, comprising: —a parallel preprocessing module; —a lightweight multimodal inference module (including early termination and lightweight fusion layers); —a parallel postprocessing module; —a consistency verification module; —a resource and memory scheduling module (circular buffer, token arbitration, and memory pool reuse).
[0087] An electronic device includes a processor and a memory, wherein the memory stores a computer program executable on the processor, and the processor executes the program to implement the method described in any of the preceding claims.
[0088] A computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the preceding claims.
[0089] A computer program product that, when run on a computer, performs the method described in any of the preceding claims.
[0090] Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing embodiments or make equivalent substitutions for some of the technical features. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.
Claims
1. A lightweight, multimodal audiovisual conversion edge-side inference method, characterized in that, include: A) Obtain the student model, which is obtained by training the teacher model through structured pruning, quantization and knowledge distillation. The structured pruning is performed on the convolutional channels, attention heads or equivalent structures, and operator folding and computation graph optimization are performed before deployment. B) Perform region processing and normalization on the visual stream, and extract time-frequency features from the audio stream in parallel. The two streams are entered into independent queues and organized into batch inputs through a shared buffer. C) The batch input is fed into a lightweight multimodal network containing a visual subnetwork and an audio subnetwork. Feature alignment and fusion are performed in the lightweight fusion layer, and an early departure branch is set in the intermediate layer to output in advance when the confidence reaches the threshold. The lightweight fusion layer is one of cross attention, gated weighting, or other deployable lightweight fusion structures. D) Perform post-processing in parallel with inference, including result decoding, trajectory or keypoint smoothing and formatting, and reduce peak memory usage through memory reuse; E) When the network is available, submit low-dimensional embeddings or summaries to the consistency verification service for samples or periodically sampled samples in the confidence interval [L,H] for shadow inference. Based on the comparison results, obtain the consistency deviation and return the threshold or scale calibration amount to update the end-side judgment. The consistency deviation is calculated based on one of the probability distribution difference measure or vector distance measure, and the threshold or scale calibration amount is updated accordingly. The consistency verification only submits the dimension-limited embeddings or summaries and the minimum necessary context without uploading the original audiovisual data.
2. The lightweight multimodal audiovisual conversion method, apparatus, and system according to claim 1, characterized in that: The knowledge distillation includes distillation of one or more of the teacher's soft target output, selected intermediate features, and attention distribution.
3. The lightweight multimodal audiovisual conversion method, apparatus, and system according to claim 1, characterized in that: The visual stream includes one or more of in-screen rendering frames, sampling frames, composite frames, or acquisition frames, and is not limited to originating from any external acquisition device.
4. A training method for obtaining the student model of claim 1, characterized in that, include: 1) Determine the teacher model and construct the student model skeleton; 2) Apply structured pruning to the student model to obtain a sparse structure; 3) Perform quantization-aware training or post-training quantization on the sparse structure and perform dynamic range calibration; 4) Using teacher soft target output and / or intermediate features and / or attention distribution as supervision, minimize the combined distillation loss to obtain the student model; 5) Set up a distillation head for the early exit branch to ensure the accuracy of the output path in advance.
5. The training method for a student model according to claim 4, characterized in that: The pruning objective is to rank the importance of convolutional channels, attention heads, or equivalent structures, and the pruning ratio shall not exceed a preset upper limit. The quantization uses an integer bit width b∈[2,8] and is compatible with the end-side integer matrix multiplication and addition instruction set.
6. The lightweight multimodal audiovisual conversion method, apparatus, and system according to claim 4, characterized in that: The quantization uses an integer bit width b∈[2,8] and is compatible with the end-side integer matrix multiplication and addition instruction set; The dynamic range calibration is based on the scaling factor and zero point of each layer obtained by statistical analysis of the calibration set.
7. A lightweight multimodal audiovisual conversion system for implementing the method described in any one of claims 1-6, characterized in that, include: —Parallel preprocessing module; —Lightweight multimodal inference module (including early termination and lightweight fusion layer); —Parallel postprocessing module; —Consistency verification module; —Resource and memory scheduling module (ring buffer, token arbitration, memory pool reuse).
8. An electronic device, characterized in that: The method includes a processor and a memory, wherein the memory stores a computer program that can run on the processor, and the processor executes the program to implement the method of any one of claims 1–6.
9. A computer-readable storage medium, characterized in that: It stores a computer program that, when executed by a processor, implements the method described in any one of claims 1–6.
10. A computer program product, characterized in that: When run on a computer, the method described in any one of claims 1–6 is performed.