Multi-modal compression model training method, multi-modal feature extraction method, electronic equipment and storage medium

Abstract

The invention provides a multi-modal compression model training method, a multi-modal feature extraction method, electronic equipment and a storage medium, and belongs to the technical field of Internet of Vehicles. The method comprises the steps that at least two sub-models are constructed respectively, each sub-model is used for extracting features of modal data of a corresponding type, and the structures of the sub-models are the same; performing joint training on each sub-model to obtain the weight of each output channel in each layer of each sub-model; generating a global mask of each output channel in each layer of each sub-model through pruning processing based on the weight of each output channel in each layer of each sub-model; the global masks of the same output channels of the same layer of each sub-model are the same; for each output channel of each layer in each sub-model, compressing the weight of each output channel in each layer of each sub-model according to the global mask corresponding to each output channel; and generating a quantization table for recording the compression strategy and the compression parameter corresponding to each weight. The computing power redundancy can be reduced, and the power consumption and the inter-frame synchronization delay are reduced.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of Internet of Vehicles, in particular to a multi-modal compression model training method, a multi-modal feature extraction method, an electronic device and a storage medium. BACKGROUND

[0002] The compression and deployment of the current vehicle-road cooperative multi-modal perception model mainly follow the technical path of independent compression-late fusion-layered inference, that is, single-modal model compression, multi-modal fusion, edge / vehicle inference.

[0003] In related technologies, the single-modal isolated compression and layered cascade fusion paradigm leads to visual, radar, ground Figure Three Although the modal network is independently compressed, a large number of semantic redundant convolution / attention calculation units are still retained. The vehicle-side ECU (Electronic Control Unit) needs to sequentially or in parallel perform three times of heterogeneous forward calculation, the algorithm power consumption is linearly superimposed, the power consumption is >30W, and the inter-frame synchronization delay is >15ms. SUMMARY

[0004] Therefore, the present application provides a multi-modal compression model training method, a multi-modal feature extraction method, an electronic device and a computer readable storage medium to solve the problems of algorithm power redundancy, high power consumption and long delay in multi-modal data calculation.

[0005] To achieve the above purpose, the first aspect of the present application provides a multi-modal compression model training method, which comprises:

[0006] At least two sub-models are constructed respectively, wherein each sub-model is used to extract the features of the corresponding type of modal data, and the structures of each sub-model are the same;

[0007] Each sub-model is jointly trained to obtain the weights of each output channel in each layer of each sub-model;

[0008] Based on the weights of each output channel in each layer of each sub-model, a global mask corresponding to each output channel in each layer of each sub-model is generated through pruning processing; wherein the global masks of the same output channels in the same layer of each sub-model are the same;

[0009] For each output channel of each layer in each sub-model, the weights of each output channel in each layer of each sub-model are compressed according to the global mask corresponding to each output channel; a quantization table is generated, which is used to record the compression strategy and compression parameters corresponding to each weight; the quantization table and each weight are parameters of a multi-modal compression model, and the multi-modal compression model is a fusion model of each sub-model.

[0010] In some embodiments, the joint training of each of the sub-models obtains the weights of each output channel in each layer of each of the sub-models, including:

[0011] According to the multi-modal sample data, the joint global sparse regularization training of each of the sub-models obtains the weights of each output channel in each layer of each of the sub-models; wherein the multi-modal sample data includes sample data of the corresponding category modal data of each of the sub-models.

[0012] In some embodiments, the joint global sparse regularization training of each of the sub-models according to the multi-modal sample data obtains the weights of each output channel in each layer of each of the sub-models, including:

[0013] Loss calculation step: based on each of the sub-models, the loss of each modal sample data is calculated according to the multi-modal sample data; according to the loss of each modal sample data and the current weights of each output channel in each layer of each of the sub-models, the total loss of the multi-modal sample data is calculated;

[0014] Weight update step: the gradient of the total loss with respect to each of the weights is calculated, and the current weights of each output channel in each layer of each of the sub-models are updated by a gradient descent algorithm;

[0015] The loss calculation step and the weight update step are repeatedly executed until a preset number of times are reached, and the weights of each output channel in each layer of each of the sub-models are obtained.

[0016] In some embodiments, the calculation of the total loss of the multi-modal sample data according to the loss of each modal sample data and the current weights of each output channel in each layer of each of the sub-models includes:

[0017] According to the loss of each modal sample data and the preset loss coefficient corresponding to each of the modal sample data, a weighted sum is performed to obtain a total loss of each modal;

[0018] The sum of the L1 norms of each of the weights is calculated, and the product of the sum of the L1 norms of each of the weights and a preset coefficient regularization coefficient is calculated to obtain a regularization loss;

[0019] The sum of the total loss of each modal and the regularization loss is calculated to obtain the total loss of the multi-modal sample data.

[0020] In some embodiments, the generation of the global mask corresponding to each output channel in each layer of each of the sub-models by pruning processing based on the weights of each output channel in each layer of each of the sub-models includes:

[0021] For each output channel in each layer of each sub-model, the sum of the absolute values ​​of the weights of the output channels in each sub-model is calculated to obtain the importance score of the output channel;

[0022] The importance scores of each output channel in each layer of each sub-model are sorted from high to low to obtain an importance score sequence;

[0023] Based on preset pruning parameters, the importance score sequence is pruned to generate a global mask corresponding to each output channel in each layer of each sub-model.

[0024] In some embodiments, after generating the quantization table, the method further includes:

[0025] In response to an update to the multimodal compression model, a first target weight that has been updated is determined, and the first target weight is sent to the target device.

[0026] Secondly, this invention provides a multimodal feature extraction method, the method comprising:

[0027] Multimodal data is input into a multimodal compression model, wherein different types of modal data in the multimodal data are input into different sub-models in the multimodal compression model; the multimodal compression model is a fusion model of the sub-models, obtained by jointly training the sub-models.

[0028] For each of the sub-models, the weights of each output channel in each layer of the sub-model are obtained by quantization and restoration according to the quantization table in the multimodal compression model; the target modal data input to the sub-model is feature extracted according to the weights of each output channel in each layer of the sub-model to obtain the features of the target modal data.

[0029] In some embodiments, the method further includes:

[0030] The system resources of the multimodal compression model are monitored, and the system resources include at least one of the following: computing power, power consumption, and network bandwidth;

[0031] The step of extracting features from the target modal data input to the sub-model based on the weights of each output channel in each layer of the sub-model to obtain the features of the target modal data includes:

[0032] In response to the system resources not meeting the preset conditions, a second target weight is determined, and feature extraction is performed on the target modal data input to the sub-model based on the second target weight to obtain the features of the target modal data; wherein, the second target weight is a preset proportion of the weights corresponding to all output channels of the sub-model.

[0033] Thirdly, the present invention provides an electronic device comprising:

[0034] One or more processors;

[0035] A memory storing one or more programs that, when executed by one or more processors, enable the one or more processors to implement the multimodal compressed model training method as described above; or the multimodal feature extraction method as described above.

[0036] One or more I / O interfaces are connected between the processor and the memory and configured to enable information interaction between the processor and the memory.

[0037] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the multimodal compressed model training method as described above; or, the multimodal feature extraction method as described above.

[0038] The present invention has the following advantages:

[0039] This invention provides a multimodal compression model training method, comprising: constructing at least two sub-models, wherein each sub-model is used to extract features of a corresponding type of modality data, and the sub-models have the same structure; jointly training the sub-models to obtain the weights of each output channel in each layer of each sub-model; generating a global mask corresponding to each output channel in each layer of each sub-model through pruning based on the weights of each output channel in each layer of each sub-model; wherein the global mask of the same output channel in the same layer of each sub-model is the same; compressing the weights of each output channel in each layer of each sub-model according to the global mask corresponding to each output channel; and generating a quantization table, which is used to record the compression strategy and compression parameters corresponding to each weight. This invention, by establishing sub-models with the same structure and sharing the pruned global mask for each output channel in each layer of each sub-model, can achieve synchronous output of multimodal features from multimodal data through a single forward computation, thereby reducing computational redundancy, power consumption, and inter-frame synchronization latency.

[0040] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0041] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used together with the embodiments of the invention to explain the invention and do not constitute a limitation thereof. The above and other features and advantages will become more apparent to those skilled in the art from the detailed description of exemplary embodiments with reference to the accompanying drawings, in which:

[0042] Figure One This is a flowchart of the training process for a multimodal compression model provided in an embodiment of the present invention;

[0043] Figure Two This is a flowchart of multimodal feature extraction provided in an embodiment of the present invention;

[0044] Figure Three A schematic diagram of multimodal feature extraction and adaptive inference scheduling provided for a specific embodiment of the present invention;

[0045] Figure Four This is a block diagram of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] The specific embodiments of the present invention will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit the present invention.

[0047] As used in this invention, the term "and / or" includes any and all combinations of one or more of the associated enumerated entries.

[0048] The terminology used in this invention is for describing particular embodiments only and is not intended to limit the invention. As used herein, the singular forms "a" and "the" are also intended to include the plural forms, unless the context clearly indicates otherwise.

[0049] When the terms “comprising” and / or “made of” are used in this invention, the presence of the said feature, integral, step, operation, element and / or component is specified, but the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or groups thereof is not excluded.

[0050] Unless otherwise specified, all terms used in this invention (including technical and scientific terms) have the same meaning as commonly understood by one of ordinary skill in the art. It will also be understood that terms such as those defined in common dictionaries should be interpreted as having the meaning consistent with their meaning in the context of the relevant art and the invention, and will not be interpreted as having an idealized or overly formal meaning unless expressly so defined by the invention.

[0051] The current compression and deployment of vehicle-road cooperative multimodal perception models mainly follow a technical path of independent compression-post-fusion-layered inference, namely single-modal model compression, multimodal fusion, and edge / vehicle-mounted inference, which mainly includes the following three schemes:

[0052] 1. Single-modal model compression: Through methods such as structured pruning, channel pruning, or low-order quantization, parameters are reduced only for a single network such as vision or radar.

[0053] 2. Multimodal fusion: First, the visual, radar, and map modalities are compressed separately, and then feature stitching or attention fusion is performed at a higher level. There is a lack of unified compression across modalities.

[0054] 3. Edge / Vehicle Inference: Model splitting or layered inference is adopted to offload some computations to the vehicle and complete the rest at the edge. However, multiple models still need to be maintained, resulting in redundant computing power and storage.

[0055] The three approaches described above suffer from problems such as redundant computing power, complex model management, and insufficient cross-modal resource sharing. Specifically, the three-modal networks perform forward computation independently, and similar convolution / attention operations are repeatedly executed, resulting in a waste of approximately 35%-50% of computing power. The combined size of the independent model files leads to a >1GB usage of automotive flash memory and high OTA (Over-The-Air) bandwidth consumption, making it difficult to meet automotive-grade differential update requirements (<100MB). Pruning / quantization strategies cannot achieve three-modal feature acquisition in a single computation, resulting in poor perceptual consistency and high decision latency.

[0056] To address the aforementioned technical problems, in a first aspect, embodiments of the present invention provide a multimodal compression model training method, which is applied to cloud devices. Figure One The flowchart for training a multimodal compression model provided in this embodiment of the invention is as follows: Figure One As shown, the multimodal compression model training method includes the following steps:

[0057] Step S11: Construct at least two sub-models, each of which is used to extract features of the corresponding type of modality data, and each sub-model has the same structure.

[0058] This invention is applied to a vehicle-road cooperative scenario. In this scenario, a visual sub-model (V-Net), a radar sub-model (R-Net), and a map sub-model (M-Net) can be constructed. The visual sub-model is used to extract visual features from image data, the radar sub-model is used to extract radar features from point cloud data, and the map sub-model is used to extract map features from map data. The visual sub-model, radar sub-model, and map sub-model have the same structure, that is, the three sub-models have the same number of layers and the same number of output channels in each layer.

[0059] The same basic convolution / transformer block can be used to construct the visual sub-model, radar sub-model, and map sub-model respectively. The number of output channels per layer of each sub-model is consistent, and the output channel numbers of each layer of each sub-model correspond.

[0060] Each sub-model (V-Net, R-Net, M-Net) is generated using the same convolutional / Transformer template (such as Conv-BN-ReLU or Transformer-Block), ensuring a strict correspondence between layer and channel numbers for each sub-model and providing a structural basis for sharing pruning masks. The numbering rules for output channels are as follows:

[0061] The hierarchical order is consistent, and the numbering of the first level is: , where n is the number of output channels in this layer of the sub-model; the second layer is numbered as follows: And so on.

[0062] Cross-modal channel mapping: For the same layer in the three sub-models, the number of output channels is the same, such as 64, 128, and 256. The k-th output channel is denoted as follows in the visual sub-model, radar sub-model, and map sub-model: V-Net: R-Net: M-Net: .

[0063] Step S12: Jointly train each sub-model to obtain the weights of each output channel in each layer of each sub-model.

[0064] This step is an "end-to-end joint training" process, in which the weights of each output channel in each layer of the three sub-models are updated synchronously in the same optimization loop step, rather than being trained separately in stages and then spliced ​​together.

[0065] Step S13: Based on the weights of each output channel in each layer of each sub-model, generate the global mask corresponding to each output channel in each layer of each sub-model through pruning; wherein, the global mask of the same output channel in the same layer of each sub-model is the same.

[0066] The global mask is a binary mask, which can be represented as M(l, k), where l represents the layer and k represents the output channel. The global mask applies to the visual sub-model, radar sub-model, and map sub-model simultaneously. During the sub-model definition phase, the same YAML (YAML Ain't Markup Language) configuration file is used to generate the weight tensors for the three sub-models, ensuring that the parameters of the output channels of each layer are identical.

[0067] After training the weights of each output channel in each layer of each sub-model, each output channel in each layer of the sub-model is traversed, and a global mask M(l, k) is generated by sorting the weights according to the absolute values ​​of the sparse regularized weights. In this way, the numbering of the same layer-channel corresponds one-to-one in the three sub-models. During pruning or quantization, only the global number (l, k) needs to be operated on, which can achieve unified compression across modalities.

[0068] By aligning the structures of each sub-model (layer-channel alignment) and using a unified decision-making mechanism, the granularity of pruning is increased from "single output channel" to "cross-modal output channel position". A pruning decision (keep or delete) will be applied simultaneously to the corresponding output channel position of all sub-models, achieving "one forward computation, output multimodal features", and significantly reducing computational redundancy.

[0069] Step S14: For each output channel of each layer in each sub-model, compress the weights of each output channel in each layer of each sub-model according to the global mask corresponding to each output channel; generate a quantization table, which is used to record the compression strategy and compression parameters corresponding to each weight; the quantization table and each weight are the parameters of the multimodal compression model, which is the fusion model of each sub-model.

[0070] In this step, for each unpruned global mask, the weights corresponding to each unpruned global mask are compressed according to a preset compression length, and a quantization table is generated. The quantization table records the compression strategy corresponding to each weight, such as the compression length, for example, 8 bits; the quantization table also records the compression parameters corresponding to each weight, such as the scaling factor and zero point. The quantization table is used to restore the weights of each output channel in each layer of each sub-model during model inference.

[0071] The quantization table and weights are parameters of a multimodal compressed model that integrates the visual sub-model, radar sub-model, and map sub-model. After generating the quantization table, the multimodal compressed model training is complete.

[0072] This invention provides a multimodal compression model training method, comprising: constructing at least two sub-models, wherein each sub-model is used to extract features of a corresponding type of modality data, and the sub-models have the same structure; jointly training the sub-models to obtain the weights of each output channel in each layer of each sub-model; generating a global mask corresponding to each output channel in each layer of each sub-model through pruning based on the weights of each output channel in each layer of each sub-model; wherein the global mask of the same output channel in the same layer of each sub-model is the same; compressing the weights of each output channel in each layer of each sub-model according to the global mask corresponding to each output channel; and generating a quantization table, which is used to record the compression strategy and compression parameters corresponding to each weight. This invention, by establishing sub-models with the same structure and sharing the pruned global mask for each output channel in each layer of each sub-model, can achieve synchronous output of multimodal features from multimodal data through a single forward computation, thereby reducing computational redundancy, power consumption, and inter-frame synchronization latency.

[0073] This invention achieves simultaneous output of multimodal features in a single forward computation of three modes by aligning the layers and output channels of each sub-model and sharing pruning masks. This reduces computing power redundancy by about 40%, resulting in a vehicle-side comprehensive computing power consumption of <10 TOPS (Tera Operations Per Second), meeting automotive-grade real-time requirements.

[0074] In some embodiments, the joint training of each sub-model to obtain the weights of each output channel in each layer of each sub-model includes the following steps: performing global sparse regularization training on each sub-model based on multimodal sample data to obtain the weights of each output channel in each layer of each sub-model; wherein, the multimodal sample data includes sample data of the corresponding type of modality data of each sub-model.

[0075] In some embodiments, the step of jointly performing global sparse regularization training on each sub-model based on multimodal sample data to obtain the weights of each output channel in each layer of each sub-model includes the following steps:

[0076] Loss calculation steps: Based on each sub-model, calculate the loss of each modal sample data according to the multimodal sample data; calculate the total loss of the multimodal sample data according to the loss of each modal sample data and the current weight of each output channel in each layer of each sub-model.

[0077] Weight update steps: Calculate the gradient of the total loss with respect to each weight, and update the current weights of each output channel in each layer of each sub-model using the gradient descent algorithm;

[0078] Repeat the loss calculation and weight update steps until the preset number of times is reached to obtain the weights of each output channel in each layer of each sub-model.

[0079] By simultaneously minimizing the loss and sparsity regularization terms of sample data from various modalities using the gradient descent algorithm, the sub-model automatically compresses the weights of output channels that contribute little to all modalities to near zero while maintaining perception accuracy. This global sparsity provides a basis for subsequent sharing of global masks; as long as an output channel has low overall importance in multimodal applications, it can be uniformly pruned, thereby achieving cross-modal computing power sharing.

[0080] In some embodiments, calculating the total loss of multimodal sample data based on the loss of various modal sample data and the current weights of each output channel in each layer of each sub-model includes the following steps: performing a weighted summation based on the loss of various modal sample data and the preset loss coefficients corresponding to various modal sample data to obtain the total loss of each modality; calculating the sum of the L1 norms of each weight, and calculating the product of the sum of the L1 norms of each weight and the preset regularization coefficient to obtain the regularization loss; calculating the sum of the total loss of each modality and the sum of the regularization loss to obtain the total loss of multimodal sample data.

[0081] The losses for various modal sample data include, but are not limited to, cross-entropy and regression error, used to measure the accuracy of perception / localization / map matching respectively. Preset loss coefficients for various modal sample data are used to balance the importance of different modalities or the imbalance of data volume; preset regularization coefficients are used to control the L1 penalty strength of the absolute value of the weights.

[0082] In some embodiments, the loss function for calculating the total loss of multimodal sample data is the following formula (1):

[0083] (1)

[0084] in, For the loss of visual sample data, For the loss of radar sample data, This represents the loss of map sample data; The loss coefficients corresponding to various preset modal sample data; These are the preset regularization coefficients; Let c represent the set of weights corresponding to all output channels of each sub-model, where c represents the channel and C represents the set of channels.

[0085] In this embodiment of the invention, the L1 regularization term is used to constrain the global weights, so that the cross-modal weights of unimportant channels approach zero, forming a shareable sparse distribution and achieving global sparse regularization.

[0086] In some embodiments, generating the global mask corresponding to each output channel of each layer of each sub-model through pruning based on the weights of each output channel in each layer of each sub-model (i.e., step S13) includes the following steps:

[0087] Step S131: For each output channel in each layer of each sub-model, calculate the sum of the absolute values ​​of the weights of the output channels in each sub-model to obtain the importance score of the output channel.

[0088] The importance score of the output channel (l, k) can be calculated according to the following formula (2):

[0089] Importance(l,k) = |W_lk^v| + |W_lk^r| + |W_lk^m|; (2)

[0090] Wherein, Importance(l,k) is the importance score of the output channel (l,k), |W_lk^v| is the absolute value of the weight of the output channel (l,k) in the visual sub-model, |W_lk^r| is the absolute value of the weight of the output channel (l,k) in the radar sub-model, and |W_lk^m| is the absolute value of the weight of the output channel (l,k) in the map sub-model.

[0091] In this step, the importance scores of all output channels in the sub-model can be calculated according to the above formula (2).

[0092] Step S132: Sort the importance scores of each output channel in each layer of each sub-model from high to low to obtain the importance score sequence.

[0093] Step S133: Prune the importance score sequence according to the preset pruning parameters to generate a global mask corresponding to each output channel in each layer of each sub-model.

[0094] In this embodiment of the invention, the pruning parameter can be the retention ratio P. In some embodiments, P can be 30%-70%.

[0095] In this step, the importance score sequence is pruned according to the retention ratio P, generating a global mask for each output channel in each layer of each sub-model. Specifically, for the output channels ranked in the top P, the global mask M(l,k)=1; for the output channels ranked in (1-P), the global mask M(l,k)=0. Output channels with a global mask of 1 are retained, and output channels with a global mask of 0 are pruned. The pruned global mask applies to all sub-models, meaning that the output channels of the visual sub-model, radar sub-model, and map sub-model are uniformly retained or pruned. This means the same global mask applies synchronously to V-Net, R-Net, and M-Net, achieving multi-modal coverage with a single decision and enabling shared computing power.

[0096] The global mask reflects the output channel's performance in images, point clouds, and ground planes. Figure Three The overall importance of each modality is determined by the global mask. If the global mask is 0, it means that the corresponding output channel is not important in the three modalities and can be pruned to achieve cross-modal computing power sharing.

[0097] In some embodiments, after generating the quantization table, the multimodal compression model training method may further include the following steps: in response to an update of the multimodal compression model, determining the updated first target weights and sending the first target weights to the target device.

[0098] In this embodiment of the invention, the target device can be an on-vehicle device. In the case of multimodal compression model updates, the cloud device can send the weights of those output channels whose global mask has changed (i.e., the first target weights) to the on-vehicle device, which then merges and completes the hot update locally.

[0099] This invention is based on a mask-driven incremental patching mechanism, which enables OTA updates to transmit only the weights corresponding to global mask changes, reducing the update packet size to 5%-10% of the full model, achieving minute-level hot updates, and significantly reducing storage costs and network bandwidth pressure.

[0100] Secondly, the present invention also provides a multimodal feature extraction method, which is applied to vehicle-mounted equipment. Figure Two The flowchart for multimodal feature extraction provided in the embodiments of the present invention is as follows: Figure Two As shown, the multimodal feature extraction method includes the following steps:

[0101] Step S21: Input the multimodal data into the multimodal compression model, wherein different types of modal data in the multimodal data are input into different sub-models in the multimodal compression model; the multimodal compression model is a fusion model of the sub-models, which is obtained by jointly training the sub-models.

[0102] Step S22: For each sub-model in each sub-model, the weights of each output channel in each layer of the sub-model are obtained by quantization and restoration according to the quantization table in the multimodal compression model; the features of the target modal data input to the sub-model are extracted according to the weights of each output channel in each layer of the sub-model to obtain the features of the target modal data.

[0103] After receiving multimodal data, the multimodal compression model inputs different modalities into their respective sub-models. For example, image data is input into the vision sub-model, point cloud data into the radar sub-model, and map data into the map sub-model. The vision, radar, and map sub-models determine the compression strategy and parameters corresponding to the weights of each channel based on a quantization table, and then restore the weights of each channel in their respective sub-models. Based on the restored weights, the vision, radar, and map sub-models perform a forward propagation on the image data, point cloud data, and map data, respectively, to obtain visual features, radar features, and map features. The multimodal compression model then outputs these visual features, radar features, and map features.

[0104] This invention establishes sub-models with identical structures and shares the pruned global mask for each output channel of each layer of each sub-model. This enables the synchronous output of multimodal features from multimodal data through a single forward computation, thereby reducing computational redundancy, power consumption, and inter-frame synchronization delay.

[0105] In some embodiments, the multimodal compression model training method may further include the following steps: monitoring the system resources of the multimodal compression model, wherein the system resources include at least one of the following: computing power, power consumption, and network bandwidth.

[0106] Accordingly, the step of extracting features from the target modal data of the input sub-model based on the weights of each output channel in each layer of the sub-model to obtain the features of the target modal data includes the following steps: in response to the system resources not meeting the preset conditions, determining the second target weight, and extracting features from the target modal data of the input sub-model based on the second target weight to obtain the features of the target modal data; wherein, the second target weight is a preset proportion of the weights corresponding to all output channels of the sub-model.

[0107] This invention utilizes an Adaptive Inference Scheduler (AIS) to monitor resources such as computing power, power consumption, and network bandwidth in real time. If these resources do not meet preset conditions, such as computing power or network bandwidth being below a threshold, or power consumption being above a threshold, then a lightweight quantum set of the multimodal compression model is loaded. This lightweight quantum set is the lightweight model, which includes the weights of some output channels, not all output channels. For example, 30% of the output channel weights are retained. It should be noted that the retained weights are the weights of the top 30% of output channels in terms of importance score (i.e., the second target weights). The multimodal compression model performs a forward calculation based on the second target weights to obtain the lightweight intermediate features of each modality. The vehicle-mounted device uploads lightweight intermediate features from each modality to an edge server via a V2X (Vehicle to Everything) network. The edge server runs a complete, unpruned full model. Upon receiving the intermediate features from the vehicle-mounted device, it starts from that feature point and continues the remaining computation (i.e., the computation portion corresponding to the "70% output channel after masking") to generate the final, high-precision perception results (such as accurate 3D bounding boxes and semantic segmentation maps). The edge server then sends the calculated final results back to the vehicle-mounted device. The vehicle-mounted device receives the remotely calculated results, fuses them with the results from its local lightweight model, and ultimately uses them for vehicle perception, planning, and decision-making.

[0108] To clearly illustrate the technical solution of the present invention, the following is combined with... Figure Three This will be illustrated with a specific example. For example... Figure Three As shown, the multimodal feature extraction process is as follows:

[0109] The instantaneous starting current I_start = 15A supplied to the ECU by the vehicle is used to estimate the peak power consumption, which is then used by the AIS to determine whether to enter "lightweight mode".

[0110] Various sensors (such as cameras and radar) synchronously acquire multimodal data, including image data, point cloud data, and high-precision map data. A hard synchronization window for the sensors is used, with Δt_sync ≤ 5 ms, to ensure the synchronization of data from cameras, radar, and ground sensors. Figure Three Modal data timestamp alignment.

[0111] Image data, point cloud data, and high-precision map data are input into the multimodal compression model for a forward computation. The global mask M(l,k) in the multimodal compression model has a sparsity of 45%, meaning that about half of the output channels are pruned. The visual sub-model, radar sub-model, and map sub-model share the pruned mask.

[0112] The forward formula for calculating features is: F=φ((X⊙M)·W_q+b);

[0113] Among them, \(W_q\) is the quantized weight tensor; the weights of the output channels with global mask = 1 in the same layer are compressed with 8 bits;

[0114] \(X\) is the input multi-modal data, namely image data, point cloud data, and map data;

[0115] \(\odot\) is the channel-wise multiplication with the mask, and the output channels to be pruned are directly set to 0 without calculation;

[0116] \(\varphi\) is the activation function, which can be activation functions such as ReLU (Rectified Linear Unit) / GELU (Gaussian Error Linear Unit), etc.;

[0117] One forward calculation can simultaneously obtain three groups of outputs: visual feature \(F_v\), radar feature \(F_r\), and map feature \(F_m\).

[0118] \(F_v\in(C\times H\times W)\), where \(C\) is the number of remaining output channels, and \(H\times W\) is the spatial resolution;

[0119] \(F_r\in(C\times N)\), where \(N\) is the number of point clouds or voxels; \(C\) is the number of remaining output channels, which is the same as the number of remaining output channels in the visual sub-model;

[0120] \(F_m\in(C\times L)\), where \(L\) is the length of the map elements (vector / raster); \(C\) is the number of remaining output channels, which also remains the same as the number of remaining output channels in the visual sub-model.

[0121] The three resources monitored in real-time by the Adaptive Inference Scheduler AIS are: available computing power \(P\), power consumption \(P_w\), and V2X link bandwidth \(B\). The thresholds for the three resources are as follows: \(P_{thr}=2\) TOPS, \(P_{w thr}=15\) W, \(B_{thr}=5\) Mbps.

[0122] The decision function is: \(d=(P < P_{thr})+(P_w > P_{w thr})+(B < B_{thr})\). If the sum of the three boolean values in the decision function \(\geq1\), it enters the collaborative inference branch.

[0123] The vehicle-side device locally runs the first 30% of the output channels of the global mask (i.e., the lightweight model), and the remaining features are completed by the RSU (Road Side Unit) / cloud, ensuring that the end-to-end delay < 30 ms.

[0124] In the case of updating the multi-modal compression model, the cloud device only downloads the weight blocks corresponding to the changes in the global mask. The OTA patch is 5% - 10% of the full model volume, and it can achieve minute-level hot updates on the vehicle side without restarting the ECU.

[0125] This invention addresses the problems of fragmented compression of multimodal perception models, redundant consumption of computing resources, and low efficiency of OTA updates in related technologies. It constructs a unified compression model that allows for cross-modal computing resource sharing, standardized compression strategies, and adaptive inference tasks, specifically solving the following three technical problems:

[0126] Technical Issue 1: Cross-modal computing power redundancy problem

[0127] In related technologies, the single-modal isolated compression and hierarchical cascade fusion paradigms lead to differences in vision, radar, and ground-based technologies. Figure Three Although the modal networks are independently compressed, they still retain a large number of semantically redundant convolutional / attention computation units. The vehicle-side ECU needs to perform three heterogeneous forward computations sequentially or in parallel, resulting in linearly accumulating computational power consumption, power consumption >30W, and inter-frame synchronization delay >15ms. This embodiment of the invention achieves simultaneous output of multimodal features with a single forward computation of three modalities through layer-channel alignment and shared pruning masks, reducing computational power redundancy by about 40% and making the overall vehicle-side computational power consumption <10 TOPS, meeting automotive-grade real-time requirements.

[0128] Technical Issue 2: Storage and OTA Bandwidth Overload

[0129] Independent compression results in the cumulative size of the three-modal model files, with onboard Flash storage exceeding 1GB and the full OTA update package exceeding 200MB, violating the stringent automotive-grade requirement of <100MB for differential updates. This invention addresses this by generating a single model file through unified compression and designing a mask-driven incremental patching mechanism. This allows OTA updates to transmit only the weight blocks corresponding to mask changes, reducing the update package size to 5%-10% of the full model size, achieving minute-level hot updates, and significantly reducing storage costs and network bandwidth pressure.

[0130] Technical Issue 3: Insufficient Perception Consistency and Adaptive Capability under Resource Constraints

[0131] Models with fixed compression ratios struggle to adapt to the dynamically changing computing power, power consumption, and V2X network bandwidth in vehicle-to-everything (V2X) scenarios. When computing power is degraded, such as when the onboard NPU (embedded neural network processor) drops to 60% TOPS or network jitter occurs, the related calculations cannot be flexibly adjusted, leading to a decrease in perception frame rate or loss of cross-modal feature synchronization, threatening driving safety. This invention addresses this by employing an Adaptive Inference Scheduler (AIS) that dynamically switches between a lightweight model and remote edge completion based on real-time resource status. This ensures that, in resource-constrained scenarios, perception latency remains <30ms and cross-modal synchronization error <5ms, improving system robustness and the safety of L3 / L4 autonomous driving.

[0132] This invention breaks through the limitations of the existing paradigm of "compression is isolation, fusion is stacking". Through a cross-modal unified compression architecture, it realizes the sharing of computing power, the concentration of storage, and the elasticity of inference of vehicle-road cooperative perception system, providing core enabling technology for the next generation of low-power, high real-time, and evolvable intelligent vehicle computing platform.

[0133] In this embodiment of the invention, after a single global sparse regularization, a binary global mask M(l,k) is uniformly generated for all output channels. The global mask is applied synchronously to V-Net, R-Net, and M-Net, eliminating redundant calculations for the same output channel in the three modes. This can reduce computing power by approximately 45%, achieving lower power consumption and higher frame rates on the same hardware platform.

[0134] This invention employs 8-bit quantization for high-importance output channels with a global mask of 1, and prunes low-importance output channels to generate a unified quantization table. Model storage is reduced from 100% to approximately 20%, and the impact of quantization error on perception accuracy is controllable. Weight storage is only about 1 / 5 of the original model, significantly reducing storage requirements and facilitating deployment on automotive ECUs / embedded platforms.

[0135] In this embodiment of the invention, a shared computation graph is executed once on the vehicle / edge hardware, which simultaneously yields visual features Fᵥ, radar features Fᵣ, and map features F. m This eliminates the overhead of multiple forward passes or cross-network feature alignments, improving the overall latency of the sensing link. Compared to three independent forward computations in related technologies, inference latency is reduced by approximately 30%, and cross-modal synchronization delay is eliminated.

[0136] This invention utilizes an Adaptive Inference Scheduler (AIS) to monitor resource metrics such as computing power, power consumption, and network bandwidth in real time. If these resource metrics do not meet preset conditions, it automatically switches between a lightweight model and remote inference. Specifically, it retains only the lightweight model representing the first 30% of the output channels of the global mask, while the remaining features are supplemented through remote inference at the edge or in the cloud, ensuring real-time perception even under resource constraints. Even if computing power drops to 60% TOPS, the system can still complete obstacle detection and path planning within 30ms, meeting the real-time requirements of autonomous driving.

[0137] In this embodiment of the invention, during multimodal compressed model updates, only the weight blocks (OTAI-Patch) corresponding to the global mask changes are issued, and the hot update is completed locally on the vehicle. The update package size is only 5%-10% of the full model, significantly reducing OTA bandwidth usage and storage pressure, while ensuring version consistency.

[0138] Thirdly, embodiments of the present invention also provide an electronic device. Figure Four A block diagram of an electronic device provided in an embodiment of the present invention, such as... Figure FourAs shown, the electronic device includes:

[0139] One or more processors 401;

[0140] The memory 402 stores one or more programs that, when executed by one or more processors, enable the one or more processors to implement either the multimodal compressed model training method or the multimodal feature extraction method described above.

[0141] One or more I / O interfaces 403 are connected between the processor and the memory and configured to enable information exchange between the processor and the memory.

[0142] Among them, processor 401 is a device with data processing capabilities, including but not limited to central processing unit (CPU); memory 402 is a device with data storage capabilities, including but not limited to random access memory (RAM, more specifically SDRAM, DDR, etc.), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory (FLASH); I / O interface (read-write interface) 403 is connected between processor 401 and memory 402, and can realize information interaction between processor 401 and memory 402, including but not limited to data bus (Bus).

[0143] In some embodiments, the processor 401, memory 402, and I / O interface 403 are interconnected via a bus, and thus connected to other components of the computing device.

[0144] Fourthly, embodiments of the present invention also provide a computer-readable medium having a computer program stored thereon. When the program is executed by a processor, it implements the multimodal compression model training method or the multimodal feature extraction method provided in this embodiment. To avoid repetition, the specific steps of the vehicle-road cooperative model training method will not be repeated here.

[0145] Fifthly, embodiments of the present invention also provide a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code. When the computer-readable code is run in a processor of an electronic device, the processor in the electronic device executes the above-described multimodal compression model training method, or the multimodal feature extraction method.

[0146] Those skilled in the art will understand that all or some of the steps, systems, or apparatuses in the methods, systems, and apparatuses described above can be implemented as software, firmware, hardware, or suitable combinations thereof. In hardware implementations, the division between functional modules / units mentioned above does not necessarily correspond to the division of physical components; for example, a physical component may have multiple functions, or a function or step may be performed collaboratively by several physical components. Some or all physical components may be implemented as software executed by a processor, such as a central processing unit, digital signal processor, or microprocessor, or as hardware, or as an integrated circuit, such as an application-specific integrated circuit (ASIC). Such software may be distributed on a computer-readable medium, which may include computer storage media (or non-transitory media) and communication media (or transient media). As is known to those skilled in the art, the term computer storage media includes volatile and non-volatile, removable and non-removable media implemented in any method or technology for storing information (such as computer-readable instructions, data structures, program modules, or other data). Computer storage media include, but are not limited to, RAM, ROM, EEPROM, flash memory or other memory technologies, CD-ROM, digital versatile disc (DVD) or other optical disc storage, magnetic cartridges, magnetic tape, disk storage or other magnetic storage devices, or any other medium that can be used to store desired information and can be accessed by a computer. Furthermore, it is well known to those skilled in the art that communication media typically contain computer-readable instructions, data structures, program modules, or other data in modulated data signals such as carrier waves or other transmission mechanisms, and may include any information delivery medium.

[0147] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element.

[0148] Those skilled in the art will understand that although some embodiments described herein include certain features that are included in other embodiments but not others, combinations of features from different embodiments are meant to be within the scope of this embodiment and form different embodiments.

[0149] It is understood that the above embodiments are merely exemplary implementations used to illustrate the principles of the present invention, and the present invention is not limited thereto. For those skilled in the art, various modifications and improvements can be made without departing from the spirit and essence of the present invention, and these modifications and improvements are also considered to be within the scope of protection of the present invention.

Claims

1. A method for training a multimodal compressed model, characterized in that, The method includes: At least two sub-models are constructed respectively, wherein each sub-model is used to extract features of the corresponding type of modality data, and the sub-models have the same structure; The sub-models are jointly trained to obtain the weights of each output channel in each layer of each sub-model; Based on the weights of each output channel in each layer of each sub-model, a global mask corresponding to each output channel in each layer of each sub-model is generated through pruning; wherein, the global mask of the same output channel in the same layer of each sub-model is the same; For each output channel of each layer in each sub-model, the weights of each output channel in each layer of each sub-model are compressed according to the global mask corresponding to each output channel; a quantization table is generated, which is used to record the compression strategy and compression parameters corresponding to each weight; the quantization table and each weight are the parameters of the multimodal compression model, which is the fusion model of each sub-model.

2. The method according to claim 1, characterized in that, The joint training of the sub-models to obtain the weights of each output channel in each layer of each sub-model includes: Based on the multimodal sample data, the sub-models are jointly trained using global sparse regularization to obtain the weights of each output channel in each layer of each sub-model; wherein, the multimodal sample data includes sample data of the corresponding type of modality data of each sub-model.

3. The method according to claim 2, characterized in that, The step of jointly training each of the sub-models with global sparse regularization based on multimodal sample data to obtain the weights of each output channel in each layer of each sub-model includes: Loss calculation steps: Based on each sub-model, calculate the loss of each modal sample data according to the multimodal sample data; calculate the total loss of the multimodal sample data according to the loss of each modal sample data and the current weight of each output channel in each layer of each sub-model. Weight update steps: Calculate the gradient of the total loss with respect to each of the weights, and update the current weights of each output channel in each layer of each of the sub-models using the gradient descent algorithm; Repeat the loss calculation step and the weight update step until a preset number of times are reached to obtain the weights of each output channel in each layer of each sub-model.

4. The method according to claim 3, characterized in that, The step of calculating the total loss of the multimodal sample data based on the loss of the various modal sample data and the current weights of each output channel in each layer of each sub-model includes: Based on the loss of the various modal sample data and the preset loss coefficients corresponding to the various modal sample data, a weighted sum is performed to obtain the total loss of each modality; Calculate the sum of the L1 norms of each weight, and calculate the product of the sum of the L1 norms of each weight and the preset regularization coefficient to obtain the regularization loss; The total loss of the multimodal sample data is obtained by summing the sum of the losses of each modality and the regularization loss.

5. The method according to claim 1, characterized in that, The process of generating a global mask corresponding to each output channel of each layer of each sub-model through pruning based on the weights of each output channel in each layer of each sub-model includes: For each output channel in each layer of each sub-model, the sum of the absolute values ​​of the weights of the output channels in each sub-model is calculated to obtain the importance score of the output channel; The importance scores of each output channel in each layer of each sub-model are sorted from high to low to obtain an importance score sequence; Based on preset pruning parameters, the importance score sequence is pruned to generate a global mask corresponding to each output channel in each layer of each sub-model.

6. The method according to claim 1, characterized in that, After generating the quantization table, the process also includes: In response to an update to the multimodal compression model, a first target weight that has been updated is determined, and the first target weight is sent to the target device.

7. A multimodal feature extraction method, characterized in that, The method includes: Multimodal data is input into a multimodal compression model, wherein different types of modal data in the multimodal data are input into different sub-models in the multimodal compression model; the multimodal compression model is a fusion model of the sub-models, obtained by jointly training the sub-models. For each of the sub-models, the weights of each output channel in each layer of the sub-model are obtained by quantization and restoration according to the quantization table in the multimodal compression model; the target modal data input to the sub-model is feature extracted according to the weights of each output channel in each layer of the sub-model to obtain the features of the target modal data.

8. The method according to claim 7, characterized in that, Also includes: The system resources of the multimodal compression model are monitored, and the system resources include at least one of the following: computing power, power consumption, and network bandwidth; The step of extracting features from the target modal data input to the sub-model based on the weights of each output channel in each layer of the sub-model to obtain the features of the target modal data includes: In response to the system resources not meeting the preset conditions, a second target weight is determined, and feature extraction is performed on the target modal data input to the sub-model based on the second target weight to obtain the features of the target modal data; wherein, the second target weight is a preset proportion of the weights corresponding to all output channels of the sub-model.

9. An electronic device, characterized in that, include: One or more processors; A memory having stored one or more programs thereon, which, when executed by one or more processors, cause the one or more processors to implement the multimodal compressed model training method as described in any one of claims 1-6, or the multimodal feature extraction method as described in any one of claims 7-8; One or more I / O interfaces are connected between the processor and the memory and configured to enable information interaction between the processor and the memory.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the multimodal compressed model training method as described in any one of claims 1-6, or the multimodal feature extraction method as described in any one of claims 7-8.

Images