Intelligent vehicle networking multi-modal learning method and related device

Abstract

The application discloses an intelligent vehicle networking multi-modal learning method and related devices, including: first, in order to reconstruct the missing perception mode, the application designs a knowledge extraction driven cross-node modal reconstruction network. Secondly, in order to select nodes with high quality contribution, the application designs an efficient model contribution evaluation module based on clustering Shapley value. Finally, in order to further alleviate the heterogeneity of cross-node modal, the application introduces a knowledge contribution perception aggregation rule for robust aggregation. Overall, the application solves the problem of missing mode in intelligent vehicle networking while selecting high-quality vehicles to perform aggregation and learning to improve performance.

Description

Technical Field

[0001] This application relates to the fields of Internet of Things and intelligent vehicle networking technology, and in particular to an intelligent vehicle networking multimodal learning method and related devices. Background Technology

[0002] In today's rapidly developing fields of the Internet of Things (IoT) and intelligent connected vehicles, mobile sensing systems have become a key technology. These systems can collect vast amounts of data from various sensors, such as cameras, microphones, accelerometers, and GPS. This data typically exhibits multimodal characteristics, meaning that the data types from different sensors vary. For example, video data is a visual modality, audio data is an auditory modality, and location information is a spatial modality. With the increasing number and types of sensors, the importance of multimodal data analysis is becoming increasingly prominent. However, multimodal data often faces the problem of missing modalities. For example, due to sensor malfunctions, transmission delays, or bandwidth limitations, some vehicles may not be able to provide complete data modalities. To address this issue, Federated Learning (FL), a distributed machine learning method, allows multiple nodes to collaboratively train a model without sharing the original data. Due to its advantages in privacy protection and communication efficiency, it is increasingly being applied to privacy-preserving learning analysis of multimodal data.

[0003] In traditional multimodal federated learning frameworks, incomplete modal data is mainly processed through single-modal subsystem training and interpolation techniques. However, these methods often ignore the possibility of sharing the latent feature space across nodes and fail to effectively evaluate and distinguish the quality of nodes, resulting in limitations on the robustness and performance of the overall system.

[0004] In existing technical solutions, some specific research and implementations attempt to address the problems of the aforementioned traditional methods. These mainly include: Multimodal Federated Learning with Missing Modalities: This proposes a multimodal FL method for missing modalities, performing feature alignment and modality completion on the server side. However, this method relies heavily on global model optimization and lacks sufficient capacity to handle heterogeneous nodes. Data Imputation Techniques for Federated Learning: This uses data interpolation techniques to fill in missing data modalities. While it can improve model integrity to some extent, its effectiveness depends on the quality of the interpolation method and fails to utilize potential feature sharing across nodes. Node Contribution Evaluation in Federated Learning: This proposes a node contribution evaluation mechanism that optimizes the global model by evaluating node data quality and model update contribution. However, this method is mainly designed for single-modal data and has poor adaptability to multimodal data. Clearly, all of these methods have their limitations, primarily in failing to effectively combine cross-node multimodal information and lacking a comprehensive evaluation mechanism for node quality. Summary of the Invention

[0005] This application provides a multimodal learning method and related apparatus for intelligent vehicle networks, which can solve the problem of modality missing in intelligent vehicle networks, while selecting high-quality vehicles to perform aggregation and learning to improve performance.

[0006] In view of this, the first aspect of this application provides a multimodal learning method for intelligent vehicle networking, the method comprising:

[0007] Each node receives the initialized feature generator and global model sent by the server, and obtains a local feature generator and a local global model. Based on the local feature generator, the missing modalities are imputed by reconstructing the induced distribution in a latent space, thereby realizing the iterative reconstruction of the feature information of the missing modalities using the common feature subspace across nodes. Here, the node is a vehicle.

[0008] The local feature generator is optimized, and the knowledge of each modality is learned by authorizing the induced distribution to align the missing modalities. Finally, coupled training is performed to complete the modality reconstruction, and the local feature generator parameters and the updated local global model are sent to the server.

[0009] The server performs a quality assessment on the received feature generators using the improved Shapley Value algorithm to obtain the parameter weights of the feature generators used for aggregation, and divides the node's contribution into local contribution and global contribution. The local contribution and global contribution are determined by calculating the local and global contributions, thereby obtaining the node contribution.

[0010] Design knowledge contribution perception aggregation rules, and based on the knowledge contribution perception aggregation rules, perform node contribution perception aggregation according to the node contribution, thereby outputting the final global model.

[0011] Optionally, the process of performing coupled training to complete modal reconstruction is represented as:

[0012] ;

[0013] In the formula, For the node, For the feature generator, For the global model, Represents a node Local model parameters, This refers to the optimization process for optimizing the local feature generator. For modality The learning process in the alignment missing modality Modal The learning process in aligning missing modalities.

[0014] Optionally, the calculation process of the improved Shapley Value algorithm is expressed as follows:

[0015] F i ( ℱ ) = ∑ S ⊆ N ∖ { ψ ^ i } | S | ! ( | N | − | S | − 1 ) ! | N | ! [ ℱ ( ψ S ( y S ) ∪ { ψ ^ i } ) − ℱ ( ψ S ( y S ) ) ] ;

[0016] In the formula, Represents the global model. The labels are randomly generated.

[0017] Optionally, the knowledge contribution perception aggregation rule is formalized as follows:

[0018] ;

[0019] In the formula, , is a binary flag variable. For the global model, Represents a node Local model parameters.

[0020] A second aspect of this application provides an intelligent vehicle networking multimodal learning system, the system comprising:

[0021] The interpolation unit is used to receive the initialized feature generator and global model sent by the server through each node, obtain the local feature generator and local global model, and interpolate the missing modalities by reconstructing the induced distribution in a latent space based on the local feature generator, thereby realizing the iterative reconstruction of the feature information of the missing modalities using the common feature subspace across nodes, wherein the node is a vehicle;

[0022] The reconstruction unit is used to optimize the local feature generator and learn the knowledge of each modality by authorizing the induced distribution to align the missing modalities. Finally, it performs coupled training to complete the modality reconstruction and sends the local feature generator parameters and the updated local global model to the server.

[0023] The computation unit is used to perform quality evaluation on the received feature generator through the server according to the improved Shapley Value algorithm to obtain the parameter weights of the feature generator for aggregation, and to divide the node's contribution into local contribution and global contribution. The local contribution and global contribution are determined by calculating the local and global contributions, thereby obtaining the node contribution.

[0024] The aggregation unit is used to design knowledge contribution perception aggregation rules, and based on the knowledge contribution perception aggregation rules, perform node contribution perception aggregation according to the node contribution, thereby outputting the final global model.

[0025] Optionally, the process of performing coupled training to complete modal reconstruction is represented as:

[0026] ;

[0027] In the formula, For the node, For the feature generator, For the global model, Represents a node Local model parameters, This refers to the optimization process for optimizing the local feature generator. For modality The learning process in the alignment missing modality Modal The learning process in aligning missing modalities.

[0028] Optionally, the calculation process of the improved Shapley Value algorithm is expressed as follows:

[0029] F i ( ℱ ) = ∑ S ⊆ N ∖ { ψ ^ i } | S | ! ( | N | − | S | − 1 ) ! | N | ! [ ℱ ( ψ S ( y S ) ∪ { ψ ^ i } ) − ℱ ( ψ S ( y S ) ) ] ;

[0030] In the formula, Represents the global model. The labels are randomly generated.

[0031] Optionally, the knowledge contribution perception aggregation rule is formalized as follows:

[0032] ;

[0033] In the formula, , is a binary flag variable. For the global model, Represents a node Local model parameters.

[0034] A third aspect of this application provides an intelligent vehicle networking multimodal learning device, the device comprising a processor and a memory:

[0035] The memory is used to store program code and transmit the program code to the processor;

[0036] The processor is configured to execute the steps of the intelligent vehicle-to-everything (V2X) multimodal learning method as described in the first aspect above, according to the instructions in the program code.

[0037] A fourth aspect of this application provides a computer-readable storage medium for storing program code for executing the intelligent vehicle networking multimodal learning method described in the first aspect above.

[0038] As can be seen from the above technical solutions, this application has the following advantages:

[0039] 1) Cross-node feature reconstruction:

[0040] This application can reconstruct the features of missing modalities by utilizing multimodal feature information across nodes. Compared with existing methods that only apply interpolation techniques or train single-mode subsystems at the node level, this application more effectively captures the potential correlations between different modalities, thereby improving the overall performance and robustness of the model.

[0041] 2) Node evaluation of knowledge contribution perception:

[0042] This application introduces a rigorous node contribution evaluation mechanism to dynamically adjust the weight of nodes in model aggregation. By evaluating the data quality and model update effectiveness of nodes, high-quality and low-quality nodes can be effectively distinguished, enhancing the system's resilience to modal heterogeneity.

[0043] 3) Aggregation rules for knowledge contribution perception:

[0044] During model aggregation, this application employs a knowledge contribution-aware aggregation rule, assigning weights based on the knowledge contribution of each node to optimize global model performance. This approach ensures that high-quality nodes have a greater impact on the global model, effectively improving the overall system performance.

[0045] 4) Efficient knowledge transfer mechanism:

[0046] This application prioritizes knowledge transfer from local to global models, leveraging cross-node multimodal feature information to guide local model updates and feature reconstruction, thereby enhancing the model's learning capabilities. Even with up to 90% lack of modal information, it maintains robust learning performance.

[0047] 5) No additional computational or communication overhead:

[0048] This application achieves robust multimodal federated learning without introducing additional computational or communication overhead. The system architecture design and algorithm optimizations make it more feasible and efficient in practical applications.

[0049] 6) Higher robustness and performance:

[0050] By combining the above-mentioned innovations, this application demonstrates superior performance on several widely recognized multimodal benchmark datasets, outperforming state-of-the-art baseline methods even with high modality information loss rates, showcasing its robustness and performance advantages. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating a multimodal learning method for intelligent vehicle networking provided in the embodiments of this application. Figure one ;

[0052] Figure 2 This is a flowchart illustrating a multimodal learning method for intelligent vehicle networking provided in the embodiments of this application. Figure 2 ;

[0053] Figure 3 This is a schematic diagram of the structure of an intelligent vehicle networking multimodal learning system provided in an embodiment of this application. Detailed Implementation

[0054] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present application, and not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present application.

[0055] Please see Figure 1 and 2 The present application provides a multimodal learning method for intelligent vehicle networking, comprising:

[0056] Step 101: By receiving the initialized feature generator and global model sent by the server at each node, a local feature generator and a local global model are obtained. Based on the local feature generator, the missing modalities are interpolated by reconstructing the induced distribution in a latent space, thereby realizing the iterative reconstruction of the feature information of the missing modalities using the common feature subspace across nodes, where the node is a vehicle.

[0057] It should be noted that, in specific implementation, step 101 proceeds as follows:

[0058] Step 1011: The server initializes the feature generator. and global model Send to each vehicle k;

[0059] Step 1012, Imputing missing modal features:

[0060] Unlike existing works that focus on reconstructing local missing modalities while ignoring cross-node feature information, this application aims to synergistically utilize the common feature subspace across nodes to iteratively reconstruct the feature information of missing modalities.

[0061] Specifically, to gain a deeper understanding of the data distribution of missing modalities among nodes and to use this understanding to guide the training of local models for incomplete modalities, this application uses conditional distributions to characterize the modal data distribution of each node. Let... This represents the conditional distribution described above, which is tailored to each node and consistent with the distribution of ground-based data. This distribution contains the necessary knowledge to guide multi-modal training of incomplete modalities:

[0062] Q k = a r g m a x Q k :  k →  k  y ~ p ( y k )  x ~ Q k ( X k ∣ y k ) [ l o g p ( y ∣ x ; oh k ) ] (1)

[0063] in, and Let represent the prior and posterior distributions of the basic facts for the target label, respectively, and \ This represents the data point in mode m of node k.

[0064] Step 1013, Privacy-protected missing modal imputation:

[0065] Since step 1012 requires sharing information about the real data with the generator on the server, this could lead to privacy breaches. Therefore, this application adopts a more feasible alternative: reconstructing the induced distribution. In a potential space. This potential space is more compact than the original data space, which helps alleviate certain privacy-related issues:

[0066] ψ k = a r g m a x ψ k :  k →  k  y ~ p ^ ( y k )  z ~ ψ k ( Z k ∣ y k ) [ l o g p ( y ∣ z ; oh k ) ] (2)

[0067] Step 102: Optimize the local feature generator and learn the knowledge of each modality through authorized induced distribution to align the missing modalities. Finally, perform coupled training to complete modality reconstruction and send the local feature generator parameters and the updated local global model to the server.

[0068] It should be noted that, in specific implementation, step 102 proceeds as follows:

[0069] Step 1021, Local Feature Generator Optimization:

[0070] Nodes are generated from parameterized condition generators. take Gain insights by extracting knowledge from missing modal data. The optimization process is as follows:

[0071] m i n oh G k J ( oh G k ) = m i n  y ~ p ( y r )  z ~ ψ k ( z ∣ y r ) [ ℒ ( s ( g ( z ; i k ) ) , y ) ] (3)

[0072] in, This represents a set of random tags generated from local tags. This represents the logits output of the predictor. This represents the nonlinear activation applied to these logits.

[0073] Step 1022: Align missing modalities:

[0074] On the other hand, it is necessary to refine the feature subspace to more accurately encapsulate the local knowledge of nodes. For example, considering a bimodal task, this application can utilize labels... Export the generated potential space: ,in and This represents the latent features of each modality. Assume... To represent the missing modality, the objective of this application is to further grant... By absorbing knowledge from various modalities, the integrity and generalization ability of the feature space are enhanced. For modalities... The learning process can be represented as follows:

[0075] ℒ K L m 0 ( oh G k ; oh k ) = m i n oh G k ∑ i = 1 B  x ~  k [ D K L [ ( f 0 ( x i m 0 ; oh k m 0 ) ‖  i m 0 ) ] ] (4)

[0076] ℒ K L m 1 ( oh G k ; oh k ) = m i n oh G k ∑ i = 1 I  x ~  k [ D K L [ ( f 1 ( x i m 1 ; oh k m 1 ) ‖  i m 1 ) ] ] (5)

[0077] Where B represents the data of a training batch, and I represents the number of data in a complete modality.

[0078] Step 1023, Coupled Training:

[0079] Based on the steps above, we can see that the overall optimization goal for each FL client is:

[0080] (6)

[0081] Step 103: The server performs a quality assessment on the received feature generator according to the improved Shapley Value algorithm to obtain the parameter weights of the feature generator used for aggregation, and divides the node's contribution into local contribution and global contribution. The local contribution and global contribution are determined by calculating the local and global contributions, thereby obtaining the node contribution.

[0082] It should be noted that, in specific implementation, step 103 proceeds as follows:

[0083] Step 1031: Evaluate the contribution of the feature generator:

[0084] Considering the inherent heterogeneity of data between nodes and the various modality loss scenarios that frequently occur on a single node, a local distribution model for knowledge transfer is proposed. The naive aggregation of ideas may inadvertently lead to a general shift in the realm of collective knowledge, resulting in counterproductive outcomes.

[0085] To mitigate this problem, this application uses the Shapley Value (SV) method to quantitatively evaluate each distinct value. Marginal contribution to the overall learning task. However, directly applying SV to calculate the marginal contribution of a single node is computationally intensive, especially in FL scenarios involving hundreds of mobile devices.

[0086] To address this challenge, this application incorporates the K-means clustering algorithm to reduce the computational complexity of SV calculations. Specifically, this application uses the K-means clustering algorithm to... generated Perform clustering to obtain the included Multiple clusters are obtained. Then, the generator parameters in the clusters are averaged to obtain... As a representative node of the cluster. In this way, this application can obtain... Node representations are used as sets in SV. Therefore, the final calculation of SV can be expressed as:

[0087] F i ( ℱ ) = ∑ S ⊆ N ∖ { ψ ^ i } | S | ! ( | N | − | S | − 1 ) ! | N | ! [ ℱ ( ψ S ( y S ) ∪ { ψ ^ i } ) − ℱ ( ψ S ( y S ) ) ] (7)

[0088] in, Represents the global model. These are randomly generated labels. Subsequently, standardized... The weights are used as parameters for generator aggregation.

[0089] Step 1032, Evaluate node contribution:

[0090] To summarize node contributions in a fine-grained manner, this application divides the contribution of each node in each round into local contribution and global contribution. Local contribution represents the node's performance in the local training of that round, while global contribution represents the node's impact on model aggregation. This application can calculate local and global contributions as follows:

[0091] (8)

[0092] in, Represents the global model. Represents the test dataset. This represents the local model parameters of node k. This application requires traversing all nodes to calculate the above contribution values.

[0093] Step 104: Design knowledge contribution perception aggregation rules, and based on the knowledge contribution perception aggregation rules, perform node contribution perception aggregation according to node contributions, thereby outputting the final global model.

[0094] It should be noted that, in specific implementation, step 104 proceeds as follows:

[0095] Node contribution-aware aggregation strategy:

[0096] After determining the global and local contributions, this application strives to adaptively incorporate them into the selection process of clients participating in model aggregation, thereby temporarily excluding clients with lower update quality.

[0097] To achieve this goal, this application introduces a binary flag variable, represented as A contribution metric has been designed to evaluate the flag parameters based on whether certain predefined criteria are met. The value of . This ensures that only high-performance nodes are included in the aggregation process.

[0098] The calculation of contribution metrics is detailed below: For the t-th round of evaluation, this application sets the global model performance evaluation result for the (t-1)-th round as follows: Then the judgment condition for the flag parameter can be expressed as: if Then set Otherwise set .when If a node is selected in a given round, it is retained for aggregation; otherwise, it is excluded from this round. Therefore, the node contribution-aware aggregation rule can be formalized as follows:

[0099] (9)

[0100] This application provides a multimodal learning method for intelligent vehicle networks (V2V) in scenarios with missing modalities. First, to reconstruct the missing perceptual modalities, a knowledge extraction-driven cross-node modality reconstruction network is designed. Second, to select nodes with high-quality contributions, an efficient model contribution evaluation module based on Shapley clustering values ​​is designed. Finally, to further alleviate the heterogeneity of cross-node modalities, a knowledge contribution-aware aggregation rule for robust aggregation is introduced. Overall, this application solves the modality missing problem in intelligent V2V while selecting high-quality vehicles for aggregation and learning, thereby improving performance.

[0101] The above is a multimodal learning method for intelligent vehicle networking provided in the embodiments of this application. The following is a multimodal learning system for intelligent vehicle networking provided in the embodiments of this application.

[0102] Please see Figure 3 The present application provides an intelligent vehicle networking multimodal learning system, comprising:

[0103] The interpolation unit 201 is used to receive the initialized feature generator and global model sent by the server through each node, obtain the local feature generator and local global model, and interpolate the missing modalities by reconstructing the induced distribution in a latent space based on the local feature generator, thereby realizing the iterative reconstruction of the feature information of the missing modalities using the common feature subspace across nodes, where the node is a vehicle.

[0104] Reconstruction unit 202 is used to optimize the local feature generator, learn the knowledge of each modality through authorized induced distribution to align the missing modalities, and finally perform coupled training to complete modality reconstruction. At the same time, the local feature generator parameters and the updated local global model are sent to the server.

[0105] The calculation unit 203 is used to obtain the parameter weights of the feature generator for aggregation by performing quality evaluation on the received feature generator through the server according to the improved Shapley Value algorithm, and to divide the node contribution into local contribution and global contribution. The local contribution and global contribution are determined by calculating the local and global contributions, thereby obtaining the node contribution.

[0106] Aggregation unit 204 is used to design knowledge contribution perception aggregation rules, and based on the knowledge contribution perception aggregation rules, perform node contribution perception aggregation according to node contributions, thereby outputting the final global model.

[0107] Furthermore, this application embodiment also provides an intelligent vehicle networking multimodal learning device, the device including a processor and a memory:

[0108] The memory is used to store program code and transmit the program code to the processor;

[0109] The processor is used to execute the steps of the intelligent vehicle network multimodal learning method as described in the above method embodiments, according to the instructions in the program code.

[0110] Furthermore, this application embodiment also provides a computer-readable storage medium for storing program code, which is used to execute the intelligent vehicle network multimodal learning method described in the above method embodiment.

[0111] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the system and unit described above can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0112] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented, for example, in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0113] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0114] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection between apparatuses or units through some interfaces, and may be electrical, mechanical, or other forms.

[0115] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0116] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0117] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes: USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, optical disks, and other media capable of storing program code.

[0118] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A multimodal learning method for intelligent vehicle networking, characterized in that, include: Each node receives the initialized feature generator and global model sent by the server, and obtains a local feature generator and a local global model. Based on the local feature generator, the missing modalities are imputed by reconstructing the induced distribution in a latent space, thereby realizing the iterative reconstruction of the feature information of the missing modalities using the common feature subspace across nodes. Here, the node is a vehicle. The local feature generator is optimized, and knowledge of each modality is learned by constraining the induced distribution to align the missing modalities. Finally, coupled training is performed to complete the modality reconstruction. At the same time, the parameters of the local feature generator and the updated local global model are uploaded to the server. The server performs a quality assessment on the received feature generators using the improved Shapley Value algorithm to obtain the parameter weights of the feature generators used for aggregation, and divides the node contributions into local contributions and global contributions. By calculating the local contributions and global contributions, the node contributions are obtained. Design knowledge contribution perception aggregation rules, and based on the knowledge contribution perception aggregation rules, perform node contribution perception aggregation according to the node contribution, thereby outputting the final global model; The process of performing coupled training to complete modality reconstruction is represented as follows: ； In the formula, For the node, For the feature generator, For the global model, Represents a node Local model parameters, This refers to the optimization process for optimizing the local feature generator. For modality The learning process in the alignment missing modality Modal The learning process in aligning missing modalities; The calculation process of the improved Shapley Value algorithm is expressed as follows: ； In the formula, This represents the global model learning method.

2. The intelligent vehicle-to-everything (V2X) multimodal learning method according to claim 1, characterized in that, The knowledge contribution perception aggregation rule is formalized as follows: ； In the formula, , is a binary flag variable. For the global model, Represents a node Local model parameters.

3. A multimodal learning system for intelligent vehicle networking, characterized in that, include: The interpolation unit is used to receive the initialized feature generator and global model sent by the server through each node, obtain the local feature generator and local global model, and interpolate the missing modalities by reconstructing the induced distribution in a latent space based on the local feature generator, thereby realizing the iterative reconstruction of the feature information of the missing modalities using the common feature subspace across nodes, wherein the node is a vehicle; The reconstruction unit is used to optimize the local feature generator and learn the knowledge of each modality by constraining the induced distribution to align the missing modalities. Finally, it performs coupled training to complete the modality reconstruction and uploads the local feature generator parameters and the updated local global model to the server. The computation unit is used to perform quality evaluation on the received feature generator through the server according to the improved Shapley Value algorithm to obtain the parameter weights of the feature generator for aggregation, and to divide the node's contribution into local contribution and global contribution. The local contribution and global contribution are determined by calculating the local and global contributions, thereby obtaining the node contribution. An aggregation unit is used to design knowledge contribution perception aggregation rules, and based on the knowledge contribution perception aggregation rules, perform node contribution perception aggregation according to the node contribution, thereby outputting the final global model. The process of performing coupled training to complete modality reconstruction is represented as follows: ； In the formula, For the node, For the feature generator, For the global model, Represents a node Local model parameters, This refers to the optimization process for optimizing the local feature generator. For modality The learning process in the alignment missing modality Modal The learning process in aligning missing modalities; The calculation process of the improved Shapley Value algorithm is expressed as follows: ； In the formula, This represents the global model.

4. The intelligent vehicle networking multimodal learning system according to claim 3, characterized in that, The knowledge contribution perception aggregation rule is formalized as follows: ； In the formula, , is a binary flag variable. For the global model, Represents a node Local model parameters.

5. A smart vehicle networking multimodal learning device, characterized in that, The device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is used to execute the intelligent vehicle network multimodal learning method according to any one of claims 1-2 according to the instructions in the program code.

6. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store program code for executing the intelligent vehicle network multimodal learning method according to any one of claims 1-2.

Images