A multi-source data efficient collaborative learning method and system fusing prior knowledge

Abstract

This invention relates to the field of intelligent driving technology and discloses a method and system for efficient collaborative learning of multi-source data that integrates prior knowledge. The method includes: identifying candidate regions of traffic participants in an RGB image using a target detection network, cropping the regions to obtain image patches; inputting the image patches into a pre-trained ResNet backbone network to extract a prior knowledge tensor; applying dynamic gating weights to each prior semantic feature vector in the prior knowledge tensor to obtain a modulated prior knowledge tensor; injecting the modulated prior knowledge tensor as additional keys and values ​​into a Transformer encoder to obtain a fused attention matrix; identifying a set of hazardous object indices based on the attention weights and relative distances corresponding to each traffic participant in the fused attention matrix; inputting this hazardous object index set into an avoidance decision module to output control commands. This invention enhances the environmental understanding and driving safety of intelligent driving systems in complex traffic scenarios.

Description

Technical Field

[0001] This invention relates to the field of intelligent driving technology, specifically to a method and system for efficient collaborative learning of multi-source data that integrates prior knowledge. Background Technology

[0002] In the field of intelligent driving, the effective use of prior knowledge is crucial for improving the performance of autonomous driving systems. As autonomous driving technology develops, systems need to maintain stable and reliable operation in complex and ever-changing environments. This prior knowledge not only influences the selection of perception algorithms and planning strategies but is also closely related to all modules of the entire system.

[0003] Traditional intelligent driving systems face several key challenges in utilizing prior knowledge: First, the representation and fusion mechanisms of prior knowledge lack systematic design, making it difficult to efficiently integrate prior information such as target categories and traffic rules with multi-source observation data like RGB images and millimeter-wave radar point clouds, thus failing to fully leverage the guiding role of knowledge in perception. Second, in terms of data quality control, existing systems struggle to effectively handle domain differences between multi-source data (especially simulation and real data) and lack online evaluation of data quality across different modalities, impacting model training and fusion effectiveness. Third, the injection of prior knowledge often employs static and fixed methods, making it difficult to dynamically modulate different priors based on the current traffic scenario (e.g., traffic light status) and target category, resulting in a lack of adaptive enhancement of the perception features of key targets. These problems are particularly pronounced in high-risk scenarios such as complex road conditions, extreme weather, and emergency obstacle avoidance, hindering the system's environmental understanding and driving safety.

[0004] For example, Chinese patent document CN115438788A discloses a method and system for data cognitive computation constrained by prior knowledge. This patent primarily addresses the problem of training on small sample data by constructing a specific scenario-target task-prior knowledge matrix and constraining feature learning based on a snake active contour recognition model. However, this patent only uses prior knowledge as a constraint condition for model training and fails to achieve a systematic processing of knowledge representation at the system architecture level. Summary of the Invention

[0005] To address the aforementioned technical problems, this invention provides a method and system for efficient collaborative learning of multi-source data that integrates prior knowledge. The method simultaneously acquires RGB images of the vehicle's forward-looking view, millimeter-wave radar point clouds, and traffic light status signals at the same sampling time, and preprocesses and assesses the quality of each modal data. Prior semantic features of each traffic participant are extracted using a pre-trained target detection network and a ResNet backbone network to construct a prior knowledge tensor. Dynamic gating modulation is applied to each prior semantic feature vector in the prior knowledge tensor based on the traffic light status and target category. The multi-source data are then encoded into word sequences, and combined with the quality assessment results, the modulated prior knowledge tensor is injected as additional keys and values ​​into a Transformer encoder to obtain a fused attention matrix. Finally, based on the attention weights and relative distances of each traffic participant in the fused attention matrix, a set of hazardous object indices is identified, and avoidance control commands are output accordingly. This invention enables adaptive and dynamic injection of prior knowledge in the multi-source fusion process, and alleviates the quality and domain differences of multi-source data through quality assessment and domain alignment, thereby improving the collaborative perception efficiency, dangerous object identification accuracy and driving safety of the system in complex traffic scenarios.

[0006] To solve the above-mentioned technical problems, the present invention adopts the following technical solution: In a first aspect, the present invention provides an efficient collaborative learning method for multi-source data that integrates prior knowledge, comprising: Simultaneously acquire multimodal observation data, including vehicle-mounted forward-looking RGB images, millimeter-wave radar point clouds, and traffic light status signals; perform data preprocessing and quality assessment on each modal observation data; the status signal values ​​0, 1, and 2 correspond to red, yellow, and green lights, respectively; The pre-trained object detection network identifies candidate regions for traffic participants in RGB images. After cropping, each image patch and its corresponding category label are obtained. Each image patch is then input into the pre-trained ResNet backbone network to extract prior semantic feature vectors, which form a prior knowledge tensor. Based on the state signal and category label, dynamic gating weights are applied to each prior semantic feature vector in the prior knowledge tensor to obtain the modulated prior knowledge tensor. After the RGB image, millimeter-wave radar point cloud, and state signal are encoded by their respective encoders, they are combined with the quality assessment results to form a multimodal word sequence. The modulated prior knowledge tensor is injected as an additional key and value into the Transformer encoder to obtain the fused attention matrix. Based on the attention weights and relative distances of each traffic participant in the fusion attention matrix, a set of hazardous object indices is identified. This set of hazardous object indices is then input into the avoidance decision module, which outputs control commands.

[0007] In one embodiment, the data preprocessing and quality assessment of the modal observation data specifically includes: For any modal observation data Execute in sequence: Preprocessing removes outliers and missing values ​​to obtain preprocessed data. ; Calculate the temporal correlation coefficient based on data from T consecutive sampling times. ; Based on the integrity of preprocessed data Correlation coefficient with time series Calculate quality score Where λ1 and λ2 are weighting coefficients; the quality score The scaling factor used for the corresponding modal lexical sequence.

[0008] In one embodiment, the step of inputting each map tile into a pre-trained ResNet backbone network to extract prior semantic feature vectors and construct a prior knowledge tensor specifically includes: Prior semantic feature vector The ResNet backbone network The outputs from different levels are fused using a feature pyramid to obtain: ; in, This indicates the ResNet backbone network's... Hierarchical output, To project the output of this level onto a projection function of a uniform dimension, For the learnable first The fusion weights of the hierarchy, and satisfying ; All prior semantic feature vectors constitute the prior knowledge tensor. : ; The number of traffic participants or blocks.

[0009] In one embodiment, the step of applying dynamic gating weights to each prior semantic feature vector in the prior knowledge tensor based on the state signal and category label to obtain the modulated prior knowledge tensor specifically includes: Based on traffic light status signals For prior knowledge tensors Each prior semantic feature vector in By corresponding category label By applying dynamic gating weights, we obtain the modulated prior knowledge tensor. The gating weights of the i-th prior semantic feature vector for: ; in, It is the Sigmoid activation function. For learnable parameters, Indicates one-hot encoding; ; This represents element-wise multiplication. Let represent the modulated i-th prior semantic feature vector.

[0010] In one embodiment, the step of encoding the RGB image, millimeter-wave radar point cloud, and state signal respectively through their respective encoders, and then concatenating them into a multimodal word sequence based on the quality assessment results, specifically includes: RGB image Input the Vision Transformer encoder to get Image block words; millimeter-wave radar point cloud Input the PointNet encoder to get Each point cloud word element; For state signals After one-hot encoding and linear projection, we obtain There are state words; and they satisfy the following conditions: ; Represents multi-source lexical sequences The total number of lexical units; Will Image block words, Point cloud terminology and Each state lexical unit is concatenated into a multi-source lexical sequence. Each modal lexical unit is multiplied by the quality score obtained through quality assessment for the corresponding modality before concatenation.

[0011] In one embodiment, the step of injecting the modulated prior knowledge tensor as additional keys and values ​​into the Transformer encoder to obtain the fused attention matrix specifically includes: Modulated prior knowledge tensor As additional key-value pairs, they are injected into the Transformer encoder to obtain the fused attention matrix. : ; in, , , , This represents the query matrix obtained by projecting the multi-source word sequence X onto the query matrix. This represents the bond matrix obtained by projecting X onto the bond matrix. Let represent the additional bond matrix obtained by key projection from the modulated prior knowledge tensor. To query the projection weight matrix, The key projection weight matrix, The feature dimension representing attention, This indicates a concatenation operation along the lexical sequence dimension. This represents the softmax function. This indicates transpose.

[0012] In one embodiment, identifying the dangerous object index set based on the attention weights and relative distances of each traffic participant in the fused attention matrix specifically includes: The fusion attention matrix A is expanded along the spatial dimension into a multidimensional heatmap matrix of safe distance distribution. The value at coordinates (u,v) in the heatmap matrix of safe distance distribution Defined as: ; in, This represents the detection bounding box region of the i-th traffic participant in the image plane. Here, it is an indicator function; Let be the attention weight of the i-th traffic participant at its corresponding position in A. It is a monotonically decreasing distance inverse correlation function. Let be the relative distance between the i-th traffic participant and the vehicle; The number of traffic participants; Heatmap matrix of safe distance distribution The value exceeds the preset threshold The index of traffic participants corresponding to the connected regions Composition of hazardous object index set Since the detection frames of each traffic participant do not overlap, within the detection frame of the i-th traffic participant, the value of the safety distance distribution heatmap matrix H is equal to the attention weight corresponding to that traffic participant. Inverse correlation function with distance Therefore, for H that exceeds the preset threshold, the product of the two is used. Finding the index of the corresponding traffic participant in a connected region is equivalent to the following equation: .

[0013] In one embodiment, inputting the hazardous object index set into the avoidance decision module and outputting control commands specifically includes: The avoidance decision module is based on a set of hazardous object indexes. Output control commands ,in, For longitudinal acceleration, It is a lateral steering angle command, and satisfies: ; in, Let i be the position vector of the i-th traffic participant relative to the vehicle. This is a preset control strategy function.

[0014] In one embodiment, the Transformer encoder is trained using a mixture of real road data and simulation data: Real-world road data includes triples composed of RGB images collected by vehicle sensors, millimeter-wave radar point clouds, and traffic light status signals, along with manually labeled hazardous object indexes. Simulation data includes triples with the same structure as real-world road data generated by the simulation platform, along with corresponding labels. During training, a source identifier is added to each training sample. ,when When r is 0, the training samples are real road data; when r is 1, the training samples are simulated data; this is achieved through joint loss. Optimize the Transformer encoder: ; in, Binary cross-entropy loss for identifying hazardous objects. For domain alignment loss, For prior constraint loss, These are weight hyperparameters; ; ; in, The Transformer encoder predicts the hazard probability of traffic participants. These are genuine hazard labels manually marked. E represents the probability that the training samples are derived from simulation data for the domain discriminator; E denotes the expectation value on the training samples. The status signals of traffic lights satisfy the following prior constraints: When the traffic light status signal When the traffic light is red, the gating weights corresponding to the tiles categorized as pedestrians and non-motorized vehicles are determined. The expected value is not less than the value corresponding to the tile with the category label "motor vehicle". Expected value; when When the traffic light is green, the tile labeled "Motor Vehicle" corresponds to... The expected value is not less than the value of the tile with the category label "pedestrian". Expected value; Prior constraint loss for: ; in, Preset prior constraint triples Set, where s represents the values ​​of the traffic light status signal. , For the two categories of traffic participants, values ​​are assigned to the labels. To constrain the interval hyperparameter, This indicates the calculation of expectation. Let be the category label for the i-th tile.

[0015] In a second aspect, the present invention provides a computer system including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps of the method of any embodiment of the first aspect.

[0016] Compared with the prior art, the beneficial technical effects of the present invention are: By fusing vehicle-mounted RGB images, millimeter-wave radar point clouds, and traffic light status signals at the same sampling time, this approach first extracts prior semantic features of each traffic participant using target detection and a ResNet network. Then, it dynamically gates and modulates prior knowledge based on traffic light status and target category. Subsequently, the multi-source data is encoded into a word sequence, and the modulated prior knowledge is injected as an additional key value into the Transformer encoder. This allows the attention mechanism to enhance key perception features based on current traffic rules (such as the meaning of traffic lights) and object categories. Finally, based on aggregated attention weights and relative distance, it accurately identifies dangerous objects and outputs avoidance control commands. This scheme achieves adaptive and dynamic injection of prior knowledge in the multi-source fusion process, significantly improving collaborative perception efficiency and the accuracy of dangerous object identification. At the same time, it provides timely and reliable input for downstream decision-making, thereby enhancing the environmental understanding and driving safety of intelligent driving systems in complex traffic scenarios. Attached Figure Description

[0017] Figure 1 This is a flowchart of the method of the present invention. Detailed Implementation

[0018] A preferred embodiment of the present invention will now be described in detail with reference to the accompanying drawings.

[0019] like Figure 1 As shown, an efficient collaborative learning method for multi-source data that integrates prior knowledge, as presented in this invention, includes the following steps: S1, synchronously acquires multimodal observation data, including vehicle-mounted forward-looking RGB images, millimeter-wave radar point clouds, and traffic light status signals; performs data preprocessing and quality assessment on each modal observation data; the status signal values ​​0, 1, and 2 correspond to red light, yellow light, and green light, respectively; S2, the pre-trained object detection network identifies traffic participant candidate regions in the RGB image, and after cropping, each patch and its corresponding category label are obtained; each patch is input into the pre-trained ResNet backbone network to extract prior semantic feature vectors and form a prior knowledge tensor. S3. Based on the state signal and category label, apply dynamic gating weights to each prior semantic feature vector in the prior knowledge tensor to obtain the modulated prior knowledge tensor. S4. The RGB image, millimeter-wave radar point cloud, and state signal are encoded by their respective encoders and concatenated into a multimodal word sequence. Combined with the quality assessment results, the modulated prior knowledge tensor is injected as an additional key and value into the Transformer encoder to obtain the fused attention matrix. S5 identifies a set of hazardous object indices based on the attention weights and relative distances of each traffic participant in the fusion attention matrix, inputs this set of hazardous object indices into the avoidance decision module, and outputs control commands.

[0020] The present invention will be described in detail below in several parts.

[0021] 1. Data acquisition.

[0022] Vehicle-mounted multimodal observation data were acquired at the same sampling time. The multimodal observation data included vehicle-mounted forward-looking RGB images. Millimeter-wave radar point cloud and the traffic light status signal corresponding to the current lane , where 0, 1, and 2 represent red light, yellow light, and green light, respectively; Represents the height and width of an RGB image. This represents the lateral and longitudinal coordinates of the k-th radar point in the vehicle coordinate system, as well as its radial velocity relative to the vehicle. K is the number of points in this frame of the point cloud. It represents the set of real numbers, and the form with dimension superscripts represents the real number space of the corresponding dimension.

[0023] This also includes data preprocessing and quality assessment of the observation data for each modality. For any given modality observation data... Execute in sequence: Preprocessing: Remove outliers and missing values ​​to obtain preprocessed data. ; Timing consistency check: based on data from T consecutive sampling times Calculate the time series correlation coefficient ; Quality assessment: based on the completeness of preprocessed data Correlation coefficient with time series Calculate the quality score: ; in, Weighting coefficients; quality score Used as a scaling factor for the corresponding modal token sequence.

[0024] 2. Construct a prior knowledge tensor.

[0025] Through a pre-trained target detection network N candidate regions for traffic participants were identified, and after cropping, a set of tiles was obtained. Each tile Carry category labels ,in ; each tile Input the pre-trained ResNet backbone network to extract prior semantic feature vectors. All prior semantic feature vectors constitute the prior knowledge tensor: ; In a preferred embodiment, the target detection network may be a YOLO series (such as YOLOv5, YOLOv8), Faster R-CNN, or DETR, etc.

[0026] Among them, the prior semantic feature vector The ResNet backbone network The outputs from different depth levels are fused using a feature pyramid to obtain the following: ; in, This indicates the ResNet backbone network's... Hierarchical output, To project the output of this level to a unified dimension The projection function, For learnable hierarchical fusion weights, and satisfying .

[0027] 3. Modulate the prior knowledge tensor.

[0028] Based on the acquired traffic light status signals For prior knowledge tensors Each line in By its category label By applying dynamic gating weights, we obtain the modulated prior knowledge tensor. The gating weights in the i-th row satisfy: ; ; in It is the Sigmoid activation function. For learnable parameters, This indicates element-wise multiplication.

[0029] 4. Obtain the fusion attention matrix.

[0030] Will , , After passing through their respective modality-specific encoders, the multi-source token sequence is obtained by concatenation. Modulated prior knowledge tensor Injected as additional key and value information into the Transformer encoder, resulting in a fused attention matrix. ;in , , In this process, each modal lexical unit is multiplied by its corresponding modal quality score as a scaling factor before concatenation, thereby incorporating the data quality assessment results into the multi-source lexical sequence X.

[0031] Dynamic gating weights The following prior constraints must be satisfied: when When the light is red, the corresponding tile for pedestrians and non-motorized vehicles... The expected value is not less than the value corresponding to the tile of category motor vehicle. Expected value; when When the light is green, the corresponding tile for the category of motor vehicle... The expected value is not less than the value corresponding to the patch of category pedestrian. The expected value.

[0032] By introducing a constraint loss term during the Transformer encoder training phase Implementing prior constraints: ; in, For the pre-defined set of prior constraint triples, This is the constraint interval hyperparameter.

[0033] The Transformer encoder is trained using a mixture of real road data and simulation data, where: Real-world road data includes data collected by onboard sensors. Three-element and manually labeled hazardous object collection labels; The simulation data is generated by the simulation platform into triples with the same structure and corresponding labels.

[0034] During training, a source identifier is attached to each sample. and with joint losses Optimize: ; in, Binary cross-entropy loss for identifying hazardous objects. For prior constraint loss, For domain alignment loss, For weight hyperparameters.

[0035] The binary cross-entropy loss and domain alignment loss for hazardous object identification are as follows: ; ; in, The Transformer encoder predicts the hazard probability of traffic participants. These are genuine hazard labels manually marked. The probability that the training samples predicted by the domain discriminator originate from the simulation data is given by r, which is the source identifier of the sample. During training, the gradient inversion layer is used to align the feature distribution of the real data with that of the simulation data. E represents the expectation value on the training samples. The modal encoder encodes each source data in the following ways: For RGB images The Vision Transformer encoder is used to divide it into... Image block words; millimeter-wave radar point clouds Output using PointNet encoder A point cloud token; Traffic light status After one-hot encoding, linear projection is used to obtain... A status token; And satisfy .

[0036] 5. Identify hazardous objects and output control commands.

[0037] Based on the attention weights of each traffic participant at its corresponding location in the fusion attention matrix A, a set of hazardous object indices is identified. : ; in, Let be the attention weight of the i-th traffic participant at its corresponding position in A. The relative distance of the participating object to the vehicle. It is a monotonically decreasing distance inverse correlation function. To preset a danger threshold; and to set up a dangerous object index set. Input downstream avoidance decision module output control commands .

[0038] When identifying the set of hazardous object indices, the fused attention matrix A is expanded along the spatial dimension into a multidimensional heatmap matrix of safe distance distribution. The value at coordinates (u, v) in the heatmap matrix of safe distance distribution is defined as: ; in, This represents the detection bounding box region of the i-th traffic participant in the image plane; The indicator function is set to 1 if the condition within the parentheses is met, and to 0 if the condition within the parentheses is not met; Safety distance distribution heatmap matrix. The value exceeds the preset threshold The index of traffic participants corresponding to the connected regions Composition of hazardous object index set In the above formula, the product of the attention weight and the distance inverse correlation function is equal to the value of the safe distance distribution heatmap matrix H within the i-th traffic participant detection box region.

[0039] The downstream avoidance decision module is based on a set of hazardous object indexes. Output control commands Where 'a' is the longitudinal acceleration command. It is a lateral steering angle command, and satisfies: ; in, Let i be the position vector of the i-th traffic participant relative to the vehicle. This is a preset control strategy function.

[0040] The method of the present invention is executed cyclically on an in-vehicle computing platform at a frequency of not less than 10Hz, and the parameters of the pre-trained ResNet backbone network and Transformer encoder are frozen during the inference phase.

[0041] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the invention. The terms “comprising,” “including,” etc., as used herein indicate the presence of the stated features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0042] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple steps or stages, which are not necessarily completed at the same time, but may be executed at different times, and the execution order of these steps or stages is not necessarily sequential, but may be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0043] In one embodiment, the present invention provides a computer system, which may be a server. The computer system includes a processor, memory, and a network interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database stores data used in the methods described above. The network interface communicates with external terminals via a network connection. The computer program is executed by the processor to implement the methods described above.

[0044] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0045] It will be apparent to those skilled in the art that the present invention is not limited to the details of the exemplary embodiments described above, and that the invention can be implemented in other specific forms without departing from its spirit or essential characteristics. Therefore, the embodiments should be considered in all respects as exemplary and non-limiting, and the scope of the invention is defined by the appended claims rather than the foregoing description. Thus, all variations falling within the meaning and scope of equivalents of the claims are intended to be included within the present invention, and no reference numerals in the claims should be construed as limiting the scope of the claims.

[0046] Furthermore, it should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This narrative style is merely for clarity. Those skilled in the art should consider the specification as a whole, and the technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

Claims

1. A method for efficient collaborative learning of multi-source data that integrates prior knowledge, characterized in that, include: Simultaneously acquire multimodal observation data, including vehicle-mounted forward-looking RGB images, millimeter-wave radar point clouds, and traffic light status signals; Data preprocessing and quality assessment are performed on the observation data of each modality; the state signal values ​​of 0, 1, and 2 correspond to red light, yellow light, and green light, respectively. The pre-trained object detection network identifies candidate regions for traffic participants in RGB images. After cropping, each image patch and its corresponding category label are obtained. Each image patch is then input into the pre-trained ResNet backbone network to extract prior semantic feature vectors, which form a prior knowledge tensor. Based on the state signal and category label, dynamic gating weights are applied to each prior semantic feature vector in the prior knowledge tensor to obtain the modulated prior knowledge tensor. After the RGB image, millimeter-wave radar point cloud, and state signal are encoded by their respective encoders, they are combined with the quality assessment results to form a multimodal word sequence. The modulated prior knowledge tensor is injected as an additional key and value into the Transformer encoder to obtain the fused attention matrix. Based on the attention weights and relative distances of each traffic participant in the fusion attention matrix, a set of hazardous object indices is identified. This set of hazardous object indices is then input into the avoidance decision module, which outputs control commands.

2. The efficient collaborative learning method for multi-source data integrating prior knowledge as described in claim 1, characterized in that, The data preprocessing and quality assessment of the observation data for each modality specifically includes: For any modal observation data Execute in sequence: Preprocessing removes outliers and missing values ​​to obtain preprocessed data. ; Calculate the temporal correlation coefficient based on data from T consecutive sampling times. ; Based on the integrity of preprocessed data Correlation coefficient with time series Calculate quality score Where λ1 and λ2 are weighting coefficients; the quality score The scaling factor used for the corresponding modal lexical sequence.

3. The efficient collaborative learning method for multi-source data integrating prior knowledge as described in claim 1, characterized in that, The step of inputting each image patch into a pre-trained ResNet backbone network to extract prior semantic feature vectors and construct a prior knowledge tensor specifically includes: Prior semantic feature vector The ResNet backbone network The outputs from different levels are fused using a feature pyramid to obtain: ； in, This indicates the ResNet backbone network's... Hierarchical output, To project the output of this level onto a projection function of a uniform dimension, For the learnable first The fusion weights of the hierarchy, and satisfying ; All prior semantic feature vectors constitute the prior knowledge tensor. : ； The number of traffic participants or blocks.

4. The efficient collaborative learning method for multi-source data integrating prior knowledge as described in claim 1, characterized in that, The step of applying dynamic gating weights to each prior semantic feature vector in the prior knowledge tensor based on the state signal and category label to obtain the modulated prior knowledge tensor specifically includes: Based on traffic light status signals For prior knowledge tensors Each prior semantic feature vector in By corresponding category label By applying dynamic gating weights, we obtain the modulated prior knowledge tensor. The gating weights of the i-th prior semantic feature vector for: ； in, It is the Sigmoid activation function. For learnable parameters, Indicates one-hot encoding; ； This represents element-wise multiplication. Let represent the modulated i-th prior semantic feature vector.

5. The efficient collaborative learning method for multi-source data integrating prior knowledge according to claim 1, characterized in that, The process of encoding the RGB image, millimeter-wave radar point cloud, and state signal separately using their respective encoders, and then concatenating them with the quality assessment results to form a multimodal word sequence, specifically includes: RGB image Input the Vision Transformer encoder to get Image block words; millimeter-wave radar point cloud Input the PointNet encoder to get Each point cloud word element; For state signals After one-hot encoding and linear projection, we obtain There are state words; and they satisfy the following conditions: ; Represents multi-source lexical sequences The total number of lexical units; Will Image block words, Point cloud terminology and Each state lexical unit is concatenated into a multi-source lexical sequence. Each modal lexical unit is multiplied by the quality score obtained through quality assessment for the corresponding modality before concatenation.

6. The efficient collaborative learning method for multi-source data integrating prior knowledge according to claim 1, characterized in that, The step of injecting the modulated prior knowledge tensor as additional keys and values ​​into the Transformer encoder to obtain the fused attention matrix specifically includes: Modulated prior knowledge tensor As additional key-value pairs, they are injected into the Transformer encoder to obtain the fused attention matrix. : ； in, , , , This represents the query matrix obtained by projecting the multi-source word sequence X onto the query matrix. This represents the bond matrix obtained by projecting X onto the bond matrix. Let represent the additional bond matrix obtained by key projection from the modulated prior knowledge tensor. To query the projection weight matrix, The key projection weight matrix, The feature dimension representing attention, This indicates a concatenation operation along the lexical sequence dimension. This represents the softmax function. This indicates transpose.

7. The efficient collaborative learning method for multi-source data integrating prior knowledge according to claim 1, characterized in that, The process of identifying a set of hazardous object indices based on the attention weights and relative distances of each traffic participant in the fused attention matrix specifically includes: The fusion attention matrix A is expanded along the spatial dimension into a multidimensional heatmap matrix of safe distance distribution. The value at coordinates (u,v) in the heatmap matrix of safe distance distribution Defined as: ； in, This represents the detection bounding box region of the i-th traffic participant in the image plane. Here, it is an indicator function; Let be the attention weight of the i-th traffic participant at its corresponding position in A. It is a monotonically decreasing distance inverse correlation function. Let be the relative distance between the i-th traffic participant and the vehicle; The number of traffic participants; Heatmap matrix of safe distance distribution The value exceeds the preset threshold The index of traffic participants corresponding to the connected regions Composition of hazardous object index set Since the detection frames of each traffic participant do not overlap, within the detection frame of the i-th traffic participant, the value of the safety distance distribution heatmap matrix H is equal to the attention weight corresponding to that traffic participant. Inverse correlation function with distance Therefore, for H that exceeds the preset threshold, the product of the two is used. Finding the index of the corresponding traffic participant in a connected region is equivalent to the following equation: 。 8. The efficient collaborative learning method for multi-source data integrating prior knowledge according to claim 7, characterized in that, The step of inputting the hazardous object index set into the avoidance decision module and outputting control commands specifically includes: The avoidance decision module is based on a set of hazardous object indexes. Output control commands ,in, For longitudinal acceleration, It is a lateral steering angle command, and satisfies: ； in, Let i be the position vector of the i-th traffic participant relative to the vehicle. This is a preset control strategy function.

9. The efficient collaborative learning method for multi-source data integrating prior knowledge according to claim 1, characterized in that, The Transformer encoder is trained using a mixture of real road data and simulation data. Real-world road data includes triples composed of RGB images collected by vehicle sensors, millimeter-wave radar point clouds, and traffic light status signals, along with manually labeled hazardous object indexes. Simulation data includes triples with the same structure as real-world road data generated by the simulation platform, along with corresponding labels. During training, a source identifier is added to each training sample. ,when When r is 0, the training samples are real road data; when r is 1, the training samples are simulated data; this is achieved through joint loss. Optimize the Transformer encoder: ； in, Binary cross-entropy loss for identifying hazardous objects. For domain alignment loss, For prior constraint loss, These are weight hyperparameters; ; ; in, The Transformer encoder predicts the hazard probability of traffic participants. These are genuine hazard labels manually marked. E represents the probability that the training samples are derived from simulation data for the domain discriminator; E denotes the expectation value on the training samples. The status signals of traffic lights satisfy the following prior constraints: When the traffic light status signal When the traffic light is red, the gating weights corresponding to the tiles categorized as pedestrians and non-motorized vehicles are determined. The expected value is not less than the value corresponding to the tile with the category label "motor vehicle". Expected value; when When the traffic light is green, the tile labeled "Motor Vehicle" corresponds to... The expected value is not less than the value of the tile with the category label "pedestrian". Expected value; Prior constraint loss for: ； in, Preset prior constraint triples Set, where s represents the values ​​of the traffic light status signal. , For the two categories of traffic participants, values ​​are assigned to the labels. To constrain the interval hyperparameter, This indicates the calculation of expectation. Let be the category label for the i-th tile.

10. A computer system comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 9.

Images