Intelligent automobile multi-modal data fusion processing method based on large model

Abstract

The present application relates to the technical field of intelligent vehicle data processing, and more particularly to a multi-modal data fusion processing method for intelligent vehicles based on large models, which comprises obtaining a perception image sequence and business operation data and recording a time identifier; converting the perception image sequence into a perception feature matrix and mapping the business operation data into a business feature vector; extracting a feature gradient value to generate a proportion value; when the proportion value is greater than a feature interference critical threshold, converting a feature weight parameter into a channel attention mask and multiplying it by the perception feature matrix, setting unmatched values to zero, and outputting a de-noised perception feature matrix; and cascading the de-noised perception feature matrix and the business feature vector to input a prediction network and output a prediction result, which can prevent high-dimensional perception noise from masking business features.

Description

Technical Field

[0001] This invention relates to the field of intelligent vehicle data processing technology, and in particular to a method for multimodal data fusion processing of intelligent vehicles based on large models. Background Technology

[0002] The operation of driverless ride-hailing fleets heavily relies on adaptive billing and dynamic dispatching. Conducting these billing and dispatching operations requires the fusion processing of heterogeneous multimodal data. This heterogeneous multimodal data specifically includes high-dimensional perception data and low-dimensional operational data. High-dimensional perception data refers to real-time road environment image sequences collected by onboard sensors, while low-dimensional operational data includes regional order supply-demand ratios and congestion surcharge rates. Existing technologies generally employ large-scale neural network models to process multi-source heterogeneous data. By utilizing feature extraction networks to transform data information from different modalities, each modality is mapped into an independent-dimensional feature vector. Multiple sets of feature vectors are then input into a pre-defined unified feature latent space for alignment and fusion. The resulting comprehensive feature vector then flows to the business prediction module, which finally outputs the fare amount and vehicle dispatch coordinates based on the comprehensive feature vector.

[0003] Existing technologies face a feature extraction imbalance in processing multi-source heterogeneous data. High-dimensional perceptual data includes a large number of dense pixel-level environmental features, while low-dimensional business operation data consists of sparse numerical features. Large-scale neural network models perform forced alignment operations on feature vectors of different dimensions in the feature latent space, requiring feature vectors at different levels to match a completely consistent distribution scale. High-dimensional perceptual data contains a large amount of non-critical pixel noise, which, under a uniform distribution scale, interferes with sparse low-dimensional business features, causing the economic numerical laws inherent in low-dimensional business operation features to be completely masked by high-dimensional perceptual noise.

[0004] Taking the use of autonomous ride-hailing fleets to handle peak traffic after large events as an example, the high-dimensional perception data from the vehicles reflects complex road conditions with dense traffic and a large number of pedestrians, while the low-dimensional business operation data in the background simultaneously indicates the existence of high dynamic premium amounts. After a large-scale neural network model directly integrates road condition data and premium amount data, the pixel noise corresponding to a large number of congested road conditions completely covers the high premium value features. The data fusion result causes the business prediction network to generate incorrect logical judgments that certain areas are impassable, thereby interrupting the dispatch of autonomous ride-hailing vehicles to areas with high demand, causing the vehicle dispatch system to lose business operation opportunities. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method for multimodal data fusion processing of intelligent vehicles based on large models. This invention solves the technical problem that low-dimensional business operation features are masked by high-dimensional perceptual noise due to the forced alignment of heterogeneous multimodal data by large models.

[0006] To solve the above-mentioned technical problems, the specific contents of the present invention are as follows: The present invention provides a method for multimodal data fusion processing of intelligent vehicles based on large models, comprising: Acquire the perception image sequence and commercial operation data of the unmanned ride-hailing fleet, and record the corresponding time markers; extract the preset feature interference threshold; The perceived image sequence is converted into a perceived feature matrix using a preset visual neural network; the business operation data is mapped into a business feature vector using a preset lightweight model; and the activation parameters output by the network layer inside the lightweight model are extracted as feature weight parameters. Extract the first feature gradient value representing pixel changes from the perception feature matrix, and extract the second feature gradient value representing numerical fluctuations from the commercial feature vector; align the first feature gradient value and the second feature gradient value according to the time identifier, and calculate the ratio of the first feature gradient value to the second feature gradient value. If the ratio value is greater than the feature interference threshold, the feature weight parameters in the commercial feature vector are converted into a channel attention mask through a preset nonlinear activation function. Multiply the channel attention mask by the channel dimension of the perceptual feature matrix; set the channel values ​​in the perceptual feature matrix that do not match the corresponding elements of the channel attention mask to zero, retain the channel values ​​that match the corresponding elements of the channel attention mask, and output the denoised perceptual feature matrix. The denoised perception feature matrix and the business feature vector are concatenated to construct a comprehensive business decision vector; the comprehensive business decision vector is input into a preset prediction network to output the prediction result, which includes the billing amount and the coordinates of the scheduling target.

[0007] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention acquires the perception image sequence and commercial operation data of an unmanned ride-hailing fleet and records the corresponding time stamps, including: The road environment sequence data transmitted via the vehicle communication bus is received as the perceived image sequence, and the road environment sequence data includes dynamic obstacles; The system calls upon pre-defined backend service nodes to record business operation data, which includes regional order supply and demand ratios and congestion premium rates. The first time identifier attached to the perceived image sequence is parsed, and the second time identifier attached to the business operation data is parsed; The first time identifier and the second time identifier are used to perform time difference calculation to generate an aligned absolute time record as the time identifier.

[0008] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention utilizes a preset visual neural network to convert the perceived image sequence into a perceived feature matrix; and utilizes a preset lightweight model to map the business operation data into a business feature vector, including: Using the multi-layer spatial convolutional layers in the visual neural network, tensor multiplication operations are performed on the pixel matrix included in the perceived image sequence to extract road spatial dimension features and output the perceived feature matrix; Using the fully connected layer in the lightweight model, nonlinear operations are performed on the regional order supply-demand ratio and the congestion premium rate in the business operation data to output the dimensionality-reduced business feature vector.

[0009] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention extracts the first feature gradient value representing pixel changes in the perception feature matrix and extracts the second feature gradient value representing numerical fluctuations in the commercial feature vector, including: Spatial difference operation is performed on the perceptual feature matrix to calculate the spatial numerical fluctuation amplitude between adjacent feature elements in the perceptual feature matrix; the maximum value of the spatial numerical fluctuation amplitude is output as the first feature gradient value. Perform time series differencing on the business feature vector to calculate the time series numerical fluctuation derivatives corresponding to adjacent time markers in the business feature vector; output the time series numerical fluctuation derivatives as the second feature gradient values.

[0010] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention, when the ratio value is greater than the feature interference critical threshold, converts the feature weight parameters in the commercial feature vector into a channel attention mask through a preset nonlinear activation function, including: The proportional value is compared with the feature interference threshold; if the proportional value is greater than the feature interference threshold, a business feature retention instruction is triggered, and the transmission of the perceptual feature matrix to the preset feature latent space is terminated. The activation parameters of the fully connected layer output in the lightweight model are extracted and used as the feature weight parameters; The feature weight parameters are input into the nonlinear activation function composed of the Sigmoid function for interval compression, generating a floating-point matrix whose numerical distribution is restricted to a preset continuous interval, which serves as the channel attention mask.

[0011] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention multiplies the channel attention mask by the channel dimension of the perception feature matrix; sets the channel values ​​in the perception feature matrix that do not match the corresponding elements of the channel attention mask to zero, retains the channel values ​​that match the corresponding elements of the channel attention mask, and outputs a denoised perception feature matrix, including: Using the tensor broadcasting mechanism, the dimensionality of the channel attention mask is expanded to match the dimensionality of the perceptual feature matrix, thereby generating an expanded channel attention mask. The extended channel attention mask is multiplied with the perceptual feature matrix at the matrix element level to generate the product result. The channel values ​​whose product results in the perceptual feature matrix are lower than a preset retention threshold are set to zero, and the channel values ​​whose product results in the perceptual feature matrix are greater than or equal to the preset retention threshold are retained, and the denoised perceptual feature matrix is ​​output.

[0012] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention concatenates the denoised perception feature matrix with the business feature vector to construct a comprehensive business decision vector; the comprehensive business decision vector is input into a preset prediction network to output a prediction result, the prediction result including a billing amount and scheduling target coordinates, including: Perform a flattening operation on the denoised sensing feature matrix to generate a one-dimensional sensing sequence; The one-dimensional perception sequence and the business feature vector are concatenated in the preset feature dimension direction to construct the comprehensive business decision vector; The integrated business decision vector is input into the multi-branch multilayer perceptron network, which serves as the prediction network, for forward propagation, and the billing amount and the scheduling target coordinates are output.

[0013] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention, after aligning the first feature gradient value and the second feature gradient value according to the time identifier and calculating the ratio of the first feature gradient value to the second feature gradient value, further includes: Read the load status data of the unmanned ride-hailing fleet corresponding to the time identifier; The load condition data is input into a pre-stored fault tolerance adjustment database for matching and querying; Extract the dynamic tolerance value corresponding to the load state data; The dynamic tolerance value is used to numerically correct the feature interference critical threshold, thereby generating an updated feature interference critical threshold.

[0014] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention outputs a prediction result. This prediction result includes, after the billing amount and the scheduling target coordinates, also: The billing amount is sent to a preset passenger terminal for display via a preset communication network; Convert the scheduling target coordinates into chassis motion control commands; The chassis motion control command is sent to the actuator of the corresponding vehicle in the unmanned ride-hailing fleet to control the vehicle to move to the dispatch target coordinates.

[0015] Furthermore, the intelligent vehicle multimodal data fusion processing method based on a large model described in this invention, after aligning the first feature gradient value and the second feature gradient value according to the time identifier and calculating the ratio of the first feature gradient value to the second feature gradient value, further includes: Establish a baseline sliding alignment window with a preset fixed time step; Within the baseline sliding alignment window, count the number of times the ratio value is greater than the feature interference threshold. The occurrence count is compared with a preset fault tolerance threshold. If the occurrence count is greater than the preset fault tolerance threshold, it is determined that there is abnormal message data in the sensing image sequence. The communication port for receiving the sensing image sequence is closed, and the remaining data corresponding to the sensing image sequence in the preset buffer queue is cleared.

[0016] Beneficial effects of this invention: The multimodal data fusion processing method for intelligent vehicles based on a large model provided by this invention first extracts the first feature gradient value representing pixel changes from the perception feature matrix, and simultaneously extracts the second feature gradient value representing numerical fluctuations from the business feature vector. Subsequently, the system uses the first and second feature gradient values ​​to perform mathematical calculations to generate a ratio value, thereby establishing a quantitative comparison mechanism between fluctuations in perceived environmental data and fluctuations in business operation data.

[0017] When the proportional value exceeds the feature interference threshold, the system inputs the feature weight parameters from the commercial feature vector into a preset nonlinear activation function, generating a channel attention mask through feature transformation. Next, a tensor multiplication operation is performed, multiplying the channel attention mask by the channel dimension of the perceptual feature matrix. Based on the multiplication result, the system sets the channel values ​​of elements in the perceptual feature matrix that do not match the channel attention mask to zero, while retaining the channel values ​​of elements that match the channel attention mask, ultimately outputting a denoised perceptual feature matrix.

[0018] The steps of extracting feature gradient values ​​and calculating proportional values ​​are coordinated with the step of applying channel attention masks to set channel values ​​to zero. This collaborative mechanism changes the data fusion logic of existing technologies, avoiding the forced alignment of multiple sets of feature vectors input into a pre-defined unified feature latent space. By reconstructing the data fusion logic, this invention overcomes the defects caused by the forced alignment of heterogeneous multimodal data in large-scale neural network models, preventing sparse low-dimensional business operation features from being covered by a large amount of non-critical pixel noise. The system uses the ratio of spatial dimension gradient to temporal dimension gradient for dynamic trigger control, constructing a nonlinear environmental interference isolation mechanism. This specific environmental interference isolation mechanism solves the feature weight collapse problem caused by global mapping in the multimodal fusion stage of existing attention networks, achieving the unexpected technical effect of accurately locking high-premium scheduling coordinates even in extremely dense road conditions such as traffic jams without increasing redundant computing resources, significantly improving the scenario generalization ability of the data fusion algorithm.

[0019] Leveraging a collaborative processing mechanism, the system concatenates the denoised sensing feature matrix with the business feature vector to construct a comprehensive business decision vector. This constructed comprehensive business decision vector effectively eliminates interference from redundant pixel data. Finally, the prediction network receives the comprehensive business decision vector and outputs the billing amount and scheduling target coordinates based on it. This data concatenation operation, which removes environmental noise, improves the accuracy of the prediction network's output prediction results. Attached Figure Description

[0020] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on the drawings without creative effort.

[0021] Figure 1 This is a flowchart illustrating the multimodal data fusion processing method for intelligent vehicles based on a large model, as described in this invention. Detailed Implementation

[0022] To make the technical solution of the present invention clearer, the present invention will be clearly and completely described below with reference to specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Please see Figure 1 The present invention provides a method for multimodal data fusion processing of intelligent vehicles based on large models, comprising: Step 1: Obtain the perception image sequence and commercial operation data of the unmanned ride-hailing fleet, and record the corresponding time markers; extract the preset feature interference threshold. Step 2: Use a preset visual neural network to convert the perceived image sequence into a perceived feature matrix; use a preset lightweight model to map the business operation data into a business feature vector; extract the activation parameters output by the network layer inside the lightweight model as feature weight parameters. Step 3: Extract the first feature gradient value representing pixel changes from the perception feature matrix, and extract the second feature gradient value representing numerical fluctuations from the commercial feature vector; align the first feature gradient value and the second feature gradient value according to the time identifier, and calculate the ratio of the first feature gradient value to the second feature gradient value. Step 4: If the ratio value is greater than the feature interference threshold, the feature weight parameters in the commercial feature vector are converted into a channel attention mask through a preset nonlinear activation function. Step 5: Multiply the channel attention mask by the channel dimension of the perceptual feature matrix; set the channel values ​​in the perceptual feature matrix that do not match the corresponding elements of the channel attention mask to zero, retain the channel values ​​that match the corresponding elements of the channel attention mask, and output the denoised perceptual feature matrix. Step 6: Concatenate the denoised perception feature matrix with the business feature vector to construct a comprehensive business decision vector; input the comprehensive business decision vector into a preset prediction network and output the prediction result, which includes the billing amount and the coordinates of the scheduling target.

[0024] The system acquires perception image sequences and operational data from an autonomous ride-hailing fleet, recording the corresponding timestamps. Simultaneously, a preset threshold for feature interference is extracted. The perception image sequence specifically includes a continuous RGB image frame sequence with a resolution of 1920×1080, captured by the onboard camera. The operational data specifically includes regional dynamic pricing rates and real-time passenger waiting statistics received from a cloud server. The timestamps use a unified UNIX timestamp format. The preset threshold for feature interference is set to 0.85, which measures the degree of interference from environmental noise on the operational data.

[0025] A pre-defined visual neural network is used to convert the perceived image sequence into a perceptual feature matrix. Simultaneously, a pre-defined lightweight model is used to map business operation data into business feature vectors. The activation parameters output from the fully connected network layers of the lightweight model are then extracted as feature weight parameters. The visual neural network employs a ResNet-50 residual network model, resulting in a 256×64×64 three-dimensional tensor for the perceptual feature matrix. To adapt to different computing power requirements of in-vehicle computing units, the visual neural network can be replaced with a VGG convolutional neural network model or a Vision Transformer visual feature extraction model, as long as the output perceptual feature matrix includes channel dimensions.

[0026] The lightweight model employs the MobileNet network model, resulting in a 128-dimensional floating-point one-dimensional vector for business features. To adapt to different formats of business operational data, the lightweight model can be replaced with either the ShuffleNet or EfficientNet model. A pre-defined dimension-aligned projection layer adaptively aligns the number of dimensions of the extracted feature weight parameters to perfectly match the number of channels in the perceptual feature matrix. This ensures that the channel attention mask can perform product operations with perceptual feature matrices of any number of channels, thus providing support for general model architecture combinations. The extracted activation parameters are the output weights of the second-to-last neuron in the MobileNet network model.

[0027] Regarding the lightweight model construction and feature dimension mapping mechanism, after receiving commercial operation data, the pre-defined lightweight model first uses a built-in feature extension layer to map the one-dimensional regional order supply-demand ratio and congestion premium rate into a pseudo-image tensor with spatial dimension. This pseudo-image tensor is then input into the backbone feature extraction network of the MobileNet network model for layer-by-layer forward propagation. The penultimate layer of the MobileNet network model outputs a 128-dimensional initial feature vector and 128-dimensional initial activation parameters. To address the feature dimension differences during multimodal data fusion, the system constructs a dimension alignment projection layer at the end of the MobileNet network model. This dimension alignment projection layer uses a linear fully connected network with 256 nodes.

[0028] The initial activation parameters of 128 dimensions are input into the dimension-aligned projection layer for linear transformation, outputting a 256-dimensional floating-point one-dimensional vector. This 256-dimensional floating-point one-dimensional vector is then used as the feature weight parameters. The dimension of the feature weight parameters is exactly the same as the 256 channel dimensions of the perceptual feature matrix, thus providing a dimension matching basis for the subsequent generation of equal-length channel attention masks.

[0029] Regarding the construction mechanism of the multi-branch multilayer perceptron network, a pre-defined prediction network receives the cascaded integrated service decision vector. This integrated service decision vector first undergoes nonlinear feature fusion through a shared fully connected layer containing 512 neurons and a ReLU activation function. The fused feature vector is then input to the first and second multilayer perceptron branches. The first multilayer perceptron branch uses a hidden layer containing 256 neurons and a linear regression output layer to generate continuous billing amount values. The second multilayer perceptron branch uses a hidden layer containing 256 neurons and a fully connected output layer to generate longitude and latitude floating-point coordinate values ​​in a pre-defined coordinate system.

[0030] The first feature gradient value, representing pixel changes, and the second feature gradient value, representing numerical fluctuations, are extracted from the perceptual feature matrix. Then, the first and second feature gradient values ​​are aligned according to a time marker, and the ratio between them is calculated. Specifically, the ratio is the quotient obtained by dividing the first feature gradient value by the sum of the second feature gradient value and a preset zero-smoothing correction term constant.

[0031] When the proportion value is greater than the feature interference threshold, the feature weight parameters in the business feature vector are converted into channel attention masks using a preset non-linear activation function. A multiplication operation is performed, multiplying the channel attention mask by the channel dimension of the perceptual feature matrix. After multiplication, the channel values ​​of elements in the perceptual feature matrix that do not match the channel attention mask are set to zero, while the channel values ​​of elements that match the channel attention mask are retained, thus outputting the denoised perceptual feature matrix. When the proportion value is less than or equal to the feature interference threshold, it is determined that the high-dimensional perceptual data does not interfere with the low-dimensional business operation features. The system skips the step of converting the feature weight parameters into channel attention masks and directly outputs the perceptual feature matrix as the denoised perceptual feature matrix.

[0032] The denoised perceptual feature matrix and the business feature vector are concatenated to construct a comprehensive business decision vector. This comprehensive business decision vector is then input into a pre-defined prediction network to output the prediction results. The prediction results include the billing amount and the scheduling target coordinates. The billing amount is in RMB, and the scheduling target coordinates are latitude and longitude information in the WGS84 coordinate system.

[0033] The process of acquiring the perception image sequences and commercial operation data of the autonomous ride-hailing fleet and recording the corresponding time markers includes the following operations: The system receives road environment sequence data transmitted via the vehicle-mounted communication bus as a perception image sequence, and this road environment sequence data includes dynamic obstacles. The vehicle-mounted communication bus uses a CAN FD bus, and the dynamic obstacles include moving pedestrians and vehicles. Simultaneously, it accesses pre-set backend service nodes that record commercial operation data, including regional order supply-demand ratios and congestion premium rates.

[0034] After receiving the data, the first time identifier attached to the perceived image sequence is parsed, and the second time identifier attached to the business operation data is parsed. Finally, the first and second time identifiers are subjected to a time difference calculation to generate an aligned absolute time record as the time identifier. The time difference calculation is used to eliminate the time delay caused by network transmission, ensuring that the two types of data from different modalities are fused at exactly the same time.

[0035] The processing logic, which uses a pre-defined visual neural network to convert a sequence of perceived images into a perceptual feature matrix and uses a pre-defined lightweight model to map business operation data into business feature vectors, includes the following computational steps: By utilizing multi-layer spatial convolutional layers in a visual neural network, tensor multiplication operations are performed on the pixel matrix of a perceived image sequence to extract spatial dimension features of the road and output a perceptual feature matrix. The multi-layer spatial convolutional layers include at least three different kernel sizes, including 3×3, 5×5, and 7×7 kernels. These kernels are used to extract spatial dimension features from local edges to global contours.

[0036] In addition, by utilizing the fully connected layer in the lightweight model, nonlinear operations are performed on the regional order supply and demand ratio and congestion premium rate values ​​in the business operation data, and the dimensionality-reduced business feature vector is output.

[0037] The specific steps for extracting the first feature gradient value representing pixel changes from the perceptual feature matrix and the second feature gradient value representing numerical fluctuations from the commercial feature vector are as follows: Spatial difference operations are performed on the perceptual feature matrix to calculate the spatial numerical fluctuation amplitude between adjacent feature elements in the perceptual feature matrix, and the maximum value of the spatial numerical fluctuation amplitude is output as the first feature gradient value. The spatial difference operation uses the Sobel operator to perform sliding window operations on the perceptual feature matrix in the horizontal and vertical directions.

[0038] Similarly, a time-series differencing operation is performed on the business feature vector to calculate the temporal numerical fluctuation derivative corresponding to adjacent time markers within a preset business update period. The time span used in the time-series differencing operation is the preset business update period. Finally, the temporal numerical fluctuation derivative is output as the second feature gradient value.

[0039] After extracting the activation parameters from the fully connected network layers of the lightweight model as feature weight parameters, the feature weight parameters in the commercial feature vector are converted into channel attention masks through a preset non-linear activation function, including the following mechanism: The proportional value is compared with the feature interference threshold. If the proportional value exceeds the feature interference threshold, a commercial feature preservation instruction is triggered, and the transmission of the perceptual feature matrix to the preset feature latent space is terminated. Simultaneously, the activation parameters output by the fully connected layers in the lightweight model are extracted as feature weight parameters.

[0040] Subsequently, the feature weight parameters are input into a non-linear activation function composed of a sigmoid function for interval compression, generating a floating-point matrix whose numerical distribution is restricted to a preset continuous interval as the channel attention mask. The preset continuous interval is limited to a closed interval from 0 to 1. The channel attention mask is a one-dimensional floating-point array, and the array length is exactly equal to the number of channels in the perceptual feature matrix.

[0041] Furthermore, the channel attention mask is multiplied by the channel dimension of the perceptual feature matrix. The channel values ​​of the elements in the perceptual feature matrix that do not match the channel attention mask are set to zero, while the channel values ​​of the matching elements are retained. This process is then used to output the calculation flow of the denoised perceptual feature matrix. The channel attention mask is expanded using a tensor broadcasting mechanism until its dimensions match those of the perceptual feature matrix, generating an expanded channel attention mask. The expanded channel attention mask and the perceptual feature matrix are then multiplied element-wise to produce the product result.

[0042] Based on the product result, channel values ​​in the perceptual feature matrix whose product result is lower than a preset retention threshold are set to zero. Simultaneously, channel values ​​in the perceptual feature matrix whose product result is greater than or equal to the preset retention threshold are retained, thus outputting a denoised perceptual feature matrix. The preset retention threshold is set to 0.5. Image regions corresponding to channel values ​​lower than 0.5 are defined as non-critical environmental noise and directly removed to prevent high-weight commercial features from being masked by environmental noise.

[0043] To output the final instruction, the denoised perceptual feature matrix and the business feature vector are concatenated to construct a comprehensive business decision vector. The comprehensive business decision vector is then input into a pre-defined prediction network to output the prediction result. The operation includes the following sequence: A flattening operation is performed on the denoised perceptual feature matrix to generate a one-dimensional perceptual sequence. This one-dimensional perceptual sequence is then concatenated with the business feature vector along a preset feature dimension to construct a comprehensive business decision vector. Specifically, the concatenation operation is performed along the last dimension of the tensor.

[0044] Subsequently, the integrated business decision vector is input into a multi-branch, multi-layer perceptron network, which serves as the prediction network, for forward propagation, outputting the billing amount and the coordinates of the scheduling target. The multi-branch, multi-layer perceptron network consists of two independent output branches. The first branch uses a linear regression layer to output continuous billing amount values, and the second branch uses a fully connected layer to output coordinate positions.

[0045] Before the multi-branch, multi-layer perceptron network performs forward propagation and outputs prediction results, the system pre-executes a supervised joint training step. Specifically, the system extracts multiple sets of perception image sequences and commercial operation data within the historical vehicle operation cycle, and uses the extracted data as training input samples; simultaneously, it acquires the real billing amount and real dispatch coordinates with the same time identifier as the training input samples, and uses the real billing amount and real dispatch coordinates as annotation benchmark samples.

[0046] After obtaining the labeled baseline samples, the system constructs a multi-task joint loss function for the prediction network. This multi-task joint loss function includes a billing regression loss branch and a coordinate positioning loss branch. In the specific calculation process, the billing regression loss branch uses the mean squared error function to calculate the numerical dispersion between the billing amount output by the prediction network and the actual billing amount; the coordinate positioning loss branch uses the L2 norm function to calculate the spatial distance offset between the scheduling target coordinates output by the prediction network and the actual scheduling coordinates.

[0047] Subsequently, the system summarizes the calculation results of the billing regression loss branch and the coordinate positioning loss branch, and generates the total loss value. After obtaining the total loss value, the system uses the adaptive moment estimation optimizer to perform error backpropagation on the total loss value. Finally, based on the gradient value of backpropagation, the system synchronously updates the node weight parameters of the preset visual neural network, the preset lightweight model, and the multi-branch multilayer perceptron network.

[0048] The configuration phase of the training operation includes setting multiple network hyperparameters. The system sets the initial learning rate to 0.0001 and uses a cosine annealing algorithm to non-linearly decay the learning rate during iteration; at the same time, the batch size of samples per iteration is configured to 64, and the total number of training iterations is configured to 200.

[0049] Through the backpropagation computation mechanism of the multi-task joint loss function, the joint training step drives the multi-branch multilayer perceptron network to establish a mapping relationship between the dimensionality-reduced business economic features and the road spatial dimension features during the weight update phase. The established mapping relationship enables the multi-branch multilayer perceptron network to automatically align the numerical distribution scale with the spatial location scale during the training convergence process, thereby endowing the prediction network with the generalization inference ability to process multimodal fusion features.

[0050] After aligning the first feature gradient value with the second feature gradient value based on the time identifier and calculating the ratio value, the system will also introduce a fault tolerance value correction mechanism: The system reads the load status data of the autonomous ride-hailing fleet corresponding to the time stamp. This load status data is acquired through vehicle suspension sensors. After obtaining the data, it is input into a pre-stored tolerance adjustment database for matching and querying, and the corresponding dynamic tolerance value is extracted.

[0051] Finally, the dynamic fault tolerance value is used to numerically correct the feature interference critical threshold, generating an updated feature interference critical threshold. When the load status data indicates that the vehicle is fully loaded, the system will extract a higher dynamic fault tolerance value to improve the anti-interference capability of the underlying calculation.

[0052] After outputting the prediction results, including the billing amount and the coordinates of the scheduling target, the system will execute data transmission and motion control command issuance: The fare amount is sent to a pre-defined passenger terminal for display via a pre-defined communication network. The pre-defined communication network uses a 5G cellular network, and the pre-defined passenger terminal includes a smartphone.

[0053] Simultaneously, the target coordinates are converted into chassis motion control commands. These commands include steering wheel angle commands and braking deceleration commands. After the command conversion is complete, the chassis motion control commands are sent to the actuators of the corresponding vehicles in the autonomous ride-hailing fleet to control the vehicles to move to the target coordinates. The actuators include an electronic power steering motor and an electro-hydraulic braking system.

[0054] To prevent interference from anomalous data, after aligning the first feature gradient values ​​with the second feature gradient values ​​according to the time identifier and calculating the ratio, corresponding security verification steps will be performed: Establish a baseline sliding alignment window with a preset fixed time step. The preset fixed time step is set to 5 seconds. Within the baseline sliding alignment window, count the number of times the statistical proportion value exceeds the characteristic interference critical threshold.

[0055] The number of occurrences is compared with a preset fault tolerance threshold. If the number of occurrences exceeds the preset fault tolerance threshold, it is determined that there is illegal message data in the perceived image sequence. The communication port receiving the perceived image sequence is then closed, and any remaining data corresponding to the perceived image sequence in the preset buffer queue is cleared. The preset fault tolerance threshold is set to 10 occurrences, thus effectively defending against high-frequency injection of abnormal visual environment data.

[0056] Spatial difference operation formula: In the formula, This represents the gradient value of the first feature. This represents the spatial numerical fluctuation amplitude of the perceptual feature matrix in the horizontal direction. This represents the spatial numerical fluctuation amplitude of the perceptual feature matrix in the vertical direction. This represents the index number of the horizontal feature element in the perceptual feature matrix. Indicates the index number of the vertical feature element of the perceptual feature matrix.

[0057] Time series differencing formula: In the formula, This represents the gradient value of the second feature. This represents the value of the business feature vector corresponding to the current time identifier. This represents the value of the business feature vector corresponding to the adjacent previous time identifier. This represents the time span corresponding to the preset business update cycle. The formula for calculating the proportion is: In the formula, Indicates a proportional value. This represents the gradient value of the first feature. This represents the gradient value of the second feature. This represents the preset zero-smoothing correction term constant.

[0058] Formula for nonlinear activation function: In the formula, This represents the mask value corresponding to the channel attention mask. Represents the base of the natural logarithm. This represents the feature weight parameters.

[0059] Formula for element-wise multiplication of matrices: In the formula, This represents the channel value corresponding to the product result. This represents the mask value corresponding to the expanded channel attention mask. This represents the unprocessed channel values ​​in the perceptual feature matrix. This represents the channel dimension index number of the perceptual feature matrix. This represents the index number of the horizontal feature element in the perceptual feature matrix. Indicates the index number of the vertical feature element of the perceptual feature matrix.

[0060] Multi-task joint loss function formula: In the formula, This represents the total loss value. This represents the total number of training input samples. This indicates the current index of the training input sample. This represents the actual billing amount corresponding to the training input sample. This represents the billing amount output by the multi-branch, multi-layer perceptron network. This represents the loss weight balancing parameter. This represents the x-coordinate of the actual scheduling coordinates corresponding to the training input sample. This represents the x-coordinate of the scheduling target coordinates output by the multi-branch, multi-layer perceptron network. This represents the ordinate of the actual scheduling coordinates corresponding to the training input sample. The vertical coordinate of the scheduling target output by the multi-branch multilayer perceptron network represents the position of the target coordinate.

[0061] Cosine annealing algorithm formula: In the formula, This represents the learning rate in the current training iteration cycle. This represents the minimum learning rate set. This represents the initial learning rate set. This indicates the value of the current iteration cycle. This indicates the total number of training iterations set. It represents pi (π).

[0062] Feature interference critical threshold correction formula: In the formula, This represents the updated feature interference threshold. This represents the preset threshold for characteristic interference. This represents the extracted dynamic fault tolerance value.

[0063] The technical solution of this invention involves multiple numerical thresholds for performing quantization control. A preset feature interference critical threshold is used to measure the degree to which environmental noise overwhelms business data, and the specific value is set to 0.85. When the ratio value is greater than 0.85, the system triggers a business feature retention instruction, thereby preventing non-critical pixel noise in high-dimensional perceived data from interfering with sparse low-dimensional business features.

[0064] To filter channel values ​​in the perceptual feature matrix, the system introduces a preset retention threshold of 0.5. During data fusion processing, the system sets channel values ​​with product results below 0.5 to zero and defines image regions associated with channel values ​​below 0.5 as environmental noise, removing them. Applying the preset retention threshold effectively prevents high-weight commercial features from being overwhelmed by environmental noise.

[0065] The preset fault tolerance threshold is set to 10 times to defend against high-frequency injection of spoofed visual environment data. If the number of occurrences exceeds 10, the system determines that there is illegal message data in the perceived image sequence. In conjunction with the preset fault tolerance threshold, the system synchronously triggers a security verification step, thereby closing the communication port receiving the perceived image sequence and clearing any remaining data corresponding to the perceived image sequence.

[0066] Taking the dynamic scheduling and billing application scenario of unmanned ride-hailing fleets during peak periods after large-scale sporting events as an example, the vehicle-mounted communication bus receives road environment sequence data and uses it as a perception image sequence. This perception image sequence is acquired by cameras installed around the vehicle at a frequency of 60 frames per second and includes a high-resolution RGB image frame sequence. The high-resolution RGB image frame sequence includes dynamic obstacles such as moving pedestrians and slowly moving vehicles. Simultaneously, the background service node records commercial operation data, which the system accesses. This commercial operation data includes the real-time regional order supply-demand ratio and congestion premium rate generated in the sporting event area. The regional order supply-demand ratio is represented as a 10:1 demand ratio, and the congestion premium rate is represented as a 3.5 times rate. Subsequently, the system parses the time stamps corresponding to the perception image sequence and the commercial operation data. The time stamps corresponding to the perception image sequence increase with an inter-frame interval of 0.0167 seconds, while the time stamps corresponding to the commercial operation data increase with a fixed step size of a preset commercial update cycle of 300 seconds. The system uses the time stamps to perform time difference calculations to achieve absolute time alignment.

[0067] The pre-defined visual neural network adopts the ResNet-50 model architecture, which is constructed through supervised pre-training on a complex urban traffic flow dataset. The visual neural network receives a sequence of perceptual images as input and performs tensor multiplication operations using multiple spatial convolutional layers. These layers contain convolutional kernels of sizes 3×3, 5×5, and 7×7. By performing tensor multiplication operations, the visual neural network extracts road spatial dimensional features representing crowd density and obstacle distribution, and outputs a perceptual feature matrix with dimensions of 256×64×64.

[0068] When processing business operation data, the preset lightweight model adopts the MobileNet model structure, which is constructed using historical travel order features from mobile devices. The lightweight model receives business operation data input and uses the fully connected layers within its internal network to perform non-linear dimensionality reduction on the numerical features. After non-linear dimensionality reduction, the lightweight model outputs a 128-dimensional business feature vector. Simultaneously, the system extracts the activation parameters from the second-to-last layer of neurons in the lightweight model's internal network as feature weight parameters.

[0069] To eliminate the interference of non-critical pixel noise on high-premium commercial features, the system extracts the first feature gradient value representing pixel changes in the perceptual feature matrix and the second feature gradient value representing numerical fluctuations in the commercial feature vector.

[0070] After obtaining two sets of feature gradient values, the system aligns the first and second feature gradient values ​​according to a time marker and calculates the ratio of the first to the second feature gradient values. Specifically, the calculation is performed using a preset business update cycle. Set to 300 seconds, within a sampling period where business operation data remains unchanged, the value of the business feature vector corresponding to the current time marker. The commercial feature vector value corresponding to the adjacent previous time identifier They are completely equal, so the value of the second characteristic gradient is calculated. The value is 0; the system introduces a specific value setting as follows. The preset zero-smoothing correction term constant Using the first feature gradient value Divide by the second feature gradient value With preset zero-smoothing correction term constant The sum of these values ​​is used to calculate the proportion. To prevent computational crashes caused by a denominator of zero, when the feature interference threshold of 0.85 is reached, the system inputs the feature weight parameters into a preset nonlinear activation function. This nonlinear activation function is composed of the Sigmoid function. The feature weight parameters undergo interval compression through the nonlinear activation function to generate a channel attention mask. The channel attention mask is represented as a one-dimensional floating-point array with values ​​distributed in a continuous interval between 0 and 1. The system multiplies the channel attention mask by the channel dimension of the perceptual feature matrix. The system sets the channel values ​​of elements in the perceptual feature matrix that do not match the corresponding channel attention mask to zero; this zeroing operation filters out data representing non-critical environmental noise. Simultaneously, the system retains the channel values ​​of elements that match the corresponding channel attention mask, ultimately outputting a denoised perceptual feature matrix.

[0071] After feature denoising, the system concatenates the denoised perceptual feature matrix and the business feature vector along the preset feature dimensions to construct a comprehensive business decision vector. Subsequently, the system inputs this comprehensive business decision vector into a preset prediction network. The prediction network employs a multi-branch, multi-layer perceptron network structure and is trained using historical ride-hailing operation trajectories and billing logs. The prediction network includes a first multi-layer perceptron branch and a second multi-layer perceptron branch. The first multi-layer perceptron branch uses a linear regression layer to perform forward propagation and outputs the billing amount, specifically 150 RMB. The second multi-layer perceptron branch uses a fully connected layer to perform forward propagation and outputs the scheduling target coordinates corresponding to hotspot pick-up points, using the WGS84 coordinate system format.

[0072] Ultimately, the onboard motion control module converts the target coordinates into specific chassis motion control commands. These commands include target steering wheel angle commands and braking deceleration commands. Subsequently, the system sends these commands to the actuators of the corresponding vehicles in the autonomous ride-hailing fleet. These actuators include electronic power steering motors and electro-hydraulic braking systems. Upon receiving the commands, the actuators control the vehicles to actively turn and smoothly decelerate to avoid dynamic obstacles, ensuring the vehicles accurately reach the high-premium pick-up coordinates. This chassis control process effectively avoids erroneous logical judgments caused by excessive noise in high-dimensional image perception, leading to zero-passage road segments and ensuring the control system accurately executes commercial dispatching tasks in complex traffic flows.

[0073] To enable those skilled in the art to fully understand the technical solution of the present invention, the technical features involved in the specific embodiments are explained as follows.

[0074] Some technical features involved in the various specific embodiments of the present invention need to be described in detail so that those skilled in the art can fully understand the technical solutions provided by the present invention: Low-dimensional business features refer to the dimensionality-reduced business feature vectors obtained by performing nonlinear operations on the regional order supply-demand ratio and congestion premium rate using the fully connected layers in the MobileNet network model. Because business operation data has smaller feature dimensions and a sparser distribution compared to perceptual image sequences containing a large amount of pixel information, it is defined as low-dimensional business features.

[0075] High-dimensional perception data refers to real-time sequential data of the road environment captured by vehicle-mounted cameras. Because each frame of the image sequence contains millions of pixels, and these pixels are highly dense in spatial dimension, it is defined as high-dimensional perception data.

[0076] The feature interference threshold is a numerical benchmark used to evaluate the impact of perceived environmental noise on business features. When the proportion exceeds the feature interference threshold, it means that the noise intensity of non-critical pixels in the high-dimensional perceived data has exceeded the system's preset tolerance, thereby triggering the generation and application of channel attention masks.

[0077] The perceptual feature matrix refers to a multidimensional tensor representing the spatial dimensional features of a road, output by performing tensor product operations on a sequence of perceived images using multiple spatial convolutional layers in a visual neural network. The perceptual feature matrix preserves the spatial structure information in the image and transforms it into a matrix data structure suitable for feature alignment and fusion by the network.

[0078] A business feature vector (BMV) is a one-dimensional floating-point numerical array formed after business operation data has been mapped using a lightweight model. The BMV compresses the original business operation data into a fixed-dimensional numerical representation through a fully connected layer, facilitating cascaded computation by subsequent prediction networks.

[0079] The channel attention mask is a floating-point matrix compressed using the sigmoid function. This floating-point matrix has the exact same number of channels as the perceptual feature matrix. Through element-wise multiplication, the channel attention mask applies weights to each channel feature in the perceptual feature matrix. Elements with channel values ​​greater than a set retention threshold are retained, while elements with values ​​less than the threshold are set to zero, thus filtering out non-critical pixel noise.

[0080] The tolerance adjustment database is a lookup table storing the mapping relationship between different load state data and dynamic tolerance values. Load state data represents the real-time load values ​​obtained by the autonomous ride-hailing fleet vehicles using vehicle suspension sensors. The system uses the load state data as an index to query the corresponding dynamic tolerance value in the tolerance adjustment database, and then performs adaptive correction on the characteristic disturbance critical threshold.

Claims

1. A method for multimodal data fusion processing of intelligent vehicles based on large models, characterized in that, include: Acquire the perception image sequences and commercial operation data of the unmanned ride-hailing fleet, and record the corresponding time stamps; Extract the preset feature interference threshold; The perceived image sequence is converted into a perceived feature matrix using a preset visual neural network; The business operation data is mapped into a business feature vector using a preset lightweight model; the activation parameters output by the network layer inside the lightweight model are extracted as feature weight parameters. The first feature gradient value representing pixel changes is extracted from the perceptual feature matrix. This first feature gradient value is obtained by performing spatial difference operation on the perceptual feature matrix, calculating the spatial numerical fluctuation amplitude between adjacent feature elements, and taking the maximum value of the fluctuation amplitude. The second feature gradient value representing numerical fluctuation is extracted from the commercial feature vector. This second feature gradient value is obtained by performing time series difference operation on the commercial feature vector and calculating the temporal numerical fluctuation derivative corresponding to adjacent time markers. Based on the time markers, the first feature gradient value and the second feature gradient value are aligned, and the ratio of the first feature gradient value to the second feature gradient value is calculated. If the ratio value is greater than the feature interference threshold, the feature weight parameters in the commercial feature vector are converted into a channel attention mask through a preset nonlinear activation function. Multiply the channel attention mask by the channel dimension of the perceptual feature matrix; set the channel values ​​in the perceptual feature matrix that do not match the corresponding elements of the channel attention mask to zero, retain the channel values ​​that match the corresponding elements of the channel attention mask, and output the denoised perceptual feature matrix. The denoised perception feature matrix and the business feature vector are concatenated to construct a comprehensive business decision vector; the comprehensive business decision vector is input into a preset prediction network to output a prediction result, which includes a billing amount and a scheduling target coordinate. Road environment sequence data transmitted by the vehicle communication bus is received as the perception image sequence, and the scheduling target coordinate is converted into a chassis motion control command to control the vehicle to move to the scheduling target coordinate.

2. The method for multimodal data fusion processing of intelligent vehicles based on a large model according to claim 1, characterized in that, Acquire the perception image sequences and commercial operation data of the autonomous ride-hailing fleet, and record the corresponding time stamps, including: The road environment sequence data transmitted via the vehicle communication bus is received as the perceived image sequence, and the road environment sequence data includes dynamic obstacles; The system calls upon pre-defined backend service nodes to record business operation data, which includes regional order supply and demand ratios and congestion premium rates. The first time identifier attached to the perceived image sequence is parsed, and the second time identifier attached to the business operation data is parsed; The first time identifier and the second time identifier are used to perform time difference calculation to generate an aligned absolute time record as the time identifier.

3. The method for multimodal data fusion processing of intelligent vehicles based on a large model according to claim 2, characterized in that, The perceived image sequence is converted into a perceived feature matrix using a preset visual neural network; The business operation data is mapped into a business feature vector using a pre-defined lightweight model, including: Using the multi-layer spatial convolutional layers in the visual neural network, tensor multiplication operations are performed on the pixel matrix included in the perceived image sequence to extract road spatial dimension features and output the perceived feature matrix; Using the fully connected layer in the lightweight model, nonlinear operations are performed on the regional order supply-demand ratio and the congestion premium rate in the business operation data to output the dimensionality-reduced business feature vector.

4. The method for multimodal data fusion processing of intelligent vehicles based on a large model according to claim 3, characterized in that, When the ratio value is greater than the feature interference threshold, the feature weight parameters in the commercial feature vector are converted into a channel attention mask through a preset nonlinear activation function, including: The proportional value is compared with the feature interference threshold; if the proportional value is greater than the feature interference threshold, a business feature retention instruction is triggered, and the transmission of the perceptual feature matrix to the preset feature latent space is terminated. The activation parameters of the fully connected layer output in the lightweight model are extracted and used as the feature weight parameters; The feature weight parameters are input into the nonlinear activation function composed of the Sigmoid function for interval compression, generating a floating-point matrix whose numerical distribution is restricted to a preset continuous interval, which serves as the channel attention mask.

5. The method for multimodal data fusion processing of intelligent vehicles based on a large model according to claim 4, characterized in that, Multiply the channel attention mask by the channel dimension of the perceptual feature matrix; set the channel values ​​in the perceptual feature matrix that do not match the corresponding elements of the channel attention mask to zero, retain the channel values ​​that match the corresponding elements of the channel attention mask, and output the denoised perceptual feature matrix, including: Using the tensor broadcasting mechanism, the dimensionality of the channel attention mask is expanded to match the dimensionality of the perceptual feature matrix, thereby generating an expanded channel attention mask. The extended channel attention mask is multiplied with the perceptual feature matrix at the matrix element level to generate the product result. The channel values ​​whose product results in the perceptual feature matrix are lower than a preset retention threshold are set to zero, and the channel values ​​whose product results in the perceptual feature matrix are greater than or equal to the preset retention threshold are retained, and the denoised perceptual feature matrix is ​​output.

6. The method for multimodal data fusion processing of intelligent vehicles based on a large model according to claim 5, characterized in that, The denoised perceptual feature matrix and the business feature vector are concatenated to construct a comprehensive business decision vector; the comprehensive business decision vector is input into a preset prediction network to output a prediction result, which includes a billing amount and scheduling target coordinates, including: Perform a flattening operation on the denoised sensing feature matrix to generate a one-dimensional sensing sequence; The one-dimensional perception sequence and the business feature vector are concatenated in the preset feature dimension direction to construct the comprehensive business decision vector; The integrated business decision vector is input into the multi-branch multilayer perceptron network, which serves as the prediction network, for forward propagation, and the billing amount and the scheduling target coordinates are output.

7. The method for multimodal data fusion processing of intelligent vehicles based on a large model according to claim 6, characterized in that, After aligning the first feature gradient value and the second feature gradient value according to the time identifier, and calculating the ratio of the first feature gradient value to the second feature gradient value, the method further includes: Read the load status data of the unmanned ride-hailing fleet corresponding to the time identifier; The load condition data is input into a pre-stored fault tolerance adjustment database for matching and querying; Extract the dynamic tolerance value corresponding to the load state data; The dynamic tolerance value is used to numerically correct the feature interference critical threshold, thereby generating an updated feature interference critical threshold.

8. The method for multimodal data fusion processing of intelligent vehicles based on a large model according to claim 7, characterized in that, The output prediction result, after including the billing amount and the scheduling target coordinates, also includes: The billing amount is sent to a preset passenger terminal for display via a preset communication network; Convert the scheduling target coordinates into chassis motion control commands; The chassis motion control command is sent to the actuator of the corresponding vehicle in the unmanned ride-hailing fleet to control the vehicle to move to the dispatch target coordinates.

9. The method for multimodal data fusion processing of intelligent vehicles based on a large model according to claim 8, characterized in that, After aligning the first feature gradient value and the second feature gradient value according to the time identifier, and calculating the ratio of the first feature gradient value to the second feature gradient value, the method further includes: Establish a baseline sliding alignment window with a preset fixed time step; Within the baseline sliding alignment window, count the number of times the ratio value is greater than the feature interference threshold. The occurrence count is compared with a preset fault tolerance threshold. If the occurrence count is greater than the preset fault tolerance threshold, it is determined that there is abnormal message data in the sensing image sequence. The communication port for receiving the sensing image sequence is closed, and the remaining data corresponding to the sensing image sequence in the preset buffer queue is cleared.

Images