End-to-end automatic driving control method and device based on multi-camera fusion

Abstract

The embodiment of the invention provides an end-to-end automatic driving control method and device based on multi-camera fusion, and multi-view target detection and tracking are realized through spatial transformation and coordinate mapping by combining front wide-angle camera information and left and right wide-angle camera information. A multi-view feature fusion network architecture is designed, the multi-view feature fusion network architecture comprises three sub-networks of feature extraction, dynamic weight distribution and feature fusion, and the fusion weight is dynamically adjusted based on image definition, detection confidence and view overlapping degree. A geometric consistency constraint between visual angles and a reconstruction loss function are introduced, a deep neural network model is constructed, abnormal conditions such as camera shielding are effectively handled, and an accurate control instruction is output. According to the method, the defects of the traditional technology in the aspects of multi-view information fusion, shielding processing and the like are overcome, and the sensing ability and the control reliability of the automatic driving system are remarkably improved.

Description

Technical Field

[0001] This application relates to the field of intelligent driving, and specifically relates to an end-to-end autonomous driving control method and device based on multi-camera fusion. Background Art

[0002] Existing end-to-end autonomous driving control methods have obvious deficiencies. Traditional methods mainly rely on a single camera view, lacking a collaborative processing mechanism for multi-view information, and it is difficult to ensure the perception accuracy and control stability in complex scenarios.

[0003] In addition, there are bottlenecks in multi-camera feature fusion in the existing technology. Most systems adopt simple feature stitching or average fusion strategies, failing to fully consider the image quality and target detection reliability of different views, which affects the fusion effect.

[0004] Existing systems have technical shortcomings in occlusion handling and robustness. Lack of effective view consistency constraints and dynamic weight adjustment mechanisms, and it is difficult to handle abnormal situations such as partial occlusion of the camera. Solving these problems is of great significance for improving the perception ability and control accuracy of autonomous driving systems. Summary of the Invention

[0005] In view of the problems in the existing technology, this application provides an end-to-end autonomous driving control method and device based on multi-camera fusion, which can solve the deficiencies of traditional technologies in multi-view information fusion, occlusion handling, etc., and significantly improve the perception ability and control reliability of autonomous driving systems.

[0006] To solve at least one of the above problems, this application provides the following technical solutions:

[0007] In a first aspect, this application provides an end-to-end autonomous driving control method based on multi-camera fusion, including:

[0008] Input the front road condition images collected by the vehicle front camera and the side environment images collected by the left and right wide-angle cameras into the preprocessing module, perform spatial transformation and scale normalization on the multi-channel images, establish the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters, calculate the depth and position information of the target objects within the field of view of each camera, use the nearest neighbor search method to identify the associated targets within the overlapping area of the fields of view of different cameras, establish a multi-target tracker to record the target movement trajectories, and construct a multi-modal data set containing multi-view target positions, motion states, and image features;

[0009] Train a multi-view feature fusion network based on the multi-modal dataset. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight allocation sub-network, and a feature fusion sub-network. The feature extraction sub-network processes multiple image inputs respectively using a parallel convolution structure. The dynamic weight allocation sub-network calculates the fusion weight coefficients of each view feature based on image sharpness, object detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on multiple features according to the fusion weight coefficients to generate a fused feature representation;

[0010] Construct a deep neural network model based on multi-view consistency constraints. Input the fused feature into the deep neural network model. The deep neural network model sets a geometric consistency loss function and a reconstruction loss function between views. Optimize the network parameters by minimizing the loss function, dynamically adjust the feature fusion weights when there is partial occlusion among different cameras, output three control quantities of steering angle, acceleration, and braking force, and convert the control quantities into vehicle execution instructions to complete the automatic driving control of the vehicle.

[0011] Further, it also includes: performing downsampling on the front road condition image collected by the front camera and the side environment images collected by the left and right wide-angle cameras, performing white balance correction and histogram equalization processing on multiple images, using the bilinear interpolation algorithm to scale images with different resolutions to a unified size, and performing pixel normalization processing on the scaled images to generate standardized image data;

[0012] Use the Zhang calibration method to obtain the internal parameter matrix and distortion coefficients of multiple cameras, calculate the rotation matrix and translation vector between cameras based on the calibration image sequence collected by the checkerboard calibration board, use the rotation matrix and translation vector as the external parameter matrix, establish the coordinate transformation relationship between the image coordinate system and the vehicle body coordinate system in combination with the internal parameter matrix, and perform perspective transformation on the standardized image data to obtain a bird's-eye view.

[0013] Further, it also includes: inputting each camera image into a target detection network to extract target region candidate boxes and class labels, calculating the depth value of each pixel point in the image based on a monocular depth estimation network, and converting the image coordinates of the target region candidate boxes to the vehicle body coordinate system in combination with the coordinate transformation relationship to generate a target state vector containing target position, size, class, and depth information;

[0014] Calculate the similarity of the target state vectors detected by different cameras, construct an association cost matrix based on target position, size, and appearance features, use the Hungarian algorithm to perform matching optimization on the association cost matrix to obtain the target association result, use the Kalman filter to predict and update the motion state of the associated target, and save the target motion trajectory, state vector, and image features to the dataset.

[0015] Further, it also includes: dividing the multi-modal dataset into a training set and a validation set, performing data augmentation operations such as randomly cropping, rotating, mirror flipping, and brightness adjustment on the image data in the training set, constructing a batch data loader to read training samples, inputting the training samples into the feature extraction sub-network, where the feature extraction sub-network uses a residual convolutional structure to extract multi-scale spatial features, and adding a batch normalization layer and an activation function layer after each convolutional layer;

[0016] Performing channel attention calculation and spatial attention calculation on the feature map output by the feature extraction sub-network, weighting the feature map based on the attention weights, inputting the weighted feature map into a global average pooling layer and a fully connected layer to generate a feature vector with a fixed dimension, constructing the training objective of the feature extraction sub-network using a cross-entropy loss function and a contrastive learning loss function, and using the stochastic gradient descent method to optimize the network parameters.

[0017] Further, it also includes: calculating the clarity evaluation index of each path of images, performing a Laplacian of Gaussian transform on the images to extract an edge intensity map, counting the number of non-zero pixel points and the intensity mean of the edge intensity map, inputting the edge intensity features and the object detection confidence scores into a weight prediction network, where the weight prediction network uses a multi-layer perceptron structure to output the initial weight value of each perspective, and correcting the initial weight value based on the field of view overlap matrix to obtain the fusion weight coefficient;

[0018] Inputting the multi-path feature vectors output by the feature extraction sub-network into a feature fusion sub-network, where the feature fusion sub-network uses a self-attention mechanism to calculate the correlation scores between the feature vectors, weighting and summing the feature vectors in combination with the fusion weight coefficient, and inputting the weighted result into a feature transformation module for dimensionality reduction and non-linear transformation to generate a fused feature representation.

[0019] Further, it also includes: constructing a deep neural network model with an encoder-decoder structure, where the encoder uses a multi-layer convolutional network to perform feature extraction and dimensionality reduction on the fused features, the decoder uses a transposed convolutional layer to restore the encoded features to a feature map with the same size as the input image, calculating the projection consistency score between the feature maps of different perspectives based on the geometric transformation relationship between the cameras, and using the projection consistency score as the optimization target of the inter-view geometric consistency loss function;

[0020] Inputting the feature map output by the decoder into a reconstruction branch network, where the reconstruction branch network performs upsampling and pixel-level prediction on the feature map to generate a reconstructed image, calculating the mean square error between the reconstructed image and the original input image as the reconstruction loss function, using the weighted sum of the inter-view geometric consistency loss function and the reconstruction loss function as the network optimization target, and using the backpropagation algorithm to update the network parameters.

[0021] Further, it also includes: constructing a gradient descent optimizer, calculating the loss value of the network model after each round of training based on the validation set, recording the optimal model parameters using the early stopping strategy, conducting an occlusion simulation experiment on the input image, randomly adding an occlusion mask to the image area, calculating the proportion of the occluded area in the image, inputting the proportion of the occluded area into the weight adjustment network, where the weight adjustment network outputs a compensation value for the feature fusion weight, and dynamically updating the original fusion weight according to the compensation value;

[0022] Input the output features of the deep neural network model into the control prediction branch. The control prediction branch includes multiple fully connected layers, which respectively predict the steering angle, acceleration, and braking force values. Perform numerical normalization and smoothing processing on the prediction results, convert the processed control quantities into low-level execution instructions through the vehicle control interface, and perform safety constraints on the execution instructions according to the road type and driving speed.

[0023] In a second aspect, the present application provides an end-to-end autonomous driving control device based on multi-camera fusion, including:

[0024] A multi-camera fusion module for inputting the front road condition images collected by the vehicle front camera and the side environment images collected by the left and right wide-angle cameras into the preprocessing module, performing spatial transformation and scale normalization on the multiple images, establishing the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters, calculating the depth and position information of the target objects within the field of view of each camera, using the nearest neighbor search method to identify the associated targets within the overlapping area of the fields of view of different cameras, establishing a multi-target tracker to record the target movement trajectory, and constructing a multi-modal dataset including multi-view target positions, motion states, and image features;

[0025] A fusion network construction module for training a multi-view feature fusion network based on the multi-modal dataset. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight assignment sub-network, and a feature fusion sub-network. The feature extraction sub-network uses a parallel convolution structure to process the multiple image inputs respectively. The dynamic weight assignment sub-network calculates the fusion weight coefficients of each view feature based on the image clarity, target detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on the multiple features according to the fusion weight coefficients to generate a fused feature representation;

[0026] An autonomous driving control module is used to construct a deep neural network model based on multi - perspective consistency constraints, input the fused features into the deep neural network model. The deep neural network model is set with an inter - perspective geometric consistency loss function and a reconstruction loss function. By minimizing the loss function, the network parameters are optimized. When there is partial occlusion in different cameras, the feature fusion weights are dynamically adjusted, and three control quantities, namely the steering angle, acceleration, and braking force, are output. The control quantities are converted into vehicle execution instructions to complete the autonomous driving control of the vehicle.

[0027] In a third aspect, the present application provides an electronic device, including a memory, a processor, and a computer program stored on the memory and executable on the processor. When the processor executes the program, the steps of the end - to - end autonomous driving control method based on multi - camera fusion described above are implemented.

[0028] In a fourth aspect, the present application provides a computer - readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the end - to - end autonomous driving control method based on multi - camera fusion described above are implemented.

[0029] In a fifth aspect, the present application provides a computer program product, including a computer program / instructions. When the computer program / instructions are executed by a processor, the steps of the end - to - end autonomous driving control method based on multi - camera fusion described above are implemented.

[0030] As can be seen from the above technical solutions, the present application provides an end - to - end autonomous driving control method and device based on multi - camera fusion. By combining the information of the front - mounted and left - and - right wide - angle cameras, multi - perspective object detection and tracking are achieved through spatial transformation and coordinate mapping. A multi - perspective feature fusion network architecture is designed, which includes three sub - networks: feature extraction, dynamic weight allocation, and feature fusion. The fusion weights are dynamically adjusted based on image sharpness, detection confidence, and field - of - view overlap. An inter - perspective geometric consistency constraint and a reconstruction loss function are introduced to construct a deep neural network model, effectively dealing with abnormal situations such as camera occlusion and outputting accurate control instructions. This method solves the deficiencies of traditional technologies in multi - perspective information fusion, occlusion processing, etc., and significantly improves the perception ability and control reliability of the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for the description of the embodiments or the prior art. Obviously, the drawings in the following description are some embodiments of the present application. For those of ordinary skill in the art, other drawings can be obtained based on these drawings without creative efforts.

[0032] Figure 1Schematic flowchart of the end-to-end autonomous driving control method based on multi-camera fusion in the embodiments of the present application;

[0033] Figure 2 Structural diagram of the end-to-end autonomous driving control device based on multi-camera fusion in the embodiments of the present application;

[0034] Figure 3 Schematic structural diagram of the electronic device in the embodiments of the present application.

[0035] Reference numerals:

[0036] Electronic device 9600, central processing unit 9100, memory 9140, communication module 9110, input unit 9120, audio processor 9130, display 9160, power supply 9170, buffer memory 9141, application / function storage unit 9142, data storage unit 9143, driver program storage unit 9144, antenna 9111, speaker 9131, microphone 9132. Detailed implementation manners

[0037] To make the objectives, technical solutions, and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present application. Apparently, the described embodiments are some, but not all, of the embodiments of the present application. All other embodiments obtained by those of ordinary skill in the art based on the embodiments of the present application without creative efforts shall fall within the protection scope of the present application.

[0038] In the technical solutions of the present application, the acquisition, storage, use, processing, etc. of data all comply with the relevant regulations of national laws and regulations.

[0039] Considering the problems existing in the prior art, the present application provides an end-to-end autonomous driving control method and device based on multi-camera fusion. By combining the information of the front and left and right wide-angle cameras, multi-view target detection and tracking are realized through spatial transformation and coordinate mapping. A multi-view feature fusion network architecture is designed, which includes three sub-networks: feature extraction, dynamic weight allocation, and feature fusion. The fusion weight is dynamically adjusted based on image clarity, detection confidence, and field-of-view overlap. The geometric consistency constraint between views and the reconstruction loss function are introduced to construct a deep neural network model, which can effectively handle abnormal situations such as camera occlusion and output accurate control instructions. This method solves the deficiencies of the traditional technology in multi-view information fusion, occlusion processing, etc., and significantly improves the perception ability and control reliability of the autonomous driving system.

[0040] In order to solve the deficiencies of traditional technologies in aspects such as multi-view information fusion and occlusion processing, and significantly improve the perception ability and control reliability of the autonomous driving system, this application provides an embodiment of an end-to-end autonomous driving control method based on multi-camera fusion. Refer to Figure 1 The end-to-end autonomous driving control method based on multi-camera fusion specifically includes the following content:

[0041] Step S101: Input the front road condition images collected by the vehicle's front camera and the side environment images collected by the left and right wide-angle cameras into the preprocessing module. Perform spatial transformation and scale normalization on the multi-channel images, establish the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters, calculate the depth and position information of the target objects within the field of view of each camera, use the nearest neighbor search method to identify the associated targets within the overlapping area of the fields of view of different cameras, establish a multi-target tracker to record the target movement trajectories, and construct a multi-modal dataset containing multi-view target positions, motion states, and image features.

[0042] Optionally, this embodiment designs a comprehensive image preprocessing process for the vehicle-mounted multi-camera system. The front camera uses a high-definition camera with a 120-degree field of view angle, mainly used to capture key information such as the front road conditions, traffic signs, and other vehicles. The left and right sides use 170-degree ultra-wide-angle cameras, which expand the observation range on the sides of the vehicle and effectively solve the problem of the blind spot of the field of view existing in traditional single cameras. The image data of the three cameras are synchronously collected by the vehicle-mounted computing unit at a sampling frequency of 30 frames per second, ensuring the timing consistency of the multi-channel data.

[0043] This embodiment realizes an efficient image preprocessing mechanism. First, perform distortion correction on the original image. Use a polynomial model to describe the lens distortion and calculate the distortion coefficients through iterative optimization. The spatial transformation of the image uses the perspective transformation method to uniformly transform the images of the three cameras to the top view plane in the vehicle body coordinate system. During the scale normalization process, the bilinear interpolation algorithm is used to adjust the images of different resolutions to the same size and perform pixel value normalization processing to eliminate the brightness and contrast differences between different cameras.

[0044] This embodiment establishes an accurate coordinate system conversion relationship. Obtain the internal parameter matrix of each camera through the Zhang's calibration method, including parameters such as the focal length and the principal point coordinates. The calibration of the external parameter matrix uses a checkerboard calibration board. Collect the calibration image sequences at different positions and calculate the relative pose relationship between the cameras through feature point matching. Based on the calibration parameters, establish the mapping relationship from the image coordinate system to the vehicle body coordinate system, realizing the geometric alignment of multi-view images.

[0045] This embodiment designs a reliable depth estimation method. A monocular depth estimation network based on deep learning is adopted. This network uses an encoder-decoder structure and improves the accuracy of depth prediction through multi-scale feature fusion. The network is trained using a dataset with ground truth depth, and the loss function comprehensively considers the depth prediction error and the depth map smoothness constraint. In practical applications, the network can output pixel-level depth estimation results in real time.

[0046] This embodiment implements an innovative target association strategy. In the overlapping area of different camera fields of view, a nearest neighbor search method based on appearance and position is used to identify associated targets. Appearance features are extracted through a deep convolutional network, and the position information is calculated based on the predicted depth value and the coordinate transformation relationship. The similarity score is calculated for each pair of candidate targets, an association cost matrix is constructed, and the optimal matching scheme is solved through the Hungarian algorithm.

[0047] This embodiment constructs a complete multi-object tracking framework. The tracker uses the Kalman filter algorithm, and the state vector contains the position, velocity, and acceleration information of the target. The motion model considers the non-linear motion characteristics of the vehicle, and the observation model fuses the visual detection results and depth information. The target ID is dynamically maintained during the tracking process to handle situations such as the appearance, disappearance, and occlusion of the target. To improve the stability of tracking, a trajectory smoothing constraint is introduced to suppress the trajectory jitter caused by detection errors.

[0048] This embodiment establishes a standardized dataset organization method. Each sample in the multi-modal dataset contains information such as multi-view images, object detection boxes, depth maps, and object trajectories collected synchronously. The dataset adopts a hierarchical storage structure to support efficient data reading and retrieval. At the same time, a data augmentation mechanism is implemented to expand the training samples through operations such as random cropping and rotation, improving the generalization ability of the model.

[0049] Through the above technological innovations, this embodiment effectively solves the key problems in the multi-camera autonomous driving system: imperfect image preprocessing, inaccurate coordinate transformation, unreliable target association, etc. In practical applications, this solution significantly improves the integrity and accuracy of environmental perception, especially the object detection and tracking performance in complex traffic scenarios. Compared with the single-camera solution, the multi-camera system has a larger perception range and stronger robustness, and can provide more reliable environmental information for subsequent driving decisions. The systematicness and innovativeness of this technical solution enable it to adapt to various road conditions and weather environments, providing an important guarantee for the safety of the autonomous driving system.

[0050] Step S102: Train a multi-view feature fusion network based on the multi-modal dataset. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight allocation sub-network, and a feature fusion sub-network. The feature extraction sub-network processes multiple image inputs respectively using a parallel convolution structure. The dynamic weight allocation sub-network calculates the fusion weight coefficients of each view feature based on image sharpness, object detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on multiple features according to the fusion weight coefficients to generate a fused feature representation;

[0051] Optionally, in this embodiment, an innovative multi-view feature fusion network architecture is first constructed. The feature extraction sub-network adopts a three-branch parallel structure, corresponding to the front, left, and right camera inputs respectively. Each branch uses ResNet-50 as the backbone network. Based on the standard ResNet structure, in this embodiment, dilated convolutions are introduced in each residual block, and the receptive field range is expanded by setting different dilation rates to effectively capture road environment features at different scales. At the same time, a feature pyramid module is added after the high-level feature map to fuse multi-scale features and improve the detection ability for near and far targets.

[0052] This embodiment implements an accurate feature extraction mechanism. Each branch network contains five stages. In the initial stage, a 7×7 convolutional layer with a stride of 2 is used to downsample the input image. The subsequent four stages respectively contain 3, 4, 6, and 3 residual blocks. The residual block adopts a "bottleneck" structure, and the computational complexity is reduced through a combination of 1×1 convolution for dimensionality reduction, 3×3 convolution for feature extraction, and 1×1 convolution for dimensionality increase. A batch normalization layer and a ReLU activation function are added after each convolutional layer to improve the convergence speed and non-linear expression ability of the network.

[0053] This embodiment designs an innovative dynamic weight allocation mechanism. The weight allocation sub-network first calculates the sharpness score of each image, extracts the image edges through the Gaussian-Laplacian operator, and statistically analyzes the edge intensity distribution characteristics. At the same time, the image is input into the object detection network to obtain the confidence scores of the detection boxes, and the area ratio of the field of view overlap region between different views is calculated. These three types of features are input into a three-layer fully connected network after normalization to predict the weight coefficients of each view. The weight prediction process considers three key factors: image quality, object reliability, and view complementarity.

[0054] This embodiment constructs a reliable feature fusion strategy. The feature fusion sub-network uses a self-attention mechanism to process multiple feature vectors, and calculates the correlation scores between features through matrix multiplication of the query matrix, key matrix, and value matrix. The correlation scores are multiplied by the value matrix after softmax normalization to generate a feature representation considering context dependence. At the same time, a residual connection is introduced to retain the original feature information, and finally the feature dimension is adjusted through 1×1 convolution to obtain the fused feature.

[0055] This embodiment realizes a stable data pre - processing process. For training data, enhancement methods such as random cropping, horizontal flipping, and brightness and contrast adjustment are used to improve the generalization ability of the model. Considering the particularity of the autonomous driving scenario, the cropping area is set in the lower half of the image to retain more road area information. At the same time, different lighting conditions are simulated by modifying brightness and contrast to enhance the model's adaptability in complex weather.

[0056] This embodiment establishes a complete training optimization mechanism. A phased training strategy is adopted. First, the parameters of the feature extraction network are fixed, and only the weight assignment network and the feature fusion network are trained. After the fusion module converges, the feature extraction network is unfrozen for end - to - end fine - tuning. The loss function consists of three parts: classification loss, detection loss, and feature consistency loss. The classification loss adopts the form of cross - entropy, the detection loss uses the smooth - L1 form, and the feature consistency loss calculates the cosine similarity of the features in the overlapping region.

[0057] This embodiment designs an innovative verification and evaluation system. In the verification stage, the robustness of the model is tested by adding random occlusions and adjusting the exposure. The verification metrics include multiple dimensions such as object detection accuracy, feature matching precision, and discriminability of the fused features. At the same time, the weight assignment situation of each perspective in different scenarios is recorded to analyze the model's adaptability to different degradation situations.

[0058] Through the above - mentioned technological innovations, this embodiment effectively solves the key problems in multi - camera fusion: insufficient feature extraction, unreasonable weight assignment, unstable fusion strategy, etc. In practical applications, this solution can adapt to different weather conditions and road scenarios, maintaining stable feature extraction and fusion effects. It is especially suitable for autonomous driving tasks in complex traffic environments. Through the dynamic fusion of multi - perspective features, the accuracy and robustness of environmental perception are significantly improved. The systematicness and innovativeness of this solution enable it to support the autonomous driving requirements in different scenarios, and through continuous feature learning and optimization, the stable improvement of the perception ability is achieved.

[0059] Step S103: Construct a deep neural network model based on multi - perspective consistency constraints, input the fused features into the deep neural network model. The deep neural network model sets the inter - perspective geometric consistency loss function and the reconstruction loss function, optimizes the network parameters by minimizing the loss function, dynamically adjusts the feature fusion weights when partial occlusions occur in different cameras, outputs three control quantities: steering angle, acceleration, and braking force, and converts the control quantities into vehicle execution instructions to complete the autonomous driving control of the vehicle.

[0060] Optionally, in this embodiment, an innovative deep neural network architecture is first designed. The main body of the network adopts an encoder-decoder structure. The encoder uses a multi-layer convolutional network to gradually extract the deep representation of the fused features, and the decoder restores the spatial structure of the features through deconvolution layers. During the encoding process, batch normalization and the LeakyReLU activation function are connected after each convolution layer, and max pooling is used for downsampling. During the decoding process, a skip connection structure is adopted to fuse the feature maps of the corresponding layers of the encoder with the upsampled features to retain the detailed information.

[0061] This embodiment realizes strict geometric consistency constraints. Based on the calibrated camera parameters, the projection transformation matrix between different perspectives is calculated. For the feature map output by the decoder, the features of one perspective are projected to another perspective through projection transformation, and the consistency loss between the projected features and the actual features is calculated. The consistency loss adopts a combined form of the structural similarity index (SSIM) and the L1 distance, considering both the structural information and the pixel-level differences of the features.

[0062] This embodiment constructs a reliable feature reconstruction mechanism. The reconstruction branch network adopts a multi-scale structure, including multiple parallel deconvolution layers, which respectively generate reconstruction feature maps of different resolutions. The feature maps of each resolution are fused through weighted fusion to obtain the final reconstruction result. The reconstruction loss function comprehensively considers the pixel-level error, the perceptual loss, and the adversarial loss. Among them, the perceptual loss is calculated by extracting features using a pre-trained VGG network, and the adversarial loss is used to evaluate the reconstruction quality through a discriminator network.

[0063] This embodiment designs an innovative occlusion processing strategy. By setting an attention module to dynamically evaluate the reliability of the features of each perspective, when a camera occlusion is detected, the fusion weight of the features of that perspective is reduced. The occlusion detection is based on the image quality assessment and the object detection results, and the persistence of the occlusion is judged by combining the temporal information. At the same time, a feature compensation mechanism is realized, and the information of other perspectives is used to infer and reconstruct the occluded area.

[0064] This embodiment realizes accurate control quantity prediction. The control branch network adopts a multi-layer perceptron structure to predict the steering angle, acceleration, and braking force respectively. Each prediction branch considers the current state and historical information, and models the temporal dependence relationship through LSTM units. Constraint conditions are introduced during the prediction process to ensure that the control quantity meets the vehicle dynamics limitations and safety boundaries. At the same time, a smoothing processing mechanism is realized to suppress the sudden change of the control quantity and provide a comfortable driving experience.

[0065] This embodiment establishes a complete training optimization process. An end-to-end training method is adopted to optimize three modules: feature extraction, geometric consistency, and control prediction simultaneously. During the training process, the curriculum learning strategy is used to gradually increase the scene complexity and occlusion difficulty. The weight coefficients of each component in the loss function are dynamically adjusted according to the performance on the validation set to maintain the training balance of different tasks.

[0066] This embodiment constructs a reliable execution control mechanism. The control quantity is converted into specific execution instructions through the vehicle dynamics model, including the steering wheel angle, throttle opening, and braking force. The execution instructions consider the current state of the vehicle and road conditions, and set safety thresholds to prevent excessive operations. At the same time, a smooth transition mechanism is implemented to gradually adjust the control quantity when switching control modes, avoiding discomfort caused by sudden changes.

[0067] This embodiment designs an innovative safety guarantee mechanism. Multiple safety checks are set at the control execution level, including control quantity range verification, state prediction verification, and collision risk assessment. When potential risks are detected, an emergency handling process is initiated, which may include deceleration, suspension of execution, or manual takeover. At the same time, a control right switching mechanism is implemented to support the driver to take over vehicle control at any time.

[0068] Through the above technological innovations, this embodiment effectively solves the key problems in end-to-end autonomous driving control: incomplete feature expression, unstable multi-view fusion, inaccurate control prediction, etc. In practical applications, this solution can adapt to complex and changing driving environments and maintain stable and reliable control effects. It is particularly suitable for complex scenarios such as urban roads. Through multi-view perception and consistency constraints, the accuracy and reliability of driving decisions are significantly improved. The systematicness and innovativeness of this solution enable it to handle various driving scenarios and abnormal situations, and through continuous optimization and verification, the overall autonomous driving ability is improved.

[0069] As can be seen from the above description, the end-to-end autonomous driving control method based on multi-camera fusion provided by the embodiments of this application can achieve multi-view object detection and tracking by combining the information of the front and left and right wide-angle cameras through spatial transformation and coordinate mapping. A multi-view feature fusion network architecture is designed, which includes three sub-networks: feature extraction, dynamic weight allocation, and feature fusion. The fusion weights are dynamically adjusted based on image clarity, detection confidence, and field of view overlap. The geometric consistency constraint and reconstruction loss function between views are introduced to construct a deep neural network model, which can effectively handle abnormal situations such as camera occlusion and output accurate control instructions. This method solves the deficiencies of traditional technologies in multi-view information fusion, occlusion processing, etc., and significantly improves the perception ability and control reliability of the autonomous driving system.

[0070] In an embodiment of the end-to-end autonomous driving control method based on multi-camera fusion of the present application, the following specific content may also be included:

[0071] Step S201: Downsample the front road condition image collected by the front camera and the side environment images collected by the left and right wide-angle cameras, perform white balance correction and histogram equalization processing on the multi-channel images, use the bilinear interpolation algorithm to scale the images with different resolutions to a unified size, and perform pixel normalization processing on the scaled images to generate standardized image data;

[0072] Step S202: Use the Zhang's calibration method to obtain the internal parameter matrix and distortion coefficients of multiple cameras, calculate the rotation matrix and translation vector between the cameras based on the calibration image sequence collected by the checkerboard calibration board, use the rotation matrix and translation vector as the external parameter matrix, establish the coordinate transformation relationship between the image coordinate system and the vehicle body coordinate system in combination with the internal parameter matrix, and perform perspective transformation on the standardized image data to obtain a bird's-eye view.

[0073] Optionally, this embodiment first realizes a comprehensive image preprocessing process. The front camera uses a high-definition sensor with a resolution of 2560×1440, and the left and right wide-angle cameras use sensors with a resolution of 1920×1080. The image resolution is reduced to a level suitable for real-time processing through 4-fold downsampling. The Gaussian pyramid structure is used in the downsampling process, and Gaussian blur is performed before downsampling to effectively suppress the aliasing and moiré effects. This design significantly reduces the computational burden of subsequent processing while maintaining key visual information.

[0074] This embodiment designs a reliable image correction mechanism. The white balance correction uses the gray world assumption algorithm. By calculating the average values of the RGB three channels, the color temperature characteristics of the scene light source are obtained, and then gain adjustment is performed on each channel. The histogram equalization processing uses an adaptive local enhancement method, which processes the image in blocks and performs interpolation between blocks, avoiding the over-enhancement problem that may be caused by traditional global equalization. These correction steps effectively eliminate the influence of different weather conditions and lighting environments on the image quality.

[0075] This embodiment realizes an accurate image scaling process. The bilinear interpolation algorithm is used to unify the image size. This algorithm performs linear interpolation in the horizontal and vertical directions respectively, and calculates the weighted average value of the four adjacent source pixels corresponding to the target pixel position. The weight coefficients are calculated based on the distance relationship between pixels, ensuring the smoothness of the interpolation result. For input images with different resolutions, the scaling ratio is first calculated, and then the sampling position of each target pixel is determined through the mapping relationship.

[0076] This embodiment constructs a standardized pixel processing flow. In the normalization process, the mean and standard deviation of each image are first calculated, and then the pixel values are transformed into a distribution with zero mean and unit variance. This processing method makes the images collected by different cameras have similar numerical distribution characteristics, which is beneficial to the feature extraction of subsequent deep learning models. At the same time, an outlier truncation mechanism is implemented to limit the influence of extreme pixel values on the normalization result.

[0077] This embodiment designs an innovative camera calibration method. The improved Zhang's calibration algorithm is adopted, and a 9×6 checkerboard calibration board is used to collect multiple groups of calibration images at different positions and poses. The intersection coordinates of the checkerboard are extracted by a sub-pixel level corner detection algorithm, and the correspondence between image points and spatial points is established. The least squares optimization method is used in the calibration process, and the influence of radial distortion and tangential distortion is considered simultaneously, improving the calibration accuracy.

[0078] This embodiment realizes an accurate coordinate transformation mechanism. Based on the internal parameter matrix and distortion coefficients obtained by calibration, the image is first corrected for distortion. Then, using the relative pose relationship between multiple cameras, a unified vehicle body coordinate system is established. The spatial relationship of each camera relative to the vehicle body coordinate system is described by a rotation matrix and a translation vector, realizing the geometric alignment of images from different perspectives. This transformation relationship provides a geometric basis for subsequent multi-view feature fusion.

[0079] This embodiment constructs a reliable bird's-eye view generation mechanism. Based on the calibration parameters, a perspective transformation matrix is calculated to project the image onto the ground plane centered on the vehicle. The tilt and undulation of the ground are considered during the transformation process, and the accuracy of the transformation is improved by dynamically adjusting the projection parameters. At the same time, an image stitching function is realized to seamlessly fuse the bird's-eye views from multiple perspectives to generate a complete panoramic view covering the area around the vehicle.

[0080] This embodiment designs a complete quality assessment mechanism. The output of each processing step is inspected for quality, including image sharpness assessment, color restoration assessment, and geometric transformation accuracy assessment. When an anomaly is detected, the system can automatically adjust the processing parameters or trigger a re-acquisition process. At the same time, a real-time monitoring mechanism is realized to continuously evaluate the image processing quality and ensure the stable operation of the system.

[0081] Through the above technological innovations, this embodiment effectively solves the key problems in the multi-camera autonomous driving system: unstable image quality, inaccurate coordinate alignment, unreliable perspective conversion, etc. In practical applications, this solution can adapt to different driving environments and weather conditions, providing stable and high-quality visual input. It is especially suitable for complex road scenarios. Through the preprocessing and calibration of multi-perspective images, it provides a reliable data basis for subsequent environmental perception and decision-making control. The systematicness and innovativeness of this solution enable it to support all-weather autonomous driving requirements. Through continuous image optimization and geometric correction, the stable operation of the perception system is achieved.

[0082] In an embodiment of the end-to-end autonomous driving control method based on multi-camera fusion of the present application, the following content may also be specifically included:

[0083] Step S301: Input each camera image into the target detection network to extract the target region candidate boxes and class labels, calculate the depth value of each pixel point in the image based on the monocular depth estimation network, and combine the coordinate transformation relationship to convert the image coordinates of the target region candidate boxes into the vehicle body coordinate system, generating a target state vector containing target position, size, class, and depth information;

[0084] Step S302: Calculate the similarity of the target state vectors detected by different cameras, construct an association cost matrix based on the target position, size, and appearance features, use the Hungarian algorithm to match and optimize the association cost matrix to obtain the target association result, use the Kalman filter to predict and update the motion state of the associated targets, and save the target motion trajectory, state vector, and image features to the data set.

[0085] The efficient image preprocessing process is first realized in this embodiment. Among the multi-cameras, the front camera uses a resolution of 2560×1440, and the left and right wide-angle cameras use a resolution of 1920×1080. Downsampling the original image by a factor of 4 can significantly reduce the computational load while retaining sufficient detail information. The white balance correction uses the gray world algorithm, assuming that the average value of all colors in the image approaches gray, and calculates the gain coefficient for each color channel. Histogram equalization remaps the pixel values through the cumulative distribution function, enhancing the image contrast and enhancing the details in the dark areas.

[0086] This embodiment designs an innovative image normalization scheme. The bilinear interpolation algorithm considers the weighted contributions of the four adjacent pixels during the scaling process, and the weight coefficients are calculated based on the relative positions of the target pixel and the original pixels. All images are uniformly scaled to a resolution of 640×360, which achieves a balance between computational efficiency and information retention. Pixel normalization maps the RGB values to the interval [-1, 1], improving the stability of network training. At the same time, adaptive gamma correction is implemented to dynamically adjust the gamma value according to the overall brightness of the image.

[0087] This embodiment constructs an accurate camera calibration framework. Zhang's calibration method collects multiple groups of images through a planar checkerboard calibration board, extracts the corner coordinates, and establishes a non-linear equation system. By solving the equations, the internal parameter matrix K (including the focal lengths fx, fy and the principal point coordinates cx, cy) and the distortion coefficients (including the radial distortions k1, k2 and the tangential distortions p1, p2) are obtained. During the calibration process, an iterative optimization algorithm is used to minimize the reprojection error and improve the accuracy of parameter estimation.

[0088] This embodiment implements a reliable external parameter calibration mechanism. The calibration board is placed within the common field of view of different cameras, and the corresponding relationships are established through feature matching. Based on the PnP algorithm, the relative poses between the cameras are solved to obtain the rotation matrix R and the translation vector t. To improve the calibration accuracy, the Bundle Adjustment algorithm is used to globally optimize all parameters and minimize the reprojection error of all observation points.

[0089] This embodiment establishes an accurate coordinate system transformation relationship. The origin of the vehicle body coordinate system is set at the center of the rear axle, the x-axis points to the front of the vehicle, the y-axis points to the left, and the z-axis points upward. The transformation relationship between the image point P(u, v) and the vehicle body coordinate system point X(x, y, z) is: s[u, v, 1]T = K[R|t]X, where s is the scale factor. This relationship supports the projection and inverse projection transformation of points between different coordinate systems.

[0090] This embodiment designs an efficient target detection process. The YOLOv5 network is used for real-time target detection, and the network consists of three parts: a CSP feature extraction backbone, an FPN feature pyramid, and a detection head. To adapt to the autonomous driving scenario, the network structure is optimized: an attention module is added to enhance the features of key regions, and a small target detection branch is added to improve the detection performance of distant targets. The detection results include the target box coordinates, class probabilities, and confidence scores.

[0091] This embodiment realizes an innovative depth estimation method. The monocular depth estimation network adopts an encoder-decoder structure. The encoder extracts multi-scale features, and the decoder gradually restores the spatial resolution through skip connections. During the training process, depth consistency loss and smooth constraints are used, and semantic information is introduced to guide depth prediction. The network outputs a pixel-level depth map, and the actual distance of the target can be restored by combining camera parameters.

[0092] This embodiment constructs a complete target tracking system. The similarity calculation comprehensively considers the spatial position distance, size ratio difference, and cosine similarity of appearance features. The construction of the association cost matrix introduces temporal information and considers the motion continuity of the target. The Hungarian algorithm realizes global optimal target association through bipartite graph matching. The Kalman filter adopts a constant velocity motion model, and the state vector contains position, velocity, and acceleration information.

[0093] Through the above technological innovations, this embodiment effectively solves the key problems in multi-camera autonomous driving systems: unstable image quality, insufficient calibration accuracy, unreliable target tracking, etc. In practical applications, this solution can adapt to different lighting conditions and road scenarios, providing stable and reliable environmental perception results. It is particularly suitable for complex traffic environments. Through multi-camera collaborative perception and precise calibration, the accuracy of target detection and tracking is significantly improved. The systematicness and innovativeness of this solution enable it to support advanced autonomous driving functions, and through continuous optimization and verification, the comprehensive improvement of the perception system is achieved.

[0094] In an embodiment of the end-to-end autonomous driving control method based on multi-camera fusion of this application, the following content may also be specifically included:

[0095] Step S401: Divide the multi-modal dataset into a training set and a validation set, perform data augmentation operations such as random cropping, rotation, mirror flipping, and brightness adjustment on the image data in the training set, construct a batch data loader to read training samples, input the training samples into the feature extraction sub-network. The feature extraction sub-network adopts a residual convolution structure to extract multi-scale spatial features, and a batch normalization layer and an activation function layer are added after each convolutional layer;

[0096] Step S402: Perform channel attention calculation and spatial attention calculation on the feature map output by the feature extraction sub-network, weight the feature map based on the attention weights, input the weighted feature map into the global average pooling layer and the fully connected layer to generate a feature vector with a fixed dimension, and use the cross-entropy loss function and the contrastive learning loss function to construct the training objective of the feature extraction sub-network, and use the stochastic gradient descent method to optimize the network parameters.

[0097] Optionally, this embodiment first implements a scientific dataset partitioning strategy. The multimodal dataset is partitioned into a training set and a validation set using the stratified sampling method, with a ratio of 8:2, ensuring that the two datasets contain scene types with similar distributions. The partitioning process takes into account key attributes such as weather conditions, lighting conditions, and road types to ensure that the validation set can comprehensively evaluate the model performance. At the same time, a scene difficulty assessment mechanism is implemented to grade the scenes according to factors such as target density and occlusion degree.

[0098] This embodiment designs a comprehensive data augmentation scheme. The random cropping operation randomly selects cropping positions from the four corners and the central region while maintaining the original image ratio. The rotation augmentation randomly selects an angle within the range of [-10, 10] degrees, and at the same time, boundary padding is performed to maintain the integrity of the image. The mirror flip randomly performs a horizontal flip to increase the model's adaptability to different driving directions. The brightness adjustment is achieved through gamma transformation, and the transformation parameters are randomly selected within the range of [0.8, 1.2].

[0099] This embodiment constructs an efficient data loading mechanism. The batch data loader adopts a multi-threaded design, supporting asynchronous data prefetching and preprocessing. Each batch contains 32 training samples, and the organization of the samples takes into account scene diversity to ensure that each batch contains different types of driving scenes. At the same time, a memory management mechanism is implemented to improve the training efficiency through data preloading and caching optimization.

[0100] This embodiment implements an innovative feature extraction structure. The residual convolutional network consists of five stages, and the number of residual blocks in each stage is 3, 4, 6, and 3 respectively. The residual block adopts a bottleneck design, realizing the reduction and increase of the number of channels through 1×1 convolution, and using 3×3 convolution in the middle to extract features. Each residual block also contains an identity branch, ensuring the effective propagation of gradients through a shortcut connection. This design significantly improves the network's feature extraction ability.

[0101] This embodiment designs a dual attention mechanism. The channel attention module obtains channel statistical features through global average pooling and generates channel weights through two fully connected networks. The spatial attention module uses spatial convolution to generate a two-dimensional attention map to highlight the key regions in the image. The outputs of the two attention mechanisms are dynamically fused through a gating unit to achieve adaptive enhancement of features.

[0102] This embodiment constructs a reliable feature transformation process. The weighted feature map is compressed in the spatial dimension through global average pooling, and then the feature dimension is adjusted to 512 dimensions through a fully connected layer. The dropout mechanism is introduced in the process of generating feature vectors, randomly discarding 50% of the neurons to enhance the generalization ability of the model. At the same time, feature normalization is implemented to convert the feature vector into the form of unit norm.

[0103] This embodiment realizes an innovative design of the loss function. The cross-entropy loss is used for the supervised learning task to ensure the discriminability of features. The contrastive learning loss adopts the InfoNCE form, and enhances the expressive ability of features through the contrastive learning of positive and negative sample pairs. The positive sample pairs are constructed from different perspectives of the same scene, and the negative sample pairs are randomly sampled from different scenes. The temperature parameter is set to 0.07 to adjust the scale of feature similarity.

[0104] This embodiment establishes a complete optimization strategy. Stochastic gradient descent uses an optimizer with momentum, the initial learning rate is set to 0.01, and the momentum coefficient is 0.9. The learning rate scheduling adopts the cosine annealing strategy to gradually reduce the learning rate during training. At the same time, a gradient clipping mechanism is implemented to limit the gradient norm within a reasonable range to avoid unstable training.

[0105] Through the above technological innovations, this embodiment effectively solves the key problems in autonomous driving feature learning: insufficient feature expression, unreasonable attention allocation, unstable training optimization, etc. In practical applications, this solution can extract stable and reliable feature representations from multi-perspective images and adapt to different driving scenarios and environmental conditions. It is particularly suitable for complex traffic environments. Through multi-scale feature extraction and attention enhancement, it significantly improves the accuracy and robustness of environmental perception. The systematicness and innovativeness of this solution enable it to support advanced autonomous driving functions, and through continuous feature learning and optimization, it realizes a comprehensive improvement in perception ability.

[0106] In an embodiment of the end-to-end autonomous driving control method based on multi-camera fusion of the present application, the following content may also be specifically included:

[0107] Step S501: Calculate the clarity evaluation index of each road image, perform Laplacian of Gaussian transformation on the image to extract the edge intensity map, count the number of non-zero pixel points and the intensity mean of the edge intensity map, input the edge intensity feature and the object detection confidence score into the weight prediction network, and the weight prediction network uses a multi-layer perceptron structure to output the initial weight value of each perspective, and correct the initial weight value based on the field of view overlap matrix to obtain the fusion weight coefficient;

[0108] Step S502: Input the multi-channel feature vectors output by the feature extraction sub-network into the feature fusion sub-network. The feature fusion sub-network uses the self-attention mechanism to calculate the correlation scores between the feature vectors, combines the fusion weight coefficients to perform weighted summation on the feature vectors, and inputs the weighted result into the feature transformation module for dimensionality reduction and non-linear transformation to generate the fused feature representation.

[0109] Optionally, this embodiment first implements an innovative image quality assessment mechanism. The sharpness evaluation uses a multi-scale Laplacian of Gaussian operator, which detects edge and detail information by calculating the second derivative of the image. In the specific implementation, the image is first smoothed by Gaussian, and then the Laplacian operator response is calculated. The standard deviation of the Gaussian kernel is set to 1.6, and the convolution kernel size is 5×5. These parameters have been verified to achieve a good balance between noise suppression and edge detail retention.

[0110] This embodiment designs a reliable edge feature extraction method. The edge intensity map is obtained by taking the absolute value of the Laplacian of Gaussian response, and the intensity value of each pixel reflects the edge saliency of the local region. During the statistical process, an adaptive threshold is used to binarize the edge intensity map, and the threshold value is dynamically determined by the OTSU algorithm. The number of non-zero pixel points reflects the overall clarity of the image, and the intensity mean characterizes the contrast level of the edges.

[0111] This embodiment constructs an accurate weight prediction network. The network adopts a four-layer fully connected structure. The input layer receives edge features and object detection confidence. The hidden layer uses the ReLU activation function to introduce non-linear characteristics. The network training uses the mean squared error loss function, and the parameters are optimized by the backpropagation algorithm. To improve the robustness of the prediction, image degradation samples with different degrees, such as blur, noise, and illumination changes, are added to the training data.

[0112] This embodiment implements an innovative weight correction mechanism. The field of view overlap matrix describes the overlap relationship of the fields of view between different cameras. The matrix element values are in the range of [0,1], indicating the proportion of the common observation area of two perspectives in the total field of view. The weight correction process takes into account the information redundancy caused by the field of view overlap. When a certain area is observed by multiple cameras, the weights of each perspective are redistributed according to the field of view overlap.

[0113] This embodiment designs an efficient feature extraction sub-network. The network adopts a parallel residual structure, and each branch corresponds to a camera input. The residual block contains 1×1 convolution for channel dimension reduction, 3×3 convolution for extracting spatial features, and 1×1 convolution for restoring the channel dimension. A spatial attention module is added after the key layer to enhance the feature expression of important regions. The spatial information continuity is maintained during the feature extraction process, which is convenient for subsequent feature fusion.

[0114] This embodiment constructs a reliable feature fusion strategy. The self-attention mechanism realizes the dynamic association between features through the query matrix Q, the key matrix K, and the value matrix V. The correlation score is calculated by the matrix multiplication of Q and K and is normalized by softmax. The score matrix is multiplied by V to generate a feature representation considering context dependence. This mechanism can adaptively capture the mutual relationship between features from different perspectives.

[0115] This embodiment realizes an accurate weighted fusion mechanism. The fusion weight coefficients reflect the reliability and importance of each perspective feature, and the weight values are dynamically adjusted according to the image quality and target detection effect. The weighted summation process takes into account the spatial correspondence of features to ensure that the fusion result maintains spatial consistency. At the same time, a residual connection is introduced to add the original features and the fusion features, retaining the unique information of each perspective.

[0116] This embodiment designs an innovative feature transformation module. This module first adjusts the feature dimension through 1×1 convolution, and then uses a non-linear activation function to enhance the expressive ability of the features. In order to maintain the discriminability of the features, a batch normalization layer is added during the transformation process to reduce the internal covariate shift. Finally, a compact feature representation is generated through global average pooling.

[0117] Through the above technological innovations, this embodiment effectively solves the key problems in multi-perspective feature fusion: inaccurate image quality assessment, unreasonable weight assignment, insufficient feature fusion, etc. In practical applications, this solution can adapt to the changes in image quality in different scenarios and maintain the stability of feature fusion through dynamic weight adjustment. It is particularly suitable for complex traffic environments, and through the collaborative expression of multi-perspective features, it significantly improves the integrity and accuracy of environmental perception. The systematicness and innovativeness of this solution enable it to support advanced autonomous driving functions, and through continuous feature learning and optimization, it realizes a comprehensive improvement in perception ability.

[0118] In an embodiment of the end-to-end autonomous driving control method based on multi-camera fusion of this application, the following content may also be specifically included:

[0119] Step S601: Construct a deep neural network model with an encoder-decoder structure. The encoder uses a multi-layer convolutional network to extract and reduce the dimensions of the fusion features. The decoder uses a transposed convolutional layer to restore the encoded features to a feature map of the same size as the input image. Calculate the projection consistency score between different perspective feature maps based on the geometric transformation relationship between the cameras, and use the projection consistency score as the optimization target of the inter-perspective geometric consistency loss function.

[0120] Step S602: Input the feature map output by the decoder into the reconstruction branch network. The reconstruction branch network performs upsampling and pixel-level prediction on the feature map to generate a reconstructed image. Calculate the mean square error between the reconstructed image and the original input image as the reconstruction loss function, and use the weighted sum of the inter-perspective geometric consistency loss function and the reconstruction loss function as the network optimization target, and use the backpropagation algorithm to update the network parameters.

[0121] Optionally, in this embodiment, an innovative network structure system is first designed. The encoder uses a deep convolutional network, which includes five downsampling stages. In each stage, two 3×3 convolutional layers are used to extract features, and the size of the feature map is reduced through 2×2 max pooling. To enhance the feature extraction ability, residual connections are introduced between convolutional layers, and at the same time, dilated convolutions are used to expand the receptive field range. This design ensures that the network can fully capture multi-scale environmental features.

[0122] This embodiment implements an efficient feature fusion mechanism. The fused features processed by the encoder come from multiple camera views. The fusion process takes into account the geometric relationships and feature importance of different views. An attention module is added after each downsampling stage to dynamically adjust the weight distribution of the feature channels. At the same time, a feature pyramid structure is adopted to fuse features of different scales through an adaptive weighting method, enhancing the network's perception ability of near and far targets.

[0123] This embodiment constructs an accurate decoder structure. The decoder uses transposed convolutional layers for upsampling, and the size of the feature map is doubled in each upsampling stage. To reduce information loss, a skip connection mechanism is introduced to fuse the feature maps of the corresponding layers of the encoder with the upsampled features. The gating mechanism is adopted in the fusion process to dynamically adjust the fusion weights according to the correlation of the features, ensuring the retention of effective features and the suppression of noise.

[0124] This embodiment designs an innovative geometric consistency constraint. Based on the calibrated camera parameters, the projection transformation matrix between different views is calculated. The feature map output by the decoder is subjected to projection transformation, and the features of one view are projected onto the image plane of another view. The projection consistency score is obtained by calculating the similarity of the corresponding features, and the similarity metric adopts a combination of the structural similarity index and the feature cosine distance.

[0125] This embodiment implements a reliable reconstruction branch network. The reconstruction network adopts a multi-resolution structure, which includes multiple parallel upsampling paths, and each path is responsible for feature reconstruction at a specific scale. A combination of bilinear interpolation and residual learning is used in the upsampling process, which not only ensures the smoothness of the reconstruction but also can restore detailed information. Pixel-level prediction adopts a multi-task learning framework, and both the reconstructed image and the semantic segmentation map are output simultaneously.

[0126] This embodiment establishes a complete loss function system. The geometric consistency loss between views includes two parts: the feature level and the pixel level. The feature-level loss calculates the similarity of the projected features, and the pixel-level loss compares the consistency of the reconstructed images. The reconstruction loss function adopts the weighted mean square error form, and the weight coefficient is dynamically adjusted according to the importance of the image region. At the same time, perceptual loss is introduced, and the pre-trained VGG network is used to extract features to calculate the reconstruction quality.

[0127] In this embodiment, a stable optimization strategy is constructed. The network parameters are updated using the Adam optimizer, and the initial learning rate is set to 0.001, which is gradually decreased through the cosine annealing strategy. To prevent overfitting, dropout and weight decay regularization are adopted during training. At the same time, a gradient clipping mechanism is implemented to limit the gradient norm and avoid unstable training. During the validation process, the weight coefficients of the loss function are dynamically adjusted to maintain the training balance of each task.

[0128] In this embodiment, an innovative training scheduling mechanism is designed. A phased training strategy is adopted. First, the encoder parameters are fixed, and the decoder and reconstruction branches are trained. After the reconstruction task converges, all parameters are unfrozen for end-to-end fine-tuning. During the training process, the weights of each loss term are dynamically adjusted according to the performance of the validation set. At the same time, a model ensemble mechanism is implemented to improve the stability of the model through the averaging of multiple checkpoints.

[0129] Through the above technological innovations, this embodiment effectively solves the key problems in the multi-view autonomous driving system: incomplete feature reconstruction, insufficient view consistency, unstable optimization process, etc. In practical applications, this solution can achieve high-quality feature reconstruction and view transformation, providing stable and reliable environmental perception results. It is particularly suitable for complex road scenarios, and through multi-view geometric constraints and feature reconstruction, it significantly improves the accuracy and integrity of environmental understanding. The systematicness and innovativeness of this solution enable it to support advanced autonomous driving functions, and through continuous optimization and verification, a comprehensive improvement of the perception system is achieved.

[0130] In an embodiment of the end-to-end autonomous driving control method based on multi-camera fusion in this application, the following content may also be specifically included:

[0131] Step S701: Construct a gradient descent optimizer, calculate the loss value of the network model after each round of training based on the validation set, record the optimal model parameters using the early stopping strategy, conduct an occlusion simulation experiment on the input image, randomly add an occlusion mask to the image area, calculate the proportion of the occluded area in the image, input the proportion of the occluded area into the weight adjustment network, the weight adjustment network outputs a compensation value for the feature fusion weight, and dynamically update the original fusion weight according to the compensation value;

[0132] Step S702: Input the output features of the deep neural network model into the control prediction branch. The control prediction branch includes multiple fully connected layers, which respectively predict the steering angle, acceleration, and braking force values. Perform numerical normalization and smoothing processing on the prediction results, convert the processed control quantities into low-level execution instructions through the vehicle control interface, and perform safety constraints on the execution instructions according to the road type and driving speed.

[0133] Optionally, this embodiment first implements an accurate optimization training mechanism. The gradient descent optimizer uses the Adam algorithm, which adaptively adjusts the learning rate of each parameter by combining first-order moment estimation and second-order moment estimation. The initial learning rate is set to 0.001, and the decay coefficients are β1 = 0.9 and β2 = 0.999. These parameter settings ensure the stability and convergence efficiency of the training process. The validation set loss value is calculated every 100 steps to evaluate the generalization performance of the model.

[0134] This embodiment designs a reliable early stopping strategy. The early stopping mechanism is triggered when the validation loss does not decrease for 5 consecutive epochs, and the model parameters with the lowest validation loss are saved. To prevent overfitting, a weight decay regularization term is introduced, and the coefficient is set to 0.0001. The model is saved using the checkpoint mechanism, which includes not only the network parameters but also the optimizer state and training epoch information, supporting resuming training from the breakpoint during the training process.

[0135] This embodiment constructs an innovative occlusion simulation scheme. The occlusion mask uses a rectangular area, and its size is randomly generated between 10% and 30% of the image area. The position is randomly selected but avoids the central area of the image. The mask value is set to the image mean to simulate the partial occlusion effect in the real scene. At the same time, multi-scale occlusion testing is implemented to evaluate the robustness of the model by changing the size and shape of the occlusion area.

[0136] This embodiment implements a dynamic weight adjustment mechanism. The weight adjustment network uses a three-layer fully connected structure. The input layer receives the feature of the occlusion area ratio, and the hidden layer uses the tanh activation function to introduce non-linearity. The calculation of the compensation value considers the impact of the occlusion degree on the feature reliability. When the occlusion area increases, the fusion weight of the corresponding view will be appropriately reduced. At the same time, a weight smooth transition mechanism is implemented to avoid the instability of fusion caused by weight mutation.

[0137] This embodiment designs a complete control prediction framework. The control prediction branch includes a four-layer fully connected network, and each layer is followed by batch normalization and the ReLU activation function. The network outputs the steering angle (range [-30, 30] degrees), acceleration (range [-2, 2] m / s2), and braking force (range [0, 1]) respectively. The prediction process considers state information such as the current vehicle speed and road curvature to ensure the continuity and smoothness of the control quantity.

[0138] This embodiment constructs a reliable control quantity processing flow. The numerical normalization uses the min-max method to map the prediction results to the effective range of the actuator. The smoothing process uses exponential moving average with a smoothing coefficient of 0.8, effectively suppressing the high-frequency oscillation of the control quantity. At the same time, control dead zone processing is implemented to keep the current control state when the prediction value changes less than the threshold, improving the stability of the control.

[0139] This embodiment realizes an innovative instruction conversion mechanism. The control quantity conversion takes into account the vehicle dynamics characteristics, and maps the high-level control quantity to the low-level execution instructions through the look-up table method. The steering angle is converted into the steering wheel angle, considering the influence of speed on the steering ratio. The acceleration and braking force are respectively converted into the throttle opening and braking pressure, while considering the engine characteristics and the braking system response.

[0140] This embodiment designs a comprehensive safety constraint system. The constraint conditions are dynamically adjusted according to the road type, with the maximum speed and acceleration restricted on urban roads and the maximum steering angle restricted on curves. At the same time, an early warning mechanism is implemented. When the predicted control quantity approaches the safety boundary, the amplitude limiting process is automatically performed. The constraint processing adopts a soft constraint method, and the control quantity is flexibly restricted through a penalty term to avoid the discomfort caused by constraint mutations.

[0141] Through the above technological innovations, this embodiment effectively solves the key problems in autonomous driving control: unstable model training, poor occlusion adaptability, discontinuous control execution, etc. In practical applications, this solution can adapt to different driving scenarios and abnormal situations, maintaining a stable and reliable control effect. It is particularly suitable for complex traffic environments. Through dynamic weight adjustment and multiple safety constraints, the safety and comfort of autonomous driving are significantly improved. The systematicness and innovativeness of this solution enable it to support the autonomous driving requirements of multiple scenarios, and through continuous optimization and verification, the overall control ability is improved.

[0142] In order to solve the deficiencies of traditional technologies in aspects such as multi-view information fusion and occlusion processing, and significantly improve the perception ability and control reliability of the autonomous driving system, this application provides an embodiment of an end-to-end autonomous driving control device based on multi-camera fusion for implementing all or part of the content of the above-mentioned end-to-end autonomous driving control method based on multi-camera fusion. Refer to Figure 2 The end-to-end autonomous driving control device based on multi-camera fusion specifically includes the following contents:

[0143] The multi-camera fusion module 10 is used to input the front road condition images collected by the vehicle front camera and the side environment images collected by the left and right wide-angle cameras into the preprocessing module, perform spatial transformation and scale normalization on the multi-channel images, establish the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters, calculate the depth and position information of the target objects within the field of view of each camera, use the nearest neighbor search method to identify the associated targets in the overlapping area of the fields of view of different cameras, establish a multi-target tracker to record the target motion trajectories, and construct a multi-modal data set including multi-view target positions, motion states, and image features.

[0144] The fusion network construction module 20 is used to train a multi-view feature fusion network based on the multi-modal data set. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight allocation sub-network, and a feature fusion sub-network. The feature extraction sub-network uses a parallel convolution structure to process multiple channels of image inputs respectively. The dynamic weight allocation sub-network calculates the fusion weight coefficients of each view feature based on image clarity, object detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on multiple channels of features according to the fusion weight coefficients to generate a fused feature representation;

[0145] The autonomous driving control module 30 is used to construct a deep neural network model based on multi-view consistency constraints, input the fused feature into the deep neural network model. The deep neural network model is provided with an inter-view geometric consistency loss function and a reconstruction loss function. The network parameters are optimized by minimizing the loss functions, and the feature fusion weights are dynamically adjusted when partial occlusion occurs in different cameras, and three control quantities, namely steering angle, acceleration, and braking force, are output. The control quantities are converted into vehicle execution instructions to complete the autonomous driving control of the vehicle.

[0146] As can be seen from the above description, the end-to-end autonomous driving control device based on multi-camera fusion provided by the embodiments of the present application can achieve multi-view object detection and tracking through spatial transformation and coordinate mapping by combining the information of the front and left and right wide-angle cameras. A multi-view feature fusion network architecture is designed, which includes three sub-networks: feature extraction, dynamic weight allocation, and feature fusion, and the fusion weights are dynamically adjusted based on image clarity, detection confidence, and field of view overlap. An inter-view geometric consistency constraint and a reconstruction loss function are introduced to construct a deep neural network model, which can effectively handle abnormal situations such as camera occlusion and output accurate control instructions. This method solves the deficiencies of traditional technologies in multi-view information fusion, occlusion processing, etc., and significantly improves the perception ability and control reliability of the autonomous driving system.

[0147] From the hardware level, in order to solve the deficiencies of traditional technologies in multi-view information fusion, occlusion processing, etc., and significantly improve the perception ability and control reliability of the autonomous driving system, the embodiments of the present application provide an electronic device for implementing all or part of the content in the end-to-end autonomous driving control method based on multi-camera fusion. The electronic device specifically includes the following content:

[0148] A processor, a memory, a communications interface, and a bus; wherein, the processor, the memory, and the communications interface complete communication with each other through the bus; the communications interface is used to implement information transmission between the end-to-end autonomous driving control device based on multi-camera fusion and related devices such as a core business system, a user terminal, and a related database, etc.; the logic controller can be a desktop computer, a tablet computer, a mobile terminal, etc., and this embodiment is not limited thereto. In this embodiment, the logic controller can be implemented with reference to the embodiments of the end-to-end autonomous driving control method based on multi-camera fusion and the embodiments of the end-to-end autonomous driving control device based on multi-camera fusion, the content of which is incorporated herein, and the repeated parts will not be elaborated again.

[0149] It can be understood that the user terminal may include a smart phone, a tablet electronic device, a network set-top box, a portable computer, a desktop computer, a personal digital assistant (PDA), a vehicle-mounted device, a smart wearable device, etc. Among them, the smart wearable device may include smart glasses, a smart watch, a smart bracelet, etc.

[0150] In practical applications, part of the end-to-end autonomous driving control method based on multi-camera fusion can be executed on the electronic device side as described above, or all operations can be completed in the client device. Specifically, it can be selected according to the processing capacity of the client device and the limitations of the user usage scenario, etc. This application does not make any limitations in this regard. If all operations are completed in the client device, the client device may further include a processor.

[0151] The above-mentioned client device may have a communication module (i.e., a communication unit), and can be communicatively connected to a remote server to achieve data transmission with the server. The server may include a server on the task scheduling center side, and in other implementation scenarios, it may also include a server on an intermediate platform, such as a server on a third-party server platform that has a communication link with the task scheduling center server. The server may include a single computer device, or may include a server cluster composed of multiple servers, or a server structure of a distributed device.

[0152] Figure 3 This is a schematic block diagram of the system composition of the electronic device 9600 according to an embodiment of the present application. As Figure 3 shown, the electronic device 9600 may include a central processing unit 9100 and a memory 9140; the memory 9140 is coupled to the central processing unit 9100. It should be noted that this Figure 3 is exemplary; other types of structures can also be used to supplement or replace this structure to implement telecommunication functions or other functions.

[0153] In one embodiment, the function of the end-to-end autonomous driving control method based on multi-camera fusion can be integrated into the central processing unit 9100. Among them, the central processing unit 9100 can be configured to perform the following control:

[0154] Step S101: Input the front road condition image collected by the vehicle front camera and the side environment images collected by the left and right wide-angle cameras into the preprocessing module. Perform spatial transformation and scale normalization on the multi-channel images. Establish the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters. Calculate the depth and position information of the target objects within the field of view of each camera. Use the nearest neighbor search method to identify the associated targets within the overlapping area of the fields of view of different cameras. Establish a multi-object tracker to record the target movement trajectories, and construct a multi-modal data set containing multi-view target positions, motion states, and image features.

[0155] Step S102: Train a multi-view feature fusion network based on the multi-modal data set. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight assignment sub-network, and a feature fusion sub-network. The feature extraction sub-network uses a parallel convolution structure to process the multi-channel image inputs respectively. The dynamic weight assignment sub-network calculates the fusion weight coefficients of each view feature based on image clarity, object detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on the multi-channel features according to the fusion weight coefficients to generate a fused feature representation.

[0156] Step S103: Construct a deep neural network model based on multi-view consistency constraints. Input the fused feature into the deep neural network model. The deep neural network model sets the geometric consistency loss function and the reconstruction loss function between views. Optimize the network parameters by minimizing the loss function. Dynamically adjust the feature fusion weights when partial occlusion occurs in different cameras. Output three control quantities: steering angle, acceleration, and braking force. Convert the control quantities into vehicle execution instructions to complete the autonomous driving control of the vehicle.

[0157] As can be seen from the above description, the electronic device provided in the embodiment of the present application combines the information of the front and left and right wide-angle cameras, and realizes multi-view target detection and tracking through spatial transformation and coordinate mapping. Design a multi-view feature fusion network architecture, which includes three sub-networks: feature extraction, dynamic weight assignment, and feature fusion. Dynamically adjust the fusion weights based on image clarity, detection confidence, and field of view overlap. Introduce the geometric consistency constraint and the reconstruction loss function between views to construct a deep neural network model, effectively handle abnormal situations such as camera occlusion, and output accurate control instructions. This method solves the deficiencies of traditional technologies in multi-view information fusion, occlusion processing, etc., and significantly improves the perception ability and control reliability of the autonomous driving system.

[0158] In another embodiment, the end-to-end autonomous driving control device based on multi-camera fusion can be separately configured from the central processing unit 9100. For example, the end-to-end autonomous driving control device based on multi-camera fusion can be configured as a chip connected to the central processing unit 9100, and the functions of the end-to-end autonomous driving control method based on multi-camera fusion can be realized through the control of the central processing unit.

[0159] As Figure 3 shown, the electronic device 9600 may further include: a communication module 9110, an input unit 9120, an audio processor 9130, a display 9160, and a power supply 9170. It should be noted that the electronic device 9600 does not necessarily have to include Figure 3 all the components shown in Figure 3 ; in addition, the electronic device 9600 may further include

[0160] As Figure 3 shown, the central processing unit 9100 is sometimes also referred to as a controller or an operation control, and may include a microprocessor or other processor devices and / or logic devices. The central processing unit 9100 receives inputs and controls the operations of the various components of the electronic device 9600.

[0161] Among them, the memory 9140 can be, for example, one or more of a buffer, a flash memory, a hard drive, a removable medium, a volatile memory, a non-volatile memory, or other suitable devices. The above information related to failures can be stored, and in addition, programs for executing relevant information can also be stored. And the central processing unit 9100 can execute the program stored in the memory 9140 to implement information storage or processing, etc.

[0162] The input unit 9120 provides inputs to the central processing unit 9100. The input unit 9120 is, for example, a key or a touch input device. The power supply 9170 is used to supply power to the electronic device 9600. The display 9160 is used to display display objects such as images and texts. The display can be, for example, an LCD display, but is not limited thereto.

[0163] The memory 9140 can be a solid-state memory, for example, a read-only memory (ROM), a random access memory (RAM), a SIM card, etc. It can also be a memory that stores information even when power is off, can be selectively erased and has more data stored. Examples of such a memory are sometimes referred to as EPROMs, etc. The memory 9140 can also be some other type of device. The memory 9140 includes a buffer memory 9141 (sometimes referred to as a buffer). The memory 9140 can include an application / function storage unit 9142, which is used to store application programs and function programs or the processes for operating the electronic device 9600 by the central processing unit 9100.

[0164] The memory 9140 can also include a data storage unit 9143, which is used to store data, such as contacts, digital data, pictures, sounds, and / or any other data used by the electronic device. The driver storage unit 9144 of the memory 9140 can include various drivers of the electronic device for communication functions and / or for performing other functions of the electronic device (such as a messaging application, an address book application, etc.).

[0165] The communication module 9110 is a transmitter / receiver that transmits and receives signals via the antenna 9111. The communication module 9110 (transmitter / receiver) is coupled to the central processing unit 9100 to provide input signals and receive output signals, which can be the same as in the case of a conventional mobile communication terminal.

[0166] Based on different communication technologies, multiple communication modules 9110 can be provided in the same electronic device, such as a cellular network module, a Bluetooth module, and / or a wireless local area network module, etc. The communication module 9110 (transmitter / receiver) is also coupled to the speaker 9131 and the microphone 9132 via the audio processor 9130 to provide an audio output via the speaker 9131 and receive an audio input from the microphone 9132, so as to implement normal telecommunication functions. The audio processor 9130 can include any suitable buffer, decoder, amplifier, etc. In addition, the audio processor 9130 is also coupled to the central processing unit 9100, so that recording can be performed on the local machine through the microphone 9132, and the sound stored on the local machine can be played through the speaker 9131.

[0167] Embodiments of the present application also provide a computer-readable storage medium capable of implementing all steps of the end-to-end autonomous driving control method based on multi-camera fusion where the execution entity in the above embodiments is a server or a client. A computer program is stored on the computer-readable storage medium. When the computer program is executed by a processor, it implements all steps of the end-to-end autonomous driving control method based on multi-camera fusion where the execution entity in the above embodiments is a server or a client. For example, when the processor executes the computer program, the following steps are implemented:

[0168] Step S101: Input the forward road condition images collected by the vehicle front camera and the side environment images collected by the left and right wide-angle cameras into the preprocessing module. Perform spatial transformation and scale normalization on the multi-channel images. Establish the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters. Calculate the depth and position information of the target objects within the field of view of each camera. Use the nearest neighbor search method to identify the associated targets within the overlapping area of the fields of view of different cameras. Establish a multi-object tracker to record the target movement trajectories, and construct a multi-modal dataset including multi-view target positions, motion states, and image features.

[0169] Step S102: Train a multi-view feature fusion network based on the multi-modal dataset. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight assignment sub-network, and a feature fusion sub-network. The feature extraction sub-network uses a parallel convolution structure to process the multi-channel image inputs respectively. The dynamic weight assignment sub-network calculates the fusion weight coefficients of the features of each view based on image sharpness, target detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on the multi-channel features according to the fusion weight coefficients to generate a fused feature representation.

[0170] Step S103: Construct a deep neural network model based on multi-view consistency constraints. Input the fused features into the deep neural network model. The deep neural network model is set with an inter-view geometric consistency loss function and a reconstruction loss function. Optimize the network parameters by minimizing the loss function. Dynamically adjust the feature fusion weights when partial occlusion occurs in different cameras, and output three control quantities: steering angle, acceleration, and braking force. Convert the control quantities into vehicle execution instructions to complete the autonomous driving control of the vehicle.

[0171] As can be seen from the above description, the computer-readable storage medium provided by the embodiments of the present application realizes multi-view object detection and tracking through spatial transformation and coordinate mapping by combining the information of the front and left and right wide-angle cameras. A multi-view feature fusion network architecture is designed, which includes three sub-networks: feature extraction, dynamic weight allocation, and feature fusion. The fusion weights are dynamically adjusted based on image clarity, detection confidence, and field of view overlap. A geometric consistency constraint between views and a reconstruction loss function are introduced to construct a deep neural network model, which can effectively handle abnormal situations such as camera occlusion and output accurate control instructions. This method solves the deficiencies of traditional technologies in multi-view information fusion, occlusion processing, etc., and significantly improves the perception ability and control reliability of the autonomous driving system.

[0172] An embodiment of the present application also provides a computer program product that can implement all the steps of the end-to-end autonomous driving control method based on multi-camera fusion with the execution subject being a server or a client in the above embodiments. When the computer program / instructions are executed by a processor, the steps of the end-to-end autonomous driving control method based on multi-camera fusion are implemented. For example, the computer program / instructions implement the following steps:

[0173] Step S101: Input the front road condition images collected by the vehicle's front camera and the side environment images collected by the left and right wide-angle cameras into the preprocessing module. Perform spatial transformation and scale normalization on the multi-channel images, establish the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters, calculate the depth and position information of the target objects within the field of view of each camera, use the nearest neighbor search method to identify the associated targets in the overlapping area of the fields of view of different cameras, establish a multi-object tracker to record the target movement trajectories, and construct a multi-modal dataset including multi-view target positions, motion states, and image features;

[0174] Step S102: Train a multi-view feature fusion network based on the multi-modal dataset. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight allocation sub-network, and a feature fusion sub-network. The feature extraction sub-network uses a parallel convolution structure to process the multi-channel image inputs respectively. The dynamic weight allocation sub-network calculates the fusion weight coefficients of each view feature based on image clarity, target detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on the multi-channel features according to the fusion weight coefficients to generate a fused feature representation;

[0175] Step S103: Construct a deep neural network model based on multi-view consistency constraints, input the fused features into the deep neural network model. The deep neural network model is set with an inter-view geometric consistency loss function and a reconstruction loss function. Optimize the network parameters by minimizing the loss functions, dynamically adjust the feature fusion weights when there is partial occlusion in different cameras, output three control quantities of steering angle, acceleration, and braking force, and convert the control quantities into vehicle execution instructions to complete the automatic driving control of the vehicle.

[0176] As can be seen from the above description, the computer program product provided by the embodiments of the present application realizes multi-view object detection and tracking through spatial transformation and coordinate mapping by combining the information of the front and left and right wide-angle cameras. Design a multi-view feature fusion network architecture, including three sub-networks of feature extraction, dynamic weight allocation, and feature fusion, and dynamically adjust the fusion weights based on image clarity, detection confidence, and field of view overlap. Introduce inter-view geometric consistency constraints and a reconstruction loss function to construct a deep neural network model, effectively handle abnormal situations such as camera occlusion, and output accurate control instructions. This method solves the deficiencies of traditional technologies in multi-view information fusion, occlusion processing, etc., and significantly improves the perception ability and control reliability of the automatic driving system.

[0177] Those skilled in the art should understand that the embodiments of the present invention can be provided as a method, a device, or a computer program product. Therefore, the present invention can adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present invention can adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0178] The present invention is described with reference to the flowcharts and / or block diagrams of methods, devices (apparatus), and computer program products according to the embodiments of the present invention. It should be understood that each flow and / or block in the flowchart and / or block diagram, and the combination of flows and / or blocks in the flowchart and / or block diagram, can be realized by computer program instructions. These computer program instructions can be provided to the processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing devices to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing devices generate a device for realizing the functions specified in one Figure 1 one flow or multiple flows and / or blocks Figure 1 one block or multiple blocks.

[0179] These computer program instructions can also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to work in a particular manner, such that the instructions stored in the computer-readable memory produce a manufacture including an instruction apparatus that implements the function specified in one or more of the processes and / or boxes Figure 1 in one or more of the processes and / or boxes Figure 1 specified in the boxes.

[0180] These computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, whereby the instructions executed on the computer or other programmable apparatus provide steps for implementing the function specified in one or more of the processes and / or boxes Figure 1 in one or more of the processes and / or boxes Figure 1 specified in the boxes.

[0181] Specific embodiments are used in the present invention to illustrate the principles and implementation manners of the present invention. The description of the above embodiments is only used to help understand the method and its core idea of the present invention; at the same time, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific implementation manners and application scopes. In summary, the content of this specification should not be construed as a limitation to the present invention.

Claims

1. An end-to-end autonomous driving control method based on multi-camera fusion, characterized in that, The method includes: Inputting the forward road condition images collected by the vehicle's front camera and the side environment images collected by the left and right wide-angle cameras into a preprocessing module, performing spatial transformation and scale normalization on the multi-channel images, establishing the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters, calculating the depth and position information of the target objects within the field of view of each camera, using the nearest neighbor search method to identify the associated targets within the overlapping areas of the fields of view of different cameras, establishing a multi-target tracker to record the target movement trajectories, and constructing a multi-modal dataset containing multi-view target positions, motion states, and image features; Training a multi-view feature fusion network based on the multi-modal dataset. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight assignment sub-network, and a feature fusion sub-network. The feature extraction sub-network uses a parallel convolution structure to process the multi-channel image inputs respectively. The dynamic weight assignment sub-network calculates the fusion weight coefficients of each view feature based on the image sharpness, target detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on the multi-channel features according to the fusion weight coefficients to generate a fused feature representation; Constructing a deep neural network model based on multi-view consistency constraints, inputting the fused feature into the deep neural network model. The deep neural network model sets a geometric consistency loss function and a reconstruction loss function between views, optimizes the network parameters by minimizing the loss function, dynamically adjusts the feature fusion weights when there is partial occlusion of different cameras, outputs three control quantities of the steering angle, acceleration, and braking force, and converts the control quantities into vehicle execution instructions to complete the autonomous driving control of the vehicle.

2. The end-to-end autonomous driving control method based on multi-camera fusion according to claim 1, wherein, The step of inputting the forward road condition images collected by the vehicle's front camera and the side environment images collected by the left and right wide-angle cameras into a preprocessing module, performing spatial transformation and scale normalization on the multi-channel images, and establishing the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters includes: Downsampling the forward road condition images collected by the front camera and the side environment images collected by the left and right wide-angle cameras, performing white balance correction and histogram equalization processing on the multi-channel images, using the bilinear interpolation algorithm to scale the images with different resolutions to a unified size, and performing pixel normalization processing on the scaled images to generate standardized image data; Using the Zhang's calibration method to obtain the internal parameter matrices and distortion coefficients of multiple cameras, calculating the rotation matrix and translation vector between the cameras based on the calibration image sequence collected by the checkerboard calibration board, taking the rotation matrix and translation vector as the external parameter matrix, establishing the coordinate transformation relationship between the image coordinate system and the vehicle body coordinate system in combination with the internal parameter matrix, and performing perspective transformation on the standardized image data to obtain a bird's-eye view.

3. The end-to-end autonomous driving control method based on multi-camera fusion according to claim 1, characterized in that The step of calculating the depth and position information of the target objects within the field of view of each camera, using the nearest neighbor search method to identify the associated targets within the overlapping areas of the fields of view of different cameras, establishing a multi-target tracker to record the target movement trajectories, and constructing a multi-modal dataset containing multi-view target positions, motion states, and image features includes: Input each camera image into the target detection network to extract the candidate bounding boxes and class labels of the target regions. Calculate the depth value of each pixel point in the image based on the monocular depth estimation network, and convert the image coordinates of the candidate bounding boxes of the target regions to the vehicle body coordinate system in combination with the coordinate transformation relationship, so as to generate a target state vector containing the target position, size, class, and depth information. Calculate the similarity of the target state vectors detected by different cameras, construct an association cost matrix based on the target position, size, and appearance features, use the Hungarian algorithm to match and optimize the association cost matrix to obtain the target association result, use the Kalman filter to predict and update the motion state of the associated targets, and save the target motion trajectories, state vectors, and image features to the dataset.

4. The end-to-end automatic driving control method based on multi-camera fusion according to claim 1, wherein, Train the multi-view feature fusion network based on the multi-modal dataset. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight assignment sub-network, and a feature fusion sub-network. The feature extraction sub-network uses a parallel convolution structure to process multiple image inputs respectively, including: Divide the multi-modal dataset into a training set and a validation set, perform data augmentation operations such as random cropping, rotation, mirror flipping, and brightness adjustment on the image data in the training set, construct a batch data loader to read the training samples, input the training samples into the feature extraction sub-network. The feature extraction sub-network uses a residual convolution structure to extract multi-scale spatial features, and adds a batch normalization layer and an activation function layer after each convolution layer. Perform channel attention calculation and spatial attention calculation on the feature maps output by the feature extraction sub-network, weight the feature maps based on the attention weights, input the weighted feature maps into the global average pooling layer and the fully connected layer to generate a feature vector with a fixed dimension, use the cross-entropy loss function and the contrastive learning loss function to construct the training objective of the feature extraction sub-network, and use the stochastic gradient descent method to optimize the network parameters.

5. The end-to-end autonomous driving control method based on multi-camera fusion according to claim 1, wherein The dynamic weight assignment sub-network calculates the fusion weight coefficients of each view feature based on the image sharpness, target detection confidence, and field of view overlap. The feature fusion sub-network weights and combines multiple features according to the fusion weight coefficients to generate a fused feature representation, including: Calculate the sharpness evaluation index of each image, perform a Laplacian of Gaussian transform on the image to extract the edge intensity map, count the number of non-zero pixel points and the intensity mean of the edge intensity map, input the edge intensity feature and the target detection confidence score into the weight prediction network. The weight prediction network uses a multi-layer perceptron structure to output the initial weight value of each view, and correct the initial weight value based on the field of view overlap matrix to obtain the fusion weight coefficient. Input the multiple feature vectors output by the feature extraction sub-network into the feature fusion sub-network. The feature fusion sub-network uses a self-attention mechanism to calculate the correlation scores between the feature vectors, combines the fusion weight coefficients to perform weighted summation on the feature vectors, and inputs the weighted result into the feature transformation module for dimensionality reduction and non-linear transformation to generate a fused feature representation.

6. The end-to-end automatic driving control method based on multi-camera fusion according to claim 1, characterized in that Construct a deep neural network model based on multi-view consistency constraints, input the fused features into the deep neural network model, and the deep neural network model is set with an inter-view geometric consistency loss function and a reconstruction loss function, including: Construct a deep neural network model with an encoder-decoder structure. The encoder uses a multi-layer convolutional network to extract and reduce the dimensions of the fused features. The decoder uses a transposed convolutional layer to restore the encoded features to a feature map of the same size as the input image. Calculate the projection consistency score between feature maps of different views based on the geometric transformation relationship between cameras, and use the projection consistency score as the optimization target of the inter-view geometric consistency loss function; Input the feature map output by the decoder into the reconstruction branch network. The reconstruction branch network performs upsampling and pixel-level prediction on the feature map to generate a reconstructed image. Calculate the mean square error between the reconstructed image and the original input image as the reconstruction loss function, and use the weighted sum of the inter-view geometric consistency loss function and the reconstruction loss function as the network optimization target, and use the backpropagation algorithm to update the network parameters.

7. The end-to-end automatic driving control method based on multi-camera fusion according to claim 1, wherein, Optimize the network parameters by minimizing the loss function, dynamically adjust the feature fusion weights when partial occlusion occurs in different cameras, output three control quantities of steering angle, acceleration, and braking force, convert the control quantities into vehicle execution instructions, and complete the autonomous driving control of the vehicle, including: Construct a gradient descent optimizer, calculate the loss value of the network model after each round of training based on the validation set, use the early stopping strategy to record the optimal model parameters, perform an occlusion simulation experiment on the input image, randomly add an occlusion mask to the image area, calculate the proportion of the occluded area in the image, input the proportion of the occluded area into the weight adjustment network, and the weight adjustment network outputs a compensation value for the feature fusion weight, and dynamically update the original fusion weight according to the compensation value; Input the output features of the deep neural network model into the control prediction branch. The control prediction branch contains multiple fully connected layers, and predicts the values of the steering angle, acceleration, and braking force respectively. Perform numerical normalization and smoothing processing on the prediction results, convert the processed control quantities into low-level execution instructions through the vehicle control interface, and perform safety constraints on the execution instructions according to the road type and driving speed.

8. An end-to-end autonomous driving control device based on multi-camera fusion, characterized in that, The device includes: A multi-camera fusion module, which is used to input the front road condition images collected by the vehicle front camera and the side environment images collected by the left and right wide-angle cameras into the preprocessing module, perform spatial transformation and scale normalization on the multi-channel images, establish the mapping relationship between the image coordinate system and the vehicle body coordinate system according to the camera calibration parameters, calculate the depth and position information of the target objects within the field of view of each camera, use the nearest neighbor search method to identify the associated targets within the overlapping area of the fields of view of different cameras, establish a multi-object tracker to record the target motion trajectories, and construct a multi-modal dataset containing multi-view target positions, motion states, and image features; A fusion network construction module, which is used to train a multi-view feature fusion network based on the multi-modal data set. The multi-view feature fusion network includes a feature extraction sub-network, a dynamic weight allocation sub-network, and a feature fusion sub-network. The feature extraction sub-network uses a parallel convolution structure to process multiple image inputs respectively. The dynamic weight allocation sub-network calculates the fusion weight coefficient of each view feature based on image clarity, object detection confidence, and field of view overlap. The feature fusion sub-network performs weighted combination on multiple features according to the fusion weight coefficient to generate a fused feature representation; An autonomous driving control module, which is used to construct a deep neural network model based on multi-view consistency constraints, input the fused feature into the deep neural network model. The deep neural network model is set with an inter-view geometric consistency loss function and a reconstruction loss function. The network parameters are optimized by minimizing the loss function. When partial occlusion occurs in different cameras, the feature fusion weight is dynamically adjusted, and three control quantities, namely steering angle, acceleration, and braking force, are output. The control quantities are converted into vehicle execution instructions to complete the autonomous driving control of the vehicle.

9. An electronic device, comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the end-to-end autonomous driving control method based on multi-camera fusion according to any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the end-to-end autonomous driving control method based on multi-camera fusion according to any one of claims 1 to 7.

Images