Automatic driving vehicle target detection system and method based on cloud edge collaboration

Abstract

The invention discloses an automatic driving vehicle target detection system and method based on cloud edge collaboration, and relates to the technical field of computer vision and artificial intelligence. According to the target detection model YOLO-Vehile provided by the invention, effective integration of text semantic information and visual features is realized through a CLIP-ViT text feature extractor and a YOLOV8 image feature extractor, and the understanding ability of the model for complex traffic scenes is remarkably improved. According to the target detection model YOLO-Vehile, high detection precision is maintained, meanwhile, light weight of the model is achieved, possibility is provided for deployment of the model on edge equipment with limited resources, and the problem that an existing large model is difficult to deploy rapidly in an actual scene is effectively solved. Through cooperative work of the edge server and the cloud server, flexible scheduling of computing resources and dynamic allocation of tasks are realized. The problem that an existing single calculation mode is unbalanced in calculation resource distribution in the face of complex and changeable traffic scenes is effectively solved, and the overall performance and reliability of the system are remarkably improved.

Description

Technical Field

[0001] The present invention relates to the field of computer vision and artificial intelligence technology, and in particular to a system and method for target detection of an autonomous driving vehicle based on cloud-edge collaboration. Background Art

[0002] As a core component of autonomous driving (AD) and intelligent transportation system (ITS), the object detection module plays a vital role in environmental perception. Efficient and accurate object detection (OD) capability is essential to ensure the safety and reliability of autonomous driving systems.

[0003] Existing object detection (OD) methods can be divided into two categories: two-stage based methods and single-stage based methods. Among them, two-stage based object detection uses the region proposal network (RPN) and the object classification network as core components, and adopts the "proposal-verify" strategy. This method first uses the RPN to generate potential candidate target regions, then performs in-depth feature extraction on these regions, and finally completes target classification and bounding box fine regression. The advantage of this method is its excellent detection accuracy, especially in complex and changing scenes. However, due to the need for two forward propagations, its real-time performance is relatively limited, and it is difficult to fully meet the demanding requirements of autonomous driving systems for real-time processing. In contrast, there are single-stage based object detection methods, representative algorithms include YOLO (You Only LookOnce) and SSD (Single Shot Detector). This type of method cleverly integrates the target localization and classification tasks into a unified network, by dividing the input image into a fixed number of grids, each grid simultaneously predicts multiple bounding boxes and their corresponding category probabilities. Finally, the final detection results are screened out through post-processing techniques such as non-maximum suppression (NMS). This end-to-end solution significantly improves the computational efficiency of the model and realizes the ability of real-time detection, so it is more suitable for application scenarios such as autonomous driving that have extremely high real-time requirements. However, the core optimization goal of the target detection task is to maximize the difference between classes, that is, to improve the distinguishability between targets of different categories. The single-stage target detection method has the problem of low detection accuracy. It can be seen that the existing methods are difficult to ensure high detection accuracy while improving the real-time performance of target detection to meet the stringent requirements of autonomous driving systems. Summary of the invention

[0004] In order to solve the above technical problems, the present invention proposes an autonomous driving vehicle target detection system and method based on cloud-edge collaboration. Through the target detection model YOLO-Vehicle specially optimized for autonomous driving scenarios, it aims to provide efficient and accurate vehicle recognition capabilities, and further improve the environmental perception capabilities of autonomous driving vehicles.

[0005] On one hand, the present invention provides an autonomous driving vehicle target detection system based on cloud-edge collaboration, including an edge server as a vehicle-mounted main controller, a cloud server, and a camera arranged on the vehicle;

[0006] The camera is used to collect road scene images in real time and transmit them to the edge server;

[0007] The edge server is used to perform target detection tasks based on the input customized traffic-related text description and the road scene image collected in real time by the camera when the computing resources are sufficient, and send the customized traffic-related text description and the road scene image collected in real time by the camera to the cloud server when the computing resources do not meet the demand; the traffic-related text description is used to describe the characteristics of the current road scene image;

[0008] The cloud server is used to receive the customized traffic-related text description sent by the edge server and the road scene image collected in real time by the camera to perform target detection tasks;

[0009] Furthermore, the edge server and cloud server use Kubernetes-based microservice orchestration and containerization technology to manage and dynamically adjust the target detection tasks and the load of other preset microservices;

[0010] Furthermore, the edge server and cloud server adopt a time-based data elimination mechanism to regularly clean up expired data;

[0011] Furthermore, a VPN tunnel is established between the edge server and the cloud server through the 5G network for data transmission;

[0012] Furthermore, the edge server and the cloud server both use the target detection model YOLO-Vehicle to perform target detection tasks;

[0013] The target detection model YOLO-Vehicle includes: an input layer, a text feature extractor, an image feature extractor, a multimodal feature fusion module, a detection head and an output layer;

[0014] The input of the input layer is a road scene image and a corresponding traffic-related text description;

[0015] The text feature extractor adopts Clip-ViT, which is used to extract features from the input traffic-related text description to obtain a text feature vector of fixed dimension as the input of the multimodal feature fusion module;

[0016] The image feature extractor adopts YOLOV8, which is used to perform multi-scale feature extraction on the input road scene image, and obtain feature maps of different scales as input of the multimodal feature fusion module;

[0017] The multimodal feature fusion module is used to fuse the text feature vector and the feature maps of different scales to obtain the fused feature representation, and then introduce the position coding mechanism to obtain the position coding and add the fused feature to the position coding PE to obtain the feature enhanced by the position coding, and finally reconstruct the feature enhanced by the position coding into the shape of the original graph feature through the reconstruction operation, and finally obtain multiple reshaped features and send them to the detection head;

[0018] The detection head is used to perform target detection tasks according to the input reshaped features, including target existence prediction, bounding box position prediction and target category prediction, and obtain target existence prediction results, bounding box position prediction results and target category prediction results;

[0019] The output layer outputs the target existence prediction result, the bounding box position prediction result and the target category prediction result;

[0020] On the other hand, the present invention provides a method for target detection of an autonomous driving vehicle based on cloud-edge collaboration, comprising the following steps:

[0021] Step 1: Obtain road scene image dataset and traffic text dataset;

[0022] The road scene image dataset includes a plurality of road scene images and annotation information corresponding to each road scene image, wherein the annotation information includes the category of an object in the road scene image, the location of the object, the size of an anchor frame, and the resolution of the road scene image;

[0023] The traffic text data set includes a plurality of traffic-related text descriptions, each of which corresponds to a road scene image and is used to describe the characteristics of the road scene image;

[0024] Step 2: preprocess the road scene images in the road scene image dataset, and divide the preprocessed road scene image dataset and the corresponding traffic text dataset into a training set and a test set according to a set ratio;

[0025] The preprocessing adopts a data enhancement strategy to perform data enhancement on the road scene image, specifically: first, uniformly adjust the size of the road scene image, adjust all the road scene images to a preset pixel size, fill the fill area with a fixed color value, and then scale the road scene image using a bilinear interpolation algorithm while maintaining the original width-to-height ratio of the road scene image, and then perform random horizontal flipping, random vertical flipping, random rotation transformation, random HSV color space enhancement, random blur processing using Gaussian blur, and random sharpening processing using the UnsharpMask method on the scaled road scene image;

[0026] Step 3: Build the target detection model YOLO-Vehicle;

[0027] The target detection model YOLO-Vehicle includes: an input layer, a text feature extractor, an image feature extractor, a multimodal feature fusion module, a detection head and an output layer;

[0028] The input of the input layer is a road scene image and a corresponding traffic-related text description;

[0029] The text feature extractor uses the pre-trained Clip-ViT. Specifically, the input traffic-related text description is first segmented by the tokenizer, and then several words obtained by the segmentation are embedded into a high-dimensional vector space to obtain several embedded vectors. These embedded vectors are then passed through several Transformer encoder layers to finally obtain a fixed-dimensional text feature vector F. text , as the input of the multimodal feature fusion module;

[0030] The image feature extractor uses YOLOV8 to extract image features of different scales from the input road scene image to obtain three feature maps of different scales F img ={f1,f2,f3}, as the input of the multimodal feature fusion module, where F img is a collection of feature maps of different scales, represents the feature map of the i-th scale, C represents the number of feature channels, and H i represents the height of the feature map of the i-th scale, W i Represents the width of the feature map of the i-th scale;

[0031] The multimodal feature fusion module is used to fuse the feature maps of three different scales output by the image feature extractor with the text feature vector respectively; the process of fusing each feature map with the text feature vector is as follows: first, a cross-modal alignment technique is used to map the text feature vector and the feature map to a unified semantic embedding space through linear projection to obtain two aligned feature information; then, a self-attention mechanism is used to calculate the cross-modal correlation to quantify the relationship between the text feature vector and the feature map; the text feature vector and the feature map are fused according to the cross-modal correlation to obtain a fused feature representation; then, a position encoding mechanism is introduced, and a fixed encoding scheme constructed by sine and cosine functions is used to encode the fused feature representation to obtain a position encoding PE; then, the fused feature is added to the position encoding PE through element-level addition to obtain a feature enhanced by the position encoding; finally, the feature enhanced by the position encoding is reconstructed into the shape of the original image feature through a reconstruction operation to obtain a reshaped feature, and finally, three reshaped features are obtained and input into the detection head for target detection;

[0032] The detection head is used to perform target detection tasks according to the input reshaped features, including target existence prediction, bounding box position prediction and target category prediction. During the detection process, the detection results are post-processed by IOU calculation and non-maximum suppression to eliminate repeated detections and obtain the final detection results, including target existence prediction results, bounding box position prediction results and target category prediction results;

[0033] The output layer outputs the target existence prediction result, the bounding box position prediction result and the target category prediction result;

[0034] Furthermore, the aligned feature information is:

[0035] F′ img =W img *F img +b img ∈R 1×512 (3)

[0036] F′ text =W text *F text +b text ∈R 1×512 (4)

[0037] Among them, F′ img is the aligned feature information obtained by linear projection of the feature map, F′ text is the aligned feature information obtained by linear projection of the text feature vector, W img and W text is the learnable weight matrix, b img and btext is the bias term;

[0038] The fused features are expressed as:

[0039]

[0040] Among them, F fused is the feature representation after fusion; σ is the sigmoid activation function, W is the learnable weight matrix for feature fusion; b is the bias term; α is the hyperparameter that controls the fusion ratio, represents element-wise multiplication, is the cross-modal correlation, where the query matrix comes from the aligned feature information F′ obtained by linear projection of the feature map img , the key matrix and value matrix come from the aligned feature information F′ obtained by linear projection of the text feature vector text , F′ img The query matrix, and F′ text The key matrix and value matrix of ;

[0041] Furthermore, the target existence prediction result is:

[0042] P obj =σ(W obj *F output +b obj ) (10)

[0043] Among them, P obj is the target existence prediction result; W obj and b obj They are the convolution weights and bias parameters used for target existence prediction, respectively; * represents the convolution operation, and σ represents the sigmoid activation function;

[0044] The bounding box position prediction result is:

[0045] B=W box *F output +b box (11)

[0046] Among them, B is the bounding box position prediction result, including four prediction values ​​(t x , t y , t w , t h ), (t x , t y ) is the center coordinate of the bounding box, t w and t h Represents the relative offset of the bounding box width and height, Wbox and b box They are the convolution weights and bias parameters for bounding box prediction, respectively;

[0047] The probability of the target category prediction is:

[0048] K = softmax(W cls *F output )+b cls (12)

[0049] Where K is an N-dimensional vector, N is the number of predefined categories, and each element represents the probability that the input road scene image belongs to the corresponding category; W cls and b cls They are the convolution weights and bias parameters used for target category prediction;

[0050] Step 4: Use the road scene images and corresponding traffic-related text descriptions in the training set to train the object detection model YOLO-Vehicle, keep the pre-trained parameters of the text feature extractor unchanged, and obtain the trained object detection model YOLO-Vehicle;

[0051] Step 5: Select several road scene images and their corresponding traffic-related text descriptions from the training set to fine-tune the text feature extractor in the trained object detection model YOLO-Vehicle to obtain the final object detection model YOLO-Vehicle;

[0052] Fine-tuning uses a contrastive learning strategy to pair traffic-related text descriptions with corresponding road scene images to form positive samples, and randomly selects unmatched traffic-related text descriptions and corresponding road scene images as negative samples. The optimization goal is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs.

[0053] The loss function of the fine-tuning process is:

[0054]

[0055] Among them, s(t, l) is the similarity score between the traffic-related text description t and the road scene image l, l + is a road scene image matched with traffic-related text descriptions, l - Road scene images that do not match traffic-related text descriptions, τ is the temperature parameter;

[0056] Step 6: Deploy the final target detection model YOLO-Vehicle to the cloud server and edge server to perform target detection and obtain target existence prediction results, bounding box position prediction results, and target category prediction results;

[0057] Step 6.1: Customize traffic-related text descriptions based on the current vehicle driving road and use the camera to capture the current road scene image;

[0058] Step 6.2: Determine whether the resources of the edge server meet the current computing requirements. If yes, use the target detection model YOLO-Vehicle deployed on the edge server to perform target detection and obtain target existence prediction results, bounding box position prediction results and target category prediction results. Otherwise, the edge server transmits the customized traffic-related text description and road scene image to the cloud server, and uses the target detection model YOLO-Vehicle deployed on the cloud server to perform target detection and obtain target existence prediction results, bounding box position prediction results and target category prediction results and transmit them back to the edge server.

[0059] Compared with the prior art, the beneficial effects of the present invention are:

[0060] (1) Multimodal feature fusion: The object detection model YOLO-Vehicle proposed in this paper innovatively combines text and image features. Through the CLIP-ViT text feature extractor and the YOLOV8 image feature extractor, it realizes the effective integration of text semantic information and visual features. This multimodal fusion strategy significantly improves the model's ability to understand complex traffic scenes and effectively solves the limitations of existing single-modal object detection methods when dealing with complex environments.

[0061] (2) Lightweight design and high performance: The target detection model YOLO-Vehicle achieves lightweight design while maintaining high detection accuracy. This efficient and lightweight design not only reduces the storage and computing resource requirements, but also makes it possible to deploy the model on resource-constrained edge devices, effectively solving the problem that existing large models are difficult to quickly deploy in actual scenarios.

[0062] (3) Cloud-edge collaborative architecture: The present invention proposes an autonomous driving vehicle target detection system based on cloud-edge collaboration, which realizes flexible scheduling of computing resources and dynamic allocation of tasks through the collaborative work of edge servers and cloud servers. It effectively solves the problem of unbalanced computing resource allocation in the existing single computing mode when facing complex and changeable traffic scenarios, and significantly improves the overall performance and reliability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] Figure 1 It is an overall flow chart of the target detection model YOLO-Vehicle in an embodiment of the present invention;

[0064] Figure 2 This is the training loss curve in the embodiment of the present invention;

[0065] Among them, (a) is the training loss curve under the COCO2017 image dataset; (b) is the training loss curve under the KITTI image dataset;

[0066] Figure 3 These are some test examples in the COCO2017 image dataset in the embodiments of the present invention;

[0067] Figure 4 These are some test examples in the KITTI image dataset in an embodiment of the present invention;

[0068] Figure 5 This is the performance monitoring panel of the edge cloud collaborative intelligent target detection system in an embodiment of the present invention. DETAILED DESCRIPTION

[0069] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution in the embodiment of the present application will be clearly and completely described below in conjunction with the drawings in the embodiment of the present application. Obviously, the described embodiment is a part of the embodiment of the present application, not all of the embodiments. It should be understood that the specific embodiments described here are only used to explain the present application and are not used to limit the present application.

[0070] On the one hand, the present invention provides an autonomous driving vehicle target detection system based on cloud-edge collaboration, which aims to solve the real-time, accuracy and scalability challenges in autonomous driving scenarios. The system combines the advantages of edge computing and cloud computing to form an efficient and flexible intelligent transportation solution. The autonomous driving vehicle target detection system based on cloud-edge collaboration adopts an innovative two-layer computing architecture, including an edge server as an on-board main controller, a cloud server and a camera set on the vehicle. The edge server is responsible for real-time data processing and decision-making, while the cloud server is responsible for more complex computing tasks. The key advantage of this layered architecture is that it can balance computing needs and energy efficiency.

[0071] The camera is used to collect road scene images in real time and transmit them to the edge server;

[0072] The edge server is used to perform target detection tasks based on the input customized traffic-related text description and the road scene image collected in real time by the camera when the computing resources are sufficient, and send the customized traffic-related text description and the road scene image collected in real time by the camera to the cloud server when the computing resources do not meet the demand; the traffic-related text description is used to describe the characteristics of the current road scene image;

[0073] The cloud server is used to receive the customized traffic-related text description sent by the edge server and the road scene image collected in real time by the camera to perform target detection tasks;

[0074] The edge server and cloud server use Kubernetes (K8s)-based microservice orchestration and containerization technology to efficiently manage and dynamically adjust the target detection tasks and the load of other preset microservices;

[0075] Through Kubernetes' microservice orchestration technology, multiple microservices can be efficiently managed and orchestrated. Among them, target detection is one of the key microservices, responsible for real-time detection of road scene images. Kubernetes provides automated deployment, expansion and management functions, so that each microservice can be dynamically adjusted according to the actual load, thereby ensuring the high availability and performance of the system. Containerization technology packages each microservice into an independent container so that it can run consistently in any environment. Through containerization, the target detection microservice can be decoupled from other microservices (such as data processing, user interface, etc.) for independent development and maintenance. In addition, containerization simplifies the deployment process, making resource scheduling between edge servers and cloud servers more flexible and efficient. This architecture not only improves the scalability of the system, but also enhances fault tolerance, ensuring stable operation under different computing resource conditions, thereby achieving efficient target detection and data processing.

[0076] The edge server and cloud server adopt a time-based data elimination mechanism to regularly clean up expired data;

[0077] A VPN tunnel is established between the edge server and the cloud server via a 5G network for data transmission;

[0078] The edge server and the cloud server both use the target detection model YOLO-Vehicle to perform target detection tasks;

[0079] The target detection model YOLO-Vehicle includes: an input layer, a text feature extractor, an image feature extractor, a multimodal feature fusion module, a detection head and an output layer;

[0080] The input of the input layer is a road scene image and a corresponding traffic-related text description;

[0081] The text feature extractor adopts Clip-ViT, which is used to extract features from the input traffic-related text description to obtain a text feature vector of fixed dimension as the input of the multimodal feature fusion module;

[0082] The image feature extractor adopts YOLOV8, which is used to perform multi-scale feature extraction on the input road scene image, and obtain feature maps of different scales as input of the multimodal feature fusion module;

[0083] The multimodal feature fusion module is used to fuse the text feature vector and the feature maps of different scales to obtain the fused feature representation, and then introduce the position coding mechanism to obtain the position coding and add the fused feature to the position coding PE to obtain the feature enhanced by the position coding, and finally reconstruct the feature enhanced by the position coding into the shape of the original graph feature through the reconstruction operation, and finally obtain multiple reshaped features and send them to the detection head;

[0084] The detection head is used to perform target detection tasks according to the input reshaped features, including target existence prediction, bounding box position prediction and target category prediction, and obtain target existence prediction results, bounding box position prediction results and target category prediction results;

[0085] The output layer outputs the target existence prediction result, the bounding box position prediction result and the target category prediction result;

[0086] On the other hand, the present invention provides a method for target detection of an autonomous driving vehicle based on cloud-edge collaboration, comprising the following steps:

[0087] Step 1: Obtain road scene image dataset and traffic text dataset;

[0088] The road scene image dataset includes a plurality of road scene images and annotation information corresponding to each road scene image, wherein the annotation information includes the category of an object in the road scene image, the location of the object, the size of an anchor frame, and the resolution of the road scene image;

[0089] The traffic text data set includes a plurality of traffic-related text descriptions, each of which corresponds to a road scene image and is used to describe the characteristics of the road scene image;

[0090] In this implementation, the road scene image dataset selects the COCO2017 image dataset and the KITTI image dataset;

[0091] The COCO2017 image dataset is a large dataset widely used in multiple computer vision tasks such as object detection, image segmentation, and key point detection. It contains 118,287 images covering 80 different types of objects, such as humans, animals, vehicles, and furniture. Each image has detailed annotation information, including the bounding box of the object, category label, and instance segmentation mask, providing complete support for training and evaluating models. Since these images come from real scenes, the COCO2017 dataset is highly diverse and complex, covering a variety of scenes and objects in the real world, which puts high demands and practical challenges on the generalization ability and robustness of the model.

[0092] The KITTI image dataset is a public dataset for autonomous driving and computer vision research, containing real-life scene images of various city streets and highways. The dataset includes images under different weather conditions, such as sunny, cloudy, and rainy days, as well as images at different time periods, such as day and night. Each image is accompanied by rich annotation information, including cars, trucks, vans, pedestrians, cyclists, seated people, trams, and miscellaneous items, a total of 8 categories, as well as their location and anchor box size information. The image resolution is 1280x384 and is saved in png format. The dataset provides a large number of image samples in real scenes, which can effectively cover various complex situations and help the model's generalization ability in real environments.

[0093] The traffic-related text description in this implementation provides a semantic description of the scene, such as at an intersection, there is a red stop sign, surrounded by two parked cars and a pedestrian crossing the road, which provides important contextual information for the model. It also covers multiple aspects such as vehicle type, color, brand, model, etc., such as "red sedan", "large freight truck", "blue electric bicycle", etc.

[0094] Step 2: preprocess the road scene images in the road scene image dataset, and divide the preprocessed road scene image dataset and the corresponding traffic text dataset into a training set and a test set according to a set ratio;

[0095] The preprocessing adopts a data enhancement strategy to enhance the road scene image data, specifically: first, the size of the road scene image is uniformly adjusted, all road scene images are adjusted to a preset pixel size, and the filling area is filled with a fixed color value, and then the road scene image is scaled using a bilinear interpolation algorithm, while maintaining the original road scene image's aspect ratio. This processing method not only ensures the consistency of the input size, but also retains the original image information to the greatest extent. Then, the scaled road scene image is randomly flipped horizontally to simulate road scenes from different perspectives; randomly flipped vertically to increase the model's recognition ability for inverted objects; randomly rotated to simulate camera angle changes; randomly enhanced HSV color space to adapt to different lighting and weather conditions; Gaussian blur is used for random blur processing to simulate motion blur or unclear focus; and the UnsharpMask method is used for random sharpening processing to enhance image detail features. The comprehensive application of these basic data enhancement strategies significantly improves the model's adaptability and generalization performance to various complex road scenes.

[0096] In this implementation, the Mixup data enhancement technology is used for the COCO2017 image dataset, with the probability set to 0.15, and the mixing coefficient β is sampled from the Beta (8,8) distribution. New training samples are created by linearly combining two images at the pixel level according to the sampled mixing ratio. This method not only improves the generalization ability of the model, but also helps to improve the performance of the model in dealing with complex target detection tasks, especially in dealing with multi-scale and multi-target scenes. At the same time, in order to maintain the consistency of the labels, the corresponding bounding box coordinates and category labels are also adjusted according to the same mixing ratio.

[0097] In this implementation, a data augmentation strategy is used for the KITTI image dataset, with the pixel size set to 1024×1024, the color value fixed to 114 (this value is a commonly used fill value in the YOLO series of algorithms. In the pixel value range of 0-255, 114 is approximately equal to 0.447 times the neutral gray value of 128. This value is selected to reduce the interference of the fill area on the model training, because this value is close to the average pixel value of natural images, which can minimize the impact of the fill area on the model feature extraction), and the probability of random horizontal flipping is set to 0. 5. The probability of random vertical flip is set to 0.1, the angle range of random rotation is limited to [-10°, 10°], the probability is 0.3, the range of hue in random HSV color space enhancement is [-0.015, 0.015], the range of saturation is [0.7, 1.3], the range of brightness is [0.4, 1.6], the range of Gaussian blur kernel size is [3, 7], the probability is 0.2, and the range of intensity of UnsharpMask method is [0.5, 1.5], the probability is 0.2;

[0098] Step 3: Build the target detection model YOLO-Vehicle;

[0099] like Figure 1 As shown in the figure, the target detection model YOLO-Vehicle is a target detection model based on the fusion of text and image information. The overall architecture of the model includes: input layer, text feature extractor, image feature extractor, multimodal feature fusion module, detection head and output layer;

[0100] The input of the input layer is a road scene image and a corresponding traffic-related text description;

[0101] In the feature extraction stage, the framework uses specialized processing modules for different modal data. Text data is processed by the text feature extractor to extract regional text features, and image data is processed by the image feature extractor to extract multi-scale feature maps. Specifically:

[0102] The text feature extractor adopts the pre-trained Clip-ViT, which is a pre-trained model based on the Vision Transformer (ViT) architecture. It uses the self-attention mechanism and the multi-head attention layer to process the input traffic-related text description, and realizes efficient text-image contrast learning. Specifically, the input traffic-related text description is firstly segmented by the tokenizer tokenization, and then several words obtained by the segmentation are respectively embedded into a high-dimensional vector space to obtain several embedding vectors, and then these embedding vectors are passed through several Transformer encoder layers to finally obtain a fixed-dimensional text feature vector as the input of the multimodal feature fusion module; each layer of the Transformer encoder layer includes a self-attention sublayer and a feedforward neural network sublayer;

[0103] In order to optimize the computational efficiency of the model and reduce the parameter complexity, a large number of experiments were conducted in this implementation. Finally, an optimal configuration that balances the expression ability and computational overhead was determined. The text feature vector output by the text feature extractor was set to (1, 512). This text feature vector captures the semantic information of the input traffic-related text description and can be expressed as:

[0104] F text = CLIP-ViT(E t (x))∈R 1×512 (1)

[0105] Among them, F text represents the text feature vector, E t represents the word embedding function of the text, x is the input traffic-related text description, and CLIP-ViT represents text feature extraction;

[0106] The core self-attention mechanism of CLIP-ViT is:

[0107]

[0108] Among them, Attention(Q, K, V) represents the self-attention mechanism, Q, K, V represent the query matrix, key matrix and value matrix respectively, d k is the dimension of the key vector;

[0109] The image feature extractor adopts YOLOV8, which uses its advanced CSPDarknet53 backbone network and feature pyramid structure to extract multi-scale visual features. YOLOV8 integrates spatial and channel attention mechanisms to enhance feature expression capabilities. It extracts image features of different scales from the input road scene image and obtains three feature maps of different scales F img={f1, f2, f3}, as the input of the multimodal feature fusion module, where F img is a collection of feature maps of different scales, represents the feature map of the i-th scale, C represents the number of feature channels, indicating how many features are extracted at each feature position, and H i represents the height of the feature map of the i-th scale, W i Represents the width of the feature map of the i-th scale, which is used to detect vehicle targets of different sizes.

[0110] Feature fusion is the core of this framework. Through the multimodal feature fusion module, the semantic information from the text and the visual features from the road scene image are integrated into a more comprehensive fusion feature. This fusion strategy enables the model to simultaneously utilize the contextual information of the text and the enhanced visual features, thereby better understanding the scene and improving the ability to detect objects in complex environments and severe weather conditions.

[0111] The multimodal feature fusion module is used to achieve effective integration of heterogeneous modal features by using an efficient multimodal feature fusion strategy. Specifically, the three feature maps of different scales output by the image feature extractor are fused with the text feature vector respectively; the fusion process of each feature map is as follows: first, the cross-modal alignment technology is used to map the text feature vector and the feature map to a unified semantic embedding space through linear projection to obtain two aligned feature information; then, the self-attention mechanism of formula (2) is used to calculate the cross-modal correlation to quantify the relationship between the text feature vector and the feature map. This correlation index provides the necessary context information for the fusion process, thereby ensuring the effective combination of different modal features; the text feature vector and the feature map are fused according to the cross-modal correlation to obtain the fused feature representation; then, an efficient position encoding mechanism is introduced, and a fixed encoding scheme constructed by sine cosine function is used to encode the fused feature representation to obtain the position encoding PE; then, the fused feature is added to the position encoding PE by element-wise addition to obtain the feature enhanced by the position encoding; finally, the feature enhanced by the position encoding is reconstructed into the shape of the original image feature by reconstruction operation to obtain the reshaped feature. After the above fusion process, three reshaped features are obtained. These three features correspond to different spatial scales, and they are input into the detection head for object detection.

[0112] The feature information after alignment is:

[0113] F′ img =W img *F img +b img ∈R 1×512 (3)

[0114] F′ text =W text *F text +b text ∈R 1×512 (4)

[0115] Among them, F′ img is the aligned feature information obtained by linear projection of the feature map, F′ text is the aligned feature information obtained by linear projection of the text feature vector, W img and W text is the learnable weight matrix, b img and b text is the bias term;

[0116] The fused features are expressed as:

[0117]

[0118] Among them, F fused is the fused feature representation, which combines the information of image and text modalities to improve the performance of the model; σ is the sigmoid activation function, W is the learnable weight matrix for feature fusion, which adjusts the importance of fused features; b is the bias term, which helps the model fit the data better; α is a hyperparameter that controls the fusion ratio, which is used to balance the contribution of different modalities. represents element-wise multiplication, is the cross-modal correlation, where the query matrix comes from the aligned feature information F′im obtained by linear projection of the feature map g , the key matrix and value matrix come from the aligned feature information F′ obtained by linear projection of the text feature vector text , F′ img The query matrix, and F′ text The key matrix and value matrix of ;

[0119] For the fused feature representation F fused ∈R 1×512 , position encoding PE∈R 1×512 Each element of is defined by the following formula:

[0120] PE (pos，2j) =sin(pos / 10000 2j / 512 ) (6)

[0121] PE (pos，2j+1) =cos(pos / 10000 2j / 512 ) (7)

[0122] Among them, PE (pos，2j) and PE (pos，2j+1) The elements of even and odd dimensions, which are position encodings, are generated by sine and cosine functions respectively. pos represents the position index on the feature map, and j is the dimension index. The advantage of this encoding method is that it can generate a unique representation for each position while maintaining the linear relationship of the relative positions, so as to further enhance the model's understanding and processing capabilities of spatial information.

[0123] The features after position encoding enhancement are:

[0124] F final =F fused +PE (8)

[0125] Among them, F final The feature is enhanced by position encoding. The introduction of this position encoding enables the model to capture spatial information more accurately on the basis of multimodal feature fusion, thereby improving the accuracy and effect of target detection. The model effectively injects spatial information into the fused features, which not only enables the subsequent attention mechanism to utilize position correlation, but also maintains the translation equivariance of the features.

[0126] The reshaped features are:

[0127] F output =Reshape(F final , (C, H, W)) (9)

[0128] Among them, F output is the reshaped feature, and Reshape is the reconstruction operation; C represents the number of feature channels, which reflects the number of different types of features extracted at each feature position. The increase in the number of channels means that the model can extract and express richer image feature information; H represents the height of the feature map, which represents the spatial resolution of the feature map in the vertical direction. A larger height value helps to retain more detailed target position and shape information; W represents the width of the feature map, which represents the spatial resolution of the feature map in the horizontal direction. The appropriate width setting ensures that the feature map can fully express the horizontal characteristics of the target; this feature reshaping operation maintains the integrity of the feature and adapts it to the input requirements of the subsequent processing module;

[0129] The detection head is used to perform target detection tasks according to the input reshaped features, including target existence prediction, bounding box position prediction and target category prediction. During the detection process, the detection results are post-processed by IOU (intersection over union) calculation and non-maximum suppression (NMS) to eliminate repeated detection and obtain the final detection results, including target existence prediction results, bounding box position prediction results and target category prediction results. This processing method ensures the accuracy and uniqueness of the detection results and effectively avoids the problem of repeated detection that may be caused by multi-scale features;

[0130] The target existence prediction result is:

[0131] P obj =σ(W obj *F output +b obj ) (10)

[0132] Among them, P obj is the target existence prediction result, when P obj When it is close to 1, it means that there is a high probability of a target at that location, and when it is close to 0, it means that the location may be the background; W obj and b obj are the convolution weights and bias parameters used for target existence prediction, respectively; * represents the convolution operation, and σ represents the sigmoid activation function, which is used to compress the output into the interval [0, 1] so that it represents the probability of target existence;

[0133] The bounding box position prediction result is:

[0134] B=W box *F output +b box (11)

[0135] Among them, B is the bounding box position prediction result, including four prediction values ​​(t x , t y , t w , t h ), (t x , t y ) is the center coordinate of the bounding box, t w and t h Represents the relative offset of the bounding box width and height, W box and b box are the convolution weights and bias parameters for bounding box prediction, respectively. This parameterization method enables the model to flexibly adjust the position and size of the bounding box to better adapt to targets of different sizes and shapes.

[0136] The probability of the target category prediction is:

[0137] K = softmax(W cls *F output )+b cls (12)

[0138] Where K is an N-dimensional vector, N is the number of predefined categories, and each element represents the probability that the input road scene image belongs to the corresponding category; the softmax function is used to convert the output into a probability distribution to ensure that the sum of all category probabilities is 1; W cls and b cls They are the convolution weights and bias parameters used for target category prediction;

[0139] The output layer outputs the target existence prediction result, the bounding box position prediction result and the target category prediction result;

[0140] Step 4: Use the road scene images and corresponding traffic-related text descriptions in the training set to train the object detection model YOLO-Vehicle, keep the pre-trained parameters of the text feature extractor unchanged, and obtain the trained object detection model YOLO-Vehicle;

[0141] In this embodiment, during the training process of the COCO2017 image dataset, the present invention carefully tunes the parameters of the YOLO-Vehicle model, and the batch size is set to 32 to achieve a balance between training efficiency and memory usage. The setting of the loss function weight reflects the importance of different tasks, where the bounding box regression loss (loss_bbox) weight is 7.5, the classification loss (loss_cls) weight is 0.5, and the distribution focus loss (loss_dfl) weight is 0.375. This weight configuration emphasizes the importance of bounding box positioning while taking into account classification accuracy. The learning rate strategy uses an initial value of 0.001, with a learning rate factor of 0.01, to ensure that the model converges stably and avoids falling into local optimality. Figure 2 The LOSS loss function values ​​of the YOLOV6, YOLOV8n, YOLOV8s, YOLO-World, and YOLO-Vehicle models trained on the COCO dataset are shown;

[0142] On the KITTI image dataset, a similar training strategy as the COCO2017 image dataset was used, but fine-tuned according to the characteristics of the dataset. The model was trained on 7481 training images and the validation set contained 7518 images. Figure 2 Shows the LOSS loss function values ​​of the YOLOV6, YOLOV8n, YOLOV8s, YOLO-World, and YOLO-Vehicle models trained on the KITTI dataset;

[0143] Step 5: Select several road scene images and their corresponding traffic-related text descriptions from the training set to fine-tune the text feature extractor in the trained object detection model YOLO-Vehicle to adapt to the specific vehicle detection task, and obtain the final object detection model YOLO-Vehicle;

[0144] In this embodiment, the criteria for selecting road scene images are: road scene images with obvious object features, a large number of objects, and obstructions;

[0145] In order to further improve the performance of the CLIP-ViT model in the field of traffic, the present invention has carried out targeted fine-tuning on it. The fine-tuning adopts a contrastive learning strategy, which pairs traffic-related text descriptions with corresponding road scene images to form positive samples, and randomly selects unmatched traffic-related text descriptions and corresponding road scene images as negative samples. The optimization goal is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs.

[0146] The loss function of the fine-tuning process is:

[0147]

[0148] Among them, s(t,l) is the similarity score between the traffic-related text description t and the road scene image l, l + is a road scene image that matches the traffic-related text description, l_ is a road scene image that does not match the traffic-related text description, and τ is a temperature parameter; the role of the temperature parameter τ is to control the smoothness of the softmax distribution, adjust the contrast intensity between positive and negative sample pairs, and affect the discriminability of the model learning features. The value of τ is usually greater than 0, and is generally set between 0.1 and 1.0. When τ is small, the difference between similarity scores will be amplified, making the model more inclined to pay attention to the most similar sample pairs, which is helpful for learning more discriminative features; when τ is large, the difference between similarity scores will be reduced, making the model pay attention to multiple sample pairs, which is helpful for learning smoother feature representations.

[0149] Empirical analysis shows that the fine-tuned object detection model YOLO-Vehicle has improved its accuracy by 15.3% in the task of matching traffic-related text descriptions to road scene images, and its F1 score in the task of predicting object categories by 12.7%. These improvements have significantly improved the overall performance of the object detection model YOLO-Vehicle.

[0150] The results in Table 1 show that the YOLO-Vehicle model performs well in multiple key indicators. In terms of detection accuracy, the model's mAP reaches 45.8%, and mAP50 and mAP75 are 62.5% and 49.7% respectively, which are better than other models in the comparison table. This shows that YOLO-Vehicle can maintain high detection accuracy at different IoU thresholds. In terms of processing speed, the model reaches 346FPS, second only to YOLOV8n, but significantly better than other models such as YOLO-World. The single-frame inference time is 10ms, which is comparable to YOLOV8s and much better than YOLO-World's 28ms. It is worth noting that YOLO-Vehicle achieves lightweight model while maintaining high performance. Its model size is 43.2MB, which is significantly smaller than YOLOV6-s6's 87.3MB and YOLO-World's 523MB. This compact design not only reduces storage requirements, but also makes it possible to deploy the model on resource-constrained edge devices.

[0151] Table 1 Comparison of the method of the present invention and previous object detection networks on the COCO2017 dataset

[0152]

[0153]

[0154] The results in Table 2 show that the YOLO-Vehicle model performs well in multiple key indicators. In terms of detection accuracy, the model's mAP reaches 92.1%, and mAP50 and mAP75 are 93.5% and 81.1% respectively, which are better than other models in the comparison table. This shows that YOLO-Vehicle can maintain high detection accuracy at different IoU thresholds. In terms of processing speed, the model reaches 226FPS, second only to YOLOV8n, but significantly better than YOLO-World and YOLOV6 models. It is worth noting that YOLO-Vehicle achieves lightweight model while maintaining high performance. Its model size is 43.2MB, which is 20 times the size of YOLO-World's 700MB model. In order to verify the adaptability of the model on different computing platforms, this paper also conducts deployment tests on embedded devices (such as JetsonNano). The results show that even in an environment with limited computing resources, YOLO-Vehicle can still maintain a processing speed of nearly 30FPS, while the detection accuracy is only slightly reduced.

[0155] Table 2 Comparison of the method of the present invention and previous object detection networks on the KITTI dataset

[0156]

[0157] Figure 3 The display shows the detection results of the YOLO-Vehicle model under the COCO image dataset. Figure 4 The display shows the detection results of the YOLO-Vehicle model on the KITTI image dataset.

[0158] Step 6: Deploy the final target detection model YOLO-Vehicle to the cloud server and edge server to perform target detection and obtain target existence prediction results, bounding box position prediction results, and target category prediction results;

[0159] Step 6.1: Customize traffic-related text descriptions based on the current vehicle driving road and use the camera to capture the current road scene image;

[0160] Step 6.2: Determine whether the resources of the edge server meet the current computing requirements. If yes, use the target detection model YOLO-Vehicle deployed on the edge server to perform target detection, and obtain the target existence prediction result, bounding box position prediction result, and target category prediction result. Otherwise, the edge server transmits the customized traffic-related text description and road scene image to the cloud server, and uses the target detection model YOLO-Vehicle deployed on the cloud server to perform target detection, obtain the target existence prediction result, bounding box position prediction result, and target category prediction result, and transmit them back to the edge server;

[0161] like Figure 5 The present invention proposes an autonomous driving vehicle target detection system based on cloud-edge collaboration. The core of the system is composed of a mobile platform that integrates multiple sensors and computing units to ensure the vehicle's comprehensive perception of the surrounding environment.

[0162] from Figure 5It can be observed that the availability of an autonomous vehicle target detection system based on cloud-edge collaboration has reached 99.777%. This high availability indicator fully demonstrates the stability and reliability of the system architecture. At the same time, the system implements efficient distributed computing and resource management, and adopts microservice orchestration and containerization technology based on Kubernetes (K8s). This design enables the system to flexibly respond to different computing needs. The total availability shown in the figure is 99.672%, which further verifies the stable performance of the system in complex environments. The data processing flow of the system includes image preprocessing and feature extraction by edge nodes, and the processed data is serialized and stored in local persistent volumes. The memory usage and CPU usage curves in the figure show the dynamic changes of system resources and reflect the system's resource allocation capabilities when handling different loads. Especially during peak hours, the system can still maintain stable operation, thanks to its adaptive task allocation mechanism. Security is another important aspect of system design. A secure VPN tunnel is established between the cloud and the edge through the 5G network to ensure the security of data transmission. The system also implements a time-based data elimination mechanism to regularly clean up expired data and optimize the use of storage resources. These measures jointly ensure the data security and storage efficiency of the system. In order to cope with different application scenarios and network conditions, the system has designed a flexible solution path. This design enables the system to dynamically adjust its working mode according to actual needs, which can not only meet the complex scene requirements of high-precision processing, but also adapt to simple scenes with high real-time requirements. The response time (Duration) and request frequency (Requests) data in the figure reflect the performance of the system under different loads, proving its good adaptability.

Claims

1. An autonomous driving vehicle target detection system based on cloud-edge collaboration, characterized in that: It includes an edge server as a vehicle-mounted main controller, a cloud server, and a camera installed on the vehicle; The camera is used to collect road scene images in real time and transmit them to the edge server; The edge server is used to perform target detection tasks based on the input customized traffic-related text description and the road scene image collected in real time by the camera when the computing resources are sufficient, and send the customized traffic-related text description and the road scene image collected in real time by the camera to the cloud server when the computing resources do not meet the demand; the traffic-related text description is used to describe the characteristics of the current road scene image; The cloud server is used to receive the customized traffic-related text description sent by the edge server and the road scene image collected in real time by the camera to perform target detection tasks.

2. The cloud-edge collaboration-based autonomous driving vehicle target detection system according to claim 1, characterized in that: The edge server and cloud server use Kubernetes-based microservice orchestration and containerization technology to manage and dynamically adjust target detection tasks and the load of other preset microservices; The edge server and cloud server adopt a time-based data elimination mechanism to regularly clean up expired data; A VPN tunnel is established between the edge server and the cloud server via the 5G network for data transmission.

3. The cloud-edge collaboration-based autonomous driving vehicle target detection system according to claim 1, characterized in that: The edge server and the cloud server both use the target detection model YOLO-Vehicle to perform target detection tasks; The target detection model YOLO-Vehicle includes: an input layer, a text feature extractor, an image feature extractor, a multimodal feature fusion module, a detection head and an output layer; The input of the input layer is a road scene image and a corresponding traffic-related text description; The text feature extractor adopts Clip-ViT, which is used to extract features from the input traffic-related text description to obtain a text feature vector of fixed dimension as the input of the multimodal feature fusion module; The image feature extractor adopts YOLOV8, which is used to perform multi-scale feature extraction on the input road scene image, and obtain feature maps of different scales as input of the multimodal feature fusion module; The multimodal feature fusion module is used to fuse the text feature vector and the feature maps of different scales to obtain the fused feature representation, and then introduce the position coding mechanism to obtain the position coding and add the fused feature to the position coding PE to obtain the feature enhanced by the position coding, and finally reconstruct the feature enhanced by the position coding into the shape of the original graph feature through the reconstruction operation, and finally obtain multiple reshaped features and send them to the detection head; The detection head is used to perform target detection tasks according to the input reshaped features, including target existence prediction, bounding box position prediction and target category prediction, and obtain target existence prediction results, bounding box position prediction results and target category prediction results; The output layer outputs a target existence prediction result, a bounding box position prediction result, and a target category prediction result.

4. The method for detecting a target of an autonomous driving vehicle based on cloud-edge collaboration implemented by the target detection system for an autonomous driving vehicle based on cloud-edge collaboration according to claim 1 is characterized in that: The following steps are involved: Step 1: Obtain road scene image dataset and traffic text dataset; The road scene image dataset includes a plurality of road scene images and annotation information corresponding to each road scene image, wherein the annotation information includes the category of an object in the road scene image, the location of the object, the size of an anchor frame, and the resolution of the road scene image; The traffic text data set includes a plurality of traffic-related text descriptions, each of which corresponds to a road scene image and is used to describe the characteristics of the road scene image; Step 2: preprocess the road scene images in the road scene image dataset, and divide the preprocessed road scene image dataset and the corresponding traffic text dataset into a training set and a test set according to a set ratio; Step 3: Build the target detection model YOLO-Vehicle; Step 4: Use the road scene images and corresponding traffic-related text descriptions in the training set to train the object detection model YOLO-Vehicle, keep the pre-trained parameters of the text feature extractor unchanged, and obtain the trained object detection model YOLO-Vehicle; Step 5: Select several road scene images and their corresponding traffic-related text descriptions from the training set to fine-tune the text feature extractor in the trained object detection model YOLO-Vehicle to obtain the final object detection model YOLO-Vehicle; Step 6: Deploy the final target detection model YOLO-Vehicle to the cloud server and edge server to perform target detection and obtain target existence prediction results, bounding box position prediction results, and target category prediction results.

5. The method for target detection of an autonomous driving vehicle based on cloud-edge collaboration according to claim 4, characterized in that: The preprocessing described in step 2 uses a data enhancement strategy to perform data enhancement on the road scene image, specifically: first, the size of the road scene image is uniformly adjusted, all road scene images are adjusted to a preset pixel size, and the fill area is filled with a fixed color value, and then the road scene image is scaled using a bilinear interpolation algorithm while maintaining the original width-to-height ratio of the road scene image, and then the scaled road scene image is randomly horizontally flipped, randomly vertically flipped, randomly rotated, randomly enhanced in HSV color space, randomly blurred using Gaussian blur, and randomly sharpened using the UnsharpMask method.

6. The method for target detection of an autonomous driving vehicle based on cloud-edge collaboration according to claim 4, characterized in that: The target detection model YOLO-Vehicle in step 3 includes: an input layer, a text feature extractor, an image feature extractor, a multimodal feature fusion module, a detection head and an output layer; The input of the input layer is a road scene image and a corresponding traffic-related text description; The text feature extractor uses the pre-trained Clip-ViT. Specifically, the input traffic-related text description is first segmented by the tokenizer, and then several words obtained by the segmentation are embedded into a high-dimensional vector space to obtain several embedded vectors. These embedded vectors are then passed through several Transformer encoder layers to finally obtain a fixed-dimensional text feature vector F. text , as the input of the multimodal feature fusion module; The image feature extractor uses YOLOV8 to extract image features of different scales from the input road scene image to obtain three feature maps of different scales F img ={f1,f2,f3}, as the input of the multimodal feature fusion module, where F img is a collection of feature maps of different scales, represents the feature map of the i-th scale, C represents the number of feature channels, and H i represents the height of the feature map of the i-th scale, W i Represents the width of the feature map of the i-th scale; The multimodal feature fusion module is used to fuse the feature maps of three different scales output by the image feature extractor with the text feature vector respectively; the process of fusing each feature map with the text feature vector is as follows: first, a cross-modal alignment technique is used to map the text feature vector and the feature map to a unified semantic embedding space through linear projection to obtain two aligned feature information; then, a self-attention mechanism is used to calculate the cross-modal correlation to quantify the relationship between the text feature vector and the feature map; the text feature vector and the feature map are fused according to the cross-modal correlation to obtain a fused feature representation; then, a position encoding mechanism is introduced, and a fixed encoding scheme constructed by sine and cosine functions is used to encode the fused feature representation to obtain a position encoding PE; then, the fused feature is added to the position encoding PE through element-level addition to obtain a feature enhanced by the position encoding; finally, the feature enhanced by the position encoding is reconstructed into the shape of the original image feature through a reconstruction operation to obtain a reshaped feature, and finally, three reshaped features are obtained and input into the detection head for target detection; The detection head is used to perform target detection tasks according to the input reshaped features, including target existence prediction, bounding box position prediction and target category prediction. During the detection process, the detection results are post-processed by IOU calculation and non-maximum suppression to eliminate repeated detections and obtain the final detection results, including target existence prediction results, bounding box position prediction results and target category prediction results; The output layer outputs a target existence prediction result, a bounding box position prediction result, and a target category prediction result.

7. The method for target detection of an autonomous driving vehicle based on cloud-edge collaboration according to claim 6, characterized in that: The feature information after alignment is: F' img =W img *F img +b img ∈R 1×512 (3) F' text =W text *F text +b text ∈R 1×512 (4) Among them, F' img is the aligned feature information obtained by linear projection of the feature map, F' text is the aligned feature information obtained by linear projection of the text feature vector, W img and W text is the learnable weight matrix, b img and b text is the bias term; The fused features are expressed as: Among them, F fused is the feature representation after fusion; σ is the sigmoid activation function, W is the learnable weight matrix for feature fusion; b is the bias term; α is the hyperparameter that controls the fusion ratio, represents element-wise multiplication, is the cross-modal correlation, where the query matrix comes from the aligned feature information F' obtained by linear projection of the feature map img , the key matrix and value matrix come from the aligned feature information F' obtained by linear projection of the text feature vector text , F' img The query matrix, and F' text The key matrix and value matrix of .

8. The method for target detection of an autonomous driving vehicle based on cloud-edge collaboration according to claim 6, characterized in that: The target existence prediction result is: P obj =σ(W obj *F output +b obj ) (10) Among them, P obj is the target existence prediction result; W obj and b obj They are the convolution weights and bias parameters used for target existence prediction, respectively; * represents the convolution operation, and σ represents the sigmoid activation function; The bounding box position prediction result is: B=W box *F output +b box (11) Among them, B is the bounding box position prediction result, including four prediction values ​​(t x ,t y ,t w ,t h ), (t x ,t y ) is the center coordinate of the bounding box, t w and t h Represents the relative offset of the bounding box width and height, W box and b box They are the convolution weights and bias parameters for bounding box prediction, respectively; The probability of the target category prediction is: K=softmax(W cls *F output )+b cls (12) Where K is an N-dimensional vector, N is the number of predefined categories, and each element represents the probability that the input road scene image belongs to the corresponding category; W cls and b cls are the convolution weights and bias parameters for target category prediction, respectively.

9. The method for target detection of an autonomous driving vehicle based on cloud-edge collaboration according to claim 4, characterized in that: The fine-tuning described in step 4 adopts a contrastive learning strategy to pair traffic-related text descriptions with corresponding road scene images to form positive samples, and randomly selects unmatched traffic-related text descriptions and corresponding road scene images as negative samples. The optimization goal is to maximize the similarity of positive sample pairs and minimize the similarity of negative sample pairs. The loss function of the fine-tuning process is: Among them, s(t,l) is the similarity score between the traffic-related text description t and the road scene image l, l + is the road scene image that matches the traffic-related text description, l_ is the road scene image that does not match the traffic-related text description, and τ is the temperature parameter.

10. The method for target detection of an autonomous driving vehicle based on cloud-edge collaboration according to claim 4, characterized in that: Step 6 specifically includes: Step 6.1: Customize traffic-related text descriptions based on the current vehicle driving road and use the camera to capture the current road scene image; Step 6.2: Determine whether the resources of the edge server meet the current computing requirements. If yes, use the target detection model YOLO-Vehicle deployed on the edge server to perform target detection and obtain target existence prediction results, bounding box position prediction results and target category prediction results. Otherwise, the edge server transmits the customized traffic-related text description and road scene image to the cloud server, and uses the target detection model YOLO-Vehicle deployed on the cloud server to perform target detection and obtain target existence prediction results, bounding box position prediction results and target category prediction results and transmit them back to the edge server.

Images