Intelligent automobile scene understanding method based on edge data learning enhancement

Abstract

The invention belongs to the technical field of automatic driving testing, and particularly relates to an intelligent automobile scene understanding method based on edge data learning enhancement. Comprising the following steps of 1, collecting natural driving data; 2, performing scene structure representation and semantic label processing; step 3, edge scene extraction and automatic labeling correction; 4, scene information prediction matching and data set construction; and step 5, large model architecture design and structure fine adjustment. Step 6, scene understanding application and model comparison evaluation; the method can be used for understanding and cognition in complex and edge scenes, and the interpretability of an intelligent automobile algorithm is improved.

Description

Technical Field

[0001] The present invention belongs to the technical field of autonomous driving testing, and specifically is a smart car scene understanding method based on edge data learning enhancement. Background Art

[0002] With the widespread application of deep learning, machine learning, and complex algorithms in intelligent vehicle systems, vehicles can demonstrate capabilities that surpass human drivers in terms of autonomous driving, road decision-making, and risk warning. However, behind this high level of intelligence lies the "black box" effect: even if the intelligent vehicle system can effectively perform various driving tasks, its decision-making process often lacks transparency and understandability, which is crucial to the safety and reliability of intelligent vehicle algorithms. Therefore, developing intelligent vehicle algorithms that are self-explanatory, easy to understand, and easy to debug has become a major issue faced by scientific researchers and technology developers around the world.

[0003] Scene understanding uses computers to actively perceive, analyze and interpret driving scenes. For smart car algorithms, it can clearly show users the external perception environment and internal decision-making logic, thereby improving the interpretability of the algorithm and enhancing users' trust and acceptance of smart cars. At the same time, through scene understanding, smart cars can prompt users when the scene is abnormal so that users can take timely measures.

[0004] Big models use massive amounts of data to train artificial intelligence models. They have powerful image and natural language understanding capabilities and have significant potential advantages in intelligent vehicle scene understanding. However, due to the long-tail effect of natural driving data, the data sets used for big model training often only contain basic information. Therefore, existing big models can only understand basic scene features and are difficult to be directly used for complex, edge intelligent vehicle scene understanding tasks. Edge scene data has rich and diverse scene features. How to efficiently apply them to the big model training process and expand the application field of big models is a scientific problem that needs to be solved urgently. Specifically, there are three challenges: (1) How to obtain and utilize those rare, complex edge driving scene data that account for a small proportion of conventional training sets; (2) How to design an effective data enhancement strategy so that the big model can learn rich scene features from limited edge scene data; (3) How to optimize the complex edge scene understanding tasks of intelligent vehicles while ensuring the original performance of the big model, thereby promoting the practical application of big models in the field of intelligent vehicles. Summary of the invention

[0005] To solve the above problems, the present invention provides a smart car scene understanding method based on edge data learning enhancement, which extracts edge scene data by analyzing road collection data; constructs a structured template based on the scene ontology, and combines the YOLO architecture and graph neural network to effectively deconstruct the test scene image, extract the element features in the scene, and improve the scene detail description; based on the extraction of future scene information and the annotation of the main vehicle's response measures, it can effectively build a complete data flow from scene cognition to functional application and construct a model training data set; based on the open source large model architecture, model fine-tuning training is performed, and finally the smart car scene understanding based on edge data learning enhancement is realized, which can be used for understanding and cognition in complex and edge scenes, and improve the interpretability of smart car algorithms.

[0006] The technical solution of the present invention is described as follows in conjunction with the accompanying drawings: A method for intelligent vehicle scene understanding based on edge data learning enhancement includes the following steps: Step 1: Based on the real vehicle visual acquisition platform, collect natural driving conditions in the urban area; Step 2: Build a structured template based on the scene ontology; use the YOLO visual processing algorithm to perform target detection and semantic label classification; use the graph neural network to fill and map the scene image into the predefined scene template to complete the conversion from image to structured data; Step 3: Develop and apply specific indicators to automatically screen out challenging edge scenarios. For cases where the classification accuracy is lower than the set threshold, introduce a secondary review to improve data quality; Step 4: Match possible future changes based on scene semantic labels and mark the actions that the vehicle should take under the current situation; pre-process the original image and encode it into a vector format to construct a training data set, with scene vector data as input and structural template representation, future change data, and the main vehicle's response measures as output; Step 5: Use the mature Qwen-VL architecture and fine-tune it for edge scene datasets so that the model can learn to extract features from images and make accurate future predictions; in order to maintain the generalization of the trained model, add the cross entropy of the two model outputs to the loss function to quantify the difference between the original model and the new model output, ensuring that the learning of the new model does not deviate significantly from the performance of the original model; regularly verify the model performance and adjust the training parameters to ensure the generalization ability of the model until a satisfactory performance level is achieved; Step 6: Select a large open source model as a benchmark, compare its performance in basic cognition, future change prediction, and vehicle behavior analysis, and evaluate the actual application value of the model; analyze the performance difference between the new model and the original model on the same test set to confirm whether the new model has successfully retained the original task understanding ability.

[0007] Furthermore, the specific method of step one is as follows: S11. Build a test vehicle data acquisition platform based on the Horizon Vision Kit; S12. Real-time capture of diverse driving scenarios: Through the camera equipment installed on the smart car, the visual information of the vehicle driving in the urban area is recorded in real time; S13. When selecting the vehicle driving area, ensure that the data covers a variety of driving scenarios.

[0008] Furthermore, the specific method of step 2 is as follows: S21. Construct a scenario structured template. Based on ontology, the driving scenario is abstracted into a series of structured templates, including terminology layer, attribute layer and association layer. By annotating according to this template, the spatial relationship and interaction rules between different elements can be defined. S22, classify the scene semantic labels and use the visual processing algorithm YOLO to perform in-depth analysis on the images obtained during the natural driving data collection phase; S23, fill and transform images, map scene elements with semantic labels to preset scene structured templates, and transform the originally unstructured scene images into structured representations with clear semantic meanings and positional relationships based on graph neural networks, and transform natural driving data into a form that can be efficiently used by large models; Furthermore, the specific method of S22 is as follows: the collected data is processed using the YOLO-v8 architecture, the terms and basic attributes in the ontology architecture diagram are extracted, each target in the image is located and identified, a corresponding semantic label is assigned to each target, and the category probability, center point coordinates (x, y), width (w), and height (h) of each type of bounding box are output, and converted into a bounding box representation.

[0009] Furthermore, the specific method of S23 is as follows: S231, taking each detected object as a node, and adding edges to express the spatial relationship or semantic relationship between the objects; S232. Create a graph neural network model to capture the interactions between nodes and propagate contextual information; define a node embedding layer to encode the original features of each object, i.e., the boundary information output by YOLO, into a high-dimensional vector representation; design a graph convolutional layer and a message passing mechanism suitable for structured representation learning, so that the network can learn the complex relationships between nodes; S233. Training the graph neural network model can learn and infer deeper semantic relationships and structured information from the graph structure; after training, the YOLO detection results are input into the graph neural network model, and the model will output updated node representations that can be mapped to concepts and their relationships in the ontological framework, thereby forming a structured representation.

[0010] Furthermore, the specific method of step three is as follows: S31. Definition of edge scenarios: There is no clear definition of edge scenarios in academia. Considering that edge scenarios may pose challenges to existing intelligent algorithms, the present invention defines edge scenarios as those situations that pose challenges to current intelligent algorithms, which specifically include the following two parts: ① Unclear identification: The target detection confidence output by the YOLO algorithm is lower than the set threshold (set to 0.7), indicating that the model is uncertain about whether the detected object belongs to a certain category; ②Poor structuring: After processing by the graph neural network (GNN), the node classification accuracy failed to reach the predetermined standard (set to 90%), which means that the elements in the image were not correctly classified and associated; S32. Automatic extraction of edge scenes: Through the above-mentioned scene edge judgment index system, automatically screen out those edge driving scene images that have not been accurately identified or have poor structured effects. These edge scenes may include rare road conditions, special traffic events, views under extreme weather conditions, etc.; S33. Automated data improvement: For automatically extracted edge scene images, due to the poor performance of the target detection algorithm and the scene structuring algorithm, it is difficult to generate accurate and complete scene structured data. For this reason, the system introduces automated labeling technology to improve these data to ensure that the model can learn and adapt to complex scenes more effectively. Through weakly supervised learning methods, various types of targets in the image are secondary labeled and classified. Through the iterative learning process, the system can gradually improve the accuracy of understanding elements in complex scenes, ensuring that all elements of each frame of the image are labeled, thereby improving the model's knowledge base when processing such complex scenes; Furthermore, the specific method of step 4 is as follows: S41, predicting and matching scene information; analyzing the original acquired video or image sequence based on the scene semantic label, extracting the state change of the scene in a continuous time (single sequence time length 5s), and constructing scene prediction information; S42, labeling the main vehicle behavior; for the current scene, combining road traffic rules and safe driving principles, the above rules are converted into prompt words through the pre-trained large language model, so as to automatically generate the driving strategy or action that the main vehicle should take in the specific scene; in addition, based on the above rules, the generated strategy is modified to delete the behavior that does not conform to the rules; S43, preprocessing the image, performing a series of preprocessing operations on the collected original image data, specifically including four steps of color space conversion, image cropping, normalization, and size scaling; S44, feature vectorization, converting the preprocessed image into a numerical feature vector that can be understood by the computer, using a convolutional neural network to extract image features, and mapping image features from a high-dimensional pixel space to a low-dimensional feature space; S45. Construct a data set. Combined with the above processing results, a complete data set containing scene information is constructed; each sample includes image data that has been preprocessed and encoded as a feature vector and the corresponding scene semantic label, predicted future state change data, and an ideal response measure label that the main vehicle should take in the current scene.

[0011] Furthermore, the specific method of step five is as follows: S51. Select and initialize the model architecture: select the qwen-vl open source large model architecture; initialize the new model based on the pre-trained qwen-vl weights; S52, application dataset: use the dataset containing edge scene information prediction constructed in step 4 for training to enhance the model's understanding of complex and rare driving conditions; Using the Lora (Low-Rank Adaptation) method, only the part of the model used to handle the new task is adjusted, especially the low-rank decomposition matrix in the fully connected layer. This method allows the network to focus on learning features related to the new task without changing the original pre-trained weights. According to the format of the image feature vector, the dimension of the input layer is adjusted to be consistent with the feature vector in the aforementioned preprocessing stage. In order to maintain the generalization of the model, a cross entropy term is added to the standard loss function to quantify the difference between the output of the original model and the new model. This ensures that the learning of the new model does not deviate significantly from the performance of the original model, while improving the accuracy of future state predictions; S53. Performance monitoring and adjustment: During the entire training cycle, regularly (every two rounds of training) evaluate the performance of the model on an independent validation set, focusing on the performance indicators of the model on specific tasks (such as scene understanding accuracy, future state prediction error, and rationality score of the main vehicle's response measures). Dynamically adjust the training strategy based on the evaluation results, and use the cosine annealing learning rate scheduler to automatically adjust the learning rate based on the performance on the validation set. The learning rate gradually decreases from the initial value to the minimum value, and then increases again, forming a cyclical change. According to the overfitting situation, appropriately adjust the L2 regularization coefficient to prevent the model from overfitting the training data. The adjustment process gradually increases by 10% until the overfitting phenomenon disappears; Among them, the L2 regularization coefficient is the attenuation of the loss weight, and the loss function is calculated as follows: In the formula, is the original loss function, is the L2 regularization coefficient, is the weight coefficient of the model.

[0012] Furthermore, the specific method of step six is ​​as follows: S61. Application test model: Apply the qwen-vl model obtained through fine-tuning training in step 5 and other open source typical image recognition large models, such as EfficientNet, ResNet, etc., to the edge scene datasets constructed and processed in steps 1 to 4, and perform scene understanding analysis; S62, cognitive evaluation scenario: compare the performance of each model in basic scene recognition, and judge whether each model can accurately identify vehicles, pedestrians, traffic signs, road conditions in the scene, and whether it can grasp the overall layout and structural characteristics of the scene; the specific indicators are designed as: perception category accuracy; S63, predict and compare future changes: analyze the ability of each model to predict future changes in the scene and compare the accuracy of different models; the specific indicators are designed as: prediction accuracy of moving subjects and collision risk assessment accuracy; S64, Simulate and evaluate vehicle behavior: For each scenario, evaluate the effectiveness of each model in simulating the reasonable behavior that the autonomous driving vehicle should take, reflecting the decision-making response under complex road conditions (whether it complies with traffic regulations and safe driving guidelines; the specific indicators are designed as follows: reasonableness of expected behavior; S65. Comprehensively evaluate model performance: Summarize the above evaluation results, compare the advantages and disadvantages of different models in edge scene understanding and prediction tasks from multiple dimensions, find the large model that is most suitable for smart car application scenarios, and at the same time explore the shortcomings of the model to provide direction for further optimizing model performance; set the weight of each indicator to 0.4, 0.3, and 0.3, and take the weighted average of the score of each indicator.

[0013] The beneficial effects of the present invention are: 1. The present invention constructs a structured template based on the scene ontology, and combines the YOLO architecture and graph neural network to effectively deconstruct the test scene image, extract the element features in the scene, and improve the scene detail description; 2. Based on the extraction of future scene information and the annotation of the main vehicle's response measures, the present invention can effectively build a complete data flow from scene cognition to functional application and construct a model training data set; at the same time, the extracted edge scene images are subjected to high-quality preprocessing and feature vectorization, which improves the consistency of the data and reduces the need for manual intervention; 3. The present invention conducts model fine-tuning training based on the open source big model architecture, and finally realizes intelligent vehicle scene understanding based on edge data learning enhancement; selects the open source big model as the benchmark, regularly compares its performance in basic cognition, future change prediction and vehicle behavior analysis, evaluates the practical application value of the new model, and builds a complete evaluation system for existing scene understanding tasks; 4. Maintain the original task understanding ability: During the fine-tuning process, a cross entropy term is added to the loss function to quantify the difference between the output of the original model and the new model, ensuring that the learning of the new model does not significantly deviate from the performance of the original model. This not only improves the accuracy of future state predictions, but also ensures the generalization ability of the model on new and old tasks; 5. By automatically screening out challenging edge scenarios and applying advanced automation technology to improve data, the present invention can effectively improve the performance of the model in handling rare or extreme conditions and enhance the system's ability to understand complex driving environments. It can be used to understand and recognize complex and edge scenarios and improve the interpretability of smart car algorithms; 6. Combining road traffic rules, safe driving principles and historical driving data, the system of the present invention can automatically generate a safe and reasonable driving strategy that the main vehicle should adopt, ensuring the accuracy and safety of autonomous driving decisions and reducing the risk of traffic accidents. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0015] Figure 1 It is the overall flow chart of the present invention; Figure 2 Describe the ontology architecture diagram for the scenario; Figure 3 Build an architectural diagram for the dataset; Figure 4 Schematic diagram for model comparison and verification. DETAILED DESCRIPTION

[0016] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0017] See also Figure 1-Figure 4 , this embodiment provides a smart car scene understanding method based on edge data learning enhancement, including the following steps: Step 1: Natural driving data collection. Based on the real vehicle visual acquisition platform, collect natural driving conditions in urban areas, as follows: S11. Driving data that is rich enough and close to the real world is the key for the big model to understand the real scene. Data collected from real vehicles can meet the authenticity requirements to the greatest extent. Based on the Horizon Vision Kit, a test vehicle data collection platform is built.

[0018] Visual sensors provided by Horizon Robotics are installed on the test vehicles, including high-definition cameras, stereo cameras, and surround-view cameras. These sensors ensure that they cover all-round views around the vehicle and can capture rich visual information. The on-board computers are configured to process and store the collected data in real time.

[0019] S12. Real-time capture of diverse driving scenarios: Through the camera equipment installed on the smart car, the visual information of the vehicle in the urban area is recorded in real time, including but not limited to various complex road traffic elements such as road environment, traffic signs, pedestrians, other vehicles, weather conditions, road conditions, etc.

[0020] S13. When selecting the vehicle driving area, ensure that the data covers a variety of driving scenarios, including but not limited to normal driving, turning, lane changing, crossing intersections, obstacle avoidance, responding to emergencies (such as pedestrians suddenly crossing the road), special weather conditions (rain, snow, fog), etc., to reflect the complexity of real-world driving.

[0021] Study and identify various driving scenarios that smart cars may face, including but not limited to normal city streets, highways, rural roads, complex intersections, narrow lanes, curves, ramps, etc. Analyze the dynamic behaviors of various traffic participants, including the behavior patterns of pedestrians, bicycles, motorcycles, and other vehicles. At the same time, consider different climate and light conditions, including rain, snow, fog, night, dawn and dusk, strong sunlight reflection, etc. Therefore, design a route cycle driving that includes typical scenarios to ensure the adequacy and diversity of data collection.

[0022] Step 2: Scene structure representation and semantic label processing. Based on the scene ontology model, build a scene structured template; based on the YOLO visual processing algorithm, realize the scene semantic label classification function; train the graph neural network to fill and convert the scene image into the basic elements in the template, as follows: S21, scene structured template construction, based on ontology, abstracts and summarizes the complex driving scenes into a series of structured templates, such as Figure 2 As shown in the figure, it specifically includes the terminology layer, attribute layer, association layer, etc. By annotating according to this template, the spatial relationship and interaction rules between different elements can be defined, providing a standardized scenario framework for subsequent intelligent analysis.

[0023] The terminology layer defines the basic element types and basic states in the scenario; the attribute layer expands the basic states; the association layer describes the association relationship between elements and the behavioral norms and constraints of each element. Through this type of ontology, the test scenario can be described in detail.

[0024] S22, scene semantic label classification, using the visual processing algorithm YOLO, to conduct in-depth analysis of the images obtained during the natural driving data collection phase.

[0025] The mature and universal YOLO-v8 architecture is used to process the collected data, and the image features are extracted through the convolutional neural network (CNN). The anchor box mechanism (a set of preset bounding boxes with different sizes and aspect ratios, covering all positions in the input image) is used to predict the position and category probability of the bounding box. The terms and basic attributes in the ontology architecture diagram are extracted to quickly locate and identify each target in the image, assign corresponding semantic labels to each target, and output the category probability, center point coordinates (x, y), width (w), height (h) and other information of each type of bounding box, and convert it into a bounding box representation. Specifically, for an input image , the model outputs a series of bounding box predictions , where each bounding box contains the following information: ,in represents the probability distribution of the category, is the probability distribution of the bounding box parameters, including the center point coordinates and width and height .

[0026] S23, image filling and conversion, maps scene elements with semantic labels to preset scene structured templates, and transforms the originally unstructured scene images into structured representations with clear semantic meanings and positional relationships based on graph neural networks, converting natural driving data into a form that can be efficiently used by large models, facilitating in-depth understanding and learning of large models.

[0027] Specifically, follow the steps below: ① Treat each detected object as a node , where each node Corresponds to a bounding box ,Node attributes include but are not limited to object category, location, and size;,edges are added to express spatial or semantic relationships between,objects; ② Create a graph convolutional network (GCN) model to capture the interactions between nodes and propagate contextual information. Define a node embedding layer to encode the original features of each object (the boundary information output by YOLO) into a high-dimensional vector representation, that is, the initial feature representation of the node , ,in Represents a multi-layer perceptron in a neural network. Design a graph convolutional layer GCL to update node representations through a message passing mechanism: in, is the level index, Representation Node The neighbor set of and are the weight matrix and bias vector respectively, is the activation function. and is a node feature.

[0028] Suitable for structured representation learning, allowing the network to learn complex relationships between nodes; ③ Train the graph neural network model so that it can learn and infer deeper semantic relationships and structured information from the graph structure. After training, the YOLO detection results are input into the graph neural network model, and the model outputs updated node representations, which can be mapped to concepts and their relationships in the ontology framework to form a structured representation.

[0029] Step 3: Edge scene extraction and automatic annotation correction. Develop and apply specific indicators to automatically screen out challenging edge scenes. For cases where the classification accuracy is lower than the set threshold, introduce a secondary review to improve data quality. The details are as follows: S31. Definition of edge scenarios: There is no clear definition of edge scenarios in academia. Considering that edge scenarios may pose challenges to existing intelligent algorithms, the present invention defines edge scenarios as those situations that pose challenges to current intelligent algorithms, which specifically include the following two parts: ① Unclear identification: The target detection confidence output by the YOLO algorithm is lower than the set threshold (set to 0.7), indicating that the model is uncertain about whether the detected object belongs to a certain category.

[0030] ②Poor structuring: After processing by the graph neural network (GNN), the node classification accuracy failed to reach the predetermined standard (set to 90%), which means that the elements in the image were not correctly classified and associated.

[0031] Specifically, the quantitative indicators are designed as follows: ① Confidence of target recognition: reflects the model’s confidence that the object in the bounding box is indeed the target object and not the background. The calculation formula is the product of the category probability p(c) and the intersection over union (IoU) between the predicted box and the true box.

[0032] Specifically, the YOLO category confidence reflects the model's confidence that the currently detected bounding box is indeed the target object rather than the background. It is usually the product of the probability that the target box contains the object and the bounding box position accuracy. The calculation formula is as follows: In the formula, is the class probability; is the intersection-over-union ratio of the predicted box and the true box; ② Scene structured accuracy: Measures the performance of GNN in correctly classifying each region or node in the image into predefined categories. It is evaluated by comparing the predicted labels with the known true value labels, and using the confusion matrix to count TP, FP, and FN to calculate the accuracy.

[0033] Specifically, the image is structured using graph neural networks, and the image is divided into multiple nodes or regions, each of which has a predicted category label. For each node, its predicted category label is compared with the manually annotated true value label. If they are consistent, it is counted as a correct classification. The total number of correct classifications is calculated, recorded as TP (True Positive), and the number of misclassifications is counted, recorded as FP (False Positive, i.e., nodes that are misclassified as a certain category) and FN (False Negative, i.e., nodes that are missed). The accuracy calculation formula is as follows: ; S32. Automatic extraction of edge scenes: Through the above-mentioned scene edge judgment index system, those edge driving scene images that cannot be accurately identified or have poor structuring effects are automatically screened out. These edge scenes may include rare road conditions, special traffic events, views under extreme weather conditions, etc.

[0034] S33. Automated data improvement: For automatically extracted edge scene images, it is difficult to generate accurate and complete scene structured data due to the poor performance of the target detection algorithm and the scene structuring algorithm. For this reason, the system introduces automated annotation technology to improve these data to ensure that the model can learn and adapt to complex scenes more effectively. Through weakly supervised learning methods, various types of targets in the image are secondary labeled and classified. Through the iterative learning process, the system can gradually improve the accuracy of understanding elements in complex scenes, ensuring that all elements of each frame of the image are labeled, thereby improving the model's knowledge base when processing such complex scenes; Step 4: Scene information prediction matching and data set construction. Match possible future changes based on scene semantic labels and mark the actions that the vehicle should take under the current situation. Perform preliminary preprocessing on the original image and encode it into a vector format for constructing a training data set (scene vector data as input, structural template representation, future change data, and vehicle response measures as output). The details are as follows: S41, scene information prediction and matching. Analyze the original captured video or image sequence based on the scene semantic label, extract the state changes of the scene over a period of time, including the motion characteristics of elements such as vehicles, pedestrians, and traffic signs, and possible scene changes, and thus construct scene prediction information.

[0035] S42, vehicle behavior labeling. For the current scenario, combined with road traffic rules and safe driving principles, the pre-trained large language model converts the above rules into prompt words, thereby automatically generating the driving strategy or action that the vehicle should take in a specific scenario, including basic tasks such as changing lanes, slowing down, and stopping and waiting. In addition, the generated strategy is modified based on the above rules to delete behaviors that do not comply with the rules.

[0036] S43, image preprocessing, performs a series of preprocessing operations on the collected raw image data, including color space conversion, image cropping, normalization (adjusting pixel values ​​to between 0 and 1), and resizing (unifying the image size required for input to the neural network) to improve model training and inference efficiency.

[0037] S44, feature vectorization, converts the preprocessed image into a numerical feature vector that can be understood by the computer, uses a convolutional neural network (CNN) to extract image features, and maps them from high-dimensional pixel space to low-dimensional feature space.

[0038] S45, data set construction, combining the above processing results, to construct a complete data set containing scene information. Each sample includes not only the image data that has been preprocessed and encoded into a feature vector, but also the corresponding scene semantic label, the predicted future state change data, and the ideal response measure label that the main vehicle should take in the current scene.

[0039] Step 5. Large model architecture design and structural fine-tuning. Use the mature Qwen-VL architecture and fine-tune it for edge scene datasets so that the model can learn to extract features from images and make accurate future predictions. In order to maintain the generalization of the trained model, add the cross entropy of the two model outputs to the loss function to quantify the difference between the original model and the new model output to ensure that the learning of the new model does not deviate significantly from the performance of the original model. Regularly verify model performance and adjust training parameters to ensure model generalization capabilities until a satisfactory performance level is achieved. Details are as follows: S51. Model architecture selection and initialization: We selected an advanced and mature open source large model architecture such as qwen-vl, which laid a solid foundation for subsequent scene understanding and prediction tasks based on its powerful generalization ability and rich parameters. We initialized the new model based on the pre-trained qwen-vl weights.

[0040] S52. Application dataset: Use the dataset constructed in step 4 containing edge scene information prediction for training to enhance the model's understanding of complex and rare driving conditions.

[0041] Using the Lora (Low-Rank Adaptation) method, only the part of the model used to handle the new task is adjusted, especially the low-rank decomposition matrix in the fully connected layer. This method allows the network to focus on learning features related to the new task without changing the original pre-trained weights. According to the format of the image feature vector, the dimension of the input layer is adjusted to be consistent with the feature vector in the aforementioned preprocessing stage. In order to maintain the generalization of the model, a cross entropy term is added to the standard loss function to quantify the difference between the output of the original model and the new model. This ensures that the learning of the new model does not deviate significantly from the performance of the original model, while improving the accuracy of future state predictions.

[0042] S53. Performance monitoring and adjustment: During the entire training cycle, the performance of the model is evaluated regularly (every two rounds of training) on ​​an independent validation set, focusing on the performance indicators of the model on specific tasks (such as scene understanding accuracy, future state prediction error, and rationality score of the main vehicle's response measures). The training strategy is dynamically adjusted according to the evaluation results. The cosine annealing learning rate scheduler is used to automatically adjust the learning rate according to the performance on the validation set. The learning rate gradually decreases from the initial value to the minimum value, and then increases again, forming a periodic change. According to the overfitting situation, the L2 regularization coefficient is appropriately adjusted to prevent the model from overfitting the training data. The adjustment process gradually increases by 10% until the overfitting phenomenon disappears.

[0043] Among them, the L2 regularization coefficient is the attenuation of the loss weight, and the loss function is calculated as follows: In the formula, is the original loss function, is the L2 regularization coefficient, is the weight coefficient of the model. If it is too large, the penalty for large weights will be heavier, resulting in a simpler model and thus underfitting. If it is too large, there will be less restriction on weights, and the model will become too complex, resulting in overfitting; therefore, it needs to be gradually increased according to the steps described above.

[0044] Step 6: Scene understanding application and model comparison and evaluation. Select a typical open source image recognition model, understand and analyze the collected scene data, compare the model's analysis results in scene basic cognition, future scene changes, and expected vehicle behavior, and evaluate the application effect of the model, as follows: S61. Model application test: The qwen-vl model obtained through fine-tuning training in step 5 and other open source typical image recognition models, such as EfficientNet and ResNet, are applied to the edge scene datasets constructed and processed in steps 1 to 4 for scene understanding analysis.

[0045] S62, Scene Cognition Evaluation: Compare the performance of each model in basic scene recognition, and judge whether each model can accurately identify the elements such as vehicles, pedestrians, traffic signs, road conditions, etc. in the scene, and whether it can grasp the overall layout and structural characteristics of the scene. The specific indicator is designed as: Perception Category Accuracy.

[0046] S63. Comparison of predictions of future changes: Analyze the ability of each model to predict future changes in the scene, such as the prediction of the movement status of vehicles and pedestrians, and the prediction of traffic signal change trends, and compare the accuracy of different models in this regard. The specific indicators are designed as: prediction accuracy of moving subjects and collision risk assessment accuracy.

[0047] S64, Vehicle Behavior Simulation Evaluation: For each scenario, evaluate the effectiveness of each model in simulating the reasonable behavior that the autonomous driving vehicle should take, reflecting whether the decision-making response (such as lane changing, braking, acceleration, etc.) under complex road conditions complies with traffic regulations and safe driving guidelines. The specific indicators are designed as: reasonableness of expected behavior.

[0048] S65. Comprehensive evaluation of model performance: Summarize the above evaluation results, compare the advantages and disadvantages of different models in edge scene understanding and prediction tasks from multiple dimensions, find the most suitable large model for smart car application scenarios, and explore the shortcomings of the model to provide direction for further optimization of model performance. Set the weight of each indicator to 0.4, 0.3, and 0.3, and take the weighted average of the scores of each indicator.

[0049] In summary, the present invention can be used for understanding and cognition in complex and edge scenarios, and improve the interpretability of smart car algorithms.

[0050] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A smart car scene understanding method based on edge data learning enhancement, characterized in that: The following steps are involved: Step 1: Based on the real vehicle visual acquisition platform, collect natural driving conditions in the urban area; Step 2: Build a structured template based on the scene ontology; use the YOLO visual processing algorithm to perform target detection and semantic label classification; use the graph neural network to fill and map the scene image into the predefined scene template to complete the conversion from image to structured data; Step 3: Develop and apply specific indicators to automatically screen out challenging edge scenarios; introduce secondary review for cases where the classification accuracy is lower than the set threshold; Step 4: Match possible future changes based on scene semantic labels and mark the actions that the vehicle should take under the current situation; pre-process the original image and encode it into a vector format for building a training data set; the scene vector data is used as input, and the structure template representation, future change data, and the main vehicle's response measures are used as output; Step 5: Use the mature Qwen-VL architecture and fine-tune it for edge scene datasets so that the model can learn to extract features from images and make accurate future predictions; add the cross entropy of the two model outputs to the loss function to quantify the difference between the original model and the new model output; regularly verify the model performance and adjust the training parameters until a satisfactory performance level is achieved; Step 6: Select a large open source model as a benchmark, compare its performance in basic cognition, future change prediction, and vehicle behavior analysis, and evaluate the practical application value of the model; analyze the performance difference between the new model and the original model on the same test set to confirm whether the new model has successfully retained the original task understanding ability.

2. According to claim 1, a smart car scene understanding method based on edge data learning enhancement is characterized in that: The specific method of step one is as follows: S11. Build a test vehicle data acquisition platform based on the Horizon Vision Kit; S12. Real-time capture of diverse driving scenarios: Through the camera equipment installed on the smart car, the visual information of the vehicle driving in the urban area is recorded in real time; S13. When selecting the vehicle driving area, ensure that the data covers a variety of driving scenarios.

3. According to claim 1, a smart car scene understanding method based on edge data learning enhancement is characterized in that: The specific method of step 2 is as follows: S21. Construct a scenario structured template. Based on ontology, the driving scenario is abstracted into a series of structured templates, including terminology layer, attribute layer and association layer. The spatial relationship and interaction rules between different elements are defined according to this template. S22, classify the scene semantic labels and use the visual processing algorithm YOLO to perform in-depth analysis on the images obtained during the natural driving data collection phase; S23. Fill and transform images, map scene elements with semantic labels to preset scene structured templates, and transform the originally unstructured scene images into structured representations with clear semantic meanings and positional relationships based on graph neural networks, thus transforming natural driving data into a form that can be efficiently used by large models.

4. The method for intelligent vehicle scene understanding based on edge data learning enhancement according to claim 3 is characterized in that: The specific method of S22 is as follows: use the YOLO-v8 architecture to process the collected data, extract the terms and basic attributes in the ontology architecture diagram, locate and identify each target in the image, assign a corresponding semantic label to each target, and output the category probability, center point coordinates, width, and height of each type of bounding box, and convert them into a bounding box representation.

5. The method for intelligent vehicle scene understanding based on edge data learning enhancement according to claim 3 is characterized in that: The specific method of S23 is as follows: S231, taking each detected object as a node, and adding edges to express the spatial relationship or semantic relationship between the objects; S232. Create a graph neural network model to capture the interactions between nodes and propagate contextual information; define a node embedding layer to encode the original features of each object, i.e., the boundary information output by YOLO, into a high-dimensional vector representation; design a graph convolutional layer and a message passing mechanism suitable for structured representation learning, so that the network can learn the complex relationships between nodes; S233, training the graph neural network model to learn and infer deeper semantic relationships and structured information from the graph structure; After training, the YOLO detection results are input into the graph neural network model, and the model will output updated node representations that can be mapped to concepts and relationships in the ontology framework to form a structured representation.

6. The method for intelligent vehicle scene understanding based on edge data learning enhancement according to claim 1 is characterized in that: The specific method of step three is as follows: S31. Define edge scenarios: Edge scenarios are defined as those situations that pose challenges to current intelligent algorithms, specifically including the following two parts: ① Unclear identification: When the target detection confidence output by the YOLO algorithm is lower than the set threshold, it indicates that the model is uncertain about whether the detected object belongs to a certain category; ②Poor structuring: After the graph neural network is processed, when the node classification accuracy fails to reach the predetermined standard, it means that the elements in the image are not correctly classified and associated; S32, automatic extraction of edge scenes: Through the scene edge judgment index system, automatically screen out edge driving scene images that are not accurately identified or poorly structured. Edge driving scenes include rare road conditions, special traffic events, and views under extreme weather conditions; S33. Automated data improvement: For automatically extracted edge scene images, the system introduces automated labeling technology to improve the data. Through weakly supervised learning methods, various types of targets in the image are secondary labeled and classified. Through an iterative learning process, it ensures that all elements of each frame of the image are labeled, thereby improving the model's knowledge base when dealing with such complex scenes.

7. The method for intelligent vehicle scene understanding based on edge data learning enhancement according to claim 1 is characterized in that: The specific method of step 4 is as follows: S41, predicting and matching scene information; analyzing the original acquired video or image sequence based on the scene semantic label, extracting the state change of the scene in continuous time, and constructing scene prediction information; S42, labeling the main vehicle behavior; for the current scene, combining road traffic rules and safe driving principles, the pre-trained large language model is used to convert the rules into prompt words, thereby automatically generating the driving strategy or action that the main vehicle should take in a specific scene; in addition, the generated strategy is modified based on the rules, and the behavior that does not comply with the rules is deleted; S43, preprocessing the image, performing a series of preprocessing operations on the collected original image data, specifically including four steps of color space conversion, image cropping, normalization, and size scaling; S44, feature vectorization, converting the preprocessed image into a numerical feature vector that can be understood by the computer, using a convolutional neural network to extract image features, and mapping image features from a high-dimensional pixel space to a low-dimensional feature space; S45. Construct a data set, and combine the processing results to construct a complete data set containing scene information; each sample includes image data that has been preprocessed and encoded as a feature vector and the corresponding scene semantic label, predicted future state change data, and the ideal response measure label that the main vehicle should take in the current scene.

8. The method for intelligent vehicle scene understanding based on edge data learning enhancement according to claim 7 is characterized in that: The specific method of step five is as follows: S51. Select and initialize the model architecture: select the qwen-vl open source large model architecture; initialize the new model based on the pre-trained qwen-vl weights; S52, application data set: use the data set containing edge scene information prediction constructed in step 4 for training; S53, Performance monitoring and adjustment: During the entire training cycle, regularly evaluate the performance of the model on an independent validation set, focusing on the performance indicators of the model on specific tasks; dynamically adjust the training strategy based on the evaluation results, use the cosine annealing learning rate scheduler, and automatically adjust the learning rate based on the performance on the validation set. The learning rate gradually decreases from the initial value to the minimum value, and then increases again, forming a periodic change; according to the overfitting situation, adjust the L2 regularization coefficient to prevent the model from overfitting the training data. The adjustment process gradually increases by 10% until the overfitting phenomenon disappears; Among them, the L2 regularization coefficient is the attenuation of the loss weight, and the loss function is calculated as follows: In the formula, is the original loss function; is the L2 regularization coefficient; is the weight coefficient of the model.

9. The method for intelligent vehicle scene understanding based on edge data learning enhancement according to claim 8 is characterized in that: The specific method of step six is ​​as follows: S61. Apply the test model: apply the Qwen-VL model obtained through fine-tuning training in step 5 and other open source typical image recognition large models to the edge scene datasets constructed and processed in steps 1 to 4, respectively, to perform scene understanding analysis; S62, cognitive evaluation scenario: compare the performance of each model in basic scene recognition, and judge whether each model can accurately identify vehicles, pedestrians, traffic signs, road conditions in the scene, and whether it can grasp the overall layout and structural characteristics of the scene; the specific indicators are designed as: perception category accuracy; S63, predict and compare future changes: analyze the ability of each model to predict future changes in the scene and compare the accuracy of different models; the specific indicators are designed as: prediction accuracy of moving subjects and collision risk assessment accuracy; S64, Simulate and evaluate vehicle behavior: For each scenario, evaluate the effectiveness of each model in simulating the reasonable behavior that the autonomous vehicle should take, reflecting whether the decision-making response under complex road conditions complies with traffic regulations and safe driving guidelines; the specific indicators are designed as follows: reasonableness of expected behavior; S65. Comprehensively evaluate model performance: Summarize the evaluation results, compare the advantages and disadvantages of different models in edge scene understanding and prediction tasks from multiple dimensions, find the large model that is most suitable for smart car application scenarios, set the weight of each indicator to 0.4, 0.3, and 0.3, and take a weighted average of the scores of each indicator.

Images