Scene recognition method, vehicle control method, electronic device, and storage medium

Abstract

The application provides a scene recognition method, a vehicle control method, an electronic device and a storage medium. The method comprises: acquiring current perception data of a current scene collected by an actual vehicle sensor; inputting the current perception data into a scene recognition model to output a scene type probability distribution; the scene recognition model is trained based on training data and two-dimensional labels of corresponding scene types of each basic scene and corresponding dynamic interaction information thereof; and a current scene type containing a basic scene and corresponding dynamic interaction information thereof is determined based on the scene type probability distribution. In this way, the interaction action of the current scene is quickly distinguished, thereby improving the understanding ability of complex scenes and making subsequent decisions more accurate.

Description

Technical Field

[0001] This invention relates to the field of vehicle control technology, and in particular to a scene recognition method, a vehicle control method, an electronic device, and a storage medium. Background Technology

[0002] In recent years, autonomous driving has moved from theoretical verification in laboratories to mass production and commercialization. Its functional levels are also upgrading from lower-level driver assistance to higher-level autonomous driving such as Level 3 (conditional automated driving) and Level 4 (highly automated driving). In autonomous driving systems, the Prediction, Decision, and Planning (PDP) model remains one of the core functions, directly determining the vehicle's driving safety, comfort, and efficiency.

[0003] The PDP model in related technologies relies solely on perceptual data to autonomously determine scene attributes, making it difficult to efficiently distinguish the semantic differences of the same perceptual data in different scenes. This can easily lead to confusion in decision-making logic, resulting in insufficient understanding of complex scenes by the trained model. Summary of the Invention

[0004] This application provides an improved scene recognition method, vehicle control method, electronic device, and storage medium.

[0005] This application provides a scene recognition method, including: Acquire current perception data of the current scene collected by the vehicle's sensors; The current perceived data is input into the scene recognition model to output the scene type probability distribution; the scene recognition model is trained based on training data and two-dimensional labels of each basic scene and its corresponding dynamic interaction information corresponding to the scene type. Based on the probability distribution of the scene types, the current scene type containing the basic scene and its corresponding dynamic interaction information is determined.

[0006] Furthermore, determining the current scene type, which includes the basic scene and its corresponding dynamic interaction information, based on the scene type probability distribution includes: The scene type with the highest probability in the scene type probability distribution is taken as the current scene type.

[0007] Furthermore, the step of taking the scene type with the highest probability in the scene type probability distribution as the current scene type includes: when there are multiple scene types with the highest probability in the scene type probability distribution, taking any scene type with the highest probability as the current scene type; or, The step of taking the scene type with the highest probability in the scene type probability distribution as the current scene type includes: when there are multiple scene types with the highest probability in the scene type probability distribution, taking the scene type with the highest security weight among the multiple scene types with the highest probability as the current scene type.

[0008] Furthermore, determining the current scene type containing the basic scene and its corresponding dynamic interaction information based on the scene type probability distribution includes: taking the scene type corresponding to the scene type probability greater than the confidence threshold as the current scene type; or, Based on the probability distribution of the scene types, the current scene type, which includes the basic scene and its corresponding dynamic interaction information, is determined, including: If the scene type is determined to be a scene type that is less than the confidence threshold, then the scene type that is less than the confidence threshold is taken as the current scene type.

[0009] Furthermore, the method also includes training the scene recognition model in the following manner: Acquire first training data; the first training data includes: perception data collected by at least one sensor of the actual vehicle, and used to label each basic scene with a two-dimensional label corresponding to the scene type of the dynamic interaction information; The perceived data and the two-dimensional label of the scene type are input into the scene recognition model to be trained, so that the scene recognition model to be trained learns the scene type label of the first training data, and a trained scene recognition model is obtained, which is used as the scene recognition model.

[0010] Furthermore, the scene recognition model includes a feature extraction backbone network and a scene classification head; Environmental features are extracted from the perceived data through the feature extraction backbone network. Based on the environmental features, the scene classification head outputs the probability distribution of the scene type.

[0011] Furthermore, the step of extracting environmental features from the perceived data through the feature extraction backbone network includes: Through the feature extraction backbone network, the static element tensor and dynamic element tensor of the current scene are extracted from the collected perception data; The step of outputting a scene type probability distribution based on the environmental features using the scene classification head includes: The static element tensor and the dynamic element tensor are encoded respectively to obtain static features and dynamic features; The static features and the dynamic features are fused to obtain the fused features; The fused features are mapped to a probability distribution of scene types.

[0012] Furthermore, the encoding of the static element tensor and the dynamic element tensor respectively to obtain static features and dynamic features includes: Based on the attention mechanism, the association weights between the static features and the dynamic features are determined; Based on the association weights, attention weights are applied to the static features and the dynamic features to obtain weighted static features and weighted dynamic features. The weighted static features and the weighted dynamic features are fused to obtain the fused features.

[0013] Furthermore, acquiring the current perception data of the current scene collected by the vehicle's sensors includes: Acquire current perception data of the current scene collected by multiple sensors of the actual vehicle; the current perception data includes the current scene and the current dynamic interaction information in the current scene. The step of inputting the current perceived data into the scene recognition model to output a scene type probability distribution includes: The current scene and the current dynamic interaction information from the multiple sensors are synchronously input into the scene recognition model to output a scene type probability distribution.

[0014] Furthermore, the method also includes: constructing two-dimensional labels for scene types in the following manner: Acquire perception data for various scene types; Based on the correlation between each basic scenario and its corresponding dynamic interactive information, the perception data of each scenario type are labeled with two dimensions of basic scenario and dynamic dimension to obtain labeled data. The labeled data is digitized to obtain two-dimensional labels for the scene types corresponding to the dynamic interactive information in the basic scene.

[0015] This application provides a vehicle control method, including: The current scene type determined by the scene recognition method described above is fused with the perception data to form a fused feature that includes environmental information and scene context; The fused features are input into a guidance model with scene information, so that the guidance model outputs planning and control instructions that are adapted to the current scene type based on the decision preferences of the current scene type.

[0016] Furthermore, the method also includes training the guiding model in the following manner: Based on the two-dimensional labels of scene types in the second training data and the encoding features of the perception data collected by at least one sensor of the real vehicle in the second training data, a fusion feature containing environmental information and scene context is obtained; the two-dimensional labels of scene types are used to label each basic scene and its corresponding dynamic interaction information corresponding to the scene type. The fused features are input into the guided model to be trained, so that the guided model outputs a prediction result; Determine the weights of each loss term corresponding to each scenario type in the prediction results; According to the weights of each loss item and each loss item corresponding to each scenario type, the loss value of each loss item between the predicted result and the expected result is determined to obtain the total loss value; the total loss value is used to train the guided model to be trained. When the total loss value converges to a stable value, the iteration stops, and the trained pilot model is obtained, which serves as the pilot model.

[0017] Furthermore, in the process of training to obtain the guided model, the method further includes: The scene type truth value labeled during the perception data is used as the first truth value and the second truth value; The step of inputting the fused features into the guided model to be trained, so that the guided model outputs a prediction result, includes: The first true value is input into the guided model to learn decision preferences based on perceptual data and output the prediction result. The determination of the weights of each loss term corresponding to each scenario type of the prediction result includes: The second true value is then matched with the loss item weight corresponding to the current scene type flag according to the association rules corresponding to the scene type and loss item weight.

[0018] Furthermore, the step of matching the second truth value with the loss term weight corresponding to the current scene type flag according to the association rules corresponding to the scene type and loss term weight includes: Using the scene digital label of the current sample as an index, the loss item weight corresponding to the matching current scene type flag is matched from the association rules corresponding to the scene type and loss item weight; the association rule library is used to characterize the loss item weight allocation standard corresponding to each basic scene and its corresponding dynamic interaction information to form a sub-scene.

[0019] Furthermore, the weights of each loss item corresponding to each scenario type are obtained from the association rules corresponding to the scenario type and the weights of the loss items; The method further includes: Acquire perception data for various scene types; Based on the correlation between each basic scenario and its corresponding dynamic interactive information, the perception data of each scenario type are labeled with two dimensions of basic scenario and dynamic dimension to obtain labeled data. The labeled data is digitally labeled to obtain two-dimensional labels for the scene types corresponding to the dynamic interactive information in the basic scene. The association rule base is constructed based on each basic scenario, the corresponding dynamic interactive information, and the two-dimensional labels of the scenario types.

[0020] This application provides an electronic device, including one or more processors, for implementing the scene recognition method or the vehicle control method described above.

[0021] This application provides a computer-readable storage medium having a program stored thereon, which, when executed by a processor, implements the scene recognition method or the vehicle control method described above.

[0022] This application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in any of the preceding claims.

[0023] In some embodiments, the scene recognition method of this application, through a scene recognition model trained using training data and two-dimensional labels of each basic scene and its corresponding dynamic interaction information, can identify basic scenes and their dynamic interaction information. This allows for rapid differentiation of interactive actions within the current scene, thereby improving the understanding of complex scenes and leading to more accurate subsequent decisions.

[0024] In other embodiments, the scene recognition method of this application, after identifying the current scene type through a scene recognition model, performs feature fusion and then uses a guidance model to infer planning and control instructions adapted to the current scene. Thus, the sequential process of the scene recognition model and the guidance model enables online collaboration between perceived information and scene information, allowing the guidance model to adapt to the decision-making needs of different sub-scenes in real time. Attached Figure Description

[0025] Figure 1 The diagram shown is an overall flowchart of the scene recognition method and vehicle control method according to an embodiment of this application; Figure 2 The diagram shown is a flowchart illustrating the scene recognition method provided in an embodiment of this application. Figure 3 As shown Figure 2 The diagram shows the training process of the scene recognition model in the scene recognition method. Figure 4 The diagram shown is a schematic flowchart of the vehicle control method provided in an embodiment of this application; Figure 5 As shown Figure 4 The flowchart shown is a guided model training flowchart in the vehicle control method. Figure 6 The diagram shown is a structural schematic of an electronic device provided in an embodiment of this application. Detailed Implementation

[0026] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following representations relate to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments represented in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0027] It should be noted that in other embodiments, the steps of the corresponding method are not necessarily performed in the order shown and represented in this specification. In some other embodiments, the method may include more or fewer steps than represented in this specification. Furthermore, a single step represented in this specification may be broken down into multiple steps in other embodiments; and multiple steps represented in this specification may be combined into a single step in other embodiments.

[0028] Figure 1 The diagram shown is an overall flowchart of the scene recognition method and vehicle control method according to an embodiment of this application.

[0029] like Figure 1 As shown, firstly, perception data for each scene type is acquired. This perception data refers to real-vehicle sensor data with scene labels, which includes perception data collected by at least one sensor of the real vehicle, as well as two-dimensional labels used to annotate each basic scene and its corresponding dynamic interaction information corresponding to the scene type. The above perception data can be used as either the first training data or the second training data. The "first" in the first training data and the "second" in the second training data distinguish the training data for the scene recognition model to be trained and the guidance model to be trained.

[0030] Based on this, using the first training data, the scene recognition model to be trained can be trained during the offline training phase, enabling the trained scene recognition model to output specific scene types. Simultaneously, using the second training data, the guidance model to be trained can be trained during the offline training phase, enabling the trained guidance model to output dynamic decisions adapted to specific scene types.

[0031] The aforementioned decision-making and planning model is used to make inference decisions based on preset scenario decision preferences and fusion features, and to generate planning and control instructions that meet the needs of the current scenario. For example, the aforementioned decision-making and planning model may include, but is not limited to, the PDP model.

[0032] Continue as Figure 1 As shown, during the offline training phase, the scene recognition model and the guidance model to be trained are trained simultaneously, resulting in a trained scene recognition model for outputting standardized scene labels and a trained guidance model, such as a trained PDP model, for making decisions using the standardized scene labels. Then, the trained scene recognition model and the trained PDP model are deployed collaboratively to construct an integrated online inference system that integrates the trained scene recognition model and the trained PDP model, enabling dynamic decision-making in real-world vehicle scenarios.

[0033] Thus, through the offline training phase, standardized outputs for different scene types are built and adapted to the trained PDP model and scene recognition model, respectively. Through the online phase, the current scene type and the trained PDP model are combined to form a closed loop from data to deployment.

[0034] The PDP model in related technologies relies solely on perception data to autonomously determine scene attributes, making it difficult to efficiently distinguish the semantic differences of the same perception data in different scenarios. For example, when perception data shows that a vehicle in front in another lane has turned on its turn signal, the PDP model cannot quickly determine whether the behavior is "normal lane change on a highway" (the vehicle has the option to accelerate / not decelerate—lane change is prohibited, and decelerate to yield—lane change is allowed) or "lane merging or lane change based on navigation information" (requiring advance deceleration and leaving room for avoidance). This can easily lead to confusion in decision-making logic, resulting in insufficient understanding of complex scenarios by the trained model.

[0035] To address the technical problem of insufficient understanding of complex scenes by the PDP models in the aforementioned related technologies, this application provides a scene recognition method. The scene recognition model, trained using training data and two-dimensional labels representing each basic scene and its corresponding dynamic interaction information, can identify basic scenes and their dynamic interaction information. This allows for rapid differentiation of interactive actions within the current scene, thereby improving the understanding of complex scenes and leading to more accurate subsequent decisions.

[0036] Figure 2 The diagram shown is a flowchart of the scene recognition method provided in the embodiment of this application.

[0037] like Figure 2 As shown, the scene recognition method may include, but is not limited to, the following steps 110 to 130: Step 110: Obtain the current perception data of the current scene collected by the actual vehicle sensors.

[0038] The perceived data is used to represent the actual situation of the current scene. This perceived data refers to information representing at least two dimensions: the basic scene and its corresponding dynamic interaction information.

[0039] Therefore, the subdivided scenarios formed by combining detailed road segment information represented by the basic scenario and dynamic interaction information are used as the scenario types in this application's embodiments. Details are as follows: The aforementioned basic scenarios may include, but are not limited to, one or a combination of road environment features such as road type (e.g., highways, urban expressways, ordinary urban roads, rural roads, etc.) and road topology (e.g., straight roads, curves, ramps, intersections, roundabouts, tunnels, bridges, etc.). These basic scenarios are used to represent detailed road segment information, such as at least one of the following: straight sections of urban roads, intersections, ramp sections, and construction sections. Of course, the above are merely illustrative examples; any road environment features that can reflect the current basic scenarios are within the scope of protection of this application and are not specifically limited herein.

[0040] The aforementioned dynamic interaction information is used to characterize the behavioral interactions between various traffic participants in the basic scenario. This dynamic interaction information may include, but is not limited to, at least one of the following: the motion state of traffic participants and the status information of traffic control facilities (such as traffic lights and traffic signs and markings).

[0041] The movement states of the aforementioned traffic participants may include the driving state of vehicles (such as acceleration, deceleration, constant speed, stationary, turning, changing lanes and making a U-turn, etc. at least one), the movement state of pedestrians (such as walking, running, standing and crossing the road, etc. at least one), and the driving state of non-motorized vehicles (such as riding, stopping and turning, etc. at least one).

[0042] Specifically, the aforementioned dynamic interaction information includes, but is not limited to, instances of vehicles from adjacent lanes cutting in or overtaking, slow-moving large vehicles ahead, maneuvering around static obstacles, vehicles going straight and another vehicle turning right, vehicles turning right and another vehicle going straight, vehicles turning left and another vehicle turning right, vehicles turning right and another vehicle turning left, multiple vehicles turning side-by-side, vehicles merging from a ramp onto the main road, vehicles cutting into a ramp from the main road, other vehicles cutting into a ramp, vehicles approaching construction signs, and multiple vehicles merging into a narrow passage. Of course, the above are merely illustrative examples; any dynamic interaction information reflecting the current basic scenario is within the scope of protection of this application, and no specific limitations are made here.

[0043] In this way, dynamic interactive information and basic scenarios together constitute the complex behavioral relationships among traffic participants in the current scenario, making it easy to grasp the dynamic changes within the scenario type.

[0044] Step 120: Input the current perception data into the scene recognition model to output the scene type probability distribution; the scene recognition model is trained based on the training data and the two-dimensional labels of each basic scene and its corresponding dynamic interaction information.

[0045] The scenario type probability distribution represents the probability value of the current perceived data belonging to different scenario types. For example, when the vehicle in front of the perceived data in another lane activates its turn signal, the scenario recognition model outputs the probability distribution for each scenario type, based on the basic scenario of the perceived data (such as an intersection, ramp section, construction section, lane merging section of ordinary highway, or navigation-based lane change section, etc.) and dynamic interaction information (such as the relative speed, distance, and relative position of the vehicle and other vehicles). For example, the probability of the vehicle going straight and the other vehicle turning right at an intersection, the probability of the vehicle going straight and the other vehicle turning right at an ramp section, the probability of the vehicle going straight and the other vehicle turning right at a construction section, the probability of the vehicle going straight and the other vehicle turning right at a lane merging / lane change section, and the probability of the vehicle going straight and the other vehicle turning right at a location based on navigation information, etc. These probability values ​​quantify the degree of matching between the current perceived data and each scenario type, facilitating accurate subsequent determination of the scenario type.

[0046] The scene recognition model mentioned above that inputs the current perception data refers to a trained scene recognition model, which enables online identification of the specific scene type in the current scene, i.e., the current scene type.

[0047] Step 130: Based on the probability distribution of scene types, determine the current scene type that contains the basic scene and its corresponding dynamic interaction information.

[0048] In this embodiment, by inputting the current perceived data into a pre-trained scene recognition model, the model can effectively output a scene type probability distribution through dual learning of basic scene and dynamic interaction information. Based on these probability distributions, the specific scene type of the current scene can be accurately determined, thereby avoiding the decision-making logic confusion caused by relying solely on single perceived data in related technologies and improving the recognition accuracy of complex scenes.

[0049] Combination Figure 2 As shown, step 130 above can employ at least one of the following optional embodiments to determine the current scene type, which includes the basic scene and its corresponding dynamic interaction information: In a first optional embodiment, the scene type with the highest probability in the scene type probability distribution is taken as the current scene type.

[0050] For example, if the scene recognition model outputs a scene type probability distribution for a given current perception data, and the probability of a vehicle going straight and another vehicle turning right at an intersection is 0.65, the probability of a vehicle going straight and another vehicle turning right at a ramp is 0.08, the probability of a vehicle going straight and another vehicle turning right at a construction site is 0.02, the probability of a vehicle going straight and another vehicle turning right at a lane merging / changing scenario is 0.1, the probability of a vehicle going straight and another vehicle turning right based on navigation information is 0.03, the probability of a lane merging / changing scenario is 0.12, and the probability of a lane changing scenario based on navigation information is 0.03, then the sub-scene with the highest probability, "a vehicle going straight and another vehicle turning right at an intersection," is directly identified as the current scene type. A two-dimensional label for the scene type is used as the scene type identifier.

[0051] In this embodiment, the results of probability statistics can reflect the matching degree between the current perception data and the scene type to the greatest extent. By selecting the scene type with the highest probability value in the scene type probability distribution as the current scene type, the core category of the current scene can be determined intuitively and efficiently, thereby ensuring that the current scene type has high reliability and accuracy, and facilitating subsequent vehicle control based on the scene type.

[0052] In a second optional embodiment, if there are multiple scene types with the highest probability in the scene type probability distribution, the scene type with the highest probability is taken as the current scene type.

[0053] For example, the probability of a car going straight and another car turning right at an intersection is 0.45, the probability of a car going straight and another car turning right at a ramp is 0.45, and the probability of other scenario types is all lower than 0.45.

[0054] In this embodiment, when two or more scene types with the same and highest probability values ​​appear in the scene type probability distribution output by the scene recognition model, any one of these highest-probability scene types can be randomly selected as the current scene type. This effectively addresses the special case of a flat probability distribution and avoids decision-making stagnation caused by multiple scene types having the same highest probability. By allowing arbitrary selection, the continuity and real-time nature of the scene type determination process are ensured, guaranteeing the smooth operation of the vehicle control process. Furthermore, since these selected scene types themselves have the highest probability, even random selection largely ensures the rationality of the chosen scene type.

[0055] In a third optional embodiment, if there are multiple scene types with the highest probability in the scene type probability distribution, the scene type with the highest security weight among the multiple scene types with the highest probability is taken as the current scene type.

[0056] Safety weights are pre-set for different scenario types. These safety weights are used to quantify the safety risk level of vehicle driving under different scenario types. The higher the safety weight, the higher the requirements for safe vehicle driving under the corresponding scenario type, and the more conservative or more detailed control strategies need to be adopted.

[0057] In this embodiment, when multiple scene types with the same and highest probability values ​​appear in the scene type probability distribution output by the scene recognition model, a preset safety weight table is invoked. The safety weights corresponding to these highest-probability scene types are compared, and the scene type with the highest safety weight is selected as the current scene type. This facilitates decision-making from a safety perspective, prioritizing vehicle driving safety in complex or high-risk scenarios. By proactively selecting scene types with higher safety weights, the final control is more safety-oriented, thereby further reducing potential safety hazards and improving vehicle driving reliability.

[0058] In a fourth optional embodiment, the scene type corresponding to the scene type probability greater than the confidence threshold is taken as the current scene type.

[0059] The aforementioned confidence threshold represents the minimum level of confidence that the probability of a scene type should reach. Its specific value can be preset according to the needs and accuracy requirements of the actual application scenario. For example, if the confidence threshold is set to 0.7, then when the probability value of a certain scene type output by the scene recognition model is greater than or equal to 0.7, the scene type is considered to have sufficient confidence and can be directly identified as the current scene type.

[0060] In this embodiment of the application, by setting a reasonable confidence threshold, scene types with low probability and insufficient confidence can be effectively filtered out, avoiding interference with subsequent vehicle control decisions due to ambiguous or uncertain recognition results. This enables the determination of scene types quickly and directly while ensuring recognition accuracy, thus ensuring that the vehicle control strategy is clear and reliable.

[0061] Of course, the confidence threshold mentioned above can be flexibly set and adjusted according to actual application needs. For example, in areas or time periods with extremely high requirements for driving safety, the confidence threshold can be set to a higher value to ensure that when the scene recognition model has a very high degree of certainty in judging a certain scene type, it is taken as the current scene type, thereby avoiding potential risks caused by the uncertainty of the model's judgment.

[0062] In some regular driving scenarios with higher real-time requirements and a certain margin of error, the confidence threshold can be appropriately reduced to ensure the timeliness and continuity of scenario type recognition.

[0063] In a fifth optional embodiment, if the scene type is determined to be a scene type that is less than the confidence threshold, the scene type that is less than the confidence threshold is taken as the current scene type.

[0064] If no scene type reaches the aforementioned confidence threshold, a safety fallback decision can be added. This means that when the confidence level for all scene types is below the set threshold, a preset conservative control strategy is adopted by default, such as reducing vehicle speed, increasing safe following distance, and enhancing environmental perception frequency. This conservative decision-making approach can maximize vehicle safety when scene recognition results are uncertain, avoiding dangers caused by blindly relying on low-confidence recognition results.

[0065] In this embodiment, a safety fallback decision can be added when no scene type reaches the aforementioned confidence threshold. This not only ensures that vehicle control decisions can be quickly adapted to the scene type when the scene identification result is clear and the confidence level is sufficient, but also avoids the risk of decision interruption or loss of control due to missing scene information by using the low-confidence scene type as the current scene type in special cases where the confidence level of a scene type is lower than the set threshold.

[0066] Furthermore, compared to the end-to-end real-time response of the current scenario type input to the subsequent guidance model adaptation in related technologies, this allows the guidance model to call offline trained scenario-based decisions, while avoiding the risk of scenario recognition failure through safety fallback decisions, thus comprehensively improving the decision adaptability and safety of real vehicle operation.

[0067] As an optional embodiment of this application, the above method may also include, but is not limited to, the following steps 210 to 220, to train the scene recognition model: Step 210: Obtain first training data; the first training data includes: perception data collected by at least one sensor of the actual vehicle, and is used to label each basic scene with a two-dimensional label corresponding to the scene type of the dynamic interaction information.

[0068] The aforementioned first training data is multimodal perception data used to represent basic scene and dynamic interaction information, comprehensively reflecting the detailed traffic scene in which the vehicle is located, also known as segmented scene. The first training data uses sensor data collected from real vehicles, labeled with two-dimensional tags for basic scene and dynamic interaction.

[0069] Next, for each training data point, its corresponding basic scene type is manually labeled, along with detailed annotations of the dynamic interaction information within that basic scene. Then, the basic scene type and dynamic interaction information are combined to form a two-dimensional label for the scene category; that is, the label for each training sample consists of a basic scene label and a dynamic interaction information label.

[0070] The two-dimensional labels mentioned above not only clearly define the types of each basic scenario, but also correspond to the dynamic interactive information under each basic scenario.

[0071] Next, the aforementioned two-dimensional label refers to a label obtained by combining and annotating basic scene and dynamic interaction information on images in the perceived data. This two-dimensional label represents the combination relationship between the basic scene type and the dynamic interaction information; that is, a specific traffic scene instance can be uniquely identified through the two-dimensional label. For example, the two-dimensional label includes, but is not limited to, a digital label for an image, such as one-hot encoding. The two-dimensional dimension of the two-dimensional label represents both the dimension of the basic scene and the dimension of the dynamic interaction.

[0072] For example, when the basic scene type is detailed road segment information of urban roads, and the dynamic interaction information is vehicle-pedestrian interaction, or interaction between the vehicle and other vehicles, this two-dimensional label can easily correspond to the subdivided scene of vehicle-pedestrian interaction in the urban road environment. In this way, the first training data can characterize the features of different subdivided scenes, which facilitates the training of the scene recognition model and helps the scene recognition model learn the inherent relationship between the basic scene and dynamic interaction information, thereby improving the ability to recognize detailed traffic scenes.

[0073] Step 220: Input the perceived data and the two-dimensional label of the scene type into the scene recognition model to be trained, so that the scene recognition model to be trained learns the scene type label of the first training data, and obtains the trained scene recognition model, which is used as the scene recognition model.

[0074] Figure 3 As shown Figure 2 The diagram shows the training process of the scene recognition model in the scene recognition method.

[0075] like Figure 3 As shown, firstly, the scene recognition model is trained using the first training data with two-dimensional labels, and the scene recognition result, such as the predicted scene label, is output. Based on the two-dimensional label of the scene category and the scene recognition result, the scene classification loss is determined as follows: using the loss function, the loss value between the predicted scene label and the real two-dimensional label is calculated. Based on the convergence state of the loss value between the current iteration round and the previous round, it is determined whether to continue iterative training. When the loss value converges to a stable state and no longer decreases significantly, the iteration is stopped and the training is completed.

[0076] In the first example, the scene recognition model to be trained uses a multi-layered neural network structure to perform in-depth mining and feature learning on the data, continuously adjusting the parameters of the scene recognition model to minimize the error between the predicted scene type and the labeled two-dimensional label, and finally converges to obtain a scene recognition model that can accurately identify scene types.

[0077] In the second example, during the training phase of the scene recognition model to be trained, the cross-entropy loss function is used to continuously adjust the model parameters through the backpropagation algorithm, so that the scene type probability distribution output by the scene recognition model to be trained is as close as possible to the labeled two-dimensional labels, until the scene recognition model to be trained converges, thereby obtaining a scene recognition model that can accurately identify the basic scene and its dynamic interaction information.

[0078] The scene recognition model described in this paper may include, but is not limited to, at least one of the following: Convolutional Neural Network (CNN), Recurrent Neural Network (RNN), Long Short-Term Memory (LSTM), Transformer model and its variants, hybrid models of CNN and LSTM, and fusion architectures of Transformer model and CNN. Any scene recognition model that takes perceptual data or its fused vector information as input, can be trained offline to obtain a scene recognition model, and can output scene type information online and collaborate with a guided model to solve the problems of guided models being unable to adaptively match scene decision preferences and the disconnect between perceptual and scene information, falls within the scope of protection of this invention.

[0079] In this embodiment, the first training data reflects the detailed traffic scene in which the vehicle is located, and the two-dimensional labeling method enables the scene recognition model to be trained to simultaneously learn the static attributes and dynamic interaction features of the scene, thereby more accurately understanding and recognizing complex and ever-changing traffic scenes. Then, after inputting the perception data and two-dimensional labels into the scene recognition model to be trained, a scene recognition model capable of accurately identifying scene types is obtained. Thus, the training process of the scene recognition model, using a large amount of first training data under different scenarios and dynamic interaction conditions, ensures that the trained scene recognition model has strong generalization ability and recognition accuracy in practical applications.

[0080] Furthermore, the aforementioned scene recognition model includes a feature extraction backbone network and a scene classification head.

[0081] Step 310: Extract environmental features from the perceived data through the feature extraction backbone network.

[0082] The aforementioned feature extraction backbone network can adopt mature deep learning network architectures such as ResNet and MobileNet, which possess powerful feature abstraction capabilities. These architectures can extract environmental features from complex perceptual data (such as images and point clouds) layer by layer, from lower to higher levels, as well as the spatial relationships and contextual information between these elements. Thus, the feature extraction backbone network is responsible for extracting environmental features from the sensor data, while the scene classification head can output the scene category probability distribution.

[0083] Step 320: Using the scene classification head, output the probability distribution of the scene type based on the environmental features.

[0084] The aforementioned scene classification head consists of fully connected layers or convolutional layers. It receives environmental features from the output of the feature extraction backbone network and maps them to a scene type probability distribution through activation functions such as softmax. Each element in this probability distribution corresponds to a prediction probability of a specific scene type, thereby realizing a probabilistic judgment of the scene type in which the current vehicle is located.

[0085] Furthermore, by comparing the scene type probability distribution output by the scene recognition model with a preset scene judgment threshold, the specific scene type currently in which the vehicle is located can be accurately determined. This not only effectively addresses complex and ever-changing traffic environments but also improves the accuracy and robustness of recognition through continuous optimization of model parameters, ensuring that the vehicle responds appropriately in different scenarios.

[0086] In related technologies, when the model input parameters do not include scene type labels, the model relies solely on perceived data to autonomously determine scene attributes, making it difficult to efficiently and accurately distinguish the semantic differences of the same perceived data in different scenes. For example, when perceived data shows that a vehicle in the adjacent lane has activated its turn signal, the model cannot quickly determine whether the behavior is a normal lane change on a highway (the vehicle's options are acceleration / non-deceleration driving—lane change prohibited, and deceleration to yield—allowing the other vehicle to change lanes) or a lane merge or lane change based on navigation information (requiring advance deceleration and leaving room for avoidance), which can easily lead to confusion in decision-making logic and result in insufficient understanding of complex scenes by the trained model.

[0087] Compared to related technologies that struggle to efficiently and accurately distinguish the semantic differences of the same perceived data in different scenarios, leading to confusion in decision-making logic and insufficient understanding of complex scenarios by the trained model, in this embodiment, by independently training the scene recognition model, the mapping relationship between the scene and multi-sensor features can be learned in advance, generating standardized and high-precision scene type labels.

[0088] Thus, an independent scene recognition model is trained offline, employing a supervised learning and classification loss-guided strategy. Based on multi-sensor data from real vehicles and two-dimensional labels, it learns the mapping relationship between scenes and features, outputting standardized scene type labels. This scene recognition model can pre-model scene features, avoiding the inefficiency and bias of models autonomously mining scene information from perceived data. It can distinguish semantic differences of the same perceived data in different scenes, avoiding confusion in decision-making logic, and enabling the trained scene recognition model to improve its understanding of complex scenes. Furthermore, it facilitates the training and inference of subsequent guided models and ensures that the guided models have a unified and reliable scene type.

[0089] In combination with the above Figure 3 As shown, step 310 above may further include step 311, and step 320 above may further include steps 321 to 323: Step 311: Using the feature extraction backbone network, extract the static element tensor and dynamic element tensor of the current scene from the collected perception data.

[0090] The feature extraction backbone network can employ deep convolutional neural network architectures such as, but not limited to, ResNet and EfficientNet. Through multiple layers of convolution, pooling, and non-linear activation operations, it progressively abstracts features at different levels from raw perceptual data (such as images, point clouds, radar data, etc.). In this way, the feature extraction backbone network parses static and dynamic element tensors from complex perceptual data.

[0091] Static elements are used to determine the spatial location and morphological stability of static elements, providing basic information for scene recognition. The static element tensor includes relatively fixed objects in the scene, such as road markings, traffic lights, buildings, and trees. In this way, the features of these elements have strong spatial location and morphological stability, constructing detailed road segment information of the basic scene.

[0092] (1) Specifically, the input layer consists of static element tensors:

[0093] Where Ns is the number of static elements, such as lane lines, traffic signs, curbs, etc. Fs is the number of sampling points for each static element; Fs is the static feature dynamic element tensor.

[0094] The aforementioned dynamic element tensor is used to capture the multi-dimensional dynamic attributes of moving objects, such as the speed of vehicles and the direction of pedestrian movement, reflecting the real-time changes in the current scene.

[0095] Specifically, dynamic element tensors:

[0096] in, The number of dynamic elements, such as vehicles, large vehicles, pedestrians, two-wheeled vehicles, etc. The number of sampling times for dynamic elements; This is a dynamic feature dimension, including the element's three-dimensional coordinates x, y, z, velocity v, heading angle, acceleration acc, type encoding, etc.

[0097] Step 321: Encode the static element tensor and the dynamic element tensor respectively to obtain static features and dynamic features.

[0098] In the encoding process of step 321, spatial attention mechanism can be used to encode static element tensors to highlight the feature weights of key static areas, such as stop lines and signs at road intersections; for dynamic element tensors, time series encoding methods, such as LSTM (Long Short-Term Memory) or Transformer encoder structures, can be combined to capture the motion trends and temporal dependencies of dynamic objects.

[0099] (2) The static element tensor can be encoded in the following way: Perform sampling-point-level feature aggregation and feature encoding on the static element tensor to output static features, i.e.:

[0100] in, It represents the real number field, meaning that every element in the matrix is ​​a real number; The shape of the real number field (number of rows × number of columns) is represented in this article. The superscripts in the upper right corner all indicate the shape of the real number field (number of rows × number of columns), which will not be elaborated here.

[0101] The specific formula is as follows: 1. Feature aggregation of sampling points: For each static element Perform mean pooling on the features of each sampling point to obtain element-level static features:

[0102] 2. Feature Encoding: Feature dimensionality enhancement and normalization are achieved through fully connected layers and LayerNorm.

[0103] in This is a static branch weight matrix. This is the bias vector.

[0104] Furthermore, dynamic element tensors can be encoded in the following ways: Temporal feature encoding and feature upsizing are performed on the dynamic element tensor to output dynamic features, i.e.:

[0105] The specific formula is as follows: 1. Temporal Feature Encoding: A single-layer GRU is used for each dynamic element. Time-series encoding of features at each time point:

[0106] Take the GRU hidden state at the last moment as the element's temporal feature. ,in is the dimension of the GRU hidden layer.

[0107] 2. Feature Upsizing Encoding: Temporal features are mapped to a unified dimension D through a fully connected layer.

[0108] in This is a dynamic branch weight matrix. This is the bias vector.

[0109] Step 322: The static features and the dynamic features are fused to obtain fused features.

[0110] The fusion process in step 322 can be carried out by means of feature splicing, weighted summation, attention fusion or gating mechanism to fuse static features with dynamic features.

[0111] For example, through an adaptive gating unit, the contribution ratio of static and dynamic features in the fused features can be dynamically adjusted according to the degree of dynamic change in the current scene. When there are many dynamic objects in the scene and their movements are complex, the weight of dynamic features is increased, and vice versa.

[0112] Step 323: Map the fused features to a probability distribution of scene types.

[0113] In step 323 above, the fused features are mapped to probability distributions for various specific scene types. This can be achieved using a fully connected layer in conjunction with the Softmax activation function. Each probability value in the probability distribution corresponds to the confidence level of the fused feature belonging to a specific scene type. For example, if the probability value of "being overtaken in an urban road scene" is 0.85 and the probability value of "being overtaken in a highway scene" is 0.12 after mapping, and the probability values ​​for other scenes are lower, it indicates that the current scene is most likely to be identified as an urban road scene where someone is being overtaken.

[0114] In this embodiment, by encoding the different characteristics of static and dynamic elements separately, the feature representation of key static regions and the ability to capture the motion trends of dynamic objects are effectively enhanced. The subsequent fusion step ensures that the fused features comprehensively and selectively reflect the overall information of the current scene. Finally, the confidence level of the current scene belonging to different target categories is intuitively given, thereby achieving a precise mapping from raw perception data to specific scene types, facilitating subsequent vehicle control.

[0115] Step 321 above may include, but is not limited to, the following steps 3211 to 3213: Step 3211: Based on the attention mechanism, determine the association weights between the static features and the dynamic features.

[0116] The aforementioned attention mechanism captures the intrinsic relationship between static and dynamic features.

[0117] Specifically, in step 3211, the mutual influence between static and dynamic features in different dimensions and spatial locations is analyzed through the attention mechanism, and the correlation weight between the two is calculated.

[0118] These association weights can quantify the importance of static and dynamic features to the current scene recognition.

[0119] For example, in detailed road segment information, the association weight of dynamic features (such as at least one of vehicles approaching from the side or pedestrians crossing the road) will be increased accordingly to highlight their impact on the current scene type judgment.

[0120] In relatively simple scenarios such as highways, the association weight of static features (such as at least one of road type markings and guardrail styles) accounts for a large proportion.

[0121] Step 3212: Based on the association weight, the static features and the dynamic features are weighted by attention to obtain the weighted static features and the weighted dynamic features.

[0122] For features with high correlation weights, their contribution in the subsequent fusion process will be amplified, while the contribution of features with low weights will be weakened accordingly, thereby achieving the focus on key information and the suppression of noise information.

[0123] Step 3213: The weighted static features and the weighted dynamic features are fused to obtain the fused features.

[0124] The feature extraction network in this paper is used to extract basic scene features and dynamic interaction features from perceived data. The aforementioned fusion network deeply fuses these two types of features, and the classification network is used to output a scene type probability distribution based on the fused features. Each element in this probability distribution corresponds to a probability of a two-dimensional scene type composed of basic scene and dynamic interaction information.

[0125] (3) Cross-modal attention fusion layer The association weights between static and dynamic features are calculated using an attention mechanism, and the fused features are output. ; The specific formula is as follows: 1. Attention weight calculation:

[0126] 2. Feature Weighted Fusion: After applying attention weights to static and dynamic features separately, global mean pooling is then performed.

[0127]

[0128]

[0129] (4) Classification Output Layer The fused features are mapped to probability distributions for 15 scene categories, as shown in the following formula:

[0130]

[0131] in This is the output layer weight matrix. It is the bias vector; Based on the probability distribution of the 15 scene categories in Table 1, the category corresponding to the maximum probability is taken as the final scene recognition result and is used as the current scene type.

[0132] (5) Training loss function With cross-entropy loss as the optimization objective, the formula is:

[0133] Where B is the training batch size. Let b be the one-hot vector of the scene label for the b-th sample (the position corresponding to the true category is 1). Used to avoid overflow in logarithmic calculations. for one of the.

[0134] In this embodiment, deep integration is performed based on the weighted information of both, enabling the fused feature vector to simultaneously retain the structural information of the static environment and the behavioral trend information of dynamic objects, resulting in more comprehensive and discriminative scene classification. This allows for adaptive adjustment of feature contributions based on the actual association between static and dynamic elements in the scene, further improving the quality of the fused features and the accuracy of scene recognition.

[0135] Combination Figure 2 As shown, step 110 above may include, but is not limited to, step 111, and step 120 above may include, but is not limited to, step 121: Step 111: Obtain current perception data of the current scene collected by multiple sensors of the actual vehicle; the current perception data includes the current scene and the current dynamic interaction information of the current scene.

[0136] Step 121: Synchronously input the current scene and the current dynamic interaction information from the multiple sensors into the scene recognition model to output the scene type probability distribution.

[0137] In related technologies, the scenario type is not used as model input during the real-vehicle inference stage, preventing the model from accessing the scenario and decision-making logic learned during offline training. For example, when a real vehicle approaches an "unprotected left-turn intersection," the model cannot quickly switch to a "safety-first" decision mode based on the scenario type, and instead continues to use the decision-making logic of conventional roads, which can easily lead to aggressive driving and affect the adaptability and safety of decisions in real-world scenarios.

[0138] Compared to related technologies that do not use scene type as model input, which can easily lead to aggressive driving problems and affect the decision adaptability and safety in real-world scenarios, this application embodiment achieves end-to-end real-time response for scene perception and decision adaptation, allowing the guidance model to call the offline trained scenario-based decision logic, while avoiding the risk of scene recognition failure through safety fallback decisions, thus comprehensively improving the decision adaptability and safety of real-world vehicle operation.

[0139] In this embodiment, during actual vehicle operation, multi-sensor data is synchronously input into the scene recognition model, which outputs a current scene type flag, thereby improving the accuracy of scene type recognition. Subsequently, the current scene type flag is further fed into the guidance model in real time, triggering the scene adaptation decision learned during the offline training phase of the guidance model, achieving end-to-end response for scene perception decision adaptation. Simultaneously, a confidence threshold judgment of the scene recognition result is added; when the confidence level falls below the threshold, the system automatically switches to a safety fallback decision to ensure driving safety.

[0140] Furthermore, the offline-trained scene recognition model and the guidance model are deployed collaboratively to construct an integrated online inference process. By embedding the scene recognition model into the online inference architecture, real-time acquisition and input of scene types are achieved, allowing the guidance model to directly call the scene and decision-making association logic learned offline. Real-time multi-sensor data from the vehicle is input into the scene recognition model, and the output scene markers synchronously trigger the guidance model's scene adaptation decision.

[0141] As an embodiment of this application, the method further includes: constructing a two-dimensional label for the scene type using steps 410 to 430 as follows: Step 410: Obtain perception data for each scene type. The perception data is collected from the vehicle's sensors in the current scene.

[0142] Step 420: Based on the correlation between each basic scene and its corresponding dynamic interaction information, the perceived data of each scene type are annotated in two dimensions, namely the basic scene dimension and the dynamic dimension, to obtain annotated data. Thus, based on the two-dimensional classification of basic scenes and dynamic interactions, the collected perceived data is annotated by type, first identifying the basic scene dimension, then identifying the dynamic interaction dimension, ultimately forming annotated data of basic scenes and dynamic interactions. Finally, the annotated data is digitally labeled.

[0143] Step 430: Digitize the labeled data to obtain two-dimensional labels for the scene types corresponding to the dynamic interactive information in the basic scene.

[0144] In this embodiment, by constructing two-dimensional labels for scene types, more precise characterization of subdivided scenes can be achieved. Furthermore, the two-dimensional labels provide richer and more discriminative sample data for the subsequent training of the scene recognition model, thereby improving the model's ability to recognize complex dynamic scenes. This breaks through the limitations of traditional single-dimensional scene segmentation, resulting in more comprehensive and detailed scene representation.

[0145] Figure 4 The diagram shown is a schematic flowchart of the vehicle control method provided in an embodiment of this application.

[0146] like Figure 4 As shown, the vehicle control method may include, but is not limited to, the following steps 510 to 520: Step 510: The current scene type determined by the above scene recognition method is fused with the perception data to form a fused feature that includes environmental information and scene context.

[0147] The input to the guided model contains two types of information, which are encoded and fused into a unified feature: the perception information includes the feature tensors of collected static elements (lane lines, curbs, etc.) and the feature tensors of dynamic elements (vehicles, pedestrians, etc.), which serve as the environmental information input to the guided model; scene label fusion: the combined labels of the labeled basic scene and dynamic interaction, such as the numerical labels 1 and 15 in Table 1 below, are converted into one-hot encoded features, which are then weighted and fused with the encoded features of the perception information, enabling the guided model to directly obtain the contextual information of the current scene.

[0148] Step 520: Input the fused features into the guidance model with scene information so that the guidance model can output planning and control instructions that are adapted to the current scene type based on the decision preferences of the current scene type.

[0149] The aforementioned guidance model, through learning from a large amount of scenario data, can develop corresponding decision-making logic and behavioral patterns for different scenario types. For example, in congested urban road scenarios, decision preferences may place greater emphasis on smooth vehicle driving and following safety. In highway scenarios, however, the focus is more on driving efficiency and lane-keeping stability. When fused features are input into the guidance model, the model comprehensively analyzes environmental information and scenario context based on the decision preferences corresponding to the current scenario type, thereby outputting specific planning and control commands, including vehicle acceleration, deceleration, steering, and lane changes, to achieve adaptive and intelligent vehicle control in different scenarios.

[0150] To address this, the offline-trained scene recognition network is deployed on a real-vehicle computing platform and runs in real time in collaboration with the guidance model through integrated online inference of scene recognition and guidance model.

[0151] The guiding model in this paper may include, but is not limited to, the PDP model. The technical route of the PDP model has also gradually shifted from the traditional architecture containing rules and models to a more generalizable end-to-end (E2E) network architecture as the functional hierarchy has evolved.

[0152] In this embodiment, after the current scene type is identified by the scene recognition model, feature fusion is performed, and a planning and control command adapted to the current scene is obtained through reasoning by the guidance model. Thus, the sequential process of the scene recognition model and the guidance model enables online collaboration between perceived information and scene information, allowing the guidance model to adapt to the decision-making needs of different sub-scenes in real time.

[0153] As an optional embodiment of this application, the vehicle control method may also include, but is not limited to, training the guidance model using steps 610 to 650 as follows: Step 610: Based on the two-dimensional labels of scene types in the second training data and the encoding features of the perception data collected by at least one sensor of the real vehicle in the second training data, a fusion feature containing environmental information and scene context is obtained; the two-dimensional labels of scene types are used to label each basic scene and its corresponding dynamic interaction information corresponding to the scene type.

[0154] In step 610 above, the two-dimensional labels of scene types in the second training data are converted into one-hot encoded features. Then, the one-hot encoded features are weighted and fused with the encoded features of the perceived data to obtain fused features that include environmental information and scene context.

[0155] Step 620: Input the fused features into the guided model to be trained so that the guided model outputs a prediction result.

[0156] Step 630: Determine the weights of each loss term corresponding to each scenario type of the prediction result.

[0157] Step 640: Determine the loss value of each loss item between the predicted result and the expected result according to the weight of each loss item corresponding to each loss item and each scenario type, and obtain the total loss value; the total loss value is used to train the guided model to be trained.

[0158] Step 650: When the total loss value converges to a stable value, stop the iteration and obtain the trained pilot model, which is used as the pilot model.

[0159] In step 650 above, based on the convergence state of the total loss value obtained from the loss values ​​of the current iteration round and the previous round, it is determined whether to continue iterative training. When the total loss value converges to a stable state, the iteration is stopped, and the trained pilot model is obtained as the pilot model.

[0160] In this embodiment of the application, during the guided model training process, a preset loss term weight that is adapted to the current scenario can be determined, thereby realizing the scenario-based dynamic adjustment of the loss term weight.

[0161] As an optional embodiment of this application, in the process of training the guided model, the method may include, but is not limited to, the following step 710. Correspondingly, the above-mentioned step 620 may include, but is not limited to, the following step 720, and the above-mentioned step 630 may include, but is not limited to, the following step 730: Step 710: Use the scene type truth value annotated in the perceived data as the first truth value and the second truth value.

[0162] In this context, the truth value refers to the annotation information used in the process of labeling perceptual data to clearly indicate the primary scene type to which the perceptual data belongs. This truth value reflects the scene attributes of the perceptual data in actual applications. For example, if the perceptual data is collected from an urban road environment, the truth value may be labeled as an urban road scene. Its role is to provide a clear and dominant scene type reference for guiding model training, enabling the model to grasp the overall scene tone of the data during the learning process.

[0163] Next, the aforementioned truth values ​​include the first truth value and the second truth value. The "first" in the first truth value and the "second" in the second truth value are used to distinguish between the two truth values, so as to use the truth values ​​to realize feature input and automatic matching of preset weights for each loss term.

[0164] Step 720: Input the first true value into the guided model, learn decision preferences based on the perceived data, and output the prediction result.

[0165] Step 730: Match the second true value with the loss item weight corresponding to the current scene type flag according to the association rules corresponding to the scene type and loss item weight.

[0166] Related technologies, such as fixed-weight fusion schemes that use a single weight to cover the entire scenario and posterior incremental training schemes that rely on manual experience to adjust weights, lack an automatic association mechanism between scenario labels and loss term weights. For example, when training on multi-vehicle game scenarios at intersections, the model cannot automatically match weights for high-safety loss terms, and still uses efficiency-priority weights from high-speed scenarios, resulting in insufficient training of the model's safety decision-making ability in this scenario. At the same time, manual weight adjustment is easily limited by experience and cannot cover the different needs of all sub-scenarios.

[0167] Compared to related technologies that use a single weight to cover the entire scenario, resulting in insufficient training of the model's safety decision-making ability in that scenario; and the fact that manual adjustment of weights is easily limited by experience and cannot cover the different needs of all subdivided scenarios, in the embodiments of this application, on the one hand, scene markers are used as input features to enhance the model's understanding of scene semantics and improve training results; on the other hand, through the association mechanism of automatic matching between scene markers and weights, the scene-specific adaptation of loss term weights can be achieved without manual intervention, thereby guiding the model to train offline and explicitly utilizing scene ground truth through dual-path input feature and weight matching.

[0168] Thus, on the one hand, by fusing prior features with perceptual data, the guided model can quickly establish connections between perceptual features, scene attributes, and decision logic, strengthening its understanding of scene semantics. On the other hand, it automatically matches the current scene with the weighted association rule base, dynamically determining the preset loss term weights. This solves the shortcomings of related technologies where scene information is not explicitly involved in training and weight adjustment relies on manual intervention. It allows the guided model training to have both clear scene guidance and the ability to automatically adapt preset loss term weights to the scene, improving the training effect and efficiency of the guided model, thereby enhancing its safe decision-making capabilities. Simultaneously, it is not limited by manual weight adjustment, allowing for differentiated decision control across all sub-scenes.

[0169] The above step 730 may include, but is not limited to: using the scene digital label of the current sample as an index, matching the loss item weight corresponding to the matching current scene type flag from the association rules corresponding to the scene type and loss item weight; the association rule library is used to characterize the loss item weight allocation standard corresponding to each basic scene and its corresponding dynamic interaction information to form a subdivided scene.

[0170] Based on this, and leveraging engineers' deep understanding of driving strategies across different scenarios, the aforementioned association rule base for weight quantization is constructed according to scenario types, clearly defining the weight allocation standards for loss items corresponding to each sub-scenario. Thus, during training, using the scenario numerical label of the current sample as an index, the association rule base for weight quantization is automatically adjusted to match the corresponding scenario's safety loss item weights, comfort loss item weights, and efficiency loss item weights. These weights are then used to weight and sum at least one of the safety loss item, comfort loss item, and efficiency loss item in the guided model, achieving adjustment of loss preferences under different scenarios.

[0171] Figure 5 As shown Figure 4 The flowchart shown is a guide model training flowchart in the vehicle control method.

[0172] like Figure 5 As shown, the Adam (Adaptive Moment Estimation) optimizer is used to train the PDP model with a weighted total loss as the objective. Backpropagation is then performed based on the total loss value to update the network parameters of the PDP model, completing one iteration of training.

[0173] Combination Figure 1 As shown, the rule base for associating scenario types and weight allocation constructed above is illustrated in the following example: Table 1. Annotation results of scene types

[0174] continue Figure 5As shown, the aforementioned loss items may include, but are not limited to, at least one of safety loss, comfort loss, imitation loss, and efficiency loss. Of course, the loss items here are merely examples; any other loss items related to autonomous driving, planning and control, reinforcement learning, and end-to-end imitation learning fall within the protection scope of this application's embodiments, and will not be listed here individually. Furthermore, the above example of 15 digital tags is merely for illustration; the number of digital tags is not limited here and depends on the specific sub-scenario.

[0175] Accordingly, in Table 1 above, the respective weights of safety loss, comfort loss, and efficiency loss are used to represent the pre-set weights of the loss items, as shown in Table 1. Figure 5 The empirical weights shown, such as the relative weights of safety loss, comfort loss, and efficiency loss, indicate the specific weights that are emphasized among these loss items. For example, if the numerical label is 1, the corresponding weights for safety loss, comfort loss, and efficiency loss are [0.7, 0.2, 0.1], indicating that this scenario type emphasizes safety loss more.

[0176] The aforementioned safety losses are analyzed and assessed quantitatively using at least one indicator, such as the distance between the vehicle and the obstacle, relative speed, and time to collision (TTC), to determine the losses caused by collision risk, violation of traffic rules, or being in an unsafe driving state.

[0177] The aforementioned comfort loss is achieved by monitoring and evaluating at least one parameter, such as longitudinal acceleration, lateral acceleration, and jerk, to ensure the smoothness of vehicle operation and the comfort of passengers.

[0178] The aforementioned imitation loss is calculated by comparing the deviations between the autonomous vehicle's actual driving behavior and the human driver's demonstration behavior in the same or similar scenarios by comparing the autonomous vehicle's trajectory, speed curve, steering angle, accelerator and brake pedal operation with human demonstration data.

[0179] The aforementioned efficiency loss is quantified by comparing and analyzing the driving time, average speed, energy consumption indicators with at least one corresponding indicator under the optimal or desired state. This results in situations such as increased driving time, increased fuel consumption, or increased electricity consumption due to at least one of the following: unreasonable route planning, improper speed control, or failure to effectively utilize road resources.

[0180] For example, regarding safety losses, let's take the current scenario type, represented by the number 1, as an example of a dynamic interaction dimension where a pedestrian (VRU) is detected crossing a road in a straight-through urban environment. First, based on the aforementioned association rules, a preset weight of 0.7 is assigned to the safety loss item, 0.2 to the comfort loss item, and 0.1 to the efficiency loss item. In this case, the calculation of safety losses prioritizes the distance, relative speed, and time to collision (TTC) between the vehicle and the pedestrian (VRU). Assuming that the TTC value is calculated in real-time from sensor data to be 2.5 seconds, which is less than the safety threshold of 3 seconds, a high collision risk is determined. Therefore, the safety loss item is given a larger weight, driving the autonomous driving system to prioritize emergency braking or significant deceleration to reduce the collision risk. In this case, comfort losses (such as passenger discomfort from sudden braking) and efficiency losses (such as increased travel time due to deceleration) are relatively less important.

[0181] For example, consider the scenario of comfort loss, represented by the number 4, where the current scenario is a slow-moving large vehicle traveling straight in an urban road. Based on the aforementioned association rules, the pre-defined comfort loss item is matched with a weight of 0.5, higher than the safety loss (0.4) and efficiency loss (0.1). In this case, the longitudinal acceleration of the vehicle is continuously monitored when calculating the comfort loss. Thus, the autonomous driving system will tend to choose a smoother following strategy, or, under the premise of ensuring safety, complete the overtaking maneuver with smooth acceleration, rather than sacrificing ride comfort for efficiency.

[0182] Regarding the imitation loss, taking the current scenario type represented by digital label 2 as an example of a vehicle cutting in from the adjacent lane and making a cut or in maneuver, the actual steering angle and brake pedal force change curves of the autonomous vehicle after the cut or in maneuver are compared with the demonstration data of a human driver in the same scenario. In this case, a preset safety loss item with a weight of 0.6 is matched. While ensuring safety, the control strategy is adjusted to make the vehicle's operating behavior closer to the smooth operating mode of a human driver, thereby reducing the imitation loss and improving the naturalness of driving behavior.

[0183] Regarding efficiency loss, let's take the scenario of a vehicle merging from a ramp onto the main road, represented by the numerical label 11, as an example. Based on the aforementioned association rules, a preset efficiency loss item with a weight of 0.2 is matched. The actual travel time, the average speed during the merging process, and the desired state (e.g., merging safely at a relatively high speed without affecting main road traffic flow) are compared. Then, based on the weights of safety loss (0.6) and comfort loss (0.2), the merging decision is optimized while ensuring safety and passenger comfort. For example, a more suitable acceleration timing and lane clearance are selected to reduce travel time and energy consumption, thereby reducing efficiency loss.

[0184] In this application embodiment, a weighted association rule library based on the subdivided scenarios reflected by specific scenario types is used to achieve a balanced optimization of multiple objectives such as safety, comfort, efficiency, and mimicking human driving behavior.

[0185] In practical applications, scenario types are categorized into dimensions based on the target object (e.g., vehicle interaction), human interaction, and static roadside interaction. First, the target object dimension is determined, and then further subdivided based on the specific interactive behaviors or basic scenarios within that dimension. This allows for rapid matching of scenario types.

[0186] As an optional embodiment of this application, the weights of each loss item corresponding to each scenario type are obtained from the association rules corresponding to the scenario type and the weights of the loss items; The above method may also include, but is not limited to, the following steps 810 to 840: Step 810: Obtain perception data for each scenario type. This perception data includes vehicle status data and human driver decision-making and planning data.

[0187] Step 820: Based on the association between each basic scene and its corresponding dynamic interaction information, perform two-dimensional annotation of the perception data of each scene type in terms of the basic scene dimension and the dynamic dimension to obtain the annotated data.

[0188] Step 830: Digitize the labeled data to obtain two-dimensional labels for the scene types corresponding to the dynamic interactive information in the basic scene.

[0189] Step 840: Based on each basic scenario, the corresponding dynamic interaction information, and the two-dimensional labels of the scenario type, construct the association rule library for weight quantification.

[0190] This step 840 may include iteratively optimizing the weights of the loss terms corresponding to each scenario type using machine learning algorithms (such as gradient descent, genetic algorithms, etc.) based on human drivers' decision preferences for different scenario types and the risk levels of actual traffic scenarios from a large amount of historical driving data, and finally establishing a weighted association rule library.

[0191] In this embodiment, the construction process of the weighted association rule base is based on the degree of influence of dynamic interaction information under different basic scenarios on scene type recognition. Therefore, different basic scenarios and their dynamic interaction information are combined with annotations. Subsequently, the combined-annotated data undergoes digital encoding processing, transforming the basic scene type and dynamic interaction information type into computable two-dimensional label values. For example, different combinations of numbers are used to represent specific scene types to reflect subdivided scenes. In this way, the rule base can quickly match the corresponding loss term weights based on the input two-dimensional labels, which is beneficial for guiding model training and thus improving the accuracy and reliability of scene recognition.

[0192] The vehicles mentioned in this article may include, but are not limited to, one or more of heavy trucks, light commercial vehicles, and passenger cars.

[0193] The electronic devices mentioned in this article may include, but are not limited to, in-vehicle terminals connected to a vehicle. In-vehicle terminals connected to a vehicle may be, but are not limited to, body processors, center consoles, or automotive HUDs (Head and Up Displays).

[0194] Figure 6 The diagram shown is a structural schematic of the electronic device 900 provided in an embodiment of this application.

[0195] like Figure 6 As shown, the electronic device 900 includes one or more processors 901 for implementing the scene recognition method or the vehicle control method as described above.

[0196] In some embodiments, electronic device 900 may include storage medium 909. For example, a computer-readable storage medium may store a program that can be invoked by processor 901, and may include non-volatile storage medium. In some embodiments, electronic device 900 may include memory 908 and interface 907. In some embodiments, electronic device 900 may also include other hardware depending on the specific application.

[0197] The computer-readable storage medium of this application embodiment stores a program that, when executed by processor 901, is used to implement the scene recognition method or the vehicle control method as described above.

[0198] This application provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the method described in any of the preceding claims.

[0199] This application also provides a computer program stored in a computer-readable storage medium, for example... Figure 6The storage medium 909, and when the processor executes the computer program, causes the processor 901 to execute the method described above.

[0200] This application may take the form of a computer program product implemented on one or more computer-readable storage media (including but not limited to disk storage, CD and ROM, optical storage, etc.) containing program code. Computer-readable storage media include permanent and non-permanent, removable and non-removable media, and information storage can be implemented by any method or technology. Information may be computer-readable instructions, data structures, program modules, or other data. Examples of computer-readable storage media include, but are not limited to: phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, optical disc read-only memory (CD and ROM), digital versatile optical disc (DVD) or other optical storage, magnetic tape, disk storage or other magnetic storage devices, or any other non-transfer medium that can be used to store information accessible by a computing device.

[0201] The above description is merely a preferred embodiment of this specification and is not intended to limit this specification. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of protection of this specification.

[0202] It should also be noted that the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus. Without further limitation, an element qualified by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

Claims

1. A scene recognition method, characterized in that, include: Acquire current perception data of the current scene collected by the vehicle's sensors; The current perceived data is input into the scene recognition model to output the scene type probability distribution; The scene recognition model is trained based on training data and two-dimensional labels of each basic scene and its corresponding dynamic interaction information corresponding to the scene type. Based on the probability distribution of the scene types, the current scene type containing the basic scene and its corresponding dynamic interaction information is determined.

2. The scene recognition method as described in claim 1, characterized in that, The step of determining the current scene type, which includes the basic scene and its corresponding dynamic interaction information, based on the scene type probability distribution, includes: The scene type with the highest probability in the scene type probability distribution is taken as the current scene type.

3. The scene recognition method as described in claim 2, characterized in that, The step of taking the scene type with the highest probability in the scene type probability distribution as the current scene type includes: when there are multiple scene types with the highest probability in the scene type probability distribution, taking any scene type with the highest probability as the current scene type; or, The step of taking the scene type with the highest probability in the scene type probability distribution as the current scene type includes: when there are multiple scene types with the highest probability in the scene type probability distribution, taking the scene type with the highest security weight among the multiple scene types with the highest probability as the current scene type.

4. The scene recognition method as described in claim 1, characterized in that, The step of determining the current scene type containing the basic scene and its corresponding dynamic interaction information based on the scene type probability distribution includes: taking the scene type corresponding to the scene type probability greater than the confidence threshold as the current scene type; or, Based on the probability distribution of the scene types, the current scene type, which includes the basic scene and its corresponding dynamic interaction information, is determined, including: If the scene type is determined to be a scene type that is less than the confidence threshold, then the scene type that is less than the confidence threshold is taken as the current scene type.

5. The scene recognition method according to any one of claims 1 to 4, characterized in that, The method further includes training the scene recognition model using the following method: Acquire first training data; the first training data includes: perception data collected by at least one sensor of the actual vehicle, and used to label each basic scene with a two-dimensional label corresponding to the scene type of the dynamic interaction information; The perceived data and the two-dimensional label of the scene type are input into the scene recognition model to be trained, so that the scene recognition model to be trained learns the scene type label of the first training data, and a trained scene recognition model is obtained, which is used as the scene recognition model.

6. The scene recognition method according to any one of claims 1 to 4, characterized in that, The scene recognition model includes a feature extraction backbone network and a scene classification head; Environmental features are extracted from the perceived data through the feature extraction backbone network. Based on the environmental features, the scene classification head outputs the probability distribution of the scene type.

7. The scene recognition method as described in claim 6, characterized in that, The step of extracting environmental features from the perceived data through the feature extraction backbone network includes: Through the feature extraction backbone network, the static element tensor and dynamic element tensor of the current scene are extracted from the collected perception data; The step of outputting a scene type probability distribution based on the environmental features using the scene classification head includes: The static element tensor and the dynamic element tensor are encoded respectively to obtain static features and dynamic features; The static features and the dynamic features are fused to obtain the fused features; The fused features are mapped to a probability distribution of scene types.

8. The scene recognition method as described in claim 7, characterized in that, The step of encoding the static element tensor and the dynamic element tensor respectively to obtain static features and dynamic features includes: Based on the attention mechanism, the association weights between the static features and the dynamic features are determined; Based on the association weights, attention weights are applied to the static features and the dynamic features to obtain weighted static features and weighted dynamic features. The weighted static features and the weighted dynamic features are fused to obtain the fused features.

9. The scene recognition method according to any one of claims 1 to 4, characterized in that, The acquisition of current perception data in the current scene collected by the vehicle's sensors includes: Acquire current perception data of the current scene collected by multiple sensors of the actual vehicle; the current perception data includes the current scene and the current dynamic interaction information in the current scene. The step of inputting the current perceived data into the scene recognition model to output a scene type probability distribution includes: The current scene and the current dynamic interaction information from the multiple sensors are synchronously input into the scene recognition model to output a scene type probability distribution.

10. The scene recognition method according to any one of claims 1 to 4, characterized in that, The method further includes: constructing two-dimensional labels for scene types using the following method: Acquire perception data for various scene types; Based on the correlation between each basic scenario and its corresponding dynamic interactive information, the perception data of each scenario type are labeled with two dimensions of basic scenario and dynamic dimension to obtain labeled data. The labeled data is digitized to obtain two-dimensional labels for the scene types corresponding to the dynamic interactive information in the basic scene.

11. A vehicle control method, characterized in that, include: The current scene type determined by the scene recognition method according to any one of claims 1 to 10 is fused with the perception data to form a fused feature that includes environmental information and scene context; The fused features are input into a guidance model with scene information, so that the guidance model outputs planning and control instructions that are adapted to the current scene type based on the decision preferences of the current scene type.

12. The vehicle control method as described in claim 11, characterized in that, The method further includes training the guided model in the following manner: Based on the two-dimensional labels of scene types in the second training data and the encoding features of the perception data collected by at least one sensor of the real vehicle in the second training data, a fusion feature containing environmental information and scene context is obtained; the two-dimensional labels of scene types are used to label each basic scene and its corresponding dynamic interaction information corresponding to the scene type. The fused features are input into the guided model to be trained, so that the guided model outputs a prediction result; Determine the weights of each loss term corresponding to each scenario type in the prediction results; According to the weights of each loss item and each loss item corresponding to each scenario type, the loss value of each loss item between the predicted result and the expected result is determined to obtain the total loss value; the total loss value is used to train the guided model to be trained. When the total loss value converges to a stable value, the iteration stops, and the trained pilot model is obtained, which serves as the pilot model.

13. The vehicle control method as described in claim 12, characterized in that, In the process of training to obtain the guided model, the method further includes: The scene type truth value labeled during the perception data is used as the first truth value and the second truth value; The step of inputting the fused features into the guided model to be trained, so that the guided model outputs a prediction result, includes: The first true value is input into the guided model to learn decision preferences based on perceptual data and output the prediction result. The determination of the weights of each loss term corresponding to each scenario type of the prediction result includes: The second true value is then matched with the loss item weight corresponding to the current scene type flag according to the association rules corresponding to the scene type and loss item weight.

14. The vehicle control method as described in claim 13, characterized in that, The step of matching the second true value with the loss term weight corresponding to the current scene type flag according to the association rules corresponding to the scene type and loss term weight includes: Using the scene digital label of the current sample as an index, the loss item weight corresponding to the matching current scene type flag is matched from the association rules corresponding to the scene type and loss item weight; the association rule library is used to characterize the loss item weight allocation standard corresponding to each basic scene and its corresponding dynamic interaction information to form a sub-scene.

15. The vehicle control method according to any one of claims 12 to 14, characterized in that, The weights of each loss term corresponding to each scenario type are obtained from the association rules corresponding to the scenario type and the weights of the loss terms; The method further includes: Acquire perception data for various scene types; Based on the correlation between each basic scenario and its corresponding dynamic interactive information, the perception data of each scenario type are labeled with two dimensions of basic scenario and dynamic dimension to obtain labeled data. The labeled data is digitally labeled to obtain two-dimensional labels for the scene types corresponding to the dynamic interactive information in the basic scene. The association rule base is constructed based on each basic scenario, the corresponding dynamic interactive information, and the two-dimensional labels of the scenario types.

16. An electronic device, characterized in that, It includes one or more processors for implementing the scene recognition method as described in any one of claims 1 to 10 or the vehicle control method as described in any one of claims 11 to 15.

17. A computer-readable storage medium, characterized in that, It stores a program that, when executed by a processor, implements the scene recognition method as described in any one of claims 1 to 10 or the vehicle control method as described in any one of claims 11 to 15.

Images