Model training method, automatic driving control method, device, equipment and medium

Abstract

This application discloses a model training method, an autonomous driving control method, a device, an equipment, and a medium. The model training method includes acquiring simulated vehicle state data, a simulated bird's-eye view, a simulated environment image, and preset semantic description text; performing semantic segmentation and data compression on the simulated environment image to obtain a simulated low-dimensional latent feature vector; determining target simulation features based on multiple consecutive frames of simulated low-dimensional latent feature vectors and simulated vehicle state data; extracting features from the simulated bird's-eye view and the preset semantic description text to obtain a first vector and a second vector; determining a semantic reward based on the first and second vectors; and training an Actor-Critic network based on the total reward and the target simulation features to obtain a trained target policy network, where the total reward is obtained from the semantic reward and the basic reward. This application can improve training efficiency and decision robustness during the training phase, and reduce edge-side inference latency and meet the real-time requirements of in-vehicle systems during the deployment phase.

Description

Technical Field

[0001] This application relates to the field of autonomous driving, and in particular to a model training method, an autonomous driving control method, a device, equipment, and a medium. Background Technology

[0002] Autonomous driving decision-making methods are the core of a vehicle's ability to drive autonomously.

[0003] Existing mainstream methods, such as decision-making methods based on deep reinforcement learning (DRL), can learn complex policies, but they still face two major bottlenecks in practical applications: First, it is difficult to design reward functions, making it difficult to convert high-level semantic intents (such as "safety" and "courtesy") into numerical signals, resulting in unstable policy training, especially in dynamic interactive scenarios where they are prone to getting stuck in local optima; Second, it is difficult to deploy models, as complex models designed to pursue performance (such as visual language models) have high computational overhead and are difficult to meet the real-time requirements of in-vehicle edge devices, resulting in a "simulation-reality" gap. Summary of the Invention

[0004] The purpose of this application is to provide a model training method, an autonomous driving control method, a device, equipment, and a medium that can improve training efficiency and decision robustness, reduce edge inference latency, and meet the real-time requirements of in-vehicle systems.

[0005] To achieve the above objectives, this application provides the following solution: Firstly, this application provides a model training method, including: Based on an autonomous driving simulation simulator, the simulation vehicle status data, simulation bird's-eye view, simulation environment images captured by the on-board camera during the simulation driving process are obtained, as well as the preset semantic description text corresponding to the simulation bird's-eye view. The simulation vehicle status data includes simulation vehicle control data, simulation vehicle speed, and simulation vehicle lane center offset. Semantic segmentation is performed on the simulation environment image to obtain the simulation target segmentation result, which is used to indicate the position of lanes and lane lines in the simulation environment image. The simulation target segmentation result is compressed to obtain the simulation low-dimensional latent feature vector. The simulation low-dimensional latent feature vector of multiple consecutive frames is concatenated with the simulation vehicle state data of the corresponding multiple consecutive frames to obtain the target simulation feature. Visual semantic features are extracted from the simulated bird's-eye view to obtain a first vector; textual semantic features are extracted from the preset semantic description text corresponding to the simulated bird's-eye view to obtain a second vector; the similarity between the first vector and the second vector is calculated to obtain a semantic reward. Based on the total reward and target simulation features, the Actor-Critic network is trained to obtain a trained target policy network. The policy network in the Actor-Critic network includes a temporal coding module. The input of the policy network is the target simulation features, and the output is vehicle control data. The total reward is obtained based on semantic reward and basic reward. The basic reward is used to indicate the degree to which the simulated vehicle meets the expected driving goal during the simulated driving process.

[0006] Furthermore, the predefined semantic description text includes both positive and negative text; The text semantic features of the preset semantic description text corresponding to the simulated bird's-eye view are extracted to obtain the second vector, including: extracting features from the positive text and negative text corresponding to the simulated bird's-eye view respectively to obtain the second vector, which includes a positive text vector and a negative text vector; The similarity between the first vector and the second vector is calculated to obtain a semantic reward, including: calculating the similarity between the first vector and the positive text vector to obtain a positive similarity; calculating the similarity between the first vector and the negative text vector to obtain a negative similarity; and determining the semantic reward based on the positive and negative similarities.

[0007] Furthermore, based on the total reward and target simulation characteristics, the Actor-Critic network is trained to obtain a trained target policy network, including: Based on the total reward and target simulation characteristics, the temporal difference error is determined; based on the temporal difference error, the value network parameters are updated; based on the temporal difference error, the advantage function, and the probability ratio of new and old actions, a truncated agent objective function is constructed; based on the truncated agent objective function, the policy network parameters are updated until the policy network converges.

[0008] Furthermore, the basic rewards include speed rewards, direction alignment rewards, lane centering rewards, and collision penalties.

[0009] Furthermore, the timing coding module is an LSTM module.

[0010] Secondly, this application provides an autonomous driving control method, including: Acquire real vehicle status data and real environment images captured by onboard cameras during the autonomous vehicle's operation. Real vehicle status data includes real vehicle control data, real vehicle speed, and real vehicle lane center offset. Semantic segmentation is performed on the real environment image to obtain the real target segmentation result, which is used to indicate the position of lanes and lane lines in the real environment image; the real target segmentation result is compressed to obtain the real low-dimensional latent feature vector; the real low-dimensional latent feature vector of multiple consecutive frames is concatenated with the real vehicle state data of multiple consecutive frames to obtain the real target features. Based on the target's true characteristics, and using the target policy network obtained from the model training method in the first aspect, vehicle control commands for autonomous vehicles are generated.

[0011] Thirdly, this application provides an automatic driving control device, comprising: The acquisition module is used to acquire real vehicle status data and real environment images captured by the on-board camera during the autonomous vehicle's driving process. The real vehicle status data includes real vehicle control data, real vehicle speed, and real vehicle lane center offset. The feature fusion module is used to perform semantic segmentation on real environment images to obtain real target segmentation results. The real target segmentation results are used to indicate the positions of lanes and lane lines in real environment images. The real target segmentation results are compressed to obtain real low-dimensional latent feature vectors. The real low-dimensional latent feature vectors of multiple consecutive frames are concatenated with the real vehicle state data of multiple consecutive frames to obtain the real features of the target. The processing module is used to generate vehicle control commands for autonomous vehicles based on the target policy network obtained by the model training method of the first aspect, according to the real features of the target.

[0012] Fourthly, this application provides a computer device, including: a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the autonomous driving control method described in the second aspect above.

[0013] Fifthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the steps of the autonomous driving control method of the second aspect described above.

[0014] In a sixth aspect, this application provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the autonomous driving control method described in the second aspect.

[0015] According to the specific embodiments provided in this application, the following technical effects are disclosed: This application provides a model training method that extracts features from a simulated bird's-eye view and its corresponding preset semantic description text, calculates the similarity of the corresponding features, and obtains a semantic reward. Since the simulated bird's-eye view highly matches how humans describe driving scenarios, and the semantic information is intuitive, the semantic reward obtained based on the simulated bird's-eye view and its corresponding preset semantic description text can indicate the degree of matching between the vehicle's current behavior and the preset semantic driving goal. Furthermore, by fusing the semantic reward with the basic reward, the policy network in the Actor-Critic network is optimized. The semantic reward can transform high-level semantic intent into numerical signals, thereby improving training efficiency and decision robustness, resulting in the target policy network. Through the temporal coding module in the policy network, the smoothness and stability of the generated control actions can be significantly improved. Simultaneously, during the deployment phase, only the target policy network can be retained, reducing edge-side inference latency and meeting the real-time requirements of in-vehicle systems. Attached Figure Description

[0016] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0017] Figure 1 This is a diagram illustrating the application environment of the model training method provided in the embodiments of this application. Figure 2 A schematic flowchart illustrating the model training method provided in this application embodiment; Figure 3 This is a schematic diagram of the structure of the policy network provided in the embodiments of this application; Figure 4 This is a schematic diagram of the overall training-deployment scheme provided in the embodiments of this application; Figure 5 This is a schematic diagram of the functional modules of the autonomous driving control device provided in the embodiments of this application; Figure 6 This is a schematic diagram of the structure of a computer device provided in an embodiment of this application. Detailed Implementation

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0019] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0020] The model training method provided in this application embodiment can be applied to, for example... Figure 1 In the application environment shown, terminal 102 communicates with server 104 via a network. A data storage system can store the data that server 104 needs to process. The data storage system can be set up independently, integrated into server 104, or placed in the cloud or on another server. Terminal 102 can send vehicle simulation data to server 104. After receiving the vehicle simulation data, server 104 trains the Actor-Critic network to be trained based on the total reward obtained from the autonomous driving simulation simulator and the vehicle simulation data, obtaining the trained target policy network. Server 104 can provide feedback of the vehicle simulation data to terminal 102. Furthermore, in some embodiments, the model training method can be implemented independently by server 104 or terminal 102. For example, terminal 102 can directly train the model based on vehicle simulation data, or server 104 can obtain vehicle simulation data from the data storage system and then train the model based on the vehicle simulation data.

[0021] The terminal 102 can be, but is not limited to, various desktop computers, laptops, etc. The server 104 can be implemented using a standalone server or a server cluster composed of multiple servers, or it can be a cloud server.

[0022] In one exemplary embodiment, such as Figure 2 As shown, a model training method is provided. This method is executed by a computer device, specifically, it can be executed by a computer device such as a terminal or a server alone, or it can be executed by both a terminal and a server. In this embodiment, the method is applied to... Figure 1 Taking server 104 as an example, the following steps are taken: S201 to S204.

[0023] S201. Based on the autonomous driving simulation simulator, acquire the simulation vehicle status data, simulation bird's-eye view, simulation environment images captured by the on-board camera during the simulation driving process, and acquire the preset semantic description text corresponding to the simulation bird's-eye view.

[0024] The simulated vehicle status data includes simulated vehicle control data, simulated vehicle speed, and simulated vehicle lane center offset.

[0025] For example, the autonomous driving simulation simulator can be the CARLA simulator. In the environment initialization phase, the autonomous vehicle is first generated and multiple types of sensors are bound; then, according to the experimental settings, NPC vehicles, pedestrians and random obstacles are introduced to form a road scene with dynamic disturbances.

[0026] It is understandable that the simulated vehicle state data is the underlying truth state directly provided by the CARLA simulator; the simulated bird's-eye view is a global perspective image taken vertically downwards from directly above the vehicle, which can show the vehicle's position on the road, its relative relationship with lane lines, and the distribution of surrounding traffic participants without obstruction. It is highly consistent with the way humans describe driving scenarios, and the semantic information is intuitive; the preset semantic description text is used to describe the driving state of the simulated vehicle in the corresponding simulated bird's-eye view.

[0027] For example, the simulated vehicle control data may include parameters that directly control the vehicle's movement, such as the steering wheel angle and accelerator / brake pedal opening, and there are no restrictions on this.

[0028] For example, the default semantic description text could be "The car is driving in its own lane".

[0029] S202. Perform semantic segmentation on the simulation environment image to obtain the simulation target segmentation result; compress the simulation target segmentation result to obtain the simulation low-dimensional latent feature vector; concatenate the simulation low-dimensional latent feature vector of multiple consecutive frames with the simulation vehicle state data of multiple consecutive frames to obtain the target simulation feature.

[0030] The simulation target segmentation results are used to indicate the positions of lanes and lane lines in the simulation environment image.

[0031] For example, semantic segmentation of simulation environment images can be achieved using existing, pre-trained image semantic segmentation models (such as the TwinLiteNetPlus(large) network) to obtain the segmentation results of the simulation target; the input of the image semantic segmentation model is RGB image data (i.e., simulation environment image), and the output is the segmentation results of lanes and lane lines.

[0032] For example, data compression of the simulated target segmentation results can be achieved using an existing, pre-trained variational autoencoder (VAE) to obtain a simulated low-dimensional latent feature vector; the input of the variational autoencoder is a semantic segmentation image (i.e., the simulated target segmentation result), and the output is a low-dimensional latent feature vector (95 dimensions).

[0033] It is understandable that when performing feature stitching, the number of frames corresponding to the simulated low-dimensional latent feature vector is the same as the number of frames corresponding to the simulated vehicle state data, and there is no limit to the size of the frame number.

[0034] For example, the simulated low-dimensional latent feature vectors of 5 consecutive frames can be concatenated with the simulated vehicle state data of the corresponding 5 consecutive frames to obtain the target simulation features.

[0035] S203. Extract visual semantic features from the simulated bird's-eye view to obtain a first vector; extract textual semantic features from the preset semantic description text corresponding to the simulated bird's-eye view to obtain a second vector; calculate the similarity between the first vector and the second vector to obtain a semantic reward.

[0036] For example, the first vector and the second vector can be obtained by using the image encoder (such as ViT-B / 16) and text encoder (such as Transformer) in the existing, pre-trained Vision-Language Model (VLM) (such as the OpenCLIP model) to extract visual semantic features from the simulated bird's-eye view and extract text semantic features from the preset semantic description text corresponding to the simulated bird's-eye view.

[0037] It is understandable that, besides the OpenCLIP model, other dual encoder models capable of outputting cross-modal similarity between images and text, such as ALIGN and Flava, can be used. It should be noted that, to ensure the stability and interpretability of the reward signal, contrastive learning-based dual encoder models should still be given priority.

[0038] For example, the similarity between the first vector and the second vector can be calculated using the following formula: In the formula, Represents the first vector With the second vector Similarity; This represents the Euclidean norm.

[0039] For example, the similarity between the first vector and the second vector can be directly determined as the semantic reward.

[0040] S204. Based on the total reward and target simulation characteristics, train the Actor-Critic network to obtain the trained target policy network.

[0041] The policy network in the Actor-Critic network includes a temporal coding module. The input of the policy network is the target simulation features, and the output is vehicle control data. The total reward is obtained based on semantic reward and basic reward. The basic reward is used to indicate the degree to which the simulated vehicle meets the expected driving goal during the simulation driving process.

[0042] For example, the basic reward may include a speed reward. Orientation alignment reward Lane centering reward and collision penalty Basic Rewards The calculation formula is as follows: In the formula, , , , These are the weighting coefficients for each reward, and all can be preset values.

[0043] For example, total reward It can be determined by the following formula: In the formula, For semantic reward The weighting coefficients can be preset values.

[0044] For example, taking the target simulation features as an example, the Actor-Critic network can include three layers connected in sequence: a fully connected layer, a temporal coding module (for example, the temporal coding module can be an LSTM layer), an action fully connected layer, an action output layer, a value fully connected layer, and a value output layer. The three fully connected layers have input dimensions of 500, 300, and 100 respectively. The output of the third fully connected layer is the input of the temporal coding module, which can be an LSTM layer with a hidden layer size of 64. The output of the temporal coding module is the input of the action fully connected layer (input dimension 64) and the value fully connected layer (input dimension 64). The output of the action fully connected layer is the input of the action output layer, and the output of the value fully connected layer is the input of the value output layer. The output dimension of the action output layer is 2-dimensional (steering wheel angle and accelerator pedal opening or brake pedal opening), and the output dimension of the value output layer is 1-dimensional (state value).

[0045] For example, the timing coding module can also be a gated loop unit (GRU) or a Transformer timing encoder.

[0046] It can be understood that in the aforementioned Actor-Critic network, the policy network and the value network share three fully connected layers and a temporal coding module. The structure of the policy network is as follows: Figure 3 As shown.

[0047] It should be noted that the policy network and value network in an Actor-Critic network can also be independent networks, and there are no restrictions on this.

[0048] For example, the training process of an Actor-Critic network generally includes: inputting the target simulation features (i.e., the current state) into the Actor-Critic network; the action output layer outputs actions; the autonomous driving simulation simulator executes the actions and obtains a basic reward, which is combined with the semantic reward to obtain the total reward; the value output layer outputs the state value of the current state and the state value of the next time step (i.e., after executing the action); the temporal difference error is determined based on the state value of the current state, the state value of the next time step, and the total reward; the square of the temporal difference error is minimized by gradient descent to update the value network parameters; a loss function is constructed based on the temporal difference error; the gradient of the policy network parameters is calculated by backpropagation; the policy network parameters are updated by gradient ascent until the policy network converges (i.e., the gradient approaches 0).

[0049] It should be noted that the general training process of Actor-Critic networks is a conventional technique in this field and will not be elaborated here.

[0050] For example, environmental enforceability constraints can be imposed during training. These constraints may include vehicle physical limit constraints (such as steering constraints, longitudinal control constraints, acceleration constraints, angular velocity constraints, etc.), road geometry constraints (such as road boundary constraints, lane line constraints, intersection passage constraints, etc.), safety distance constraints (such as forward vehicle safety distance, lateral safety distance, collision warning constraints, etc.), and traffic rule constraints (such as speed limit constraints, traffic light constraints, yield rules, etc.).

[0051] In some possible implementations, the Actor-Critic network is trained based on the total reward and target simulation characteristics to obtain a trained target policy network, which may include: Based on the total reward and target simulation characteristics, the temporal difference error is determined; based on the temporal difference error, the value network parameters are updated. Based on the temporal difference error, the dominance function, and the probability ratio of new and old actions, a truncated agent objective function is constructed; based on the truncated agent objective function, the policy network parameters are updated until the policy network converges.

[0052] For example, the first can be calculated by the following formula Timing difference error of the step : In the formula, As a discount factor, For parameters The value network below the first Step state The value estimate, For parameters The value network below the first Step state Value estimate.

[0053] It is understandable that the target simulation features represent the state of the first step.

[0054] For example, after obtaining the temporal difference error, the square of the temporal difference error can be minimized by gradient descent to update the value network parameters.

[0055] For example, the advantage function can be expressed as follows: In the formula, Indicates the dominance value. This represents the total number of steps in one training epoch. Indicates the number of look-ahead index steps. The attenuation coefficient is... Indicates the first The timing difference error of the step.

[0056] For example, in continuous motion control scenarios, the policy network typically outputs motion distribution parameters and defines a stochastic policy accordingly. The steering wheel angle and accelerator / brake inputs are used to construct a two-dimensional continuous motion vector. The strategy can then be written in Gaussian distribution form: in, Indicates the state The following parameters are The strategy network takes action The probability, Indicates a multivariate Gaussian distribution. Indicates the state The following parameters are The action mean vector of the policy network, In the state The following parameters are The standard deviation of the action of the policy network, This indicates the constructor for a diagonal matrix.

[0057] Based on this, the probability ratio of new and old actions is defined. for: In the formula, Indicates the state The following parameters are The policy network (i.e., the old policy network parameters from the previous step) takes action. The probability of.

[0058] For example, a truncated surrogate objective function can be constructed based on the temporal difference error, the dominance function, and the probability ratio of new to old actions. As shown in the following formula: In the formula, This represents the batch expectation operator (generally an approximation of the average value of the batch data). This represents the truncation function. Indicates will Limited to Inside, This indicates the cutoff threshold.

[0059] For example, after obtaining the truncated agent objective function, the policy network gradient can be expressed as follows: In the formula, Represents the gradient operator. Indicates the parameter Find the gradient.

[0060] It is understandable that during the policy network parameter update process, semantic rewards are first integrated with basic rewards to enter the total reward, which then affects the temporal difference error and advantage function estimation, and finally participates in the policy gradient calculation in the form of advantage weighting. By updating the truncated agent objective function through gradient ascent, it is possible to enhance good actions and suppress bad actions while restricting policy mutations, thereby gradually improving the decision consistency and training stability of the policy under semantic objective constraints.

[0061] It can be understood that the policy network after training is the target policy network.

[0062] For example, after obtaining the target policy network, the target policy network can be deployed only to the vehicle edge device to meet the vehicle real-time requirements.

[0063] This embodiment extracts features from a simulated bird's-eye view and its corresponding preset semantic description text, calculates the similarity of the corresponding features, and obtains a semantic reward. Since the simulated bird's-eye view highly matches how humans describe driving scenarios, and the semantic information is intuitive, the semantic reward obtained based on the simulated bird's-eye view and its corresponding preset semantic description text can indicate the degree of matching between the vehicle's current behavior and the preset semantic driving goal. Furthermore, by fusing the semantic reward with the basic reward, the policy network in the Actor-Critic network is optimized. The semantic reward can transform high-level semantic intent into numerical signals, thereby improving training efficiency and decision robustness, resulting in the target policy network. Through the temporal coding module in the policy network, the smoothness and stability of the generated control actions can be significantly improved. Simultaneously, during the deployment phase, only the target policy network can be retained, reducing edge-side inference latency and meeting the real-time requirements of in-vehicle systems.

[0064] It is understandable that high-level semantic intent refers specifically to the abstract, non-physically quantifiable driving expectations and behavioral guidelines described by humans in natural language. It is distinct from low-level physical targets such as "vehicle speed" and "lane deviation" and requires semantic understanding ability to interpret driving requirements.

[0065] In some possible embodiments, the preset semantic description text includes positive text and negative text; The text semantic features of the preset semantic description text corresponding to the simulated bird's-eye view are extracted to obtain the second vector, including: extracting features from the positive text and negative text corresponding to the simulated bird's-eye view respectively to obtain the second vector, which includes a positive text vector and a negative text vector; The similarity between the first vector and the second vector is calculated to obtain a semantic reward, including: calculating the similarity between the first vector and the positive text vector to obtain a positive similarity; calculating the similarity between the first vector and the negative text vector to obtain a negative similarity; and determining the semantic reward based on the positive and negative similarities.

[0066] For example, positive text can be text that describes the simulated bird's-eye view in a positive and accurate way, while negative text can be text that describes the simulated bird's-eye view in a negative and incorrect way.

[0067] For example, positive text could be "The car is driving in its own lane", while negative text could be "The car is driving in another lane".

[0068] For example, a text encoder (such as Transformer) in an existing, pre-trained visual language model (such as the OpenCLIP model) can also be used to extract features from positive and negative text respectively, to obtain positive and negative text vectors.

[0069] For example, the similarity between the first vector and the positive text vector (i.e., positive similarity) and the similarity between the first vector and the negative text vector (i.e., negative similarity) can also be calculated using the formula for calculating the similarity between the first vector and the second vector in the foregoing embodiments.

[0070] For example, the semantic reward can be calculated using the following formula. : In the formula, Indicates positive similarity. This indicates negative similarity.

[0071] This embodiment sets up positive and negative text and extracts features, combines the first vector to obtain positive and negative similarity, and determines semantic rewards based on positive and negative similarity. This allows semantic rewards to not only encourage desired behaviors but also actively punish undesirable behaviors.

[0072] This application also provides an autonomous driving control method, including: Acquire real vehicle status data and real environment images captured by onboard cameras during the autonomous vehicle's operation. Real vehicle status data includes real vehicle control data, real vehicle speed, and real vehicle lane center offset. Semantic segmentation is performed on the real environment image to obtain the real target segmentation result, which is used to indicate the position of lanes and lane lines in the real environment image; the real target segmentation result is compressed to obtain the real low-dimensional latent feature vector; the real low-dimensional latent feature vector of multiple consecutive frames is concatenated with the real vehicle state data of multiple consecutive frames to obtain the real target features. Based on the target's true characteristics, and using the target policy network obtained from the aforementioned model training method, vehicle control commands for autonomous vehicles are generated.

[0073] For example, during the deployment phase, a lightweight image semantic segmentation model can be used for semantic segmentation.

[0074] It is understandable that during the deployment phase, the actual vehicle lane center offset is not directly acquired by the vehicle's underlying sensors, but rather calculated based on real-world images captured by the onboard camera. Specifically, firstly, lane line detection or semantic segmentation is performed on the real-world image to obtain pixel results representing lane lines or lane regions (i.e., real target segmentation results). Then, the vehicle reference center and lane center are determined from these pixel results. The vehicle reference center is the center point along the image width, and the lane center is determined based on the pixel positions of the left and right lane boundaries. Finally, the lateral pixel difference between the vehicle reference center and the lane center is calculated, and this lateral pixel difference or its normalized result is used as the actual vehicle lane center offset.

[0075] This embodiment focuses on the model deployment phase, removing the visual language model used in the training phase. Autonomous vehicles can then use the trained policy network for forward inference based on real vehicle state data and real-world images captured by onboard cameras, generating control commands in real time. This "introduction during training, removal during deployment" approach ensures the policy receives high-level semantic constraints without increasing the burden of edge-side inference.

[0076] The specific implementation method and other beneficial effects of this autonomous driving control method embodiment can be found in the foregoing model training method embodiment, and will not be repeated here.

[0077] It should be noted that the entity executing the autonomous driving control method can be the vehicle control processor of the autonomous vehicle.

[0078] This application also provides an application scenario in which the above-described autonomous driving control method is applied. Specifically, the autonomous driving control method provided in this embodiment can be applied in autonomous driving scenarios. During operation, the autonomous vehicle acquires external environmental data through various sensors. The vehicle processor then determines control actions based on the external environmental data and the driving target to control the autonomous vehicle to continue driving. This application pertains to the stage of determining these control actions.

[0079] Figure 4 The schematic diagram of the overall training-deployment scheme provided in the embodiments of this application can be referred to. Figure 4 The technical solution of this application should be fully understood. It can be understood that the environmental features in the training phase are the simulated low-dimensional latent feature vectors, and the state data are the target simulated features; the environmental features in the deployment phase are the real low-dimensional latent feature vectors, and the state data are the target real features.

[0080] Based on the same inventive concept, this application also provides an autonomous driving control device for implementing the aforementioned autonomous driving control method. The solution provided by this device is similar to the implementation described in the above method; therefore, the specific limitations in one or more embodiments of the autonomous driving control device provided below can be found in the limitations of the autonomous driving control method described above, and will not be repeated here.

[0081] In one exemplary embodiment, such as Figure 5 As shown, an automatic driving control device is provided, comprising: The acquisition module 501 is used to acquire real vehicle state data and real environment images captured by the on-board camera during the driving process of the autonomous vehicle. The real vehicle state data includes real vehicle control data, real vehicle speed, and real vehicle lane center offset. The feature fusion module 502 is used to perform semantic segmentation on the real environment image to obtain the real target segmentation result. The real target segmentation result is used to indicate the position of lanes and lane lines in the real environment image. The real target segmentation result is compressed to obtain a real low-dimensional latent feature vector. The real low-dimensional latent feature vector of multiple consecutive frames is concatenated with the real vehicle state data corresponding to multiple consecutive frames to obtain the real feature of the target. The processing module 503 is used to generate vehicle control commands for autonomous vehicles based on the target policy network obtained by the aforementioned model training method, according to the target's real features.

[0082] The specific implementation methods and beneficial effects of this device embodiment can be found in the foregoing method embodiments, and will not be repeated here.

[0083] In one exemplary embodiment, a computer device is provided, which may be a server or a terminal, and its internal structure diagram may be as follows. Figure 6As shown, this computer device includes a processor, memory, input / output interfaces (I / O), and a communication interface. The processor, memory, and I / O interfaces are connected via a system bus, and the communication interface is also connected to the system bus via the I / O interfaces. The processor provides computational and control capabilities. The memory includes non-volatile storage media and internal memory. The non-volatile storage media stores the operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage media. The database stores data. The I / O interfaces are used for exchanging information between the processor and external devices. The communication interface is used for communicating with external terminals via a network. When the computer program is executed by the processor, it implements the aforementioned automatic driving control method.

[0084] Those skilled in the art will understand that Figure 6 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.

[0085] In one exemplary embodiment, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above-described method embodiments.

[0086] In one exemplary embodiment, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0087] In one exemplary embodiment, a computer program product is provided, including a computer program that, when executed by a processor, implements the steps in the above-described method embodiments.

[0088] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of the relevant data must comply with relevant regulations.

[0089] Those skilled in the art will understand that all or part of the processes in the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments described above. Any references to memory, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. By way of illustration and not limitation, RAM can take many forms, such as Static Random Access Memory (SRAM) or Dynamic Random Access Memory (DRAM).

[0090] The databases involved in the embodiments provided in this application may include at least one type of relational database and non-relational database. Non-relational databases may include, but are not limited to, blockchain-based distributed databases. The processors involved in the embodiments provided in this application may be general-purpose processors, central processing units, graphics processing units, digital signal processors, programmable logic devices, quantum computing-based data processing logic devices, etc., and are not limited to these.

[0091] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0092] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A model training method, characterized in that, The model training method includes: Based on an autonomous driving simulation simulator, the simulation vehicle status data, simulation bird's-eye view, and simulation environment images captured by the on-board camera during the simulation driving process are obtained, as well as the preset semantic description text corresponding to the simulation bird's-eye view. The simulation vehicle status data includes simulation vehicle control data, simulation vehicle speed, and simulation vehicle lane center offset. Semantic segmentation is performed on the simulation environment image to obtain the simulation target segmentation result, which is used to indicate the position of lanes and lane lines in the simulation environment image; the simulation target segmentation result is compressed to obtain the simulation low-dimensional latent feature vector; the simulation low-dimensional latent feature vector of multiple consecutive frames is concatenated with the simulation vehicle state data of the corresponding multiple consecutive frames to obtain the target simulation feature; Visual semantic features are extracted from the simulated bird's-eye view to obtain a first vector; textual semantic features are extracted from the preset semantic description text corresponding to the simulated bird's-eye view to obtain a second vector; the similarity between the first vector and the second vector is calculated to obtain a semantic reward. The Actor-Critic network is trained based on the total reward and target simulation features to obtain a trained target policy network. The policy network in the Actor-Critic network includes a temporal coding module. The input of the policy network is the target simulation features, and the output is vehicle control data. The total reward is obtained based on the semantic reward and the basic reward. The basic reward is used to indicate the degree to which the simulated vehicle meets the expected driving goal during the simulated driving process.

2. The model training method according to claim 1, characterized in that, The preset semantic description text includes positive text and negative text; The step of extracting text semantic features from the preset semantic description text corresponding to the simulated bird's-eye view to obtain the second vector includes: extracting features from the positive text and negative text corresponding to the simulated bird's-eye view respectively to obtain the second vector, wherein the second vector includes a positive text vector and a negative text vector; The step of calculating the similarity between the first vector and the second vector to obtain a semantic reward includes: calculating the similarity between the first vector and the positive text vector to obtain a positive similarity; calculating the similarity between the first vector and the negative text vector to obtain a negative similarity; and determining the semantic reward based on the positive similarity and the negative similarity.

3. The model training method according to claim 1, characterized in that, The process of training the Actor-Critic network based on the total reward and target simulation characteristics to obtain a trained target policy network includes: Based on the total reward and target simulation characteristics, the time-series difference error is determined; Update the value network parameters based on temporal difference error; Based on the time-series difference error, the dominance function, and the probability ratio of new and old actions, a truncated agent objective function is constructed. Based on the truncated agent objective function, update the policy network parameters until the policy network converges.

4. The model training method according to claim 1, characterized in that, The basic rewards include speed rewards, direction alignment rewards, lane centering rewards, and collision penalties.

5. The model training method according to claim 1, characterized in that, The timing coding module is an LSTM module.

6. An automatic driving control method, characterized in that, The autonomous driving control method includes: Acquire real vehicle status data and real environment images captured by onboard cameras during the autonomous vehicle's operation. The real vehicle status data includes real vehicle control data, real vehicle speed, and real vehicle lane center offset. Semantic segmentation is performed on the real environment image to obtain the real target segmentation result, which is used to indicate the position of lanes and lane lines in the real environment image; the real target segmentation result is compressed to obtain the real low-dimensional latent feature vector; the real low-dimensional latent feature vector of multiple consecutive frames is concatenated with the real vehicle state data corresponding to multiple consecutive frames to obtain the real feature of the target. Based on the true characteristics of the target, and using the target policy network obtained by the model training method according to any one of claims 1-5, vehicle control commands for autonomous vehicles are generated.

7. An automatic driving control device, characterized in that, The automatic driving control device includes: The acquisition module is used to acquire real vehicle status data and real environment images captured by the on-board camera during the autonomous vehicle's driving process. The real vehicle status data includes real vehicle control data, real vehicle speed, and real vehicle lane center offset. The feature fusion module is used to perform semantic segmentation on real environment images to obtain real target segmentation results, which are used to indicate the positions of lanes and lane lines in real environment images; the real target segmentation results are compressed to obtain real low-dimensional latent feature vectors; the real low-dimensional latent feature vectors of multiple consecutive frames are concatenated with the real vehicle state data corresponding to multiple consecutive frames to obtain the real features of the target. The processing module is used to generate vehicle control commands for autonomous vehicles based on the target policy network obtained by the model training method described in any one of claims 1-5, according to the target's true features.

8. A computer device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that the processor executes the computer program to implement the steps of the autonomous driving control method of claim 6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the automatic driving control method of claim 6.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the automatic driving control method of claim 6.

Images