Scene representation model training method, obstacle marking method and autonomous vehicle

Abstract

The present disclosure provides a scene representation model training method, an obstacle marking method and an autonomous vehicle, relates to the field of artificial intelligence, and particularly relates to the field of autonomous driving. The specific implementation scheme is: obtaining first scene information of a first driving scene; obtaining second scene information of a second driving scene; wherein the second driving scene and the first driving scene have a distribution difference of a key obstacle, and the key obstacle is used to change the driving risk state of a host vehicle; and training a scene representation model based on the first scene information and the second scene information. The present disclosure can improve the accuracy of the scene representation result and the accuracy of the obstacle marking result.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence, in particular to the field of autonomous driving technology, and specifically to a scene representation model training method, an obstacle marking method, and an autonomous driving vehicle. Background Art

[0002] Autonomous driving technology involves multiple aspects, including environmental perception, behavioral decision-making, trajectory planning, and motion control. Behavioral decision-making, in turn, relies on scene representation and obstacle identification within the driving scene. Currently, these scene representation and obstacle identification results are primarily obtained using manually configured computational logic.

[0003] Since the artificially set calculation logic has great limitations and cannot be flexibly applied to various actual driving scenarios, it will affect the accuracy of the scene representation results and obstacle marking results. Summary of the Invention

[0004] The present disclosure provides a scene representation model training method, an obstacle marking method, and an autonomous driving vehicle.

[0005] According to one aspect of the present disclosure, a scene representation model training method is provided, comprising:

[0006] Obtaining first scene information of a first driving scene;

[0007] Obtaining second scene information of a second driving scene; wherein the second driving scene differs from the first driving scene in distribution of key obstacles, and the key obstacles are used to change the driving risk state of the host vehicle;

[0008] A scene representation model is trained based on the first scene information and the second scene information.

[0009] According to another aspect of the present disclosure, there is provided an obstacle marking method, comprising:

[0010] Get the driving scene information of the current driving scene;

[0011] Inputting the driving scene information into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation based on the driving scene information; wherein the scene representation model is trained by any scene representation model training method;

[0012] Based on the intermediate parameters, the current obstacle in the current driving scene is marked to obtain an obstacle marking result.

[0013] According to another aspect of the present disclosure, a scene representation model training device is provided, comprising:

[0014] A first information acquisition unit, configured to acquire first scene information of a first driving scene;

[0015] A second information acquisition unit is configured to acquire second scene information of a second driving scene; wherein the second driving scene differs from the first driving scene in the distribution of key obstacles, and the key obstacles are used to change the driving risk state of the host vehicle;

[0016] The first model training unit is used to train the scene representation model based on the first scene information and the second scene information.

[0017] According to another aspect of the present disclosure, there is provided an obstacle marking device, comprising:

[0018] A current information acquisition unit, used to acquire driving scene information of the current driving scene;

[0019] an intermediate parameter acquisition unit, configured to input driving scene information into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation based on the driving scene information; wherein the scene representation model is obtained by training using any scene representation model training method;

[0020] The obstacle marking unit is used to mark the current obstacle in the current driving scene based on the intermediate parameters to obtain the obstacle marking result.

[0021] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0022] at least one processor;

[0023] a memory communicatively coupled to the at least one processor;

[0024] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any method in the embodiments of the present disclosure.

[0025] According to another aspect of the present disclosure, an autonomous driving vehicle is provided, comprising an electronic device.

[0026] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method in the embodiments of the present disclosure.

[0027] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method in the embodiments of the present disclosure when executed by a processor.

[0028] The present disclosure can improve the accuracy of scene characterization results and the accuracy of obstacle marking results.

[0029] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0031] Figure 1 A flowchart of a scene representation model training method provided in an embodiment of the present disclosure;

[0032] Figure 2A and 2B A diagram illustrating a derivation method of a second driving scenario provided in an embodiment of the present disclosure;

[0033] Figure 3A and 3B An illustration of another derivative method of the second driving scenario provided in an embodiment of the present disclosure;

[0034] Figure 4A A schematic diagram of the network structure of a scene representation model provided in an embodiment of the present disclosure.

[0035] Figure 4B A schematic diagram of the network structure of another scenario representation model provided in an embodiment of the present disclosure;

[0036] Figure 5A and 5B This is an illustration of a derivative method of a fourth driving scenario provided by an embodiment of the present disclosure;

[0037] Figure 6 A schematic diagram of a scenario for a scenario representation model training method provided in an embodiment of the present disclosure;

[0038] Figure 7 A flowchart of an obstacle marking method provided in an embodiment of the present disclosure;

[0039] Figure 8 A schematic diagram of a scenario of an obstacle marking method provided by an embodiment of the present disclosure;

[0040] Figure 9 A schematic structural block diagram of a scene representation model training device provided in an embodiment of the present disclosure;

[0041] Figure 10 A schematic structural block diagram of an obstacle marking device provided in an embodiment of the present disclosure;

[0042] Figure 11 A schematic structural block diagram of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0043] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be appreciated by those skilled in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0044] The present disclosure provides a scene representation model training method, which can be applied to electronic devices. Figure 1 The flowchart shown illustrates a scene representation model training method provided by an embodiment of the present disclosure. It should be noted that although a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in other orders.

[0045] Step S101, obtaining first scene information of a first driving scene;

[0046] Step S102: Acquire second scene information of a second driving scene; wherein the second driving scene differs from the first driving scene in distribution of key obstacles, and the key obstacles are used to change the driving risk state of the host vehicle;

[0047] Step S103: training a scene representation model based on the first scene information and the second scene information.

[0048] The first driving scenario may be a historical driving scenario of the host vehicle, and the second driving scenario may be a derived driving scenario obtained by adjusting the key obstacles in the first driving scenario. Therefore, the second driving scenario differs from the first driving scenario in the distribution of key obstacles, and the key obstacles are used to change the driving risk state of the host vehicle. The driving risk state may be any one of a safe state (no collision risk and no sudden braking risk), a collision risk, and a sudden braking risk. In addition, it should be noted that in the embodiment of the present disclosure, when the host vehicle is driving in a certain driving scenario, if the distance between the obstacle and the host vehicle is less than the safety distance threshold throughout the entire driving process, the host vehicle is considered to have no collision risk; otherwise, the host vehicle is considered to have a collision risk; when the host vehicle is driving in a certain driving scenario, if the host vehicle does not suddenly brake throughout the entire driving process, the host vehicle is considered to have no sudden braking risk; otherwise, the host vehicle is considered to have a sudden braking risk. The safety distance threshold can be set according to actual application requirements, for example, it can be set to 20 centimeters (cm), and the embodiment of the present disclosure does not impose any restrictions on this.

[0049] In the disclosed embodiment, the first scene information may include first vehicle information and first obstacle information. The first vehicle information may include speed information, acceleration information, and position information of the vehicle in the first driving scene, and the first obstacle information may include the obstacle type of the obstacle in the first driving scene, which may be a movable obstacle (e.g., an obstacle vehicle) or a fixed obstacle (e.g., a bridge pier). If the obstacle type is a movable obstacle, the first obstacle information may also include speed information, acceleration information, and position information of the obstacle in the first driving scene. If the obstacle type is a fixed obstacle, the first obstacle information may also include the location of the obstacle in the first driving scene.

[0050] Similarly, in the disclosed embodiments, the second scenario information may include information about a second vehicle and information about a second obstacle. The second vehicle information may include information about the vehicle's speed, acceleration, and position in the second driving scenario. The second obstacle information may include the type of obstacle in the second driving scenario, specifically, whether it is a movable obstacle or a fixed obstacle. If the obstacle is a movable obstacle, the second obstacle information may also include information about the obstacle's speed, acceleration, and position in the second driving scenario. If the obstacle is a fixed obstacle, the second obstacle information may also include the location of the obstacle in the second driving scenario.

[0051] After obtaining first scene information of the first driving scene and second scene information of the second driving scene, the scene representation model may be trained based on the first scene information and the second scene information.

[0052] In a specific example, the first scene information can be input into the scene representation model to obtain a first representation result output by the scene representation model; the second scene information can be input into the scene representation model to obtain a second representation result output by the scene representation model; and the scene representation model can be trained based on the first representation result and the second representation result. Wherein, "training the scene representation model based on the first representation result and the second representation result" can include: obtaining a first loss between the first representation result and the second representation result; when the first loss meets the first loss requirement, obtaining a trained scene representation model; when the first loss does not meet the first loss requirement, adjusting the parameters of the scene representation model. Wherein, the first loss can be a first cosine similarity, and the first loss requirement can be that the first cosine similarity is less than a first similarity threshold. The first similarity threshold can be set according to actual application requirements, for example, it can be set to 0.3, and the embodiments of the present disclosure are not limited to this.

[0053] The scene representation model is a learnable neural network model. The trained scene representation model can be used to output a current scene representation result based on input driving scene information. The driving scene information can be acquired based on the current driving scene during the actual driving process of the current host vehicle and can have the same data structure as the first and second scene information described above. This description is omitted here.

[0054] By adopting the scene representation model training method provided by the embodiment of the present disclosure, it is possible to obtain first scene information of a first driving scene; obtain second scene information of a second driving scene; wherein, there is a difference in the distribution of key obstacles between the second driving scene and the first driving scene, and the key obstacles are used to change the driving risk state of the main vehicle; and train the scene representation model based on the first scene information and the second scene information. After obtaining the trained scene representation model, it can be used to: output the current scene representation result based on the input driving scene information. On the one hand, since the scene representation model is a learnable neural network model, compared with the existing technology, obtaining the scene representation result based on the trained scene representation model can improve the accuracy of the scene representation result. On the other hand, since there is a difference in the distribution of key obstacles between the second driving scene and the first driving scene, and the key obstacles are used to change the driving risk state of the main vehicle, the key obstacles play a key role in the behavioral decision-making of the main vehicle and are important information for any driving scene. Based on this, in the embodiment of the present disclosure, by training the scene representation model on a pair of dissimilar scenes (the first driving scene and the second driving scene) with different distribution of key obstacles, the scene representation model's perception and processing capabilities for key obstacles can be improved, thereby increasing the proportion and accuracy of key obstacle information in the scene representation results, and further improving the accuracy of the scene representation results.

[0055] As previously mentioned, in the disclosed embodiments, the first driving scenario may be a historical driving scenario of the host vehicle, and the second driving scenario may be a derived driving scenario obtained by adjusting key obstacles in the first driving scenario. Based on this, in some optional implementations, the scenario representation model training method may further include the following steps:

[0056] Select a target obstacle from the first driving scene;

[0057] Adjust the target obstacle in the first driving scene to obtain a pending driving scene;

[0058] Performing an automatic driving simulation on the main vehicle in the pending driving scenario to obtain a simulation risk state corresponding to the main vehicle driving in the pending driving scenario;

[0059] When the simulated risk state is different from the historical risk state, the target obstacle is determined as the key obstacle, and the pending driving scenario is determined as the second driving scenario; wherein the historical risk state is the actual risk state corresponding to the main vehicle when driving in the first driving scenario.

[0060] In the disclosed embodiment, a target obstacle can be selected from the first driving scene according to the scene category of the first driving scene, or any obstacle can be selected from the first driving scene as the target obstacle. The scene categories may include a safety scene, a collision scene, and an emergency braking scene. The safety scene may be a driving scene with no collision risk and no emergency braking risk. When the main vehicle is driving in this driving scene, the driving risk state is a safe state. The collision scene may be a driving scene with a collision risk. When the main vehicle is driving in this driving scene, the driving risk state is a collision risk. The emergency braking scene may be a driving scene with an emergency braking risk. When the main vehicle is driving in this driving scene, the driving risk state is an emergency braking risk.

[0061] Among them, "adjusting the target obstacle in the first driving scene to obtain the pending driving scene" may include: adjusting the position and / or driving time of the target obstacle in the first driving scene to obtain the pending driving scene.

[0062] After obtaining the pending driving scenario, the autonomous driving simulation software can be used to simulate the autonomous driving of the host vehicle in the pending driving scenario to obtain the simulated risk state corresponding to the host vehicle driving in the pending driving scenario. If the simulated risk state differs from the historical risk state, the target obstacle is determined to be a critical obstacle, and the pending driving scenario is determined to be the second driving scenario. For example, if the simulated risk state is a safe state, while the historical risk state indicates a collision risk, the target obstacle is determined to be a critical obstacle, and the pending driving scenario is determined to be the second driving scenario.

[0063] Through the above steps, in the disclosed embodiment, after selecting a target obstacle from the first driving scenario and adjusting the target obstacle in the first driving scenario to obtain the pending driving scenario, an autonomous driving simulation can be performed on the host vehicle in the pending driving scenario to obtain a simulated risk state corresponding to the host vehicle driving in the pending driving scenario. This facilitates, when the simulated risk state differs from the historical risk state, determining the target obstacle as a critical obstacle and determining the pending driving scenario as the second driving scenario. Because the simulated risk state corresponding to the host vehicle driving in the pending driving scenario is obtained through autonomous driving simulation in the disclosed embodiment, the efficiency of obtaining the simulated risk state can be improved, thereby improving the efficiency of determining the critical obstacle and thus improving the training efficiency of the scenario representation model.

[0064] In some optional implementations, “selecting a target obstacle from the first driving scene” may include the following steps:

[0065] Determining a scene category of a first driving scene;

[0066] When the scene category is a safety scene, a first undetermined obstacle corresponding to the first a posteriori decision to avoid is selected from the first driving scene as the target obstacle; wherein the first a posteriori decision is a driving decision made by the host vehicle with respect to the first undetermined obstacle;

[0067] When the scene category is a collision scene or an emergency braking scene, any obstacle whose distance from the host vehicle is less than a first distance threshold is selected from the first driving scene as a target obstacle.

[0068] That is, in the embodiment of the present disclosure, a target obstacle may be selected from the first driving scene according to the scene category of the first driving scene.

[0069] If the scenario is a safety scenario, the first undetermined obstacle corresponding to the first a posteriori decision to avoid is selected from the first driving scenario as the target obstacle. Avoidance can be lateral or longitudinal, lateral avoidance can be left turn avoidance or right turn avoidance, and longitudinal avoidance can be acceleration avoidance in the front or deceleration avoidance in the back, which is not limited in the present embodiment.

[0070] If the scenario type is a collision scenario or an emergency braking scenario, any obstacle within the first driving scenario that is less than a first distance threshold from the host vehicle is selected as the target obstacle. The first distance threshold can be set based on actual application requirements, for example, 30 meters (m), and is not limited in this embodiment.

[0071] Through the above steps, in the embodiment of the present disclosure, a corresponding target obstacle determination strategy is provided for the first driving scene of each scene category, and a unified target obstacle determination strategy is avoided, thereby ensuring that the derived second driving scene has a different scene category from the first driving scene, that is, ensuring that the main vehicle has different driving risk states in the first driving scene and the second driving scene, so as to improve the training effect of the scene representation model, thereby further improving the accuracy of the scene representation results.

[0072] In some optional implementations, “adjusting the target obstacle in the first driving scene to obtain a pending driving scene” may include the following steps:

[0073] When the scene category is a safe scene, adjusting the position and / or driving time of the target obstacle in the first driving scene to obtain a pending driving scene;

[0074] And / or, when the scene category is a collision scene or an emergency braking scene, the target obstacle is removed from the first driving scene to obtain a pending driving scene.

[0075] Adjusting the position and / or posture of the target obstacle may involve adjusting the position and / or posture of the target obstacle to bring it closer to the host vehicle. Adjusting the travel time of the target obstacle may involve adjusting the time at which the target obstacle enters the first driving scene to bring it closer to the host vehicle, if the target obstacle is a movable obstacle. For example, if the target obstacle is located in front of the host vehicle in the first driving scene, the target obstacle may be controlled to enter the first driving scene earlier. If the target obstacle is located behind the host vehicle in the first driving scene, the time at which the target obstacle enters the first driving scene may be delayed.

[0076] Please combine Figure 2A and Figure 2B In the first driving scene 201, including the host vehicle 202, if the scene category of the first driving scene 201 is a safe scene, a first pending obstacle corresponding to the first a posteriori decision to avoid can be selected from the first driving scene 201 as the target obstacle 203. Thereafter, the position and / or travel time of the target obstacle 203 can be adjusted in the first driving scene 201 to obtain the pending driving scene 204.

[0077] Please combine Figure 3A and Figure 3B First driving scene 301 includes a host vehicle 302. If the scene category of first driving scene 301 is a collision scene or an emergency braking scene, any obstacle within first driving scene 301 that is less than a first distance threshold from the host vehicle can be selected as target obstacle 303. Because the collision risk or emergency braking risk may be caused by target obstacle 303, target obstacle 303 can be directly removed from first driving scene 301 to obtain a pending driving scene 304.

[0078] Through the above steps, in the embodiment of the present disclosure, when the scene category is a safety scene, the posture and / or driving time of the target obstacle can be adjusted in the first driving scene to obtain a pending driving scene; when the scene category is a collision scene or an emergency braking scene, the target obstacle can be directly removed in the first driving scene to obtain a pending driving scene, thereby simplifying the creation process of the pending driving scene and further improving the training efficiency of the scene representation model.

[0079] In some optional implementations, “training a scene representation model based on the first scene information and the second scene information” may include the following steps:

[0080] Inputting the target scene information into the scene representation model to extract the fusion features of the host vehicle from the target scene information and the obstacle features from the target scene information through the scene representation model; wherein the target scene information is any one of the first scene information and the second scene information;

[0081] Obtaining a scene representation result output by the scene representation model based on the correlation between the fusion features of the host vehicle and the obstacle features; wherein, when the target scene information is the first scene information, the scene representation result is the first representation result; and when the target scene information is the second scene information, the scene representation result is the second representation result;

[0082] The scene representation model is trained based on the first representation result and the second representation result.

[0083] As previously mentioned, in embodiments of the present disclosure, when the target scene information is first scene information, the first scene information may include first vehicle information and first obstacle information. In one specific example, multiple first sampling time points may be pre-set. The first vehicle information may include the speed, acceleration, and position information of the vehicle collected from the first driving scene at each first sampling time point. The first obstacle information may include the obstacle type of any obstacle in the first driving scene, specifically, whether it is a movable obstacle or a fixed obstacle. For each obstacle in the first driving scene, if the obstacle type is movable, the first obstacle information may also include the speed, acceleration, and position information of the obstacle collected from the first driving scene at each first sampling time point. If the obstacle type is fixed, the first obstacle information may also include the location of the obstacle in the first driving scene. The total number of first sampling time points may be set based on actual application requirements. For example, it may be set to 16, and the interval between two adjacent first sampling time points may be 0.1 seconds (s). This is not a limitation in embodiments of the present disclosure.

[0084] Similarly, in embodiments of the present disclosure, when the target scene information is second scene information, the second scene information may include second host vehicle information and second obstacle information. In one specific example, multiple second sampling time points may be pre-set. The second host vehicle information may include speed information, acceleration information, and position information of the host vehicle collected from the second driving scene at each second sampling time point. The second obstacle information may include the obstacle type of any obstacle in the second driving scene, specifically, whether it is a movable obstacle or a fixed obstacle. For each obstacle in the second driving scene, if the obstacle type is movable, the second obstacle information may also include the speed information, acceleration information, and position information of the obstacle collected from the second driving scene at each second sampling time point. If the obstacle type is fixed, the second obstacle information may also include the location of the obstacle in the second driving scene. The total number of second sampling time points can be set based on actual application requirements. For example, it can be set to 16, and the interval between two adjacent second sampling time points can be 0.1 seconds (s). This embodiment of the present disclosure does not impose any restrictions on this.

[0085] Please combine Figure 4A , for the scene representation model 400 , in a specific example, it may include an encoder module 401 and a first attention module 402 .

[0086] The encoder module 401 is used to encode the host vehicle information in the target scene information, extract the host vehicle's independent features from the target scene information, and directly use them as the host vehicle's fused features. It also encodes the obstacle information in the target scene information and extracts obstacle features from the target scene information. Specifically, it may include feature information corresponding to any obstacle in the driving scene. Subsequently, the first attention module obtains a scene representation result based on the correlation between the host vehicle's fused features and the obstacle features.

[0087] The encoder module 401 may be any available feature encoder, and the first attention module 402 may be a neural network module implemented based on the attention mechanism.

[0088] To further improve the accuracy of the scene representation result, in the embodiment of the present disclosure, in addition to the first main vehicle information and the first obstacle information, the first scene information can also include first traffic indication information, such as the related information of traffic indication signs such as lanes, stop lines, pedestrian crossings, and traffic signs. Specifically, it can include lane width information, stop line position information, pedestrian crossing position information, traffic sign semantic information, etc. Similarly, in addition to the second main vehicle information and the second obstacle information, the second scene information can also include second traffic indication information, such as the related information of traffic indication signs such as lanes, stop lines, pedestrian crossings, and traffic signs. Specifically, it can include lane width information, stop line position information, pedestrian crossing position information, traffic sign semantic information, etc. Based on this, please refer to the description of the first scene information and the second scene information in the above embodiment. Figure 4B For the scene representation model 400, in another specific example, in addition to the encoder module 401 and the first attention module 402, it can also include a traffic indication processing module 403 and a second attention module 404.

[0089] Among them, the encoder module 401 is used to encode the main vehicle information in the target scene information, extract the independent features of the main vehicle from the target scene information, and encode the obstacle information in the target scene information, extract the obstacle features from the target scene information, which can include the feature information of any obstacle in the driving scene. The traffic indication processing module 403 is used to encode the traffic indication information in the target scene information, extract the traffic indication features from the target scene information, which can include the feature information of traffic indication signs such as lanes, stop lines, pedestrian crossings, and traffic signs in the driving scene. Thereafter, through the second attention module 404, based on the correlation between the independent features of the main vehicle and the traffic indication features, the fusion features of the main vehicle are obtained. Finally, through the first attention module 402, based on the correlation between the fusion features of the main vehicle and the obstacle features, the scene representation result is obtained.

[0090] Among them, the encoder module 401 can be any available feature encoder, the first attention module 402 and the second attention module 404 can be neural network modules implemented based on the attention mechanism. Specifically, the first attention module 402 and the second attention module 404 can be two attention modules included in the attention network implemented based on the cross-attention mechanism. The traffic indication processing module 403 can include multiple feature extraction networks composed of convolutional neural networks (CNNs) 4031 and self-attention modules 4032, and a feature fusion module 4033. Each feature extraction network corresponds to a traffic sign, and is used to encode the traffic indication information corresponding to the traffic sign to obtain the independent features of the traffic sign. The feature fusion module 4033 is used to fuse the independent features of all traffic signs to obtain traffic indication features.

[0091] In addition, it should be noted that, in the embodiment of the present disclosure, since the first attention module 402 can be a neural network module implemented based on the attention mechanism, after obtaining the fusion features and obstacle features of the main vehicle, the first attention module 402 "obtains a scene representation result based on the correlation between the fusion features of the main vehicle and the obstacle features" may include: obtaining query parameters based on the fusion features of the main vehicle, and obtaining key parameters based on the obstacle features; calculating the correlation between the fusion features of the main vehicle and the obstacle features based on the query parameters and the key parameters; and obtaining a scene representation result based on the correlation between the fusion features of the main vehicle and the obstacle features.

[0092] Query1=X11*W1 Q

[0093] Key1=X12*W1 K

[0094] Among them, Query1 is the query parameter, X11 is the fusion feature of the main vehicle, and W 1Q is the first parameter matrix that can be learned, Key1 is the key parameter, X12 is the obstacle feature, W1 K is the learnable second parameter matrix.

[0095] In the embodiment of the present disclosure, when the target scene information is the first scene information, the scene characterization result is the first characterization result; when the target scene information is the second scene information, the scene characterization result is the second characterization result.

[0096] After obtaining the first representation result and the second representation result, a first loss between the first representation result and the second representation result can be obtained; in a case where the first loss meets a first loss requirement, a trained scene representation model is obtained; in a case where the first loss does not meet the first loss requirement, parameter adjustment is performed on the scene representation model, for example, including adjustment of the first parameter matrix and the second parameter matrix. The first loss can be a first cosine similarity, the first loss requirement can be that the first loss is less than a first similarity threshold, and the first similarity threshold can be set according to actual application requirements, for example, can be set to 0.3, and the present disclosure is not limited in this regard.

[0097] Through the above steps, in the present disclosure, target scene information can be input into the scene representation model to extract the fusion feature of the host vehicle from the target scene information through the scene representation model, and extract the obstacle feature from the target scene information, and then obtain the scene representation result output by the scene representation model based on the correlation between the fusion feature of the host vehicle and the obstacle feature. In a case where the target scene information is the first scene information, the scene representation result is the first representation result, and in a case where the target scene information is the second scene information, the scene representation result is the second representation result. Finally, the scene representation model is trained based on the first representation result and the second representation result. In this process, since the scene representation result is output by the scene representation model based on the correlation between the fusion feature of the host vehicle and the obstacle feature, the scene representation result can reflect the interaction between the feature information, so as to further improve the accuracy of the scene representation result.

[0098] In addition, as mentioned earlier, in a case where the scene representation model has a network structure as shown in Figure 4B extracting the fusion feature of the host vehicle from the target scene information can include the following steps:

[0099] extracting an independent feature of the host vehicle from the target scene information;

[0100] extracting a traffic indication feature from the target scene information;

[0101] obtaining the fusion feature of the host vehicle based on the independent feature of the host vehicle and the traffic indication feature.

[0102] The traffic indication feature can include feature information corresponding to traffic indication signs such as lanes, stop lines, pedestrian crossings, and traffic indication boards in the driving scene.

[0103] Through the above steps, in the embodiment of the present disclosure, the independent features of the main vehicle can be extracted from the target scene information, and then the traffic indication features can be extracted from the target scene information. Based on the independent features and traffic indication features of the main vehicle, the fused features of the main vehicle can be obtained, so that the fused features of the main vehicle carry not only the independent features of the main vehicle but also the traffic indication features, thereby enhancing the representability of the fused features of the main vehicle and further improving the training effect of the scene representation model.

[0104] In some optional implementations, the scene representation model training method further includes the following steps:

[0105] Obtaining third scene information of a third driving scene;

[0106] Obtaining fourth scene information of a fourth driving scene; wherein there is a difference in distribution of non-critical obstacles between the third driving scene and the fourth driving scene;

[0107] The scene representation model is trained based on the third scene information and the fourth scene information.

[0108] The third driving scenario can be a historical driving scenario for the host vehicle, and the fourth driving scenario can be a derived driving scenario obtained by adjusting the non-critical obstacles in the third driving scenario. Therefore, the fourth driving scenario differs from the third driving scenario in the distribution of non-critical obstacles, and critical obstacles do not change the driving risk status of the host vehicle.

[0109] In the disclosed embodiment, the third scenario information may include third vehicle information and third obstacle information. The third vehicle information may include speed information, acceleration information, and position information of the vehicle in the third driving scenario. The third obstacle information may include the obstacle type of the obstacle in the third driving scenario, specifically, whether it is a movable obstacle or a fixed obstacle. If the obstacle type is a movable obstacle, the third obstacle information may also include the speed information, acceleration information, and position information of the obstacle in the third driving scenario. If the obstacle type is a fixed obstacle, the third obstacle information may also include the location of the obstacle in the third driving scenario.

[0110] Similarly, in the embodiments of the present disclosure, the fourth scene information can include fourth host vehicle information and fourth obstacle information. The fourth host vehicle information can include speed information, acceleration information and pose information of the host vehicle in the fourth driving scene, and the fourth obstacle information can include an obstacle type of the obstacle in the fourth driving scene, which can be a movable obstacle or a fixed obstacle. In the case of a movable obstacle, the fourth obstacle information can further include speed information, acceleration information and pose information of the obstacle in the fourth driving scene. In the case of a fixed obstacle, the fourth obstacle information can further include a region position corresponding to the obstacle in the fourth driving scene.

[0111] To further improve the accuracy of the scene representation result, in the embodiments of the present disclosure, in addition to the third host vehicle information and the third obstacle information, the third scene information can also include third traffic indication information, such as information related to traffic indication signs such as lanes, stop lines, pedestrian crossings and traffic signs. Specifically, it can include lane width information, stop line position information, pedestrian crossing position information, and traffic sign semantic information. Similarly, in addition to the fourth host vehicle information and the fourth obstacle information, the fourth scene information can also include fourth traffic indication information, such as information related to traffic indication signs such as lanes, stop lines, pedestrian crossings and traffic signs. Specifically, it can include lane width information, stop line position information, pedestrian crossing position information, and traffic sign semantic information.

[0112] After obtaining the third scene information of the third driving scene and the fourth scene information of the fourth driving scene, the scene representation model can be trained based on the third scene information and the fourth scene information.

[0113] In a specific example, the third scene information can be input into the scene representation model to obtain a third representation result output by the scene representation model; the fourth scene information can be input into the scene representation model to obtain a fourth representation result output by the scene representation model; and the scene representation model can be trained based on the third representation result and the fourth representation result. The training of the scene representation model based on the third representation result and the fourth representation result can include: obtaining a second loss between the third representation result and the fourth representation result; obtaining a trained scene representation model when the second loss meets a second loss requirement; and adjusting the parameters of the scene representation model when the second loss does not meet the second loss requirement. The second loss can be a second cosine similarity, and the second loss requirement can be that the second cosine similarity is greater than a second similarity threshold. The second similarity threshold can be set according to actual application requirements, for example, it can be set to 0.95, and the embodiments of the present disclosure do not limit this.

[0114] In addition, it should be noted that in the embodiments of the present disclosure, the scene representation model can be first trained based on the first scene information and the second scene information, and then trained based on the third scene information and the fourth scene information, or it can be first trained based on the third scene information and the fourth scene information, and then trained based on the first scene information and the second scene information. The embodiments of the present disclosure do not limit this.

[0115] It should also be noted that, in the embodiment of the present disclosure, the process of training the scene representation model based on the third scene information and the fourth scene information can be referred to the relevant description of "training the scene representation model based on the first scene information and the second scene information" mentioned above, and will not be repeated here.

[0116] Through the above steps, in the embodiment of the present disclosure, the third scene information of the third driving scene can be obtained; the fourth scene information of the fourth driving scene can be obtained; and the scene representation model can be trained based on the third scene information and the fourth scene information. The fourth driving scene and the third driving scene have distribution differences in non-critical obstacles, so that in the embodiment of the present disclosure, the scene representation model can also be trained using a similar scene pair (the third driving scene and the fourth driving scene) with distribution differences in non-critical obstacles. This can further improve the scene representation model's ability to perceive and process critical obstacles, thereby further improving the proportion and accuracy of critical obstacle information in the scene representation results, thereby further improving the accuracy of the scene representation results.

[0117] As previously mentioned, in the disclosed embodiments, the third driving scenario may be a historical driving scenario of the host vehicle, and the fourth driving scenario may be a derived driving scenario obtained by adjusting key obstacles in the third driving scenario. Based on this, in some optional implementations, the scenario representation model training method may further include the following steps:

[0118] Select any obstacle from the third driving scene whose distance from the host vehicle is greater than the second distance threshold as the second pending obstacle;

[0119] If the second a posteriori decision corresponding to the second pending obstacle is to ignore or follow, and the driving path of the second pending obstacle in the third driving scenario does not interact with the driving path of the host vehicle in the third driving scenario, the second pending obstacle is determined as a non-critical obstacle; wherein the second a posteriori decision is the driving decision made by the host vehicle with respect to the second pending obstacle;

[0120] In the third driving scenario, non-critical obstacles are removed to obtain the fourth driving scenario.

[0121] The second distance threshold may be set according to actual application requirements, for example, it may be set to 15m, which is not limited in the embodiment of the present disclosure.

[0122] Among them, the fact that the driving path of the second pending obstacle in the third driving scene has no interaction with the driving path of the main vehicle in the third driving scene can be understood as: the driving path of the second pending obstacle in the third driving scene has no intersection with the driving path of the main vehicle in the third driving scene.

[0123] Please combine Figure 5A and Figure 5B The third driving scene 501 includes a host vehicle 502. Any obstacle in the third driving scene 501 that is spaced from the host vehicle 502 by a distance greater than a second distance threshold is selected as a second pending obstacle 503. The second a posteriori decision corresponding to the second pending obstacle 503 is to ignore, and the driving path 504 of the second pending obstacle 503 in the third driving scene 501 does not interact with the driving path 505 of the host vehicle 502 in the third driving scene 501. Therefore, the second pending obstacle 503 can be determined as a non-critical obstacle. The non-critical obstacles are then removed from the third driving scene to obtain a fourth driving scene 506.

[0124] Through the above steps, in the disclosed embodiment, on the one hand, any obstacle whose distance from the host vehicle is greater than the second distance threshold can be selected from the third driving scene as the second pending obstacle. Then, if the second posterior decision corresponding to the second pending obstacle is to ignore or follow, and the driving path of the second pending obstacle in the third driving scene does not interact with the driving path of the host vehicle in the third driving scene, the second pending obstacle can be determined as a non-critical obstacle, thereby ensuring the reliability of the non-critical obstacle and further improving the training effect of the scene representation model. On the other hand, after the second pending obstacle is determined as a non-critical obstacle, the non-critical obstacle can be directly removed from the third driving scene to obtain the fourth driving scene, thereby simplifying the creation process of the fourth driving scene and further improving the training efficiency of the scene representation model.

[0125] The following describes the complete process of a scene representation model training method provided by an embodiment of the present disclosure.

[0126] Get the first driving scene.

[0127] A target obstacle is selected from a first driving scenario; the target obstacle is adjusted in the first driving scenario to obtain a pending driving scenario; an automatic driving simulation is performed on the main vehicle in the pending driving scenario to obtain a simulated risk state corresponding to the main vehicle when driving in the pending driving scenario; when the simulated risk state is different from the historical risk state, the target obstacle is determined as a key obstacle, and the pending driving scenario is determined as a second driving scenario; wherein the historical risk state is the actual risk state corresponding to the main vehicle when driving in the first driving scenario.

[0128] Obtain first scene information of a first driving scene.

[0129] Obtain second scene information of the second driving scene.

[0130] The first scene information is input into the scene representation model to obtain a first representation result output by the scene representation model; the second scene information is input into the scene representation model to obtain a second representation result output by the scene representation model.

[0131] Obtain a first loss between the first characterization result and the second characterization result; if the first loss meets the first loss requirement, obtain a trained scene characterization model; if the first loss does not meet the first loss requirement, adjust parameters of the scene characterization model. The first loss may be a first cosine similarity, and the first loss requirement may be that the first cosine similarity is less than a first similarity threshold. The first similarity threshold may be set according to actual application requirements, for example, may be set to 0.3, and this is not limited in the present embodiment.

[0132] Get the third driving scene.

[0133] Any obstacle whose distance from the host vehicle is greater than a second distance threshold is selected from the third driving scenario as a second pending obstacle; if the second a posteriori decision corresponding to the second pending obstacle is to ignore or follow, and if the driving path of the second pending obstacle in the third driving scenario does not interact with the driving path of the host vehicle in the third driving scenario, the second pending obstacle is determined as a non-critical obstacle; wherein the second a posteriori decision is the driving decision made by the host vehicle with respect to the second pending obstacle; and the non-critical obstacle is removed from the third driving scenario to obtain a fourth driving scenario.

[0134] Obtain third scene information of the third driving scene.

[0135] Obtain fourth scene information of a fourth driving scene.

[0136] The third scene information is input into the scene representation model to obtain a third representation result output by the scene representation model; the fourth scene information is input into the scene representation model to obtain a fourth representation result output by the scene representation model.

[0137] Obtain a second loss between the third characterization result and the fourth characterization result; if the second loss meets the second loss requirement, obtain a trained scene representation model; if the second loss does not meet the second loss requirement, adjust parameters of the scene representation model. The second loss may be a second cosine similarity, and the second loss requirement may be that the second cosine similarity is greater than a second similarity threshold. The second similarity threshold can be set according to actual application requirements, for example, can be set to 0.95, and this is not limited in the present embodiment.

[0138] See also Figure 6 , which is a scene diagram of a scene representation model training method provided in an embodiment of the present disclosure.

[0139] As previously mentioned, the scene representation model training method provided by the embodiments of the present disclosure is applied to electronic devices. The term "electronic device" is intended to refer to various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants (PDAs), servers, blade servers, mainframe computers, in-vehicle computers, and other suitable computers. The term "electronic device" may also refer to various forms of mobile devices, such as personal digital assistants (PDAs), cellular phones, smartphones, wearable devices, and other similar computing devices.

[0140] Electronic devices can be used to:

[0141] Obtaining first scene information of a first driving scene;

[0142] Obtaining second scene information of a second driving scene; wherein the second driving scene differs from the first driving scene in distribution of key obstacles, and the key obstacles are used to change the driving risk state of the host vehicle;

[0143] A scene representation model is trained based on the first scene information and the second scene information.

[0144] The first driving scene may be a historical driving scene of the main vehicle, and the second driving scene may be a derived driving scene obtained by adjusting key obstacles in the first driving scene.

[0145] It should be noted that, in the embodiments of the present disclosure, Figure 6 The scene diagram shown is only for illustration and not for limitation. Those skilled in the art can Figure 6 Various obvious changes and / or substitutions may be made to the examples, and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0146] The present disclosure provides an obstacle marking method, which can be applied to electronic devices. Figure 7 The flowchart shown in FIG. 1 illustrates an obstacle marking method provided by an embodiment of the present disclosure. It should be noted that although the flowchart shows a logical order, in some cases, the steps shown or described may be performed in other orders.

[0147] Step S701, obtaining driving scene information of the current driving scene;

[0148] Step S702: Inputting the driving scene information into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation result based on the driving scene information; wherein the scene representation model is obtained by training using a scene representation model training method;

[0149] Step S703: Mark the current obstacle in the current driving scene based on the intermediate parameters to obtain an obstacle marking result.

[0150] The current driving scene may be a real driving scene of the current host vehicle.

[0151] The current scene information may include the current vehicle information and the current obstacle information. The current vehicle information may include the current vehicle's speed, acceleration, and position information in the current driving scene. The current obstacle information may include the obstacle type in the current driving scene, specifically whether it is a movable obstacle or a fixed obstacle. If the obstacle type is a movable obstacle, the current obstacle information may also include the obstacle's speed, acceleration, and position information in the current driving scene. If the obstacle type is a fixed obstacle, the current obstacle information may also include the obstacle's corresponding area location in the current driving scene.

[0152] In addition, in the embodiment of the present disclosure, the scene representation model may have the following Figure 4A The network structure shown in the figure, the scene representation model can also have the following Figure 4B The network structure is shown in Figure 2. The scene representation model has the following characteristics: Figure 4B In the case of the network structure shown, the current scene information may include not only the current main vehicle information and the current obstacle information, but also the current traffic instruction information, for example, relevant information of traffic signs such as lanes, stop lines, crosswalks, and traffic signs, which may specifically include lane width information, stop line position information, crosswalk position information, semantic information of traffic signs, etc.

[0153] In the embodiment of the present disclosure, the process of "inputting driving scene information into a trained scene representation model to obtain the scene representation model outputting a current scene representation result based on the driving scene information" can be referred to the aforementioned description of "inputting target scene information into the scene representation model to extract the fusion features of the main vehicle from the target scene information through the scene representation model, and extracting obstacle features from the target scene information; obtaining the scene representation results output by the scene representation model based on the correlation between the fusion features of the main vehicle and the obstacle features", which will not be repeated here.

[0154] After obtaining the obstacle marking result, a behavior decision for each marked current obstacle can be obtained based on the obstacle marking result, which will not be described in detail here.

[0155] The obstacle marking method provided by the embodiments of the present disclosure can obtain driving scene information for the current driving scene; input the driving scene information into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation result based on the driving scene information; wherein the scene representation model is trained using a scene representation model training method; and based on the intermediate parameters, the current obstacle in the current driving scene is marked to obtain an obstacle marking result. Because the scene representation model is a learnable neural network model, marking the current obstacle in the current driving scene based on the intermediate parameters obtained by the scene representation model to obtain an obstacle marking result can improve the accuracy of the obstacle marking result compared to the prior art.

[0156] In the disclosed embodiment, the intermediate parameters may include the fusion features of the current host vehicle extracted from the current scene information, and the features of the current obstacle extracted from the current scene information. Based on this, in some optional embodiments, "marking obstacles in the current driving scene based on the intermediate parameters" may include the following steps:

[0157] Based on the fusion features of the current main vehicle and the features of the current obstacles, multiple obstacles in the current driving scene are ranked by their criticality;

[0158] Select a target number of current obstacles with the highest criticality from multiple current obstacles as obstacles to be marked;

[0159] Mark the current obstacle to be marked and obtain the obstacle marking result.

[0160] As previously mentioned, in the disclosed embodiment, the scene representation model may include a first attention module, and the first attention module may be a neural network module implemented based on the attention mechanism. Therefore, after obtaining the fusion features of the current main vehicle and the current obstacle features, the fusion features of the current main vehicle and the current obstacle features are used as intermediate parameters, and the scene representation result obtained accordingly may include: obtaining the current query parameter based on the fusion features of the current main vehicle, and obtaining the current key parameter based on the current obstacle features; calculating the correlation between the fusion features of the current main vehicle and the current obstacle features based on the current query parameters and the current key parameters; and obtaining the current scene representation result based on the correlation between the fusion features of the current main vehicle and the current obstacle features. Wherein:

[0161] Query2 = X21 * W2 Q

[0162] Key2=X22*W2 K

[0163] Among them, Query2 is the current query parameter, X21 is the fusion feature of the current main vehicle, and W2 Q is the third parameter matrix, that is, the adjusted first parameter matrix, Key2 is the current key parameter, that is, the adjusted second parameter matrix, X22 is the current obstacle feature, W2 K is the fourth parameter matrix.

[0164] After obtaining the current query parameters and the current key parameters, the first function can be used to process the current query parameters and the current key parameters to obtain the degree of influence of each current obstacle on the host vehicle in the current driving scene. The current obstacle is the obstacle perceived by the scene representation model in the current driving scene. This process can be specifically characterized as follows:

[0165] attn_score=matmul(Query2,Key2)

[0166] Among them, attn_score is the impact degree of each current obstacle on the main vehicle in the current driving scene, and matmul() is the first functional function.

[0167] Afterwards, each obstacle in the current driving scene can be sorted by the second function from the lowest criticality. This process can be specifically characterized as follows:

[0168] importance_rank=Sort(attn_score)

[0169] Among them, importance_rank is the attention weight ranking result of each current obstacle in the current driving scene, and Sort() is the second function.

[0170] Finally, a target number of obstacles ranked high in criticality can be selected from the multiple current obstacles as obstacles to be marked, and the obstacles to be marked are marked to obtain obstacle marking results. The target number can be set according to actual application requirements, for example, it can be set to 4, and this embodiment of the disclosure does not limit this.

[0171] Through the above steps, in the disclosed embodiment, multiple current obstacles in the current driving scene can be ranked by their criticality based on the fused features of the current host vehicle and the current obstacle features. A target number of current obstacles with the highest criticality rankings are then selected from the multiple current obstacles as obstacles to be marked. These obstacles to be marked are then marked to obtain an obstacle marking result. Because the disclosed embodiment ranks multiple current obstacles in the current driving scene by their criticality based on the fused features of the current host vehicle and the current obstacle features, and takes into account the correlation between each current obstacle and the current host vehicle, the reliability of the ranking result can be improved, thereby further improving the accuracy of the obstacle marking result.

[0172] The following describes the complete process of an obstacle marking method provided by an embodiment of the present disclosure.

[0173] Get the driving scene information of the current driving scene.

[0174] The driving scene information is input into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation result based on the driving scene information; wherein the scene representation model is trained by a scene representation model training method, and the intermediate parameters may include fusion features of the current main vehicle extracted from the current scene information, and current obstacle features extracted from the current scene information.

[0175] Based on the fusion features of the current main vehicle and the features of the current obstacles, multiple current obstacles in the current driving scene are ranked by their criticality; a target number of current obstacles with high criticality rankings are selected from the multiple current obstacles as obstacles to be marked; the current obstacles to be marked are marked to obtain the obstacle marking result.

[0176] See also Figure 8 , which is a scene diagram of an obstacle marking method provided in an embodiment of the present disclosure.

[0177] As mentioned above, the obstacle marking method provided by the embodiments of the present disclosure is applied to electronic devices. The electronic devices are intended to represent various forms of digital computers, such as vehicle computers.

[0178] Electronic devices can be used to:

[0179] Get the driving scene information of the current driving scene;

[0180] Inputting the driving scene information into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation result based on the driving scene information; wherein the scene representation model is trained using a scene representation model training method;

[0181] Based on the intermediate parameters, the current obstacle in the current driving scene is marked to obtain an obstacle marking result.

[0182] The current driving scene may be a real driving scene of the current host vehicle.

[0183] In the disclosed embodiments, a perception system installed on the current host vehicle can sense the driving environment and obtain environmental perception data. The electronic device then constructs the current driving scene based on the environmental perception data and obtains driving scene information for the current driving scene. The perception system may include an imaging unit, a laser radar, a millimeter-wave radar, an ultrasonic radar, and the like, which are not limited in the disclosed embodiments.

[0184] It should be noted that, in the embodiments of the present disclosure, Figure 8 The scene diagram shown is only for illustration and not for limitation. Those skilled in the art can Figure 8 Various obvious changes and / or substitutions may be made to the examples, and the obtained technical solutions still fall within the scope of the disclosure of the embodiments of the present disclosure.

[0185] In order to better implement the scene representation model training method, the embodiment of the present disclosure also provides a scene representation model training device 900, which can be integrated into an electronic device. Figure 9 The structural diagram shown illustrates a scene representation model training device 900 provided in the disclosed embodiment.

[0186] The scene representation model training device 900 may include:

[0187] A first information acquisition unit 901 is configured to acquire first scene information of a first driving scene;

[0188] A second information acquisition unit 902 is configured to acquire second scene information of a second driving scene; wherein the second driving scene differs from the first driving scene in the distribution of key obstacles, and the key obstacles are used to change the driving risk state of the host vehicle;

[0189] The first model training unit 903 is configured to train the scene representation model based on the first scene information and the second scene information.

[0190] In some optional implementations, the first driving scene is a historical driving scene of the main vehicle, and the apparatus further includes a first scene creation unit configured to:

[0191] Select a target obstacle from the first driving scene;

[0192] Adjust the target obstacle in the first driving scene to obtain a pending driving scene;

[0193] Performing an automatic driving simulation on the main vehicle in the pending driving scenario to obtain a simulation risk state corresponding to the main vehicle driving in the pending driving scenario;

[0194] When the simulated risk state is different from the historical risk state, the target obstacle is determined as the key obstacle, and the pending driving scenario is determined as the second driving scenario; wherein the historical risk state is the actual risk state corresponding to the main vehicle when driving in the first driving scenario.

[0195] In some optional implementations, the first scene creation unit is configured to:

[0196] Determining a scene category of a first driving scene;

[0197] When the scene category is a safety scene, a first undetermined obstacle corresponding to the first a posteriori decision to avoid is selected from the first driving scene as the target obstacle; wherein the first a posteriori decision is a driving decision made by the host vehicle with respect to the first undetermined obstacle;

[0198] When the scene category is a collision scene or an emergency braking scene, any obstacle whose distance from the host vehicle is less than a first distance threshold is selected from the first driving scene as a target obstacle.

[0199] In some optional implementations, the first scene creation unit is configured to:

[0200] When the scene category is a safe scene, adjusting the position and / or driving time of the target obstacle in the first driving scene to obtain a pending driving scene;

[0201] And / or, when the scene category is a collision scene or an emergency braking scene, the target obstacle is removed from the first driving scene to obtain a pending driving scene.

[0202] In some optional implementations, the first model training unit 903 is configured to:

[0203] Inputting the target scene information into the scene representation model to extract the fusion features of the host vehicle from the target scene information and the obstacle features from the target scene information through the scene representation model; wherein the target scene information is any one of the first scene information and the second scene information;

[0204] Obtaining a scene representation result output by the scene representation model based on the correlation between the fusion features of the host vehicle and the obstacle features; wherein, when the target scene information is the first scene information, the scene representation result is the first representation result; and when the target scene information is the second scene information, the scene representation result is the second representation result;

[0205] The scene representation model is trained based on the first representation result and the second representation result.

[0206] In some optional implementations, the first model training unit 903 is configured to:

[0207] Extract independent features of the main vehicle from the target scene information;

[0208] Extracting traffic indication features from target scene information;

[0209] Based on the independent features of the main vehicle and the traffic indication features, the fusion features of the main vehicle are obtained.

[0210] In some optional implementations, the scene representation model training apparatus 900 further includes a second model training unit configured to:

[0211] Obtaining third scene information of a third driving scene;

[0212] Obtaining fourth scene information of a fourth driving scene; wherein there is a difference in distribution of non-critical obstacles between the third driving scene and the fourth driving scene;

[0213] The scene representation model is trained based on the third scene information and the fourth scene information.

[0214] In some optional implementations, the third driving scene is a historical driving scene of the main vehicle, and the apparatus further includes a second scene creation unit configured to:

[0215] Select any obstacle from the third driving scene whose distance from the host vehicle is greater than the second distance threshold as the second pending obstacle;

[0216] If the second a posteriori decision corresponding to the second pending obstacle is to ignore or follow, and the driving path of the second pending obstacle in the third driving scenario does not interact with the driving path of the host vehicle in the third driving scenario, the second pending obstacle is determined as a non-critical obstacle; wherein the second a posteriori decision is the driving decision made by the host vehicle with respect to the second pending obstacle;

[0217] In the third driving scenario, non-critical obstacles are removed to obtain the fourth driving scenario.

[0218] In the embodiment of the present disclosure, the specific functions and examples of each unit of the scene representation model training device 900 can be described by referring to the relevant descriptions of the corresponding steps in the above method embodiment, which will not be repeated here.

[0219] In order to better implement the obstacle marking method, the embodiment of the present disclosure also provides an obstacle marking device, which can be integrated into an electronic device. Figure 10The structural schematic diagram shown in FIG. 1 illustrates an obstacle marking device 1000 provided in the disclosed embodiment.

[0220] The obstacle marking device 1000 may include:

[0221] The current information acquisition unit 1001 is used to acquire driving scene information of the current driving scene;

[0222] An intermediate parameter acquisition unit 1002 is configured to input driving scene information into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation result based on the driving scene information; wherein the scene representation model is obtained by training using the method of any one of claims 1 to 8;

[0223] The obstacle marking unit 1003 is configured to mark the current obstacle in the current driving scene based on the intermediate parameters to obtain an obstacle marking result.

[0224] In some optional implementations, the intermediate parameters include fusion features of the current host vehicle extracted from the current scene information and current obstacle features extracted from the current scene information; the obstacle marking unit 1003 is configured to:

[0225] Based on the fusion features of the current main vehicle and the features of the current obstacles, multiple obstacles in the current driving scene are ranked by their criticality;

[0226] A target number of current obstacles is selected from multiple current obstacles with a higher criticality ranking as obstacles to be marked;

[0227] Mark the current obstacle to be marked and obtain the obstacle marking result.

[0228] In the embodiment of the present disclosure, the specific functions and examples of each unit of the obstacle marking device 1000 can be found in the relevant descriptions of the corresponding steps in the above method embodiment, and will not be repeated here.

[0229] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0230] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, an autonomous driving vehicle, a readable storage medium, and a computer program product.

[0231] Figure 11A schematic block diagram of an example electronic device 1100 that can be used to implement embodiments of the present disclosure is shown. The electronic device 1100 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 1100 can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0232] like Figure 11 As shown, the device 1100 includes a computing unit 1101, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1102 or a computer program loaded from a storage unit 1108 into a random access memory (RAM) 1103. Various programs and data required for the operation of the device 1100 can also be stored in the RAM 1103. The computing unit 1101, the ROM 1102, and the RAM 1103 are connected to each other via a bus 1104. An input / output (I / O) interface 1105 is also connected to the bus 1104.

[0233] Various components in device 1100 are connected to I / O interface 1105, including an input unit 1106, such as a keyboard and mouse; an output unit 1107, such as various types of displays and speakers; a storage unit 1108, such as a magnetic disk and optical disk; and a communication unit 1109, such as a network card, a modem, a wireless communication transceiver, etc. Communication unit 1109 allows device 1100 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0234] Computing unit 1101 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of computing unit 1101 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Computing unit 1101 performs the various methods and processes described above, such as the scene representation model training method and / or the obstacle marking method. For example, in some embodiments, the scene representation model training method and / or the obstacle marking method may be implemented as a computer software program tangibly embodied in a machine-readable medium, such as storage unit 1108. In some embodiments, part or all of the computer program may be loaded and / or installed onto device 1100 via ROM 1102 and / or communication unit 1109. When the computer program is loaded into RAM 1103 and executed by computing unit 1101, one or more steps of the scene representation model training method and / or obstacle marking method described above may be performed. Alternatively, in other embodiments, computing unit 1101 may be configured to perform the scene representation model training method and / or obstacle marking method in any other suitable manner (e.g., via firmware).

[0235] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0236] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0237] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by an instruction execution system, device or equipment or used in combination with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a RAM, a ROM, an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0238] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a cathode ray tube (CRT) display or a liquid crystal display (LCD)) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0239] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0240] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises from computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0241] The disclosed embodiment also provides an autonomous driving vehicle, including an electronic device 1100.

[0242] An embodiment of the present disclosure further provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute a scene representation model training method and / or an obstacle marking method.

[0243] An embodiment of the present disclosure also provides a computer program product, including a computer program, which implements a scene representation model training method and / or an obstacle marking method when executed by a processor.

[0244] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, can be executed sequentially, or can be executed in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document is not limited here. In addition, in this disclosure, relational terms such as "first", "second", "third", etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply the existence of any such actual relationship or order between these entities or operations. In addition, in this disclosure, "multiple" can be understood as at least two, and "any" can be understood as any one.

[0245] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A scene representation model training method, comprising: Obtaining first scene information of a first driving scene; Obtaining second scene information of a second driving scene; wherein the second driving scene differs from the first driving scene in the distribution of key obstacles, and the key obstacles are used to change the driving risk state of the host vehicle; Training the scene representation model based on the first scene information and the second scene information; Wherein, the first driving scene is a historical driving scene of the host vehicle, and the method further includes: Selecting a target obstacle from the first driving scene; Adjusting the target obstacle in the first driving scene to obtain a pending driving scene; Performing an automatic driving simulation on the host vehicle in the pending driving scenario to obtain a simulation risk state corresponding to the host vehicle driving in the pending driving scenario; When the simulated risk state is different from the historical risk state, the target obstacle is determined as the key obstacle, and the pending driving scenario is determined as the second driving scenario; wherein the historical risk state is the actual risk state corresponding to the main vehicle when driving in the first driving scenario.

2. The method according to claim 1, wherein The selecting a target obstacle from the first driving scene includes: determining a scene category of the first driving scene; When the scenario category is a safety scenario, selecting a first undetermined obstacle corresponding to a first a posteriori decision to avoid from the first driving scenario as the target obstacle; wherein the first a posteriori decision is a driving decision made by the host vehicle with respect to the first undetermined obstacle; When the scene category is a collision scene or an emergency braking scene, any obstacle whose distance from the host vehicle is less than a first distance threshold is selected from the first driving scene as the target obstacle.

3. The method according to claim 2, wherein: The step of adjusting the target obstacle in the first driving scene to obtain a pending driving scene includes: When the scene category is a safe scene, adjusting the position and / or driving time of the target obstacle in the first driving scene to obtain the pending driving scene; And / or, when the scene category is a collision scene or an emergency braking scene, the target obstacle is removed from the first driving scene to obtain the pending driving scene.

4. The method according to any one of claims 1 to 3, wherein The training of the scene representation model based on the first scene information and the second scene information includes: Inputting target scene information into the scene representation model, so as to extract the fusion features of the host vehicle from the target scene information and the obstacle features from the target scene information through the scene representation model; wherein the target scene information is any one of the first scene information and the second scene information; Obtaining a scene representation result output by the scene representation model based on the correlation between the fusion feature of the host vehicle and the obstacle feature; wherein, when the target scene information is the first scene information, the scene representation result is a first representation result, and when the target scene information is the second scene information, the scene representation result is a second representation result; The scene representation model is trained based on the first representation result and the second representation result.

5. The method according to claim 4, wherein The extracting the fusion feature of the main vehicle from the target scene information includes: Extracting independent features of the host vehicle from the target scene information; extracting traffic indication features from the target scene information; A fusion feature of the host vehicle is obtained based on the independent feature of the host vehicle and the traffic indication feature.

6. The method according to claim 1, further comprising: Obtaining third scene information of a third driving scene; Acquiring fourth scene information of a fourth driving scene; wherein there is a difference in distribution of non-critical obstacles between the third driving scene and the fourth driving scene; The scene representation model is trained based on the third scene information and the fourth scene information.

7. The method according to claim 6, wherein: The third driving scene is a historical driving scene of the host vehicle, and the method further includes: Selecting any obstacle from the third driving scene, the obstacle having a distance from the host vehicle greater than a second distance threshold, as a second obstacle to be determined; If the second a posteriori decision corresponding to the second pending obstacle is to ignore or follow, and the driving path of the second pending obstacle in the third driving scenario does not interact with the driving path of the host vehicle in the third driving scenario, determining the second pending obstacle as a non-critical obstacle; wherein the second a posteriori decision is the driving decision made by the host vehicle with respect to the second pending obstacle; The non-critical obstacles are removed from the third driving scene to obtain the fourth driving scene.

8. A method for marking an obstacle, comprising: Get the driving scene information of the current driving scene; Inputting the driving scene information into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation result based on the driving scene information; wherein the scene representation model is trained by the method of any one of claims 1 to 7; Based on the intermediate parameters, a current obstacle in the current driving scene is marked to obtain an obstacle marking result.

9. The method according to claim 8, wherein The intermediate parameters include a fusion feature of the current host vehicle extracted from the current scene information and a current obstacle feature extracted from the current scene information; The marking of obstacles in the current driving scene based on the intermediate parameters includes: sorting the criticality of multiple current obstacles in the current driving scene based on the fusion feature of the current host vehicle and the feature of the current obstacle; Selecting a target number of current obstacles with higher criticality rankings from the multiple current obstacles as obstacles to be marked; Mark the obstacle to be marked to obtain the obstacle marking result.

10. A scene representation model training device, comprising: A first information acquisition unit, configured to acquire first scene information of a first driving scene; A second information acquisition unit is configured to acquire second scene information of a second driving scene; wherein the second driving scene differs from the first driving scene in the distribution of key obstacles, and the key obstacles are used to change the driving risk state of the host vehicle; a first model training unit, configured to train the scene representation model based on the first scene information and the second scene information; The first driving scene is a historical driving scene of the host vehicle, and the device further includes a first scene creation unit, which is configured to: Selecting a target obstacle from the first driving scene; Adjusting the target obstacle in the first driving scene to obtain a pending driving scene; Performing an automatic driving simulation on the host vehicle in the pending driving scenario to obtain a simulation risk state corresponding to the host vehicle driving in the pending driving scenario; When the simulated risk state is different from the historical risk state, the target obstacle is determined as the key obstacle, and the pending driving scenario is determined as the second driving scenario; wherein the historical risk state is the actual risk state corresponding to the main vehicle when driving in the first driving scenario.

11. The device according to claim 10, wherein The first scene creation unit is used to: determining a scene category of the first driving scene; When the scenario category is a safety scenario, selecting a first undetermined obstacle corresponding to a first a posteriori decision to avoid from the first driving scenario as the target obstacle; wherein the first a posteriori decision is a driving decision made by the host vehicle with respect to the first undetermined obstacle; When the scene category is a collision scene or an emergency braking scene, any obstacle whose distance from the host vehicle is less than a first distance threshold is selected from the first driving scene as the target obstacle.

12. The device according to claim 11, wherein The first scene creation unit is used to: When the scene category is a safe scene, adjusting the position and / or driving time of the target obstacle in the first driving scene to obtain the pending driving scene; And / or, when the scene category is a collision scene or an emergency braking scene, the target obstacle is removed from the first driving scene to obtain the pending driving scene.

13. The device according to any one of claims 10 to 12, wherein: The first model training unit is used to: Inputting target scene information into the scene representation model, so as to extract the fusion features of the host vehicle from the target scene information and the obstacle features from the target scene information through the scene representation model; wherein the target scene information is any one of the first scene information and the second scene information; Obtaining a scene representation result output by the scene representation model based on the correlation between the fusion feature of the host vehicle and the obstacle feature; wherein, when the target scene information is the first scene information, the scene representation result is a first representation result, and when the target scene information is the second scene information, the scene representation result is a second representation result; The scene representation model is trained based on the first representation result and the second representation result.

14. The device according to claim 13, wherein The first model training unit is used to: Extracting independent features of the host vehicle from the target scene information; extracting traffic indication features from the target scene information; A fusion feature of the host vehicle is obtained based on the independent feature of the host vehicle and the traffic indication feature.

15. The apparatus according to claim 10, further comprising a second model training unit, configured to: Obtaining third scene information of a third driving scene; Obtain fourth scene information of a fourth driving scene; wherein, There is a difference in the distribution of non-critical obstacles between the third driving scene and the fourth driving scene; The scene representation model is trained based on the third scene information and the fourth scene information.

16. The device according to claim 15, wherein The third driving scene is a historical driving scene of the host vehicle, and the device further includes a second scene creation unit, which is configured to: Selecting any obstacle from the third driving scene, the obstacle having a distance from the host vehicle greater than a second distance threshold, as a second obstacle to be determined; If the second a posteriori decision corresponding to the second undetermined obstacle is to ignore or follow, and the driving path of the second undetermined obstacle in the third driving scenario does not interact with the driving path of the host vehicle in the third driving scenario, determining the second undetermined obstacle as a non-critical target obstacle; wherein the second a posteriori decision is the driving decision made by the host vehicle with respect to the second undetermined obstacle; The non-critical obstacles are removed from the third driving scene to obtain the fourth driving scene.

17. An obstacle marking device comprising: A current information acquisition unit, used to acquire driving scene information of the current driving scene; an intermediate parameter acquisition unit, configured to input the driving scene information into a trained scene representation model to obtain intermediate parameters obtained by the scene representation model in the process of outputting a current scene representation result based on the driving scene information; wherein the scene representation model is trained and obtained by the method of any one of claims 1 to 7; The obstacle marking unit is configured to mark the current obstacle in the current driving scene based on the intermediate parameters to obtain an obstacle marking result.

18. The device according to claim 17, wherein The intermediate parameters include a fusion feature of the current host vehicle extracted from the current scene information and a current obstacle feature extracted from the current scene information; The obstacle marking unit is used for: sorting the criticality of multiple current obstacles in the current driving scene based on the fusion feature of the current host vehicle and the feature of the current obstacle; Selecting a target number of current obstacles with higher criticality rankings from the multiple current obstacles as obstacles to be marked; Mark the obstacle to be marked to obtain the obstacle marking result.

19. An electronic device comprising: at least one processor; a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 9.

20. An autonomous driving vehicle comprising the electronic device according to claim 19.

21. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 9.

22. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 9.

Images