A human attention experience learning method and device for automatic driving of a vehicle

Abstract

The application discloses a human attention experience learning method and device for automatic driving of a vehicle, and comprises the following steps: acquiring current driving scene data, wherein the current driving scene data comprises at least one area data; processing the current driving scene data through an attention distribution model to obtain weight values of different area data; and sequentially processing the area data from high weight values to low weight values and generating driving decisions. After acquiring the current driving scene data of the vehicle, the weight values of different area data in the current driving scene data are obtained through the attention distribution model, that is, the automatic driving vehicle can distinguish the importance of each area data in the current driving scene data under different driving environments; and the area data with high weight values are preferentially processed, that is, the automatic driving system will focus on processing of key information with higher weight values, so that the response speed of generating driving decisions and the driving safety are improved.

Description

Technical Field

[0001] This invention relates to the field of autonomous driving technology, and in particular to a method and apparatus for learning human attention experience in autonomous driving of vehicles. Background Technology

[0002] Currently, computer vision technology is widely used in the field of autonomous driving. The most common approach relies on a large amount of labeled data and model training resources to train various mature machine learning models for autonomous driving. Cameras capture information about the driving environment, and the trained machine learning model determines the direction of travel, automatically plans the path, and avoids obstacles. However, the problem remains that the model cannot react in time to various sudden and complex situations. How to effectively learn the attention distribution characteristics of human drivers in various complex scenarios during driving, thereby improving the vehicle's understanding of various complex scenarios and enhancing the optimization level of the vehicle's autonomous driving control strategy, is one of the key challenges for autonomous vehicles in dealing with various complex "long-tail scenarios." Summary of the Invention

[0003] The technical problem to be solved by this invention is to provide a method and apparatus for learning human attention experience in autonomous driving of vehicles, thereby improving the response speed of autonomous driving systems.

[0004] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows:

[0005] A method for learning human attention experience in autonomous driving of vehicles includes:

[0006] Acquire current driving scenario data, which includes at least one region data;

[0007] The current driving scenario data is processed by an attention distribution model to obtain weight values ​​for data in different regions.

[0008] The regional data are processed sequentially from high weight values ​​to low weight values, and driving decisions are generated.

[0009] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows:

[0010] A human attention experience learning device for autonomous driving of vehicles includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0011] Acquire current driving scenario data, which includes at least one region data;

[0012] The current driving scenario data is processed by an attention distribution model to obtain weight values ​​for data in different regions.

[0013] The regional data are processed sequentially from high weight values ​​to low weight values, and driving decisions are generated.

[0014] The beneficial effects of this invention are as follows: after acquiring the current driving scenario data of the vehicle, the weight values ​​of different regions of the current driving scenario data are obtained through the attention distribution model, that is, the autonomous vehicle can distinguish the importance of each region of the current driving scenario data under different driving environments; and prioritize the processing of region data with high weight values, that is, the autonomous driving system will focus on processing key information with higher weight values, thereby improving the response speed of generating driving decisions and driving safety. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the steps of a human attention experience learning method for autonomous driving of vehicles according to an embodiment of the present invention.

[0016] Figure 2 This is a flowchart of another step in a human attention experience learning method for autonomous driving of vehicles according to an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram illustrating the construction of an attention distribution model for a human attention experience learning method for autonomous driving of vehicles, as described in an embodiment of the present invention.

[0018] Figure 4 This is a schematic diagram of the structure of a human attention experience learning device for autonomous driving of a vehicle, according to an embodiment of the present invention. Detailed Implementation

[0019] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0020] Please refer to Figure 1 A method for learning human attention experience in autonomous driving vehicles, comprising:

[0021] Acquire current driving scenario data, which includes at least one region data;

[0022] The current driving scenario data is processed by an attention distribution model to obtain weight values ​​for data in different regions.

[0023] The regional data are processed sequentially from high weight values ​​to low weight values, and driving decisions are generated.

[0024] As can be seen from the above description, the beneficial effects of the present invention are as follows: after acquiring the current driving scenario data of the vehicle, the weight values ​​of different regions of the current driving scenario data are obtained through the attention distribution model, that is, the autonomous vehicle can distinguish the importance of each region of the current driving scenario data under different driving environments; and prioritize the processing of region data with high weight values, that is, the autonomous driving system will focus on processing key information with higher weight values, thereby improving the response speed of generating driving decisions and driving safety.

[0025] Furthermore, the process of processing the current driving scenario data using the attention distribution model includes:

[0026] Acquire vehicle driving scene data and generate a three-dimensional real-time dynamic map based on the vehicle driving scene data;

[0027] Obtain the driver's current gaze.

[0028] The image region in the three-dimensional real-time dynamic image that matches the currently viewed screen is marked as the attention focus region;

[0029] The marked 3D real-time dynamic graph is used as a training set to train the original attention distribution model, thus obtaining the attention distribution model.

[0030] As described above, after generating a 3D real-time dynamic image based on vehicle driving scene data, the driver's current gaze is captured and matched with the 3D real-time dynamic image. The attention concentration area is marked in the 3D real-time dynamic image for model training. That is, the model can learn the driver's method of adjusting visual attention distribution during driving, eliminate information that is irrelevant to driving behavior decisions, and focus on the processing of key information, so that the trained model can make driving decisions quickly and smoothly.

[0031] Furthermore, the process of acquiring the driver's current gaze frame includes:

[0032] Determine whether the current vehicle driving status has changed. If so, proceed with the step of obtaining the driver's current gaze area.

[0033] As described above, by capturing the driver's current gaze only after detecting a change in the current vehicle's driving state, such as when the vehicle changes lanes, accelerates, or decelerates, the driver's current gaze can be captured in a targeted manner. This reduces unnecessary training datasets and improves model accuracy and training efficiency.

[0034] Furthermore, before processing the region data sequentially from high weight values ​​to low weight values, the process further includes:

[0035] Determine whether emergency high-priority information input has been detected; if so, generate the driving decision based on the emergency high-priority information.

[0036] If not, the driving decision is generated based on all the data from the stated regions.

[0037] As described above, when an emergency high-priority information input is detected, a driving decision is directly generated based on the emergency high-priority information. That is, the autonomous vehicle prioritizes the handling of emergency high-priority information, thereby improving the safety of the driving process.

[0038] Furthermore, the step of processing the regional data sequentially from high weight values ​​to low weight values ​​and generating driving decisions includes:

[0039] Obtain the weight value threshold;

[0040] Data in regions whose weight values ​​are lower than the weight threshold will be filtered out.

[0041] The driving decision is generated based on all the data from the filtered-out regions.

[0042] As described above, by filtering out data from regions with low weight values, information that is irrelevant or of low relevance to driving behavior decisions can be eliminated, while the focus is placed on processing key information with high relevance. This reduces the amount of data processed by the autonomous driving system, thereby improving the efficiency of data processing and the response speed and accuracy of generating driving decisions.

[0043] Another embodiment of the present invention provides a human attention experience learning device for autonomous driving of vehicles, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0044] Acquire current driving scenario data, which includes at least one region data;

[0045] The current driving scenario data is processed by an attention distribution model to obtain weight values ​​for data in different regions.

[0046] The regional data are processed sequentially from high weight values ​​to low weight values, and driving decisions are generated.

[0047] As described above, after acquiring the current driving scenario data of the vehicle, the weight values ​​of different regions of the current driving scenario data are obtained through the attention distribution model. That is, the autonomous vehicle can distinguish the importance of each region of the current driving scenario data under different driving environments; and prioritize the processing of region data with higher weight values. In other words, the autonomous driving system will focus on processing key information with higher weight values, thereby improving the response speed of generating driving decisions and driving safety.

[0048] Furthermore, the process of processing the current driving scenario data using the attention distribution model includes:

[0049] Acquire vehicle driving scene data and generate a three-dimensional real-time dynamic map based on the vehicle driving scene data;

[0050] Obtain the driver's current gaze.

[0051] The image region in the three-dimensional real-time dynamic image that matches the currently viewed screen is marked as the attention focus region;

[0052] The marked 3D real-time dynamic graph is used as a training set to train the original attention distribution model, thus obtaining the attention distribution model.

[0053] As described above, after generating a 3D real-time dynamic image based on vehicle driving scene data, the driver's current gaze is captured and matched with the 3D real-time dynamic image. The attention concentration area is marked in the 3D real-time dynamic image for model training. That is, the model can learn the driver's method of adjusting visual attention distribution during driving, eliminate information that is irrelevant to driving behavior decisions, and focus on the processing of key information, so that the trained model can make driving decisions quickly and smoothly.

[0054] Furthermore, the process of acquiring the driver's current gaze frame includes:

[0055] Determine whether the current vehicle driving status has changed. If so, proceed with the step of obtaining the driver's current gaze area.

[0056] As described above, by capturing the driver's current gaze only after detecting a change in the current vehicle's driving state, such as when the vehicle changes lanes, accelerates, or decelerates, the driver's current gaze can be captured in a targeted manner. This reduces unnecessary training datasets and improves model accuracy and training efficiency.

[0057] Furthermore, before processing the region data sequentially from high weight values ​​to low weight values, the process further includes:

[0058] Determine whether emergency high-priority information input has been detected; if so, generate the driving decision based on the emergency high-priority information.

[0059] If not, the driving decision is generated based on all the data from the stated regions.

[0060] As described above, when an emergency high-priority information input is detected, a driving decision is directly generated based on the emergency high-priority information. That is, the autonomous vehicle prioritizes the processing of emergency high-priority information, thereby improving the safety of the driving process.

[0061] Furthermore, the step of processing the regional data sequentially from high weight values ​​to low weight values ​​and generating driving decisions includes:

[0062] Obtain the weight value threshold;

[0063] Data in regions whose weight values ​​are lower than the weight threshold will be filtered out.

[0064] The driving decision is generated based on all the data from the filtered-out regions.

[0065] As described above, by filtering out data from regions with low weight values, information that is irrelevant or of low relevance to driving behavior decisions can be eliminated, while the focus is placed on processing key information with high relevance. This reduces the amount of data processed by the autonomous driving system, thereby improving the efficiency of data processing and the response speed and accuracy of generating driving decisions.

[0066] The human attention experience learning method and device for autonomous driving provided by this invention can be applied to autonomous vehicle scenarios to improve the response speed of autonomous vehicles. The following is a detailed description of the specific implementation:

[0067] Example 1

[0068] Please refer to Figure 1 as well as Figure 2 A method for learning human attention experience in autonomous driving vehicles is proposed, wherein the autonomous vehicle incorporates a well-trained attention distribution mechanism. The specific training process of the attention distribution model is as follows:

[0069] Please refer to Figure 3 S01, Build an attention mechanism distribution learning system, equip drivers with smart glasses and other smart devices, and install sensors and vehicle cameras in vehicles;

[0070] S02. Experienced drivers wear smart glasses to drive the vehicle, training the attention distribution mechanism of the autonomous driving system in actual vehicle control.

[0071] S03. Acquire vehicle driving scene data and generate a three-dimensional real-time dynamic map based on the vehicle driving scene data; that is, the learning system can use on-board cameras distributed around the vehicle to acquire 2D scene images, and use image-based three-dimensional target detection technology to depict and render a three-dimensional real-time dynamic map of the vehicle driving environment, and use sensors to sense the vehicle driving action.

[0072] S04. Determine if the current vehicle driving status has changed. If yes, proceed to S05; otherwise, proceed to S06.

[0073] S05. Obtain the driver's current gaze area and mark the image region in the 3D real-time dynamic image that matches the current gaze area as the attention focus area; for example, using image feature extraction and matching technology, mark the driver's visual range in the 3D scene image and use this as the basis for attention distribution; at this time, the area where the driving environment overlaps with the smart camera is marked as the attention focus area; among them, the processing of static images often has problems such as different viewing angles; therefore, using 3D object detection technology to map the driving environment in real time, and relying on the image captured by the smart glasses to be marked in 3D space through image feature extraction and matching technology, can overcome the problem of different viewing angles; for example, when the driver is changing... When the driver is observing vehicles approaching from the side and rear via the rearview mirror, the smart glasses capture the scene in the rearview mirror that the driver is focusing on and mark it as the area of ​​concentrated attention. The importance of the scene around the vehicle is displayed in the form of a heat map in a 3D scene map, with the highlighted areas being the areas of concentrated attention. The marked 3D real-time dynamic map is then used as a training set to train the original attention distribution model, resulting in the attention distribution model. At the same time, sensors are used to perceive changes in the vehicle's driving state, such as acceleration, deceleration, lane change, etc., to learn the vehicle's attention distribution when dealing with emergencies.

[0074] S06. Repeatedly complete the training multiple times to improve the attention distribution learning system, switch to different driving environments to enhance the robustness of the system; thereby learning how to classify the importance of each region in the three-dimensional real-time dynamic map.

[0075] The method specifically includes the following steps:

[0076] S1. Obtain current driving scenario data, which includes data from at least one region. When an autonomous vehicle is about to make a driving behavior decision, it needs to analyze the obtained current driving scenario data.

[0077] S2. Process the current driving scenario data through the attention distribution model to obtain the weight values ​​of data in different regions. If the autonomous driving system cannot distinguish the attention focus distribution areas in the current driving scenario data, the autonomous driving system assumes that each region has the same importance in the current driving scenario, that is, the weight values ​​of each region data are the same. If the autonomous driving system can identify the weight values ​​of each region data in the current driving scenario data, then proceed to step S3.

[0078] S3. Process the region data sequentially from high weight value to low weight value and generate driving decisions; in an optional implementation, by obtaining a weight value threshold, filter out region data with weight values ​​lower than the weight value threshold, and then generate driving decisions based on all the filtered region data; that is, the attention distribution mechanism will assign weights to each region of the current driving scene, and the autonomous driving system will prioritize processing the information passed in from the attention distribution region (i.e., the region with high weight value) in the current driving environment, so that it can handle complex visual tasks more effectively;

[0079] In another optional implementation, before generating a driving decision, the method further includes: determining whether an emergency high-priority information input is detected; if so, generating a driving decision based on the emergency high-priority information; that is, when the vehicle encounters a sudden situation during driving, emergency high-priority information will be generated, such as a pedestrian or non-motorized vehicle suddenly darting out, a vehicle in front braking suddenly, a tire blowout, etc.; if not, generating a driving decision based on all area data.

[0080] Example 2

[0081] Please refer to Figure 4 A human attention experience learning device for autonomous driving of vehicles includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it performs the following steps:

[0082] S1. Obtain current driving scenario data, which includes at least one area data;

[0083] S2. Process the current driving scenario data through the attention distribution model to obtain the weight values ​​of data in different regions;

[0084] S3. Process the regional data sequentially from high weight value to low weight value and generate a driving decision; in an optional implementation, by obtaining a weight value threshold, regional data with weight values ​​lower than the weight value threshold are filtered out, and then a driving decision is generated based on all the filtered regional data; in another optional implementation, before generating a driving decision, it further includes: determining whether emergency high-priority information input is detected; if so, a driving decision is generated based on the emergency high-priority information; that is, when the vehicle encounters a sudden situation during driving, emergency high-priority information will be generated, such as the sudden appearance of a pedestrian or non-motorized vehicle, the emergency braking of the vehicle in front, a tire blowout, etc.; if not, a driving decision is generated based on all regional data.

[0085] The specific training process of the attention distribution model is as follows:

[0086] S01. Build an attention mechanism distribution learning system, equip drivers with smart glasses, and install sensors and in-vehicle cameras in vehicles;

[0087] S02. Experienced drivers wear smart glasses to drive the vehicle, training the attention distribution mechanism of the autonomous driving system in actual vehicle control.

[0088] S03. Acquire vehicle driving scenario data and generate a three-dimensional real-time dynamic map based on the vehicle driving scenario data;

[0089] S04. Determine if the current vehicle driving status has changed. If yes, proceed to S05; otherwise, proceed to S06.

[0090] S05. Obtain the driver's current gaze frame and mark the image regions in the 3D real-time dynamic image that match the current gaze frame as attention concentration regions; then use the marked 3D real-time dynamic image as a training set to train the original attention distribution model to obtain the attention distribution model.

[0091] S06. Repeatedly complete the training multiple times to improve the attention distribution learning system, and switch between different driving environments to enhance the robustness of the system.

[0092] In summary, the present invention provides a method and apparatus for learning human attention experience in autonomous driving. After acquiring the current driving scenario data of the vehicle, it obtains the weight values ​​of different regions of data in the current driving scenario data through an attention distribution model. That is, the autonomous vehicle can distinguish the importance of each region of data in the current driving scenario data under different driving environments; and prioritizes the processing of region data with higher weight values. In other words, the autonomous driving system will focus on processing key information with higher weight values, thereby improving the response speed of generating driving decisions and driving safety.

[0093] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for learning human attention experience in autonomous driving of vehicles, characterized in that, include: Acquire current driving scenario data, which includes at least one region data; Acquire vehicle driving scene data and generate a three-dimensional real-time dynamic map based on the vehicle driving scene data; Obtain the driver's current gaze. The image region in the three-dimensional real-time dynamic image that matches the currently viewed screen is marked as the attention focus region; The importance of the scene surrounding the vehicle is shown in the form of a heat map in the three-dimensional real-time dynamic image. The marked 3D real-time dynamic graph is used as a training set to train the original attention distribution model, thus obtaining the attention distribution model. The current driving scenario data is processed by the attention distribution model to obtain weight values ​​for data in different regions. The regional data are processed sequentially from high weight values ​​to low weight values, and driving decisions are generated accordingly. Before acquiring the driver's current gaze area, the following steps are included: Determine whether the current vehicle driving status has changed; if so, proceed with the step of obtaining the driver's current gaze area. The process of processing the regional data sequentially from high weight values ​​to low weight values ​​and generating driving decisions includes: Obtain the weight value threshold; Data from regions whose weight values ​​are lower than the weight value threshold will be filtered out. The driving decision is generated based on all the data from the filtered-out regions.

2. The method for learning human attention experience in autonomous driving of vehicles according to claim 1, characterized in that, Before processing the region data sequentially from high weight values ​​to low weight values, the process also includes: Determine whether emergency high-priority information input has been detected; if so, generate the driving decision based on the emergency high-priority information. If not, the driving decision is generated based on all the data from the stated regions.

3. A human attention experience learning device for autonomous driving of vehicles, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it performs the following steps: Acquire current driving scenario data, which includes at least one region data; Acquire vehicle driving scene data and generate a three-dimensional real-time dynamic map based on the vehicle driving scene data; Obtain the driver's current gaze. The image region in the three-dimensional real-time dynamic image that matches the currently viewed screen is marked as the attention focus region; The importance of the scene surrounding the vehicle is shown in the form of a heat map in the three-dimensional real-time dynamic image. The marked 3D real-time dynamic graph is used as a training set to train the original attention distribution model, thus obtaining the attention distribution model. The current driving scenario data is processed by the attention distribution model to obtain weight values ​​for data in different regions. The regional data are processed sequentially from high weight values ​​to low weight values, and driving decisions are generated accordingly. Before acquiring the driver's current gaze area, the following steps are included: Determine whether the current vehicle driving status has changed; if so, proceed with the step of obtaining the driver's current gaze area. The process of processing the regional data sequentially from high weight values ​​to low weight values ​​and generating driving decisions includes: Obtain the weight value threshold; Data from regions whose weight values ​​are lower than the weight value threshold will be filtered out. The driving decision is generated based on all the data from the filtered-out regions.

4. The human attention experience learning device for autonomous driving of a vehicle according to claim 3, characterized in that, Before processing the region data sequentially from high weight values ​​to low weight values, the following steps are also included: Determine whether emergency high-priority information input has been detected; if so, generate the driving decision based on the emergency high-priority information. If not, the driving decision is generated based on all the data from the stated regions.

Images