Automatic driving scene mining method, device, equipment, medium and program

Abstract

The embodiment of the invention provides an automatic driving scene mining method and device, equipment, a medium and a program, and the method comprises the steps: obtaining a mining demand text of an automatic driving scene; target scene features in the demand text are identified and mined, a database is inquired based on the target scene features, the database stores time sequence features related to the automatic driving scene, and the time sequence features are composed of feature frames at different moments; querying a target feature frame and a single-frame moment corresponding to the target scene feature from the time sequence feature, and querying at least one time sequence segment from the time sequence feature based on the single-frame moment of the target feature frame; and generating scene mining data of the automatic driving scene according to all the queried time sequence fragments. Therefore, the problems of low working efficiency, poor flexibility and expandability and the like caused by the fact that continuous scene fragments conforming to complex spatial-temporal logic cannot be efficiently, accurately and automatically mined from massive time series data due to dependence on manual rules and static retrieval in related technologies are solved.

Description

Technical Field

[0001] This application relates to the field of data processing technology, and in particular to a method, apparatus, device, medium and program for autonomous driving scene mining. Background Technology

[0002] The research, development, testing, and verification of autonomous driving technology heavily rely on massive amounts of real-world scenario data. As large-scale road testing of autonomous vehicles progresses, petabytes of multimodal time-series data, including perception, localization, mapping, and vehicle status, are generated daily.

[0003] In related technologies, scene mining methods have the following limitations: (1) Engineers usually need to write complex and fixed logical rules to filter data based on their prior understanding of the scene. This is not only inefficient and wasteful of human resources, but also difficult to cope with complex and ever-changing scenes, and cannot achieve effective mining. It has poor flexibility and scalability. (2) It focuses on the retrieval of single frames or short-term states. It is difficult to extract discrete single frame moments that may be interrupted due to data loss, resulting in fragmented scenes that cannot reflect complete driving behavior. (3) In order to find continuous segments from long-term, multi-vehicle data, multiple loop traversals or complex self-join queries may be used. These operations have high computational complexity and long time consumption, and cannot meet the real-time or near-real-time requirements of retrieval efficiency in large-scale data environments. Summary of the Invention

[0004] This application provides an autonomous driving scene mining method, apparatus, device, medium, and program, aiming to improve the problem in related technologies that rely on manual rules and static retrieval, which cannot efficiently and accurately automatically mine continuous scene fragments that conform to complex spatiotemporal logic from massive time-series data, resulting in low work efficiency and poor flexibility and scalability.

[0005] The first aspect of this application provides a method for mining autonomous driving scenarios, comprising the following steps: obtaining mining requirement text of autonomous driving scenarios; identifying target scenario features in the mining requirement text; querying a database based on the target scenario features, wherein the database stores temporal features related to autonomous driving scenarios, and the temporal features are composed of feature frames at different times; querying the target feature frames and single frame times corresponding to the target scenario features from the temporal features; querying at least one temporal segment from the temporal features based on the single frame times of the target feature frames; and generating scenario mining data of autonomous driving scenarios based on all the queried temporal segments.

[0006] Based on the aforementioned technical means, this application embodiment can effectively capture the temporal continuity of the scene by identifying the target scene features in the mining requirement text, and querying the temporal segments based on the target feature frames and single frame timestamps corresponding to the target scene features in the database. Through a hierarchical query strategy, the target feature frames are located first, and then extended to the temporal segments, which greatly reduces unnecessary data scanning and improves the retrieval effect. Thus, accurate autonomous driving scene mining data is automatically generated based on all the queried temporal segments, ensuring the integrity of the scene information, significantly improving the accuracy of scene retrieval and data construction efficiency, and effectively solving the problems of relying on manual rule definition, difficulty in handling temporal continuity, and low retrieval efficiency in the traditional scene mining process.

[0007] Optionally, before querying the target feature frame and single frame time corresponding to the target scene feature from the temporal features, the method further includes: querying the temporal segment corresponding to the target scene feature from the temporal features; and filtering the temporal segment corresponding to the target scene feature to obtain the filtered temporal segment.

[0008] Based on the above technical means, the embodiments of this application can retrieve the time sequence segments corresponding to the target scene features from the time sequence features before querying the target feature frame and single frame time, and filter the time sequence segments. This can effectively exclude irrelevant or low-quality data segments, narrow the subsequent query range, improve the accuracy and efficiency of target feature frame positioning, avoid redundant calculations, reduce computational complexity, and make the final generated scene mining data more representative and practical, thereby enhancing the robustness and resource utilization efficiency of the overall scene mining process.

[0009] Optionally, the time-series segments corresponding to the target scene features are filtered, including: obtaining the feature duration, feature interruption duration, and segment length of the time-series segment; if the feature duration is greater than a first preset duration, the feature interruption duration is less than a second preset duration, and the segment length is less than a third preset duration, then the time-series segment is retained; otherwise, the time-series segment is filtered.

[0010] Based on the above technical means, the embodiments of this application can perform multi-dimensional filtering on the time-series segments corresponding to the target scene features based on feature duration, feature interruption duration, and segment length, retaining only time-series segments that meet the requirements of sufficiently long feature duration, short interruption time, and moderate overall segment length. This can effectively eliminate short, discontinuous, or excessively long invalid data segments, ensuring that the retained time-series segments are more in line with the needs of real autonomous driving scenarios in terms of temporal continuity and semantic integrity. This improves the reliability of subsequent feature frame localization and the quality of scene mining data, ultimately enhancing the effectiveness and practicality of scene mining results.

[0011] Optionally, querying at least one time segment from the time sequence features based on a single frame time of the target feature frame includes: determining a start time and an end time based on a single frame time of the target feature frame; and querying at least one time segment from the time sequence features based on the start time and the end time.

[0012] Based on the above technical means, the embodiments of this application can determine the start and end times based on the single frame time of the target feature frame, and extract the corresponding time segment from the time sequence features accordingly. This can effectively transform discrete single frame times into continuous scene segments with a clear start and end time range, overcoming the technical limitations of blurred scene boundaries and difficulty in ensuring temporal continuity in traditional methods. It can ensure that each output scene segment has a complete time span and behavioral context, significantly improving the accuracy of subsequent scene generation.

[0013] Optionally, determining the start and end times based on the single-frame times of the target feature frame includes: identifying the temporal order of temporal features; determining the previous and next single-frame times adjacent to the current single-frame time from the temporal order; calculating the first interval duration between the current single-frame time and the previous single-frame time; if the first interval duration is greater than a fourth preset duration, then the current single-frame time is taken as the start time; calculating the second interval duration between the current single-frame time and the next single-frame time; if the second interval duration is greater than a fifth preset duration, then the current single-frame time is taken as the end time.

[0014] Based on the above technical means, the embodiments of this application can automatically determine the start and end times of a scene segment by identifying the temporal sequence and calculating the interval between the current single frame time and the adjacent time. When the time interval with the previous or next frame exceeds a preset threshold, it indicates that the scene may be interrupted or switched. At this time, taking the current single frame time as the segment boundary can avoid cross-scene or discontinuous data from being incorrectly included in the same temporal segment, ensuring that the extracted temporal segments have temporal continuity and semantic consistency, thereby improving the accuracy and logical integrity of scene mining data.

[0015] Optionally, before determining the start and end times based on the single-frame time of the target feature frame, the method includes: if the target feature frame is the first feature frame in the time sequence features, then the single-frame time of the target feature frame is taken as the start time; if the target feature frame is the last feature frame in the time sequence features, then the single-frame time of the target feature frame is taken as the end time.

[0016] Based on the above technical means, the embodiments of this application can determine whether the target feature frame is the first or last frame in the temporal feature before determining the start and end times. If so, the single frame time can be directly used as the start or end time respectively. This can effectively handle key feature frames located at the temporal boundary, avoid the omission of boundary scenes or abnormal truncation caused by the inability to calculate the time interval due to the lack of preceding or succeeding frames, and ensure that the temporal segments corresponding to the first and last key frames can still be extracted completely and accurately, thereby improving the robustness and completeness of the overall temporal segment extraction.

[0017] A second aspect of this application provides an autonomous driving scene mining device, comprising: an acquisition module for acquiring mining requirement text of an autonomous driving scene; an identification module for identifying target scene features in the mining requirement text, querying a database based on the target scene features, the database storing temporal features related to the autonomous driving scene, the temporal features being composed of feature frames at different times; a query module for querying the target feature frame and single frame time corresponding to the target scene feature from the temporal features, and querying at least one temporal segment from the temporal features based on the single frame time of the target feature frame; and a generation module for generating scene mining data of the autonomous driving scene based on all the queried temporal segments.

[0018] A third aspect of this application provides an electronic device, including a processor and a memory, wherein the memory is used to store computer programs; and the processor is used to execute the programs stored in the memory to implement the autonomous driving scene mining method as described in the above embodiments.

[0019] A fourth aspect of this application provides a computer-readable storage medium having a computer program stored thereon, which is executed by a processor to implement the autonomous driving scene mining method as described in the above embodiments.

[0020] The fifth aspect of this application provides a computer program product, including a computer program or instructions, which, when executed, implement the autonomous driving scene mining method as described in the above embodiments. Attached Figure Description

[0021] Figure 1 This is a flowchart of an embodiment of the autonomous driving scene mining method provided in this application; Figure 2 This is a structural diagram of the autonomous driving scene discovery device provided in the embodiments of this application; Figure 3 This is a structural diagram of the electronic device provided in the embodiments of this application. Detailed Implementation

[0022] To make the technical problems, technical solutions, and beneficial effects solved by this application clearer, the following detailed description is provided in conjunction with embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0023] Scene discovery and labeling is a fundamental and crucial module in an autonomous driving engineering system. The demand for scene discovery, driven by problem-solving, is large and urgent. Currently, many scene discovery systems are inefficient and unable to meet the numerous scene requirements in a short time.

[0024] There are many engineering system solutions for data mining and retrieval. The first and most common approach is to redevelop a new set of logic for each new requirement, traverse the data, and directly label the results. The problem with this approach is low development efficiency, many repetitive tasks between different scenarios cannot be utilized, and if the results are unsatisfactory, modifying parameters requires rerunning the process, resulting in high time costs. The second approach adds database storage of results to the first approach. When a new requirement is a combination of two previously defined scenarios, such as a new requirement for rainy night, and two scenarios for rainy night have been implemented before, the results can be obtained through database association. This approach achieves a certain degree of reusability, but because it stores a highly abstract semantic feature corresponding to a clip, a significant amount of temporal precision is lost. Therefore, the probability of regenerating a new scenario based on this information is greatly reduced. The third method is an optimization of the second method, retaining more original features and finer temporal granularity. For example, it records the information of each obstacle at each moment in the database and then performs combined searches through the database. This method performs better than the second method in terms of feature reuse, but it can still only perform single-frame recall. Admittedly, expanding forward and backward based on a single frame can recall fragments, but the accuracy of a single frame is often limited. For example, when mining rainy scenes, using the classification results of the weather model based on key features for recall will result in many false positives in a single frame because the model accuracy is not 100%, leading to incorrect recall of fragments. However, weather scenes have a time sequence continuity, and filtering by increasing the duration of features can significantly improve the recall quality.

[0025] In related technologies, a method and system for mining autonomous driving perception data under arbitrary conditions are also disclosed. The method includes the following steps: S1, taking the perception data file collected by the autonomous vehicle as input, further splitting the perception object information based on the arbitrary condition query purpose, processing several tags, and generating an autonomous driving perception result detail table with the autonomous vehicle, perception acquisition time, and data corresponding to several tags as a record; S2, loading the autonomous driving perception result detail table into an analytical database based on columnar storage of autonomous vehicle, perception acquisition time, and several tags; S3, inputting a general query model with arbitrary conditions to query the analytical database based on columnar storage to obtain the required autonomous driving perception scene data. Thus, by constructing a tagged perception result detail table and utilizing a columnar storage database to support arbitrary condition queries, flexible retrieval of autonomous driving perception data is achieved. However, this method mainly relies on static tags and single-frame perception information for querying, failing to consider the dynamic evolution characteristics of the scene in the time dimension and lacking the ability to model temporal continuity and contextual correlation.

[0026] This application provides a method for mining autonomous driving scenarios, which involves: obtaining mining requirement text for autonomous driving scenarios; identifying target scenario features in the mining requirement text; querying a database based on the target scenario features, wherein the database stores temporal features related to the autonomous driving scenario, and the temporal features are composed of feature frames at different times; querying the target feature frame and single frame time corresponding to the target scenario feature from the temporal features; querying at least one temporal segment from the temporal features based on the single frame time of the target feature frame; and generating scenario mining data for the autonomous driving scenario based on all the queried temporal segments.

[0027] Compared to the aforementioned related technologies, this application focuses on temporal feature structures, using temporal segments composed of multiple frames as basic units. By identifying the specific location of target scene features within the temporal sequence and combining dynamic factors such as temporal continuity, interruption intervals, and segment boundaries, it accurately extracts autonomous driving scenes with complete semantics. Therefore, this solution incorporates temporal filtering logic through SQL statements combined with window functions in terms of logical reuse, enhancing the recall of temporal features and better meeting the requirements of continuous segments in autonomous driving scenarios.

[0028] Specifically, this application provides a method for mining autonomous driving scenarios. Please refer to [link / reference]. Figure 1 This includes the following steps: S101: Obtain the mining requirements text for autonomous driving scenarios; It should be noted that the text of the requirements can contain the core semantic elements of the scenario, such as environmental conditions (weather, road type, time), behavioral themes (vehicles, pedestrians, other traffic participants), interaction patterns (following other vehicles, changing lanes, avoiding other vehicles), risk levels (dangerous, emergency, normal), etc. These text requirements will be parsed into structured features through natural language processing technology, such as: highway features being parsed into road type features, etc., without specific limitations.

[0029] For example, the text to be mined can include: 1) Basic scenario description: mining all vehicle lane-changing scenarios on highways; 2) Complex scenario description: identifying dangerous scenarios where pedestrians suddenly cross the road in urban areas on rainy nights; 3) Specific parameterized scenarios: retrieving segments where there is a risk of conflict between a vehicle turning left at an intersection and oncoming straight-going vehicles.

[0030] S102: Identify and mine the target scene features in the demand text, and query the database based on the target scene features. The database stores time-series features related to autonomous driving scenarios. The time-series features are composed of feature frames at different times. It should be noted that the target scene feature can be a specific state or event defined by user needs, extracted from the user's needs text described in natural language, and can be understood by computers and queried in a database. For example, if the target scene feature is "the vehicle in front brakes suddenly", the system can query the time-series features stored in the database based on this feature. Each time-series feature in the database records feature frames at different moments in a continuous driving process. Each feature frame contains environmental perception information at that moment, such as the speed of the vehicle in front, the speed of the vehicle itself, the relative distance, acceleration, etc.

[0031] S103: Query the target feature frame and single frame time corresponding to the target scene feature from the temporal features, and query at least one temporal segment from the temporal features based on the single frame time of the target feature frame; Specifically, the database searches for specific frames that meet the target scene characteristics in the time-series features stored in it. Since the time-series features consist of multiple feature frames arranged in chronological order, each feature frame represents a structured description of the vehicle's perception and state at a certain moment, such as target position, speed, category, relative distance, etc.; these feature frames are traversed to find the target feature frame whose content matches the target scene characteristics, and its corresponding single frame time is recorded, where the single frame time is the timestamp of the target feature frame in the time-series features.

[0032] A single keyframe is insufficient to reconstruct the complete scene because autonomous driving decisions rely on the dynamic evolution before and after an event. Therefore, using that single frame as an anchor point, we can extend forward and backward, extracting a continuous time interval from the same temporal feature to form a temporal segment. For example, if the target feature frame appears at the 15th second, we can take 3 seconds forward and 4 seconds backward to obtain a temporal segment from the 12th to the 19th second. This segment contains the complete context, including the environmental state before the event, the event triggering process, and the subsequent vehicle response, and can be used for training, testing, or analysis.

[0033] S104: Generate scene mining data for autonomous driving scenarios based on all time-series segments retrieved.

[0034] Based on the aforementioned technical features, this application embodiment can effectively capture the temporal continuity of a scene by identifying target scene features in the mining requirement text, and querying time-series segments based on the target feature frames and single-frame timestamps corresponding to the target scene features in the database. Through a hierarchical query strategy, the target feature frames are located first, and then extended to time-series segments, significantly reducing unnecessary data scanning and improving retrieval results. Thus, accurate autonomous driving scene mining data is automatically generated based on all queried time-series segments, ensuring the integrity of scene information, significantly improving the accuracy of scene retrieval and data construction efficiency, and effectively solving the problems of relying on manual rule definition, difficulty in handling temporal continuity, and low retrieval efficiency in the traditional scene mining process.

[0035] In this embodiment of the application, before querying the target feature frame and single frame time corresponding to the target scene feature from the temporal features, the method further includes: querying the temporal segment corresponding to the target scene feature from the temporal features; and filtering the temporal segment corresponding to the target scene feature to obtain the filtered temporal segment.

[0036] Based on the above technical features, the embodiments of this application can retrieve the time-series segments corresponding to the target scene features from the time-series features before querying the target feature frame and single frame time, and filter the time-series segments. This can effectively exclude irrelevant or low-quality data segments, narrow the subsequent query range, improve the accuracy and efficiency of target feature frame positioning, avoid redundant calculations, reduce computational complexity, and thus make the final generated scene mining data more representative and practical, enhancing the robustness and resource utilization efficiency of the overall scene mining process.

[0037] In this embodiment of the application, the time segment corresponding to the target scene feature is filtered, including: obtaining the feature duration, feature interruption duration and segment length of the time segment; if the feature duration is greater than a first preset duration, the feature interruption duration is less than a second preset duration and the segment length is less than a third preset duration, then the time segment is retained; otherwise, the time segment is filtered.

[0038] The first, second, and third preset durations can all be set according to actual needs. For example, the first preset duration is 10 seconds, the second preset duration is 5 seconds, and the third preset duration is 100 seconds, without any specific limitations.

[0039] Based on the above technical features, the embodiments of this application can perform multi-dimensional filtering on the time-series segments corresponding to the target scene features, based on the feature duration, feature interruption duration, and segment length. Only time-series segments that meet the requirements of sufficiently long feature duration, short interruption time, and moderate overall segment length can be retained. This can effectively eliminate short, discontinuous, or excessively long invalid data segments, ensuring that the retained time-series segments are more in line with the needs of real autonomous driving scenarios in terms of temporal continuity and semantic integrity. This improves the reliability of subsequent feature frame localization and the quality of scene mining data, ultimately enhancing the effectiveness and practicality of scene mining results.

[0040] For example, suppose the mining requirement is to find all long-lasting following scenarios that occur on highways. Querying the time series segments corresponding to the target scenario features from the time series features will first perform a preliminary query based on highway and following features, generating many candidate time series segments, such as: Segment A: Vehicle 1 is on the highway with a vehicle ahead, lasting 30 seconds; Segment B: Vehicle 1 is on the highway with a vehicle ahead, but only lasts 3 seconds (possibly just a momentary overtaking maneuver); Segment C: Vehicle 2 is on the highway with a vehicle ahead, lasting 60 seconds, but with a 15-second data interruption (possibly due to signal loss in a tunnel); Segment D: Vehicle 3 is on the highway with a vehicle ahead, lasting up to 300 seconds (possibly encountering a prolonged traffic jam).

[0041] These candidate segments were filtered using preset rules to retain high-quality scenes: Rule 1: Feature duration less than 10 seconds (filtering excessively short segments). Segment B, with only 3 seconds, was filtered out. Rule 2: Feature allowable interruption time greater than 5 seconds (filtering segments with poor continuity). Segment C, with an interruption of 15 seconds, exceeded the allowable 5 seconds and was filtered out. Rule 3: Segment length greater than 100 seconds (filtering excessively long segments that may contain other irrelevant behaviors). Segment D, at 300 seconds, was judged to be a congestion scene rather than a typical car-following scene and was filtered out. After these three rounds of filtering, only segment A remained because it met all the conditions: sufficient duration, continuous data, and moderate segment length.

[0042] In this embodiment of the application, querying at least one time segment from time features based on a single frame time of a target feature frame includes: determining a start time and an end time based on a single frame time of the target feature frame; and querying at least one time segment from time features based on the start time and the end time.

[0043] Based on the above technical features, the embodiments of this application can determine the start and end times based on the single frame time of the target feature frame, and extract the corresponding time segments from the time sequence features accordingly. This can effectively transform discrete single frame times into continuous scene segments with a clear start and end time range, overcoming the technical limitations of blurred scene boundaries and difficulty in ensuring temporal continuity in traditional methods. It can ensure that each output scene segment has a complete time span and behavioral context, significantly improving the accuracy of subsequent scene generation.

[0044] In this embodiment of the application, determining the start and end times based on the single-frame time of the target feature frame includes: identifying the temporal order of the temporal features; determining the previous and next single-frame times adjacent to the current single-frame time from the temporal order; calculating the first interval duration between the current single-frame time and the previous single-frame time; if the first interval duration is greater than a fourth preset duration, then the current single-frame time is taken as the start time; calculating the second interval duration between the current single-frame time and the next single-frame time; if the second interval duration is greater than a fifth preset duration, then the current single-frame time is taken as the end time.

[0045] The fourth and fifth preset durations can be set according to actual needs, such as 5s or 6s, without specific limitations.

[0046] Based on the above technical features, the embodiments of this application can automatically determine the start and end times of a scene segment by identifying the temporal sequence and calculating the interval between the current single frame time and the adjacent time. When the time interval with the previous or next frame exceeds a preset threshold, it indicates that the scene may be interrupted or switched. At this time, taking the current single frame time as the segment boundary can avoid cross-scene or discontinuous data being incorrectly included in the same temporal segment, ensuring that the extracted temporal segments have temporal continuity and semantic consistency, thereby improving the accuracy and logical integrity of scene mining data.

[0047] In this embodiment of the application, before determining the start time and end time based on the single frame time of the target feature frame, the method includes: if the target feature frame is the first feature frame in the time sequence features, then the single frame time of the target feature frame is taken as the start time; if the target feature frame is the last feature frame in the time sequence features, then the single frame time of the target feature frame is taken as the end time.

[0048] Based on the above technical features, the embodiments of this application can determine whether the target feature frame is the first or last frame in the temporal feature before determining the start and end times. If so, the single frame time can be directly used as the start or end time respectively. This can effectively handle key feature frames located at the temporal boundary, avoid the omission of boundary scenes or abnormal truncation caused by the inability to calculate the time interval due to the lack of preceding or succeeding frames, and ensure that the temporal segments corresponding to the first and last key frames can still be extracted completely and accurately, thereby improving the robustness and completeness of the overall temporal segment extraction.

[0049] For example, suppose our scenario requires parking at an intersection, and the database query has found four target feature frames that meet the condition that the vehicle is at the intersection and its speed is zero. The corresponding single frame times are: T1 (10:00:01), T2 (10:00:02), T3 (10:00:10), and T4 (10:00:11).

[0050] Analyzing the relationship between each time point and its preceding and following times, we assume that if the time difference between two time points exceeds 3 seconds, they are considered to be different parking events. Specifically: the time difference between T1 (10:00:01) and the previous time point (assuming there is no earlier time point that meets the condition) is greater than 3 seconds, therefore T1 is determined to be the start time of a segment. The time difference between T1 (10:00:01) and the next time point T2 (10:00:02) is 1 second (less than 3 seconds), so T1 is not the end time.

[0051] The time difference between T2 (10:00:02) and the previous time T1 (10:00:01) is 1 second (less than 3 seconds), so T2 is not the start time. The time difference between T2 (10:00:02) and the next time T3 (10:00:10) is 8 seconds (greater than 3 seconds), so T2 is determined to be the end time of a segment.

[0052] The time difference between T3 (10:00:10) and the previous time T2 (10:00:02) is 8 seconds (greater than 3 seconds), so T3 is determined to be the start time of another segment. The time difference between T3 (10:00:10) and the next time T4 (10:00:11) is 1 second (less than 3 seconds), so T3 is not the end time.

[0053] T4 (10:00:11) differs from the previous time T3 (10:00:10) by 1 second (less than 3 seconds), therefore T4 is not the start time. T4 (10:00:11) differs from the next time (assuming there is no later time that meets the conditions) by more than 3 seconds, therefore T4 is marked as the end time.

[0054] In this embodiment of the application, before obtaining the mining requirement text of the autonomous driving scenario, the process includes: obtaining multi-source heterogeneous data collected by autonomous driving vehicles; extracting scenario features from the multi-source heterogeneous data; and classifying and storing the multi-source heterogeneous data into an autonomous driving database according to the scenario features.

[0055] Based on the above technical features, the embodiments of this application can extract scene features from multi-source heterogeneous data collected by autonomous vehicles in advance, and classify and store the data in an autonomous driving database according to these features. This transforms the raw sensor data into feature representations with clear scene semantics and organizes them in an orderly manner according to categories, significantly improving the query efficiency and semantic retrieval capabilities of the database.

[0056] It should be noted that autonomous vehicles collect multi-source heterogeneous data during actual road tests, including camera images, LiDAR point clouds, millimeter-wave radar signals, GPS positioning information, and vehicle CAN bus data. This raw data is fused and analyzed to extract semantically meaningful scene features, such as stationary obstacles ahead, vehicles cutting in from the left lane, low-adhesion road surfaces in rain, and low-light conditions at night. Based on the extracted scene features, the corresponding raw data or its structured representation is categorized and stored in the autonomous driving database. For example, all data containing features of vehicles cutting in are categorized into the traffic interaction sub-database, and rain-related data is categorized into the severe weather sub-database, preserving its complete temporal structure and timestamp information. When subsequent input requests, such as searching for scenarios of vehicles suddenly cutting in from the left on a highway, the system can efficiently retrieve matching time-series segments directly from the traffic interaction sub-database without traversing all the raw data, significantly improving mining efficiency and accuracy.

[0057] The autonomous driving scenario mining method of this application will be described in detail below, with specific steps as follows: Step 1: Extract low-dimensional time-series features and store them in the database. i. The database contains vehicle identifiers, timestamps, and feature storage fields. The feature storage fields are stored in JSON format with keys and values. If there are multiple elements in a single frame, the values ​​are stored as an array.

[0058] ii. Feature extraction: a. Extraction of structured data generated by the autonomous driving system, including perception information (obstacles, lane lines, etc.), map information (ground elements, road types, etc.), and vehicle information (speed, acceleration, steering angle, latitude and longitude, etc.). b. Based on DINO (Self-Distillation with No Labels), understand the overall scene semantics of the image, specifically: The first step requires training a DINO model using a massive amount of unlabeled images via a self-distillation method. This model learns to extract the overall semantic features of an image by making different cropped views of the same image as close as possible in the feature space, while keeping the features of different images as far apart as possible, thus eliminating the need for manual annotation.

[0059] The second step is to input the single-frame images captured by the cameras of the autonomous vehicle into this pre-trained DINO model.

[0060] The third step involves the multi-layer Transformer architecture within the DINO model encoding the image, ultimately generating a high-dimensional feature vector at the output layer. This vector compactly encodes the global scene information of the image.

[0061] The fourth step is to input the high-dimensional feature vector into a pre-trained classifier, which maps the feature vector to preset scene category labels, such as outputting city roads, highways, ramps, tunnels, etc.

[0062] c. Based on YOLO (an end-to-end real-time object detection algorithm), identify all objects of interest in the image and determine their location and category. Specifically: The first step is to input the images captured by the autonomous driving camera into the YOLO model. The model first divides the input image into an S x S grid.

[0063] The second step involves each grid cell predicting objects that will fall into that region. For each grid cell, the model predicts multiple bounding boxes simultaneously. Each bounding box contains information including the center coordinates, width, height, confidence level of the presence of an object within the box, and the probability of each object's category.

[0064] The third step involves the model generating numerous overlapping bounding boxes. At this stage, a non-maximum suppression algorithm is used to filter out the optimal results. This algorithm sorts the bounding boxes based on their confidence scores and suppresses redundant bounding boxes that have a high degree of overlap with the highest-scoring bounding box, ensuring that each object is detected only once.

[0065] The fourth step involves filtering the data, after which the model outputs a final list of all detected objects. Each object's information includes its category (e.g., car, pedestrian, traffic sign), its precise bounding box coordinates in the image, and a confidence score indicating the reliability of the detection.

[0066] d. Based on CLIP (Contrastive Language–Image Pretraining) image vector features, obtain a general, dense vector representation of the image for similarity search, specifically: The first step is to input the images captured by the autonomous driving camera into the image encoder of the pre-trained CLIP model. This encoder is typically a Vision Transformer or a large CNN (Convolutional Neural Network).

[0067] The second step involves the image encoder performing depth processing on the image, ultimately generating a high-dimensional feature vector with fixed dimensions in the output layer, such as a 512-dimensional or 768-dimensional floating-point array. This vector is a dense, abstract mathematical representation of the image.

[0068] The third step typically involves normalizing the generated high-dimensional vector, converting it into a unit vector. This step aims to make subsequent calculations of cosine similarity more efficient and accurate.

[0069] The fourth step is to store this normalized high-dimensional vector directly as a feature in the tags field of the database.

[0070] Step 2: Receive a new scenario requirement, break down the mapping of the scenario requirement to low-dimensional features, and consider temporal feature logic. Take trajectory data retrieval within a 100m radius of an intersection as an example.

[0071] a. The low-dimensional feature of this scene mapping is that the distance from the vehicle's latitude and longitude to the intersection's latitude and longitude is less than 100m.

[0072] b. Since it is necessary to recall complete intersection scene segments, the key temporal feature logic can be broken down as follows: (1) The duration of the feature is greater than 10 seconds (the normal time to pass through an intersection within 100m is generally greater than 10 seconds). (2) The feature allows for a 5-second interruption period (the entire segment will not be interrupted due to accidental data loss). (3) The segment length is less than 100 seconds (filtering data that has been stopped at the intersection for a long time). Step 3: Based on the above decomposition, first retrieve all vehicles that meet the low-dimensional feature requirements and their corresponding single-frame times by searching.

[0073] Step 4: Identify segments that satisfy the temporal sequence characteristic logic based on all single-frame moments. a. For each single frame moment found, use the lag and lead window functions to find the previous and next single frame moments that meet the requirements for the corresponding vehicle in chronological order.

[0074] It's important to note that for each row of data, the lag window function returns the timestamp field value of the row preceding it. The current row then obtains its own timestamp, as well as the timestamp of the last frame that met the conditions. If the current row is the first row in the group and there is no preceding row, the lag window function returns null.

[0075] The lead window function is the opposite of the lag window function. For each row of data, the lag window function returns the timestamp field value of the next row. In this way, the current row obtains the timestamp of the next frame that meets the conditions. If the current row is the last row in the group and there is no next row, the lag window function returns null.

[0076] b. Find the start and end times of each segment. If the time difference between the current moment and the previous frame that meets the requirements exceeds the allowed interruption time, it is considered the start time of the segment. If the time difference between the current moment and the next frame that meets the requirements exceeds the allowed interruption time, it is considered the end time of the segment. Sort the above start and end times in chronological order, with each pair forming the start and end times of a segment.

[0077] c. Based on the start and end times of each segment obtained above, determine whether the duration and segment length requirements are met. If they are met, output the segment.

[0078] In summary, for each requirement, we only need to map and select features for the scenario and set parameters for the temporal logic, and we can complete the mining output through a single set of code.

[0079] Therefore, this application presents the following data mining logic in the field of autonomous driving: 1) Feature extraction: Features from various sources involved in autonomous driving are stored in a database, preserving temporal information. Frequently changing features are managed using JSON field formats, significantly improving the development efficiency of scene labeling for data mining due to the high reusability of features and logic. 2) Feature combination and mining through temporal logic: Scene requirements are mapped to corresponding low-dimensional features, and the temporal feature logic of the scene is simultaneously refined. These low-dimensional features and temporal feature parameters are used as input parameters to a unified SQL code to obtain the final mining results. The mining logic achieves a high degree of abstraction and reusability, allowing data mining to be performed without code. Simply selecting features and changing logic parameters can produce scenes, enabling more people to participate in data mining output.

[0080] This application also provides an autonomous driving scene discovery device 10, please refer to... Figure 2 It includes: acquisition module 100, identification module 200, query module 300 and generation module 400.

[0081] The acquisition module 100 is used to acquire the mining requirement text of autonomous driving scenarios; the identification module 200 is used to identify the target scene features in the mining requirement text, and query the database based on the target scene features. The database stores time-series features related to autonomous driving scenarios, and the time-series features are composed of feature frames at different times; the query module 300 is used to query the target feature frame and single frame time corresponding to the target scene features from the time-series features, and query at least one time-series segment from the time-series features based on the single frame time of the target feature frame; the generation module 400 is used to generate scene mining data of autonomous driving scenarios based on all the time-series segments queried.

[0082] Based on the aforementioned technical features, this application embodiment can receive user-inputted mining requirement text in natural language form via an acquisition module; an identification module parses out structured target scene features from the text and uses these features as conditions to query a database of pre-stored time-series features; a query module first locates the target feature frames that satisfy the features and their individual frame timestamps, and then dynamically extracts time-series segments containing contextual information using these timestamps as anchor points; and a generation module integrates all matching time-series segments to output complete scene mining data. This is achieved through an accurate understanding of the requirement semantics, a structured organization of time-series features, and an intelligent segment extraction mechanism based on time context, thereby realizing efficient, reusable, and high-precision automated mining of autonomous driving scenarios.

[0083] This application also provides an electronic device 30, please refer to... Figure 3 It includes a processor 310 and a memory 320, wherein the memory 310 is used to store computer programs; the processor 320 is used to execute the programs stored in the memory 310 to implement the autonomous driving scene mining method described in any embodiment of this application.

[0084] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the autonomous driving scene mining method described in any embodiment of this application.

[0085] This application also provides a computer program product, including a computer program or instructions, which, when executed, implement the autonomous driving scene mining method as described in the above embodiments.

[0086] Note: After the specification is completed, check whether the specification has omitted any technical solutions and technical features from the patent disclosure. Detailed interaction diagrams and product structure diagrams provided by the inventors should be retained in the application documents and should not be simplified into simple block diagrams.

[0087] In this application, "multiple" refers to two or more.

[0088] In this application, unless otherwise expressly defined, the terms "installation," "connection," and "linking" should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral connection; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; and they can refer to the internal connection between two components. Those skilled in the art can understand the specific meaning of the above terms in this application based on the specific circumstances.

[0089] The terms “first,” “second,” “third,” “fourth,” etc., in this application (if present) are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0090] In this application, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, in this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0091] Unless otherwise specified, all steps in this application may be performed sequentially or randomly. For example, if a method includes steps A and B, it means that the method may include steps A and B performed sequentially, or it may include steps B and A performed sequentially. For example, if a method may also include step C, it means that step C may be added to the method in any order. For example, the method may include steps A, B, and C, or it may include steps A, C, and B, or it may include steps C, A, and B, etc.

[0092] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A method for mining autonomous driving scenarios, characterized in that, Includes the following steps: Obtain the mining requirements text for autonomous driving scenarios; Identify the target scene features in the mining requirement text, and query the database based on the target scene features. The database stores the temporal features related to the autonomous driving scene, and the temporal features are composed of feature frames at different times. From the temporal features, query the target feature frame and single frame time corresponding to the target scene feature, and based on the single frame time of the target feature frame, query at least one temporal segment from the temporal features; Scene mining data for the autonomous driving scenario is generated based on all the time-series segments retrieved.

2. The autonomous driving scene mining method according to claim 1, characterized in that, Before querying the target feature frame and single frame time corresponding to the target scene feature from the temporal features, the method further includes: Query the time segment corresponding to the target scene feature from the time series features; The time segments corresponding to the target scene features are filtered to obtain the filtered time segments.

3. The autonomous driving scenario mining method according to claim 2, characterized in that, The filtering of time-series segments corresponding to the target scene features includes: Obtain the characteristic duration, characteristic interruption duration, and segment length of the time segment; If the duration of the feature is greater than a first preset duration, the duration of the feature interruption is less than a second preset duration, and the length of the segment is less than a third preset duration, then the time segment is retained; otherwise, the time segment is filtered.

4. The autonomous driving scene mining method according to claim 1, characterized in that, The step of querying at least one time segment from the time sequence features based on a single frame time of the target feature frame includes: The start and end times are determined based on the single-frame times of the target feature frame; At least one time segment is queried from the time series features based on the start time and the end time.

5. The autonomous driving scene mining method according to claim 4, characterized in that, The determination of the start and end times based on the single-frame time of the target feature frame includes: Identify the temporal sequence order of the aforementioned temporal features; The previous and next frame times adjacent to the current frame time are determined from the temporal order. Calculate the first interval duration between the current single frame time and the previous single frame time. If the first interval duration is greater than a fourth preset duration, then the current single frame time is taken as the start time. Calculate the second interval duration between the current single frame time and the next single frame time. If the second interval duration is greater than the fifth preset duration, then the current single frame time is taken as the end time.

6. The autonomous driving scene mining method according to claim 4, characterized in that, Before determining the start and end times based on the single-frame time of the target feature frame, the process includes: If the target feature frame is the first feature frame in the temporal features, then the single frame time of the target feature frame is taken as the start time; If the target feature frame is the last feature frame in the temporal features, then the single frame time of the target feature frame is taken as the end time.

7. An autonomous driving scene discovery device, characterized in that, include: The acquisition module is used to acquire the mining requirements text for autonomous driving scenarios; The identification module is used to identify target scene features in the mining requirement text, and query the database based on the target scene features. The database stores time-series features related to the autonomous driving scene, and the time-series features are composed of feature frames at different times. The query module is used to query the target feature frame and single frame time corresponding to the target scene feature from the temporal features, and to query at least one temporal segment from the temporal features based on the single frame time of the target feature frame; The generation module is used to generate scene mining data for the autonomous driving scenario based on all the time-series segments retrieved.

8. An electronic device, characterized in that, Including processor and memory, among which, Memory, used to store computer programs; A processor is used to execute a program stored in memory to implement the autonomous driving scene mining method according to any one of claims 1-6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a processor, implements the autonomous driving scene mining method according to any one of claims 1-6.

10. A computer program product, comprising a computer program or instructions, characterized in that, When the computer program or instructions are executed, they implement the autonomous driving scene mining method as described in any one of claims 1-6.

Images