Automatic driving scene data retrieval method and device, and electronic device

Abstract

This disclosure provides a method for retrieving autonomous driving scene data, relating to artificial intelligence technologies such as autonomous driving, neural networks, and intelligent transportation. The method includes: obtaining embedding vectors of scene data to be processed from a target scene dataset; generating inference result records for the scene data to be processed based on the embedding vectors and attribute information of the scene data to be processed; obtaining a target inference result file corresponding to the target scene dataset based on the inference result records of the scene data to be processed; matching the target scene description information in the target inference result file; determining the target embedding vector based on the matching result; and retrieving the target scene data corresponding to the target scene description information using the target embedding vector in the full inference result file. This disclosure can improve the accuracy and efficiency of autonomous driving scene data retrieval.

Description

Technical Field

[0001] This disclosure relates to the field of internet technology, and more particularly to the fields of artificial intelligence technology such as autonomous driving, neural networks, and intelligent transportation. It provides a method, apparatus, electronic device, and readable storage medium for retrieving autonomous driving scene data. Background Technology

[0002] In the field of autonomous driving technology, massive amounts of road test data are the core resource for training and optimizing autonomous driving systems. During road testing, autonomous vehicles continuously collect data from various driving scenarios, including straight driving, left and right turns, lane changes, and pedestrian crossings, among other typical and extreme traffic situations. With the continuous development of autonomous driving technology, the amount of accumulated road test data is growing exponentially.

[0003] Existing technologies typically employ rule-based matching and traditional feature engineering methods to retrieve autonomous driving scenario data, but these methods suffer from technical problems such as poor accuracy and low efficiency in retrieving results.

[0004] Therefore, how to quickly and accurately retrieve scene data similar to the target scene from massive amounts of data has become a technical problem that urgently needs to be solved in the process of autonomous driving data management and model optimization. Summary of the Invention

[0005] According to a first aspect of this disclosure, a method for retrieving autonomous driving scene data is provided, comprising: obtaining an embedding vector of scene data to be processed in a target scene dataset, and generating an inference result record of the scene data to be processed based on the embedding vector and attribute information of the scene data to be processed; obtaining a target inference result file corresponding to the target scene dataset based on the inference result record of the scene data to be processed; matching the target inference result file with target scene description information, and determining a target embedding vector based on the matching result; and retrieving the target embedding vector in the full inference result file, and obtaining target scene data corresponding to the target scene description information based on the retrieval result.

[0006] According to a second aspect of this disclosure, an apparatus for retrieving autonomous driving scene data is provided, comprising: a generation unit, configured to acquire embedding vectors of scene data to be processed in a target scene dataset, and generate inference result records of the scene data to be processed based on the embedding vectors and attribute information of the scene data to be processed; a processing unit, configured to obtain a target inference result file corresponding to the target scene dataset based on the inference result records of the scene data to be processed; a matching unit, configured to perform matching in the target inference result file using target scene description information, and determine a target embedding vector based on the matching result; and a retrieval unit, configured to perform retrieval in the full inference result file using the target embedding vectors, and obtain target scene data corresponding to the target scene description information based on the retrieval result.

[0007] According to a third aspect of this disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method as described above.

[0008] According to a fourth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions, wherein the computer instructions are used to cause the computer to perform the method described above.

[0009] According to a fifth aspect of this disclosure, a computer program product is provided, comprising a computer program that, when executed by a processor, implements the method described above.

[0010] According to a sixth aspect of this disclosure, an autonomous vehicle is provided, including the electronic devices described above.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0012] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0013] Figure 1 This is a schematic diagram based on the first embodiment of the present disclosure;

[0014] Figure 2 This is a schematic diagram according to the second embodiment of the present disclosure;

[0015] Figure 3 This is a schematic diagram according to the third embodiment of the present disclosure;

[0016] Figure 4 This is a schematic diagram according to the fourth embodiment of the present disclosure;

[0017] Figure 5 This is a block diagram of an electronic device used to implement the autonomous driving scene data retrieval method of the present disclosure embodiments. Detailed Implementation

[0018] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and mechanisms are omitted in the following description.

[0019] Figure 1 This is a schematic diagram based on the first embodiment of this disclosure. (See diagram below.) Figure 1 As shown, the autonomous driving scenario data retrieval method in this embodiment specifically includes the following steps:

[0020] S101. Obtain the embedding vector of the scene data to be processed in the target scene dataset, and generate the inference result record of the scene data to be processed based on the embedding vector and the attribute information of the scene data to be processed.

[0021] S102. Based on the reasoning result record of the scene data to be processed, obtain the target reasoning result file corresponding to the target scene dataset;

[0022] S103. Use the target scene description information to match the target inference result file, and determine the target embedding vector based on the matching result;

[0023] S104. Use the target embedding vector to search in the full inference result file, and obtain the target scene data corresponding to the target scene description information based on the search results.

[0024] The autonomous driving scene data retrieval method of this embodiment first obtains a target inference result file based on the target scene dataset, then determines the target embedding vector based on the matching results of the target scene description information in the target inference result file, and finally obtains the target scene data corresponding to the target scene description information based on the retrieval results of the target embedding vector in the full inference result file. This embodiment achieves the purpose of obtaining the target scene data corresponding to the target scene description information more accurately and quickly from massive autonomous driving scene data by sequentially executing the construction of the target inference result file, matching the target scene description information, and querying the target embedding vector, thereby improving the retrieval accuracy and efficiency of autonomous driving scene data.

[0025] In this embodiment, the autonomous driving scenario data refers to sensor data (such as multimodal sensor data) collected by the autonomous vehicle during the driving process corresponding to different scenarios. The scenarios in this embodiment may include the autonomous vehicle's straight driving scenario, left and right turning scenario, lane changing scenario, failure to yield to pedestrians scenario, and emergency braking scenario.

[0026] In this embodiment, there are two types of scene datasets: target scene datasets and full scene datasets. The target scene dataset includes sensor data for a specific scene that has been collected recently (e.g., within the day or week), while the full scene dataset includes all sensor data collected historically.

[0027] In this embodiment, the scene dataset includes multiple frames of sensor data. Each frame of sensor data has a corresponding file name, frame number (the sequence number assigned to the sensor data frame), and timestamp. The file name is the name of the file where the sensor data frame is located (e.g., a TFRecord (TensorFlow Record) file).

[0028] It is understood that after obtaining the sensor data collected by the autonomous vehicle during driving, this embodiment can organize the obtained sensor data into a standardized data packet in TFRecord format. Each TFRecord file includes a continuous segment of sensor data, which consists of multiple frames of sensor data. Each sensor data frame is indexed according to the frame number and timestamp.

[0029] In this embodiment, when executing S101, sensor data frames in the target scene dataset can be obtained as scene data to be processed, and then the file name, frame number and timestamp of the corresponding sensor data frame can be obtained as attribute information of the scene data to be processed.

[0030] Specifically, in this embodiment, when executing S101 to obtain the embedding vector of the scene data to be processed in the target scene dataset, the following implementation method can be adopted: obtain an inference model based on the encoder and decoder in the pre-trained autonomous driving model; input the scene data to be processed into the obtained inference model to obtain the encoding vector output by the encoder and the decoding vector output by the decoder in the inference model. In this embodiment, the encoding vector and decoding vector have a fixed dimension (e.g., 288 dimensions); obtain the embedding vector of the scene data to be processed based on the obtained encoding vector and decoding vector.

[0031] In other words, this embodiment uses a pre-trained autonomous driving model to infer the scene data to be processed to obtain embedding vectors. On the one hand, it eliminates the need for additional design and training of a dedicated model, reducing the cost of obtaining embedding vectors and improving the efficiency of obtaining embedding vectors. On the other hand, it compresses the scene data to be processed into embedding vectors of fixed dimensions, so that the obtained embedding vectors achieve the best balance between feature representation and storage efficiency. Thus, it avoids the loss of semantic information due to low dimensionality and the increase in computation and storage overhead due to high dimensionality.

[0032] In this embodiment, the pre-trained autonomous driving model can output driving decisions and control commands based on the input sensor data. The autonomous driving model includes an encoder, a decoder, a prediction module, and a control module. The encoder is used to encode the input sensor data and output at least one encoded vector. The decoder is used to decode the at least one encoded vector output by the encoder and output at least one decoded vector. Therefore, the embedded vector obtained in this embodiment is an intermediate vector generated when the autonomous driving model processes the sensor data.

[0033] In this embodiment, when executing S101 to obtain the embedding vector of the scene data to be processed based on the acquired encoding vector and decoding vector, the following implementation method can be adopted: determine the scene type corresponding to the scene data to be processed. In this embodiment, the scene type may include straight-going, sudden braking, failure to yield to pedestrians, etc.; filter the acquired encoding vector and decoding vector according to the determined scene type. In this embodiment, vector filtering can be performed according to the preset correspondence between scene type and vector; obtain the embedding vector of the scene data to be processed based on the vector filtering result.

[0034] In other words, this embodiment filters the encoding and decoding vectors output by the inference model according to the scene type corresponding to the scene data to be processed, which can improve the accuracy of the obtained embedding vectors and avoid the redundancy of the embedding vectors.

[0035] For example, if the encoding vector includes a first encoding vector (vector 1) and a second encoding vector (vector 2), and the decoding vector includes a first decoding vector (vector 3) and a second decoding vector (vector 4); if the scene type corresponding to the current scene data to be processed is "failure to yield to pedestrians", and if in the preset correspondence, the vectors corresponding to the current scene type are the first encoding vector and the second decoding vector; in this embodiment, when executing S101, vector 1 and vector 4 will be selected as the embedding vectors of the current scene data to be processed.

[0036] It is understood that, in this embodiment, when executing S101, all the vectors output by the inference model can also be directly used as the embedding vectors of the scene data to be processed.

[0037] In this embodiment, after executing S101 to obtain the embedding vector of the scene data to be processed, the attribute information of the scene data to be processed, specifically the file name, frame number and timestamp of the scene data to be processed, is combined to generate an inference result record corresponding to the current scene data to be processed. That is, the generated inference result record includes four fields: file name, frame number, timestamp and embedding vector and their corresponding field values.

[0038] To further optimize the storage and transmission efficiency of the embedding vector, this embodiment can also use Base64 encoding to convert the embedding vector into a text string when executing S101 to generate the inference result record based on the embedding vector. Then, the inference result record is generated based on the obtained string and attribute information, which facilitates the reading, writing and transmission of the embedding vector on the distributed file system and is compatible with various types of data processing tools.

[0039] It is understandable that if the target scene dataset includes multiple scene data to be processed, this embodiment will generate corresponding inference result records for each scene data to be processed.

[0040] In this embodiment, after executing S101 to generate the inference result record of the scene data to be processed, S102 is executed to obtain the target inference result file of the corresponding target scene dataset based on the inference result record of the scene data to be processed.

[0041] Specifically, in this embodiment, when executing S102 to obtain the target inference result file of the corresponding target scene dataset based on the inference result record of the scene data to be processed, the implementation method can be as follows: obtain an initial inference result file based on the inference result record of the scene data to be processed; scan each line of inference result record in the obtained initial inference result file, and obtain the file name and frame number of the current inference result record based on the scan result; construct current identification information based on the obtained file name and frame number. In this embodiment, the current identification information is a tuple composed of file name and frame number; in response to the absence of the constructed current identification information in the identification information set, write the current inference result record into the target inference result file, and add the current identification information to the identification information set. In this embodiment, the identification information set can be a key set maintained by a hash set data structure, where each tuple is a key.

[0042] In other words, this embodiment removes duplicate records from the obtained inference results to avoid duplicate records in the inference results file caused by repeated inference on the same frame of data to be processed. This reduces the storage space required for the target inference results file, and each inference results record included in the target inference results file is unique. It also reduces subsequent meaningless calculations and improves matching accuracy and efficiency.

[0043] It is understood that this embodiment can adopt a similar approach to obtain inference result records based on scene data in the full scene dataset, and then deduplicate the inference result records to obtain the full inference result file corresponding to the full scene dataset; wherein, this embodiment can obtain the full inference result file corresponding to the full scene dataset in advance through offline processing.

[0044] In this embodiment, after executing S102 to obtain the target inference result file of the corresponding target scene dataset, S103 is executed to match the target scene description information in the target inference result file, and the target embedding vector is determined based on the matching result.

[0045] In this embodiment, when executing S103, the target scene description information can be read from the configuration file; the configuration file in this embodiment includes one or more scene description information specified by the corresponding user, and the scene description information includes the target file name and the target timestamp.

[0046] Therefore, in this embodiment, the embedding vector that best matches the target scene specified by the user is accurately extracted from the deduplicated target inference result file, and then the extracted embedding vector is used to perform subsequent retrieval operations.

[0047] Specifically, in this embodiment, when performing S103, matching the target scene description information in the target inference result file and determining the target embedding vector based on the matching result, the following implementation method can be adopted: Select an inference result record from the target inference result file that matches the target filename in the target scene description information as an initial inference result record. In this embodiment, the filename in the initial inference result record is the same as the target filename. Construct a time-series window based on the target timestamp in the target scene description information, and select the initial inference result record located within the constructed time-series window as a candidate inference result record. Calculate the absolute value of the time difference between the target timestamp and the timestamp in the candidate inference result record. Select the candidate inference result record with the smallest absolute value as the target inference result record, and use the embedding vector in the selected target inference result record as the target embedding vector.

[0048] In other words, since the inference frame rate and acquisition frame rate of sensor data may not be completely consistent, the target timestamp specified by the user may not have a corresponding sensor data frame in the inference result record. Therefore, this embodiment uses the above-mentioned time alignment mechanism to find the inference result record closest to the target timestamp from the deduplicated target inference result file, and then determines the target embedding vector based on the found inference result record, which can improve the accuracy of the determined target embedding vector.

[0049] In this embodiment, when executing S103, a time sequence window can be constructed based on the target timestamp and the preset window width, with the middle value of the time sequence window being the target timestamp. The initial inference result records are filtered through the constructed time sequence window, thereby excluding initial inference result records that are far from the target timestamp in time.

[0050] This embodiment combines a time-series window to determine the target embedding vector. On the one hand, an appropriate window width can cover the core time period of most autonomous driving scenario events. On the other hand, setting the window width can tolerate slight differences between the inference frame rate and the acquisition frame rate, ensuring that a valid inference result record can be found in the target inference result file to determine the target embedding vector.

[0051] In this embodiment, when executing S103, if there are multiple target scene description information, the target embedding vector corresponding to each different target scene description information will be determined, and then the determined target embedding vector will be written into the target vector file. In this embodiment, the attribute information, time deviation information, etc. of the corresponding target embedding vector can also be written into the target vector file together.

[0052] In this embodiment, after executing S103 to determine the target embedding vector, S104 is executed to use the target embedding vector to search in the full inference result file, and the target scene data corresponding to the target scene description information is obtained based on the search results.

[0053] In this embodiment, the full inference result file includes multiple inference result records, each of which is different; that is, the full inference result file is obtained after deduplication.

[0054] Specifically, in this embodiment, when performing S104 to retrieve target scene data corresponding to target scene description information from the full inference result file using the target embedding vector, the implementation method can be as follows: Calculate the cosine similarity between the target embedding vector and each candidate embedding vector in the full inference result file. In this embodiment, the candidate embedding vectors are the embedding vectors included in the inference result records in the full inference result file; determine candidate embedding vectors whose cosine similarity is greater than or equal to a preset similarity threshold. In this embodiment, the preset similarity threshold can be 0.9; obtain target sensor data based on the attribute information in the inference result record corresponding to the determined candidate embedding vector. In this embodiment, the attribute information includes filename, frame number, and timestamp; and obtain target scene data corresponding to target scene description information based on the obtained target sensor data.

[0055] In other words, this embodiment adopts an exhaustive similarity calculation strategy to retrieve all candidate embedding vectors that are semantically similar to the target embedding vector in the full inference result file. Then, based on the inference result records of the corresponding retrieved candidate embedding vectors, the target scene data with corresponding target scene description information is obtained, ensuring that no retrieval results that meet the similarity threshold are missed, thereby improving the completeness and accuracy of the obtained target scene data.

[0056] In this embodiment, the cosine similarity between vectors can be calculated using the following formula:

[0057]

[0058] In the above formula: Represents the target embedding vector; Represents the candidate embedding vector; The L2 norm of the target embedding vector is represented; This represents the L2 norm of the candidate embedding vector.

[0059] It is understandable that, in order to improve computational efficiency, this embodiment can calculate and cache the L2 norm of the vector when recording the embedding vector to avoid repeatedly calculating the L2 norm of the target embedding vector in multiple similarity calculations. In addition, this embodiment can also use a multi-process parallel computing similarity calculation method, in which multiple working processes calculate the cosine similarity between the target embedding vector and different candidate embedding vectors in parallel.

[0060] In this embodiment, when executing S104, similarity bins can be divided according to a preset similarity threshold. For example, when the preset similarity threshold is 0.9, five bins are divided: [0.90, 0.92), [0.92, 0.94), [0.94, 0.96), [0.96, 0.98), and [0.98, 1.00]. Then, based on the calculated cosine similarity, the bin in which the candidate embedding vector is located is determined. In this embodiment, when executing S104, the output can be based on the similarity bin label, cosine similarity, file name, frame number, and timestamp corresponding to the determined candidate embedding vector.

[0061] In this embodiment, when executing S104 to obtain target scene data corresponding to target scene description information based on the acquired target sensor data, the implementation method can be as follows: expand the acquired target sensor data according to the frame number of the target sensor data; obtain the target scene data corresponding to target scene description information based on the expanded sensor data sequence.

[0062] In other words, considering the temporal continuity of autonomous driving scenarios, a single frame of sensor data is usually insufficient. It is necessary to process consecutive sensor data frames within a certain range before and after the retrieved sensor data frame together to ensure the temporal integrity of the dataset. Therefore, in this embodiment, after acquiring the target sensor data, the acquired target sensor data is extended before and / or after to obtain a sensor data sequence, which can improve the accuracy of the obtained target scene data.

[0063] For example, if the frame number corresponding to the target sensor is N, then the sensor data sequence obtained by executing S104 in this embodiment can be [N-before_seq_num, N+after_seq_num]; where before_seq_num represents the number of sensor data frames before the target sensor data, and after_seq_num represents the number of sensor data frames after the target sensor data.

[0064] Figure 2 This is a schematic diagram according to the second embodiment of this disclosure. (See diagram below.) Figure 2 As shown, after executing S104 "obtaining target scene data corresponding to the target scene description information based on the search results", this embodiment may further include:

[0065] S201. Obtain data for training;

[0066] S202. Process the target scene data in the full scene dataset according to the processing method corresponding to the data training requirements;

[0067] S203. Based on the processing results, obtain the training dataset corresponding to the data training requirements.

[0068] In other words, after obtaining the target scene data with corresponding target scene description information, this embodiment can further combine the data training data to process the target scene data included in the full scene dataset, thereby obtaining a training dataset that meets the data training requirements. On the one hand, it can improve the flexibility and accuracy of obtaining the training dataset, and on the other hand, it can achieve the purpose of automatically reconstructing the training dataset, thereby seamlessly connecting the scene data retrieval results with the training pipeline of the autonomous driving model and improving the training efficiency of the autonomous driving model.

[0069] In this embodiment, the data training requirements include data retention and data filtering; that is, this embodiment supports bidirectional reconstruction of the training dataset. "Bidirectional" means supporting two complementary reconstruction modes: the filtering mode corresponds to data filtering, and the retention mode corresponds to data retention.

[0070] In this embodiment, when executing S202, if it is determined that the data training requirement is data retention, the target scene data included in the full scene dataset is retained, and other scene data is removed; if it is determined that the data training requirement is data filtering, the target scene data in the full scene dataset is removed.

[0071] For example, when a certain type of scenario (such as dirty data caused by sensor malfunctions) negatively affects model training, all similar dirty data frames are retrieved and added to a blacklist, and then removed from the full scenario dataset to obtain the training dataset. When it is necessary to perform enhanced training on a specific type of scenario (such as emergency lane change), all similar emergency lane change scenario data frames are retrieved, and then only the emergency lane change scenario data frames in the full scenario dataset are retained to obtain the training dataset.

[0072] The training dataset obtained by executing S203 in this embodiment can be used to update or iterate the autonomous driving model.

[0073] Figure 3 This is a schematic diagram according to the third embodiment of this disclosure. (See diagram below.) Figure 3 As shown in the figure, this embodiment illustrates the structure of an autonomous driving scenario data retrieval system; the retrieval system in this embodiment includes an inference module, a deduplication module, an alignment module, a retrieval module, and a reconstruction module.

[0074] The inference module is used to infer the sensor data (the scene data to be processed) to obtain the embedding vector. After obtaining the embedding vector, it can be converted into a string using Base64 encoding for storage. The deduplication module is used to deduplicat the inference result records, including the embedding vector, based on the identifier information set. The alignment module is used to determine the target embedding vector based on the target scene description information. The retrieval module is used to retrieve the target scene data based on the target embedding vector. The reconstruction module is used to obtain the training dataset based on the target scene data, in accordance with the data training requirements.

[0075] The retrieval system in this embodiment can significantly shorten the time cycle for problem investigation and data location. When a problem scenario occurs, relevant personnel only need to specify the file name and timestamp of the target scenario, and the system can retrieve the sensor data corresponding to all similar scenarios from the full scenario dataset in a short time. On the other hand, it realizes the precise management of the training dataset. Based on the embedding vector, similarity retrieval and data expansion mechanism can accurately locate each data sequence that needs to be retained or filtered. Thus, it can flexibly generate datasets that meet different training needs through bidirectional reconstruction mode. Furthermore, it has good scalability and flexibility. It can use only the inference module, deduplication module, alignment module and retrieval module for exploratory data analysis, or use the complete pipeline for automatic reconstruction of the training dataset.

[0076] Figure 4 This is a schematic diagram according to the fourth embodiment of this disclosure. (See diagram below.) Figure 4 As shown, the autonomous driving scene data retrieval device 400 of this embodiment includes:

[0077] The generation unit 401 is used to obtain the embedding vector of the scene data to be processed in the target scene dataset, and generate the inference result record of the scene data to be processed based on the embedding vector and the attribute information of the scene data to be processed.

[0078] Processing unit 402 is used to obtain a target reasoning result file corresponding to the target scene dataset based on the reasoning result record of the scene data to be processed;

[0079] Matching unit 403 is used to match the target scene description information in the target inference result file and determine the target embedding vector based on the matching result;

[0080] The retrieval unit 404 is used to perform a retrieval in the full inference result file using the target embedding vector, and obtain target scene data corresponding to the target scene description information based on the retrieval results.

[0081] The generation unit 401 can acquire sensor data frames from the target scene dataset as scene data to be processed, and then acquire the file name, frame number and timestamp of the corresponding sensor data frame as attribute information of the scene data to be processed.

[0082] Specifically, when the generation unit 401 obtains the embedding vector of the scene data to be processed in the target scene dataset, it can adopt the following implementation method: obtain an inference model based on the encoder and decoder in the pre-trained autonomous driving model; input the scene data to be processed into the obtained inference model to obtain the encoding vector output by the encoder and the decoding vector output by the decoder in the inference model; and obtain the embedding vector of the scene data to be processed based on the obtained encoding vector and decoding vector.

[0083] In other words, the generation unit 401 uses the pre-trained autonomous driving model to infer the scene data to be processed to obtain the embedding vector. On the one hand, it eliminates the need for additional design and training of a dedicated model, reducing the cost of obtaining the embedding vector and improving the efficiency of obtaining the embedding vector. On the other hand, it compresses the scene data to be processed into a fixed-dimensional embedding vector, so that the obtained embedding vector achieves the best balance between feature representation and storage efficiency. Thus, it avoids the loss of semantic information due to too low a dimension, and also avoids the increase in computation and storage overhead due to too high a dimension.

[0084] When generating unit 401 obtains the embedding vector of the scene data to be processed based on the acquired encoding vector and decoding vector, the following implementation methods can be adopted: determine the scene type corresponding to the scene data to be processed; filter the acquired encoding vector and decoding vector according to the determined scene type; and obtain the embedding vector of the scene data to be processed based on the vector filtering results.

[0085] In other words, the generation unit 401 filters the encoding vector and decoding vector output by the inference model according to the scene type corresponding to the scene data to be processed, which can improve the accuracy of the obtained embedding vector and avoid the redundancy of the embedding vector.

[0086] It is understandable that the generation unit 401 can also directly use all the vectors output by the inference model as the embedding vectors of the scene data to be processed.

[0087] After obtaining the embedding vector of the scene data to be processed, the generation unit 401 combines the attribute information of the scene data to be processed, specifically the file name, frame number and timestamp of the scene data to be processed, to generate an inference result record corresponding to the current scene data to be processed. That is, the generated inference result record includes four fields: file name, frame number, timestamp and embedding vector and their corresponding field values.

[0088] To further optimize the storage and transmission efficiency of the embedding vector, the generation unit 401 can also use Base64 encoding to convert the embedding vector into a text string when generating the inference result record based on the embedding vector. Then, the inference result record is generated based on the obtained string and attribute information, which facilitates the reading, writing and transmission of the embedding vector on the distributed file system and is compatible with various types of data processing tools.

[0089] It is understandable that if the target scene dataset includes multiple scene data to be processed, this embodiment will generate corresponding inference result records for each scene data to be processed.

[0090] In this embodiment, after the generation unit 401 generates the inference result record of the scene data to be processed, the processing unit 402 obtains the target inference result file of the corresponding target scene dataset based on the inference result record of the scene data to be processed.

[0091] Specifically, when the processing unit 402 obtains the target inference result file of the corresponding target scene dataset based on the inference result record of the scene data to be processed, the following implementation method can be adopted: obtain an initial inference result file based on the inference result record of the scene data to be processed; scan each line of inference result record in the obtained initial inference result file, and obtain the file name and frame number of the current inference result record based on the scan result; construct the current identification information based on the obtained file name and frame number; in response to the absence of the constructed current identification information in the identification information set, write the current inference result record into the target inference result file, and add the current identification information to the identification information set. In this embodiment, the identification information set can be a key set maintained by a hash set data structure, where each tuple is a key.

[0092] In other words, the processing unit 402 deduplicates the obtained inference result records to avoid the problem of duplicate records in the inference result file caused by repeated inference on the same frame of data to be processed. This can reduce the storage space required for the target inference result file, and each inference result record included in the target inference result file is unique. It can also reduce subsequent meaningless calculations and improve matching accuracy and efficiency.

[0093] It is understood that this embodiment can adopt a similar approach to obtain inference result records based on scene data in the full scene dataset, and then deduplicate the inference result records to obtain the full inference result file corresponding to the full scene dataset; wherein, this embodiment can obtain the full inference result file corresponding to the full scene dataset in advance through offline processing.

[0094] In this embodiment, after the processing unit 402 obtains the target inference result file of the corresponding target scene dataset, the matching unit 403 uses the target scene description information to match the target inference result file and determines the target embedding vector based on the matching result.

[0095] The matching unit 403 can read target scene description information from the configuration file; in this embodiment, the configuration file includes one or more scene description information specified by the corresponding user, and the scene description information includes the target file name and the target timestamp.

[0096] Therefore, in this embodiment, the embedding vector that best matches the target scene specified by the user is accurately extracted from the deduplicated target inference result file, and then the extracted embedding vector is used to perform subsequent retrieval operations.

[0097] Specifically, when the matching unit 403 performs matching in the target inference result file using the target scene description information and determines the target embedding vector based on the matching result, the following implementation method can be adopted: select the inference result record that matches the target file name in the target scene description information from the target inference result file as the initial inference result record; construct a time sequence window based on the target timestamp in the target scene description information, and select the initial inference result record located in the constructed time sequence window as the candidate inference result record; calculate the absolute value of the time difference between the target timestamp and the timestamp in the candidate inference result record; select the candidate inference result record with the smallest absolute value as the target inference result record, and use the embedding vector in the selected target inference result record as the target embedding vector.

[0098] In other words, since the inference frame rate and acquisition frame rate of sensor data may not be completely consistent, the target timestamp specified by the user may not have a corresponding sensor data frame in the inference result record. Therefore, the matching unit 403 uses the above-mentioned time alignment mechanism to find the inference result record closest to the target timestamp from the deduplicated target inference result file, and then determines the target embedding vector based on the found inference result record, which can improve the accuracy of the determined target embedding vector.

[0099] The matching unit 403 can construct a time sequence window based on the target timestamp and the preset window width, with the target timestamp as the median value of the time sequence window. The constructed time sequence window is used to filter the initial inference result records, thereby excluding the initial inference result records that are far from the target timestamp in time.

[0100] This embodiment combines a time-series window to determine the target embedding vector. On the one hand, an appropriate window width can cover the core time period of most autonomous driving scenario events. On the other hand, setting the window width can tolerate slight differences between the inference frame rate and the acquisition frame rate, ensuring that a valid inference result record can be found in the target inference result file to determine the target embedding vector.

[0101] If there are multiple target scene descriptions, the matching unit 403 will determine the target embedding vectors corresponding to different target scene descriptions, and then write the determined target embedding vectors into the target vector file. The matching unit 403 can also write the attribute information, time deviation information, etc. of the corresponding target embedding vectors into the target vector file.

[0102] In this embodiment, after the matching unit 403 determines the target embedding vector, the retrieval unit 404 uses the target embedding vector to search in the full inference result file, and obtains the target scene data corresponding to the target scene description information based on the retrieval results.

[0103] In this embodiment, the full inference result file includes multiple inference result records, each of which is different; that is, the full inference result file is obtained after deduplication.

[0104] Specifically, when the retrieval unit 404 performs a retrieval in the full inference result file using the target embedding vector and obtains the target scene data corresponding to the target scene description information based on the retrieval results, the following implementation method can be adopted: calculate the cosine similarity between the target embedding vector and each candidate embedding vector in the full inference result file; determine the candidate embedding vectors whose cosine similarity is greater than or equal to a preset similarity threshold; obtain the target sensor data based on the attribute information in the inference result record corresponding to the determined candidate embedding vector; and obtain the target scene data corresponding to the target scene description information based on the obtained target sensor data.

[0105] In other words, the retrieval unit 404 adopts an exhaustive similarity calculation strategy to retrieve all candidate embedding vectors that are semantically similar to the target embedding vector in the full inference result file. Then, based on the inference result record of the corresponding retrieved candidate embedding vectors, the target scene data with corresponding target scene description information is obtained, ensuring that no retrieval result that meets the similarity threshold is missed, thereby improving the completeness and accuracy of the obtained target scene data.

[0106] Understandably, when calculating the cosine similarity between embedding vectors, the retrieval unit 404 can improve computational efficiency by calculating and caching the L2 norm of the vector when recording the embedding vector, thus avoiding repeated calculation of the L2 norm of the target embedding vector in multiple similarity calculations. In addition, the retrieval unit 404 can also use a multi-process parallel computing similarity calculation method, in which multiple working processes calculate the cosine similarity between the target embedding vector and different candidate embedding vectors in parallel.

[0107] The retrieval unit 404 can also divide the similarity into buckets according to a preset similarity threshold, and then determine the bucket where the candidate embedding vector is located based on the calculated cosine similarity. The retrieval unit 404 can output the bucket label, cosine similarity, file name, frame number and timestamp corresponding to the determined candidate embedding vector.

[0108] When the retrieval unit 404 obtains target scene data corresponding to target scene description information based on the acquired target sensor data, the following implementation method can be adopted: expand the acquired target sensor data according to the frame number of the target sensor data; and obtain the target scene data corresponding to target scene description information based on the expanded sensor data sequence.

[0109] In other words, considering the temporal continuity of autonomous driving scenarios, a single frame of sensor data is usually insufficient. It is necessary to process consecutive sensor data frames within a certain range before and after the retrieved sensor data frame together to ensure the temporal integrity of the dataset. Therefore, after acquiring the target sensor data, the retrieval unit 404 performs pre- and / or post-expansion on the acquired target sensor data to obtain a sensor data sequence, which can improve the accuracy of the obtained target scene data.

[0110] The autonomous driving scene data retrieval device 400 in this embodiment may further include a reconstruction unit 405, which is used to perform the following: obtain data training requirements; process the target scene data in the full scene dataset according to the processing method of the corresponding data training requirements; and obtain the training dataset corresponding to the data training requirements based on the processing results.

[0111] In other words, after obtaining the target scene data with corresponding target scene description information, the reconstruction unit 405 can further combine the data training data to process the target scene data included in the full scene dataset, thereby obtaining a training dataset that meets the data training requirements. On the one hand, this can improve the flexibility and accuracy of obtaining the training dataset, and on the other hand, it can achieve the goal of automatically reconstructing the training dataset, thereby seamlessly connecting the scene data retrieval results with the training pipeline of the autonomous driving model and improving the training efficiency of the autonomous driving model.

[0112] In this embodiment, the data training requirements include data retention and data filtering; that is, the reconstruction unit 405 supports bidirectional reconstruction of the training dataset. "Bidirectional" means supporting two complementary reconstruction modes: the filtering mode corresponds to data filtering, and the retention mode corresponds to data retention.

[0113] If the data training requirement is determined to be data retention, the reconstruction unit 405 retains the target scene data included in the full scene dataset and removes other scene data; if the data training requirement is determined to be data filtering, the reconstruction unit 405 removes the target scene data from the full scene dataset.

[0114] The training dataset obtained by reconstruction unit 405 can be used to update or iterate the autonomous driving model.

[0115] The acquisition, storage, and application of various types of user personal information involved in the technical solutions disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0116] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0117] like Figure 5 The diagram shown is a block diagram of an electronic device for a method of retrieving autonomous driving scene data according to an embodiment of the present disclosure. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0118] like Figure 5 As shown, device 500 includes a computing unit 501, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 502 or a computer program loaded from storage unit 508 into random access memory (RAM) 503. RAM 503 may also store various programs and data required for the operation of device 500. The computing unit 501, ROM 502, and RAM 503 are interconnected via bus 504. Input / output (I / O) interface 505 is also connected to bus 504.

[0119] Multiple components in device 500 are connected to I / O interface 505, including: input unit 506, such as keyboard, mouse, etc.; output unit 507, such as various types of displays, speakers, etc.; storage unit 508, such as disk, optical disk, etc.; and communication unit 509, such as network card, modem, wireless transceiver, etc. Communication unit 509 allows device 500 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0120] The computing unit 501 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 501 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 501 performs the various methods and processes described above, such as methods for retrieving autonomous driving scene data. For example, in some embodiments, the method for retrieving autonomous driving scene data may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 508.

[0121] In some embodiments, part or all of the computer program may be loaded and / or installed on the device 500 via ROM 502 and / or communication unit 509. When the computer program is loaded into RAM 503 and executed by computing unit 501, one or more steps of the autonomous driving scene data retrieval method described above may be performed. Alternatively, in other embodiments, computing unit 501 may be configured to perform the autonomous driving scene data retrieval method by any other suitable means (e.g., by means of firmware).

[0122] Various implementations of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations may include: implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transferring data and instructions to the storage system, the at least one input device, and the at least one output device.

[0123] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to the processor or controller of a general-purpose computer, special-purpose computer, or other programmable autonomous driving scenario data retrieval device, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0124] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0125] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for showing information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0126] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0127] Computer systems can include clients and servers. Clients and servers are generally geographically separated and typically interact via communication networks. The client-server relationship is established by computer programs running on the respective computers and having a client-server relationship with each other. A server can be a cloud server, also known as a cloud computing server or cloud host, a hosting product within the cloud computing service ecosystem, addressing the shortcomings of traditional physical hosts and VPS (Virtual Private Server, or simply "VPS") services, such as high management difficulty and weak business scalability. Servers can also be servers for distributed systems or servers incorporating blockchain technology.

[0128] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0129] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for retrieving autonomous driving scenario data, comprising: Obtain the embedding vector of the scene data to be processed in the target scene dataset, and generate the inference result record of the scene data to be processed based on the embedding vector and the attribute information of the scene data to be processed; Based on the inference result records of the scene data to be processed, a target inference result file corresponding to the target scene dataset is obtained; The target scene description information is used to match the target inference result file, and the target embedding vector is determined based on the matching result. The target embedding vector is used to search the full inference result file, and the target scene data corresponding to the target scene description information is obtained based on the search results.

2. The method according to claim 1, wherein, The method of obtaining the embedding vector of the scene data to be processed in the target scene dataset includes: Based on the encoder and decoder in the pre-trained autonomous driving model, an inference model is obtained; The scene data to be processed is input into the inference model to obtain the encoding vector output by the encoder and the decoding vector output by the decoder in the inference model. The embedding vector of the scene data to be processed is obtained based on the encoded vector and the decoded vector.

3. The method according to claim 2, wherein, The step of obtaining the embedding vector of the scene data to be processed based on the encoded vector and the decoded vector includes: Determine the scene type corresponding to the scene data to be processed; Based on the scene type, the encoded vector and the decoded vector are filtered; Based on the vector filtering results, the embedding vector of the scene data to be processed is obtained.

4. The method according to claim 1, wherein, The step of obtaining the target inference result file corresponding to the target scene dataset based on the inference result record of the scene data to be processed includes: Based on the inference result records of the scene data to be processed, an initial inference result file is obtained; Scan each line of inference result record in the initial inference result file, and obtain the file name and frame number of the current inference result record based on the scan results; Construct the current identification information based on the file name and the frame number; In response to the absence of the current identification information in the identification information set, the current inference result record is written to the target inference result file, and the current identification information is added to the identification information set.

5. The method according to claim 1, wherein, The step of matching the target scene description information in the target inference result file and determining the target embedding vector based on the matching result includes: Select the inference result record that matches the target file name in the target scene description information from the target inference result file and use it as the initial inference result record; A time-series window is constructed based on the target timestamp in the target scene description information, and the initial inference result record located in the time-series window is selected as the candidate inference result record; Calculate the absolute value of the time difference based on the target timestamp and the timestamp in the candidate inference result record; The candidate inference result record with the smallest absolute value is selected as the target inference result record, and the embedding vector in the target inference result record is used as the target embedding vector.

6. The method according to claim 1, wherein, The step of using the target embedding vector to perform a retrieval in the full inference result file, and obtaining the target scene data corresponding to the target scene description information based on the retrieval results, includes: Calculate the cosine similarity between the target embedding vector and each candidate embedding vector in the full inference result file; Determine candidate embedding vectors whose cosine similarity is greater than or equal to a preset similarity threshold; Based on the attribute information in the inference result record corresponding to the determined candidate embedding vector, obtain the target sensor data; Based on the target sensor data, target scene data corresponding to the target scene description information is obtained.

7. The method according to claim 6, wherein, The step of obtaining the target scene data corresponding to the target scene description information based on the target sensor data includes: The target sensor data is expanded based on the frame number of the target sensor data; Based on the extended sensor data sequence, target scene data corresponding to the target scene description information is obtained.

8. The method according to claim 1, further comprising: Obtain data for training purposes; According to the processing method corresponding to the data training requirements, the target scene data in the full scene dataset is processed; Based on the processing results, a training dataset corresponding to the data training requirements is obtained.

9. A device for retrieving autonomous driving scene data, comprising: The generation unit is used to obtain the embedding vector of the scene data to be processed in the target scene dataset, and generate the inference result record of the scene data to be processed based on the embedding vector and the attribute information of the scene data to be processed. The processing unit is used to obtain a target inference result file corresponding to the target scene dataset based on the inference result record of the scene data to be processed; A matching unit is used to match the target scene description information in the target inference result file and determine the target embedding vector based on the matching result. The retrieval unit is used to perform a retrieval in the full inference result file using the target embedding vector, and obtain target scene data corresponding to the target scene description information based on the retrieval results.

10. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-8.

11. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-8.

Images