Data retrieval method and apparatus, electronic device, storage medium, and program product

Abstract

Embodiments of the present disclosure relate to a data retrieval method and device, electronic equipment, storage medium and program product. The method comprises: obtaining example scene data of a target driving scene; the example scene data comprises example data of multiple different modalities related to an autonomous vehicle; performing feature extraction processing on the example scene data to obtain a multi-modal feature vector; obtaining a plurality of target multi-modal feature vectors from a pre-set feature pool according to the multi-modal feature vector; and obtaining a plurality of target scene data corresponding to the target driving scene according to the plurality of target multi-modal feature vectors, the plurality of target scene data being used to update an autonomous driving algorithm corresponding to the target driving scene. The present method can accurately determine the required scene data from a large amount of data for updating the autonomous driving system.

Description

Technical Field

[0001] This disclosure relates to the field of autonomous driving technology, and in particular to a data retrieval method, apparatus, electronic device, storage medium, and program product. Background Technology

[0002] In the field of autonomous driving, to improve the robustness and adaptability of autonomous driving systems, it is necessary to collect data from various scenarios for continuous system updates. However, the scenarios in the autonomous driving field are diverse and ever-changing, and the collected data is massive. Therefore, in the process of updating autonomous driving systems, the problem of how to accurately determine the required scenario data from massive amounts of data for system updates urgently needs to be solved. Summary of the Invention

[0003] This disclosure provides a data retrieval method, apparatus, electronic device, storage medium, and program product that can accurately determine data for required scenarios from massive amounts of data for use in updating autonomous driving systems.

[0004] In a first aspect, embodiments of this disclosure provide a data retrieval method, the method comprising:

[0005] Obtain example scene data for the target driving scenario; this example scene data includes example data of multiple different modalities related to autonomous vehicles; perform feature extraction processing on the example scene data to obtain multimodal feature vectors; based on the multimodal feature vectors, obtain multiple target multimodal feature vectors from a preset feature pool; based on the multiple target multimodal feature vectors, obtain multiple target scene data corresponding to the target driving scenario, and use the multiple target scene data to update the autonomous driving algorithm corresponding to the target driving scenario.

[0006] In one embodiment, the feature extraction process of the example scene data to obtain a multimodal feature vector includes: determining the feature extraction algorithm corresponding to each example data according to the type of the example data corresponding to each modality in the example scene data; and performing feature extraction processing on each example data according to the corresponding feature extraction algorithm to obtain the multimodal feature vector.

[0007] In one embodiment, the example scene data includes example data of visual perception type and / or example data of driving planning and control type; the feature extraction processing of each example data according to the corresponding feature extraction algorithm to obtain the multimodal feature vector includes: fusing the example data of each visual perception type to obtain fused data, and extracting features from the fused data according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain a first multimodal feature vector; or, extracting features from the example data of each visual perception type according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain multiple first candidate features, and fusing the first candidate features to obtain a first multimodal feature vector; extracting features from the example data of each driving planning and control type according to the feature extraction algorithm corresponding to the example data of the driving planning and control type to obtain multiple second candidate features, and fusing the second candidate features to obtain a second multimodal feature vector; and obtaining the multimodal feature vector based on the first multimodal feature vector and / or the second multimodal feature vector.

[0008] In one embodiment, if the example scene data includes example data of the visual perception type and example data of the driving planning and control type, obtaining the multimodal feature vector based on the first multimodal feature vector and / or the second multimodal feature vector includes: fusing the first multimodal feature vector and the second multimodal feature vector to obtain the multimodal feature vector.

[0009] In one embodiment, the feature extraction algorithm corresponding to the example data of each visual perception type performs feature extraction on the example data of each visual perception type, including: performing feature extraction on the example data of each visual perception type based on a first deep neural network model.

[0010] In one embodiment, feature extraction is performed on the example data of each driving planning and control type according to the feature extraction algorithm corresponding to the example data of the driving planning and control type, including: performing feature extraction on each example data according to the preset manual rules corresponding to the example data of each driving planning and control type, or performing feature extraction on the example data based on the second deep neural network model.

[0011] In one embodiment, example data of the visual perception type includes at least one of image data, lidar data, and millimeter-wave radar data; example data of the driving planning and control type includes at least one of vehicle control signals, vehicle planning signals, vehicle prediction signals, vehicle attitude signals, vehicle positioning signals, vehicle driving signals, and other vehicle behavior signals.

[0012] In one embodiment, the preset feature pool includes multiple reference multimodal feature vectors corresponding to different scenarios; obtaining multiple target multimodal feature vectors from the preset feature pool based on the multimodal feature vectors includes: performing feature matching processing on the multimodal feature vectors and each reference multimodal feature vector, and taking the multiple reference multimodal feature vectors with the highest fit as the multiple target multimodal feature vectors.

[0013] In one embodiment, the preset feature pool includes multiple sub-feature pools corresponding to different scenarios, and each sub-feature pool includes multiple reference multimodal feature vectors. Obtaining multiple target multimodal feature vectors from the preset feature pool based on the multimodal feature vectors includes: determining a target sub-feature pool from the multiple sub-feature pools based on the multimodal feature vectors; performing feature matching processing on the multimodal feature vectors and each reference multimodal feature vector in the target sub-feature pool, and taking the multiple reference multimodal feature vectors with the highest fit as the multiple target multimodal feature vectors.

[0014] Secondly, embodiments of this disclosure provide a data retrieval device, the device comprising:

[0015] The first acquisition module is used to acquire example scenario data of the target driving scenario; the example scenario data includes example data of multiple different modalities related to autonomous vehicles.

[0016] The feature extraction module is used to perform feature extraction processing on the example scene data to obtain a multimodal feature vector;

[0017] The second acquisition module is used to acquire multiple target multimodal feature vectors from a preset feature pool based on the multimodal feature vector;

[0018] The third acquisition module is used to acquire multiple target scene data corresponding to the target driving scenario based on the multiple target multimodal feature vectors. The multiple target scene data is used to update the autonomous driving algorithm corresponding to the target driving scenario.

[0019] In one embodiment, the feature extraction module is specifically used to: determine the feature extraction algorithm corresponding to each example data according to the type of the example data for each modality in the example scene data; and perform feature extraction processing on each example data according to the corresponding feature extraction algorithm to obtain the multimodal feature vector.

[0020] In one embodiment, the example scene data includes example data of visual perception type and / or example data of driving planning and control type; the feature extraction module is specifically used for: fusing the example data of each visual perception type to obtain fused data, and extracting features from the fused data according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain a first multimodal feature vector; or, extracting features from the example data of each visual perception type according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain multiple first candidate features, and fusing the first candidate features to obtain a first multimodal feature vector; extracting features from the example data of each driving planning and control type according to the feature extraction algorithm corresponding to the example data of the driving planning and control type to obtain multiple second candidate features, and fusing the second candidate features to obtain a second multimodal feature vector; and obtaining the multimodal feature vector based on the first multimodal feature vector and / or the second multimodal feature vector.

[0021] In one embodiment, if the example scene data includes example data of visual perception type and example data of driving planning and control type, the feature extraction module is specifically used to: fuse the first multimodal feature vector and the second multimodal feature vector to obtain the multimodal feature vector.

[0022] In one embodiment, the feature extraction module is specifically used to: extract features from example data of each visual perception type based on a first deep neural network model.

[0023] In one embodiment, the feature extraction module is specifically used to: extract features from each example data according to preset manual rules corresponding to example data of each driving planning and control type, or to extract features from the example data based on a second deep neural network model.

[0024] In one embodiment, example data of the visual perception type includes at least one of image data, lidar data, and millimeter-wave radar data; example data of the driving planning and control type includes at least one of vehicle control signals, vehicle planning signals, vehicle prediction signals, vehicle attitude signals, vehicle positioning signals, vehicle driving signals, and other vehicle behavior signals.

[0025] In one embodiment, the preset feature pool includes multiple reference multimodal feature vectors corresponding to different scenarios; the second acquisition module is specifically used to: perform feature matching processing on the multimodal feature vector and each reference multimodal feature vector, and take the multiple reference multimodal feature vectors with the highest fit as the multiple target multimodal feature vectors.

[0026] In one embodiment, the preset feature pool includes multiple sub-feature pools corresponding to different scenarios, and each sub-feature pool includes multiple reference multimodal feature vectors; the second acquisition module is specifically used to: determine a target sub-feature pool from the multiple sub-feature pools based on the multimodal feature vectors; perform feature matching processing between the multimodal feature vectors and each reference multimodal feature vector in the target sub-feature pool, and take the multiple reference multimodal feature vectors with the highest fit as the multiple target multimodal feature vectors.

[0027] Thirdly, embodiments of this disclosure provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method described in the first aspect above.

[0028] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in the first aspect above.

[0029] Fifthly, embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect above.

[0030] The data retrieval method, apparatus, electronic device, storage medium, and program product provided in this disclosure acquire example scene data of a target driving scenario, including example data of multiple different modalities related to autonomous vehicles, and perform feature extraction processing on the example scene data to obtain multimodal feature vectors. Then, based on the multimodal feature vectors, multiple target multimodal feature vectors are obtained from a preset feature pool to acquire multiple target scene data corresponding to the target driving scenario, which are then used to update the autonomous driving algorithm corresponding to the target driving scenario. In this way, using only one example scene data, multiple target scene data that can be used to update the autonomous driving algorithm of the target driving scenario can be obtained from a preset feature pool through feature extraction and matching. That is, the available data of the required target driving scenario can be quickly determined from massive amounts of autonomous driving data, ensuring the efficiency and timeliness of autonomous driving system updates. Furthermore, the feature extraction processing method allows for the acquisition of target scene data of the target driving scenario at any time, improving the flexibility of target scene data acquisition. Attached Figure Description

[0031] Figure 1 This is a diagram illustrating the application environment of a data retrieval method in one embodiment.

[0032] Figure 2 This is a flowchart illustrating the process of determining multimodal feature vectors in one embodiment;

[0033] Figure 3 This is a schematic diagram of another process for determining multimodal feature vectors in one embodiment;

[0034] Figure 4 This is a flowchart illustrating the process of determining multiple target multimodal feature vectors in one embodiment;

[0035] Figure 5 This is a structural block diagram of a data retrieval device in one embodiment;

[0036] Figure 6 This is a diagram of the internal structure of a server in one embodiment. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings and examples. It should be understood that the specific embodiments described herein are merely illustrative of the embodiments of this disclosure and are not intended to limit the embodiments of this disclosure.

[0038] First, before introducing the technical solutions of the embodiments of this disclosure in detail, the technical background or technical evolution of the embodiments of this disclosure on which they are based will be introduced.

[0039] In the field of autonomous driving technology, a recognized challenge lies in achieving sufficient robustness for an autonomous driving system to adapt to any road surface and scenario. This is because, in real-world scenarios, autonomous driving scenarios and data are incredibly diverse and varied. No matter the scale of the dataset collected, there will always be unseen edge scenarios and corresponding data that the autonomous driving system has never encountered before. Therefore, efficiently collecting high-value autonomous driving scenario data has become a crucial issue to address.

[0040] In related technologies, a common approach to collecting corner case data is to use a trigger mechanism. A trigger is a set of artificially designed conditions deployed in an autonomous driving system. During autonomous driving, when a certain trigger condition is met, the system automatically collects data fragments before and after the trigger; these fragments constitute the corner case data. Against this backdrop, through long-term model simulation development and the collection, demonstration, and verification of experimental data, the applicant discovered that the trigger-based method can only obtain relevant data when the trigger's conditions are met, requiring the vehicle to be online. Furthermore, if algorithm developers need to update the autonomous driving algorithm for a corner case quickly, there is a problem of not being able to obtain the relevant data in a timely manner, resulting in poor flexibility and real-time performance. It should also be noted that the applicant has devoted considerable creative effort from identifying the aforementioned technical problems to the technical solutions described in the following embodiments.

[0041] The technical solutions involved in the embodiments of this disclosure will be described below in conjunction with the scenarios in which they are applied.

[0042] It should be noted that the data retrieval method provided in this application can be executed by a data retrieval device, which can be implemented as part or all of an electronic device through software, hardware, or a combination of both. The electronic device can be, but is not limited to, various personal computers, laptops, smartphones, tablets, and servers, or can be implemented through interaction between the electronic device and a server. The server can be a standalone server or a server cluster composed of multiple servers. In the following method embodiments, the execution subject is described as an electronic device.

[0043] In one embodiment, such as Figure 1 As shown, a data retrieval method is provided. Taking the application of this method to an electronic device as an example, the method includes the following steps:

[0044] Step 101: Obtain example scenario data for the target driving scenario. The example scenario data includes example data for multiple different modalities related to the autonomous vehicle.

[0045] The target driving scenario is the driving scenario corresponding to the current autonomous driving algorithm that needs to be updated. Optionally, there can be one or more target driving scenarios. Optionally, the target driving scenario can be some edge scenarios of vehicle autonomous driving, which are mainly some possible scenarios with low probability of occurrence, such as weather changes, sudden steering wheel turns by the driver, sudden braking by the driver, unstable target detection, complex intersections, and other vehicles merging into the driving route, etc., which are not fully exemplified here.

[0046] When an electronic device determines a target driving scenario, it can acquire example scenario data. Optionally, the example scenario data can be data generated by the vehicle's autonomous driving in the target driving scenario. Specifically, the example scenario data can include example data from multiple different modalities related to the autonomous vehicle.

[0047] Optionally, the electronic device may store the correspondence between different driving scenarios and example scenario data. Algorithm engineers can select a target driving scenario in the electronic device, and then the electronic device can determine the example scenario data corresponding to the target driving scenario from the correspondence.

[0048] Optionally, electronic devices can also acquire example scene data of the target driving scenario input in real time by algorithm engineers.

[0049] Optionally, the electronic device can determine a target driving scenario from multiple preset driving scenarios based on the identifier selected by the algorithm administrator, and then filter a set of scenario data for the target driving scenario from the scenario database as the example scenario data. Alternatively, the electronic device can filter multiple sets of scenario data for the target driving scenario from the scenario database, compare the completeness of these multiple sets of scenario data, and use the scenario data with the highest completeness as the example scenario data. Here, highest completeness may refer to the scenario data set containing the most modalities of example data.

[0050] Different forms of existence or information sources can all be referred to as a modality. Data composed of two or more modalities is called multimodal data. Multimodal data refers to data obtained from different directions or perspectives for the same descriptive object. In this embodiment of the disclosure, the example scene data includes example data of multiple different modalities related to autonomous vehicles. That is, it includes example data of different modalities for the target driving scenario. Therefore, the example scene data in this embodiment of the disclosure is multimodal data. Optionally, the data format of the example data of each modality can be different, such as images, text, videos, time-series signals, etc., as long as it is vehicle-related data during autonomous driving.

[0051] For example, the example scenario data may include multiple data sources such as camera signals, lidar signals, millimeter-wave radar signals, vehicle control signals, vehicle attitude signals, vehicle positioning signals, and other vehicle behavior signals acquired by the vehicle's camera.

[0052] Step 102: Perform feature extraction processing on the example scene data to obtain multimodal feature vectors.

[0053] After acquiring example scene data, the electronic device needs to perform feature extraction on the example scene data to accurately determine multiple target scene data that can be used for algorithm updates. Based on the features of each example data obtained through feature extraction, a multimodal feature vector is determined. Since this is a multimodal feature vector, meaning it includes multimodal feature data, the required target scene data can be quickly and accurately determined from massive amounts of autonomous driving data.

[0054] Step 103: Obtain multiple target multimodal feature vectors from a preset feature pool based on the multimodal feature vectors.

[0055] The preset feature pool includes multiple preset multimodal feature vectors. Each preset modal feature vector is determined based on multimodal scene data for the corresponding driving scenario. This multimodal scene data can be collected from different vehicles during historical autonomous driving processes, or it can be obtained by electronic devices simulating the autonomous driving process and driving environment in that scenario.

[0056] For example, for each driving scenario, the electronic device can acquire multimodal scene data collected at different times and from different vehicles corresponding to each driving scenario. The electronic device performs feature extraction processing on the multiple sets of multimodal scene data corresponding to each driving scenario to obtain multiple preset multimodal feature vectors corresponding to each driving scenario, and stores them in the electronic device or in the database of the server to form the preset feature pool.

[0057] Optionally, the preset multimodal feature vectors included in the preset feature pool can also be full-modal. That is, each preset multimodal feature vector is obtained by feature extraction processing based on a set of full-modal scene data for the corresponding driving scenario. Here, full-modal can refer to all vehicle-related modal data in the driving scenario.

[0058] The electronic device can determine multiple preset multimodal feature vectors that are most similar to the multimodal feature vector from a preset feature pool based on the multimodal feature vector, and use them as multiple target multimodal feature vectors.

[0059] Step 104: Obtain multiple target scene data corresponding to the target driving scenario based on multiple target multimodal feature vectors. The multiple target scene data are used to update the autonomous driving algorithm corresponding to the target driving scenario.

[0060] As mentioned above, each preset multimodal feature vector is determined based on multimodal or full-modal scene data. Correspondingly, the database of the electronic device or server stores the scene data corresponding to each preset multimodal feature vector to form a scene database.

[0061] After determining the multiple target multimodal feature vectors, the electronic device can query the scene data corresponding to each target multimodal feature vector from the scene database based on the multiple target multimodal feature vectors, and use it as the target scene data.

[0062] Optionally, each preset multimodal feature vector in the preset feature pool is identified by a preset sequence number, and similarly, each set of scene data in the scene database is identified by a preset sequence number. Thus, after determining multiple target multimodal feature vectors, the electronic device queries the scene database based on the preset sequence number corresponding to each target multimodal feature vector to determine the scene data corresponding to each target multimodal feature vector, which is then used as the target scene data.

[0063] Multiple target scenario data sets are used by algorithm engineers to update the autonomous driving algorithm corresponding to the target driving scenario. Updating the autonomous driving algorithm can refer to upgrading the autonomous driving system. Optionally, if an autonomous driving system is deployed in the electronic device, the device can also directly update the autonomous driving algorithm corresponding to the target driving scenario given multiple target scenario data sets.

[0064] Optionally, multiple target scene data can also be used to display potentially dangerous scenarios to users.

[0065] The data retrieval method provided in this disclosure acquires example scene data of a target driving scenario, including example data of multiple different modalities related to autonomous vehicles, and performs feature extraction processing on the example scene data to obtain multimodal feature vectors. Then, based on the multimodal feature vectors, multiple target multimodal feature vectors are obtained from a preset feature pool. These target multimodal feature vectors are then used to obtain multiple target scene data corresponding to the target driving scenario, which are then used to update the autonomous driving algorithm corresponding to the target driving scenario. In this way, using only one example scene data, multiple target scene data that can be used to update the autonomous driving algorithm of the target driving scenario can be obtained from a preset feature pool through feature extraction and matching. That is, it can quickly determine the available data of the required target driving scenario from massive amounts of autonomous driving data, ensuring the efficiency and timeliness of autonomous driving system updates. Furthermore, the feature extraction processing method allows for the acquisition of target scene data of the target driving scenario at any time, improving the flexibility of target scene data acquisition.

[0066] In this embodiment of the disclosure, to achieve accurate feature extraction, in one feasible approach, different feature extraction algorithms can be used to extract features from different types of example data, based on the type of each example data, to obtain a multimodal feature vector that can accurately represent the features of the example scene. The process of feature extraction processing for example scene data is described below.

[0067] In one embodiment, such as Figure 2 The diagram illustrates a flowchart of a method for determining a multimodal feature vector according to an embodiment of this disclosure. Feature extraction processing is performed on example scene data to obtain a multimodal feature vector, including:

[0068] Step 201: Determine the feature extraction algorithm corresponding to each example data according to the type of example data for each modality in the example scene data.

[0069] Step 202: Based on the corresponding feature extraction algorithm, perform feature extraction processing on each example data to obtain a multimodal feature vector.

[0070] In this context, for example data related to different modalities of autonomous vehicles, some example data are correlated, while others are not. Furthermore, the feature distributions of the example data may differ across modalities. Therefore, in this embodiment, to perform accurate feature extraction on each example data, a corresponding feature extraction algorithm is determined based on the type of example data for each modality, and feature extraction is performed on each example data according to the corresponding algorithm. Thus, the resulting multimodal feature vector can accurately characterize the feature distribution of the example scene data for the target driving scenario.

[0071] In one embodiment, the example scenario data includes example data of the visual perception type and / or example data of the driving planning and control type. That is, the data related to autonomous vehicles can be divided into visual perception type data and driving planning and control type data. Since the example scenario data is only the scenario data corresponding to the target driving scenario, the example data in the example scenario data may only include example data of multiple modalities of visual perception, or may include example data of multiple modalities of driving planning and control, or may include example data of both visual perception type and driving planning and control type. No specific limitation is made here.

[0072] In this embodiment of the disclosure, example data of the visual perception type includes at least one of image data, lidar data, and millimeter-wave radar data. Example data of the driving planning and control type includes at least one of vehicle control signals, vehicle planning signals, vehicle prediction signals, vehicle attitude signals, vehicle positioning signals, vehicle driving signals, and other vehicle behavior signals.

[0073] The example data for visual perception type consists of data acquired by various sensor devices in the vehicle, while the example data for driving planning and control type consists of vehicle-related signals.

[0074] Specifically, image data includes images acquired by camera devices in autonomous vehicles. LiDAR data includes data acquired by LiDAR devices in autonomous vehicles, such as point cloud maps. Millimeter-wave radar data includes data acquired by millimeter-wave radar devices in autonomous vehicles. Since all these data are related to the vehicle's driving scenario—for example, data acquired simultaneously by cameras, LiDAR, and millimeter-wave radar may all be related to the same environmental area—these data are all considered visual perception data.

[0075] The example data for driving planning control types are all related to the driving direction of the vehicle or other vehicles and driving prediction planning control. Vehicle control signals include, for example, control signals generated by the vehicle's central control system. Vehicle planning signals include, for example, signals that plan the vehicle's driving trajectory. Vehicle prediction signals include, for example, signals that predict the vehicle's driving trajectory. Vehicle attitude signals include, for example, data such as vehicle body angles. Other vehicle behavior signals include behavioral signals related to other vehicles within a preset range around the vehicle.

[0076] Optionally, the example data for visual perception and driving planning and control can be single-frame data or a sequence of data. Correspondingly, the example scene data can be single-frame data, a sequence of data, or complex data composed of an indefinite number of single frames and sequences of data.

[0077] The following section provides a detailed explanation of the feature extraction process for different types of example data. In one embodiment, such as... Figure 3 The diagram illustrates another flowchart for determining multimodal feature vectors provided in this embodiment. Based on the corresponding feature extraction algorithm, feature extraction processing is performed on each example data to obtain multimodal feature vectors, including:

[0078] Step 301: Perform fusion processing on the example data of each visual perception type to obtain fused data, and perform feature extraction on the fused data according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain a first multimodal feature vector; or, perform feature extraction on the example data of each visual perception type according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain multiple first candidate features, and perform fusion processing on each first candidate feature to obtain a first multimodal feature vector.

[0079] Since there are multiple example data sets, to facilitate subsequent feature matching, feature extraction is performed on each example data set to obtain a multimodal feature vector corresponding to the example scene data. Optionally, the length of the multimodal feature vector corresponding to the example scene data is fixed to facilitate subsequent feature matching. That is, the length of the multimodal feature vectors obtained by feature extraction processing on example scene data from different driving scenarios can be equal.

[0080] If the example scene data includes example data of multiple visual perception types with different modalities, as mentioned above, the example data of each visual perception type are correlated. Therefore, the electronic device can first directly fuse the example data of each visual perception type to obtain fused data, and then directly extract features from the fused data according to the feature extraction algorithm corresponding to the example data of each visual perception type to obtain the first multimodal feature vector. Alternatively, the electronic device can extract features from the example data of each visual perception type separately before fusing the data. That is, the electronic device first extracts features from the example data of each visual perception type separately according to the feature extraction algorithm corresponding to the example data of each visual perception type to obtain the first candidate features corresponding to each visual perception type, and then directly fuses the first candidate features to obtain the first multimodal feature vector.

[0081] Step 302: Based on the feature extraction algorithm corresponding to the example data of each driving planning and control type, feature extraction is performed on the example data of each driving planning and control type to obtain multiple second candidate features, and the second candidate features are fused to obtain the second multimodal feature vector.

[0082] If the example scenario data includes example data of multiple different modalities of driving planning and control, as mentioned above, the correlation between the example data of driving planning and control types is small. Therefore, the e-book device can perform feature extraction processing on the example data of each driving planning and control type separately to obtain the second candidate features corresponding to the example data of each driving planning and control type. Then, the second candidate features are directly fused to obtain the second multimodal feature vector.

[0083] Step 303: Obtain the multimodal feature vector based on the first multimodal feature vector and / or the second multimodal feature vector.

[0084] If the example scenario data only includes example data of multiple modalities of visual perception, then after performing feature extraction processing on the example data of each visual perception type to obtain the first multimodal feature vector, this first multimodal feature vector can be directly used as the multimodal feature vector corresponding to the target driving scenario. If the example scenario data only includes example data of driving planning and control type, then after performing feature extraction processing on the example data of each driving planning and control type to obtain the second multimodal feature vector, this second multimodal feature vector can be directly used as the multimodal feature vector corresponding to the target driving scenario.

[0085] If the example scenario data includes example data of visual perception type and example data of driving planning and control type, the multimodal feature vector corresponding to the target driving scenario can be obtained based on the first multimodal feature vector and the second multimodal feature vector.

[0086] Optionally, during the fusion processing of example data for each visual perception type, the fusion processing of each first candidate feature, and the fusion processing of each second candidate feature, the fusion processing method can be feature concatenation, feature addition, feature multiplication, or adaptive feature fusion processing, etc. Among them, adaptive feature fusion processing can be adaptive feature fusion processing based on an attention mechanism.

[0087] In one embodiment, obtaining a multimodal feature vector based on a first multimodal feature vector and / or a second multimodal feature vector includes: fusing the first multimodal feature vector and the second multimodal feature vector to obtain the multimodal feature vector.

[0088] After obtaining the first and second multimodal feature vectors, the electronic device needs to fuse them to obtain the multimodal feature vector corresponding to the target driving scenario. The fusion process of the first and second multimodal feature vectors can include feature concatenation, feature addition, feature multiplication, or adaptive feature fusion. The adaptive feature fusion process can be based on an attention mechanism.

[0089] Optionally, adaptive feature fusion based on attention mechanisms can be achieved by fusing two features through a self-attention layer.

[0090] It is understandable that the process of determining multiple preset multimodal feature vectors in the preset feature pool based on the corresponding scene data can be similar to the process of obtaining multimodal feature vectors through feature extraction as described above. For example, for a set of multimodal scene data, feature extraction and fusion processing are performed on data of the same type, and then the obtained feature vectors are fused to obtain the final preset multimodal feature vector.

[0091] In this embodiment of the disclosure, by analyzing multimodal example data and taking into account the fusion of multimodal data, a multimodal feature vector with comprehensive feature coverage is obtained. Consequently, the multiple target scene data determined based on this vector are highly representative, valuable, and have comprehensive coverage.

[0092] In one embodiment, feature extraction is performed on the example data of each visual perception type according to the feature extraction algorithm corresponding to the example data of the visual perception type, including: feature extraction is performed on the example data of each visual perception type based on a first deep neural network model.

[0093] The first deep neural network model is primarily used for feature extraction from example data of visual perception types. Optionally, the first deep neural network model can be iteratively trained based on a large amount of pre-collected visual perception type data to obtain the first deep neural network model.

[0094] Optionally, for example data of each visual perception type, each example data is input into the first deep neural network model to obtain the first candidate features corresponding to each example data output by the first deep neural network model.

[0095] Optionally, for example data of each visual perception type, all example data are simultaneously input into the first deep neural network model to obtain the first candidate features corresponding to each example data directly output by the first deep neural network model. Optionally, each first candidate feature can be in the form of a vector.

[0096] In one embodiment, feature extraction is performed on the fused data according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain a first multimodal feature vector, including: feature extraction on the fused data based on a first deep neural network model.

[0097] The fused data is obtained by fusing example data from various visual perception types. The first deep neural network model is also trained based on visual perception type data. Therefore, the electronic device can directly input the fused data into the first deep neural network model to obtain the first multimodal feature vector directly output by the first deep neural network model.

[0098] In one embodiment, feature extraction is performed on the example data of each driving planning and control type according to the feature extraction algorithm corresponding to the example data of each driving planning and control type, including: performing feature extraction on each example data according to the preset manual rules corresponding to the example data of each driving planning and control type, or performing feature extraction on the example data based on the second deep neural network model.

[0099] The example data for driving planning and control is a sequential signal, meaning it is a signal generated over a period of time and is time-dependent. In this embodiment, for the sequential signal, feature extraction can be performed using preset manual rules or based on a second deep neural network model.

[0100] Specifically, the electronic device can deploy a set of preset manual rules corresponding to example data of different modes of driving planning and control. For each mode of example data, the corresponding set of preset manual rules is used to extract features to obtain its corresponding second candidate feature. For example, one example data point in the example scenario data is a vehicle driving signal of the driving planning and control type. The electronic device can determine a set of preset manual rules corresponding to the vehicle driving signal. This set of preset manual rules may include one or more preset manual rules. Based on this set of preset manual rules, feature extraction is performed on the vehicle driving signal to obtain the second candidate feature corresponding to the vehicle driving signal.

[0101] For example, taking vehicle positioning signals as an example, a set of manual rules corresponding to vehicle positioning signals may include that the vehicle positioning signal is valid within a preset range, the vehicle positioning signal is valid within a preset lane range, and the intensity of the vehicle positioning signal is valid above a preset intensity threshold.

[0102] Furthermore, for each example data point of the driving planning and control type, each example data point can be simultaneously or sequentially input into the second deep neural network model to obtain the second candidate features corresponding to each example data point output by the second deep neural network model. Optionally, the sequence signal of the position and size of the target detection box of the vehicle or other vehicles changing over time can be used as the input of the target neural network, with the form type of the vehicle or other vehicles as the supervision signal, to train the target neural network, and the intermediate layer of the trained target neural network can be extracted as the second deep neural network model for feature extraction. The form type of the vehicle or other vehicles refers to, for example, whether the prediction is accurate or inaccurate, or whether the driving route is normal or deviated.

[0103] In this embodiment, feature extraction is performed using a corresponding feature extraction algorithm based on the type of each example data to achieve accurate and rapid feature extraction. This is then combined with fusion processing to obtain a fixed-length multimodal feature vector, providing a stable foundation for determining the most suitable multiple target scene data.

[0104] The process of determining multiple target multimodal feature vectors is explained below.

[0105] In one embodiment, obtaining multiple target multimodal feature vectors from a preset feature pool based on multimodal feature vectors includes: performing feature matching processing between the multimodal feature vectors and each reference multimodal feature vector, and using the multiple reference multimodal feature vectors with the highest fit as multiple target multimodal feature vectors.

[0106] The preset feature pool includes multiple reference multimodal feature vectors corresponding to different scenarios. That is, the preset feature pool includes multiple reference multimodal feature vectors, each of which corresponds to a specific autonomous driving scenario, and some reference multimodal feature vectors may correspond to the same autonomous driving scenario.

[0107] After determining the multimodal feature vector corresponding to the target driving scenario, the electronic device performs feature matching processing on the multimodal feature vector and each reference multimodal feature vector to determine which reference multimodal feature vectors have the highest fit.

[0108] Optionally, feature matching can be performed by calculating the feature distance or feature similarity between the multimodal feature vector and each reference multimodal feature vector. Optionally, the feature distance can be determined by calculating the Euclidean distance or KL divergence between the two vectors; the smaller the Euclidean distance or KL divergence, the higher the fit. Optionally, feature similarity can be determined by calculating the cosine similarity between the two vectors; the larger the cosine similarity, the higher the fit. It should be noted that other feature matching methods can also be used to determine the multiple reference multimodal feature vectors with the highest fit, as long as they can characterize the distance between the two vectors.

[0109] Based on this, the fitness scores between the multimodal feature vectors and each reference multimodal feature vector are arranged in order, and the top K reference multimodal feature vectors with the highest fitness scores are selected as multiple target multimodal feature vectors. K is a positive integer. K can be determined according to the actual situation.

[0110] In one embodiment, such as Figure 4 The diagram illustrates a flowchart of determining multiple target multimodal feature vectors according to an embodiment of this application. Based on the multimodal feature vectors, multiple target multimodal feature vectors are obtained from a preset feature pool, including:

[0111] Step 401: Determine the target sub-feature pool from multiple sub-feature pools based on the multimodal feature vector.

[0112] Step 402: Perform feature matching processing on the multimodal feature vectors and each reference multimodal feature vector in the target sub-feature pool, and take the multiple reference multimodal feature vectors with the highest fitness as multiple target multimodal feature vectors.

[0113] The preset feature pool includes multiple sub-feature pools corresponding to different scenarios, and each sub-feature pool contains multiple reference multimodal feature vectors. In other words, the preset feature pool includes multiple sub-feature pools, each corresponding to a different scenario.

[0114] Optionally, for each sub-feature pool, clustering is performed on each reference multimodal feature vector in that sub-feature pool to obtain the corresponding identifier feature vector. Optionally, clustering can be achieved using a clustering algorithm. In this way, the electronic device can perform feature matching processing between the multimodal feature vector corresponding to the target driving scenario and the identifier feature vectors corresponding to each sub-feature pool to determine the sub-feature pool with the highest fit as the target sub-feature pool.

[0115] Furthermore, by performing feature matching processing between the multimodal feature vectors and each reference multimodal feature vector in the target sub-feature pool, multiple target multimodal feature vectors can be quickly determined.

[0116] In this embodiment, considering that the reference multimodal feature vectors in the preset feature pool are massive, each reference multimodal feature vector is classified to determine multiple sub-feature pools, and each sub-feature pool is pre-identified using an identifier feature vector. Thus, by determining the target sub-feature pool, feature matching processing is only required with the reference multimodal feature vectors in the target sub-feature pool, thereby improving the efficiency of determining the target multimodal feature vector and reducing the computational power consumption of the electronic device.

[0117] In one embodiment, obtaining multiple target multimodal feature vectors from a preset feature pool based on multimodal feature vectors includes: performing dimensionality reduction processing on the multimodal feature vectors to obtain hash values ​​corresponding to the multimodal feature vectors; determining target sub-feature pools from each sub-feature pool in the preset feature pool based on the hash values; performing feature matching processing on the multimodal feature vectors and each reference multimodal feature vector in the target sub-feature pool; and using the multiple reference multimodal feature vectors with the highest fit as multiple target multimodal feature vectors.

[0118] In this system, each sub-feature pool within the preset feature pool is stored in the database of the electronic device or server using a hash index. The electronic device can perform dimensionality reduction on the multimodal feature vectors and use the resulting dimensionality reduction values ​​as the hash values ​​corresponding to the multimodal feature vectors.

[0119] Furthermore, the electronic device can match the hash value corresponding to the multimodal feature vector with the hash values ​​of each sub-feature pool, and obtain the sub-feature pool whose hash value is closest to that corresponding to the multimodal feature vector as the target sub-feature pool. Then, by simply performing feature matching processing between the multimodal feature vector and each reference multimodal feature vector in the target sub-feature pool, multiple target multimodal feature vectors can be quickly identified.

[0120] In this embodiment, each sub-feature pool is stored using an index, which facilitates quick retrieval of the target sub-feature pool based on the hash value corresponding to the multimodal feature vector, avoiding a large amount of redundant calculation. Furthermore, the target multimodal feature vector can be quickly determined directly based on each reference multimodal feature vector in the target sub-feature pool.

[0121] The following describes an embodiment of this disclosure using a specific data mining scenario. This embodiment provides a method for edge scene data mining based on multimodal retrieval. The method includes the following:

[0122] This method includes a multimodal feature extraction module, a multimodal feature fusion module, a multimodal feature matching module, and a multimodal feature storage pool.

[0123] After the algorithm engineer inputs exemplary scene data of the target edge scene to be retrieved into the electronic device, the multimodal feature extraction module extracts features from different modalities of the scene data. Then, the multimodal feature fusion module combines the features of different modalities to form a fixed-length multimodal feature vector. Next, the multimodal feature matching module sequentially matches the multimodal feature vector of the scene data with the feature vectors of the data to be mined in the multimodal feature storage pool, recording the K sets of feature vectors with the highest matching degree. Finally, the original scene data corresponding to these K sets of feature vectors is the target scene data of the mined target edge scene. This target scene data can be used by algorithm engineers to upgrade autonomous driving systems or to demonstrate potentially dangerous scenarios.

[0124] The scene data of the target edge scene can be single frame data, a sequence of data, or complex data composed of an indefinite number of single frames and sequences of data.

[0125] In the aforementioned multimodal feature extraction module, the feature extraction methods differ for different modalities. For image, LiDAR data, and millimeter-wave radar data, feature extraction can be performed based on deep neural networks. For sequential signals such as vehicle control signals, vehicle attitude signals, vehicle positioning signals, vehicle driving signals, vehicle prediction signals, vehicle planning signals, vehicle trajectory signals, and other vehicle behavior signals, feature extraction can be performed based on manual rules or using deep neural networks.

[0126] In the aforementioned multimodal feature fusion module, for image, LiDAR data, and millimeter-wave radar data, feature fusion can be performed using either pre-fusion or post-fusion methods. Pre-fusion refers to first combining scene data from different modalities, and then extracting features from the combined data based on a deep neural network. Post-fusion refers to first extracting features from scene data of different modalities separately, and then directly combining the features.

[0127] In the aforementioned multimodal feature fusion module, vehicle control signals, vehicle attitude signals, vehicle positioning signals, vehicle driving / predicted / planned trajectories, and other vehicle behavior signals are fused using a post-fusion method.

[0128] In the aforementioned multimodal feature fusion module, after performing feature fusion on the image, LiDAR data, and millimeter-wave radar data respectively, as well as feature fusion on the vehicle-related signals, the obtained features are fused again to obtain the multimodal feature vector corresponding to the target edge scene.

[0129] The aforementioned fusion methods can include feature concatenation, feature addition, feature multiplication, or adaptive feature fusion based on attention mechanisms.

[0130] In the aforementioned multimodal feature matching module, the multimodal feature vector corresponding to the target edge scene is matched sequentially with the feature vector of the data to be mined in the multimodal feature storage pool by calculating the Euclidean distance, KL divergence, or cosine similarity between the two.

[0131] In this embodiment, data is mined through data retrieval, eliminating the need for waiting and allowing for the timely acquisition of target scene data for updating and upgrading autonomous driving algorithms. By analyzing multimodal data, the value of each scene data point and its potential for mining are efficiently determined. Due to the consideration of multimodal data fusion, the mined scene data is highly representative, valuable, and comprehensive. Furthermore, the use of feature fusion and feature matching techniques for scene retrieval offers advantages such as high retrieval efficiency, high hit rate, and accurate results.

[0132] It should be understood that although the steps in the flowcharts of the embodiments described above are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the embodiments described above may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the steps or stages of other steps.

[0133] In one embodiment, such as Figure 5 As shown, a data retrieval device 500 is provided, comprising: a first acquisition module 501, a feature extraction module 502, a second acquisition module 503, and a third acquisition module 504, wherein:

[0134] The first acquisition module 501 is used to acquire example scenario data of the target driving scenario; the example scenario data includes example data of multiple different modalities related to the autonomous vehicle.

[0135] Feature extraction module 502 is used to perform feature extraction processing on the example scene data to obtain a multimodal feature vector;

[0136] The second acquisition module 503 is used to acquire multiple target multimodal feature vectors from a preset feature pool based on the multimodal feature vector;

[0137] The third acquisition module 504 is used to acquire multiple target scene data corresponding to the target driving scene based on the multiple target multimodal feature vectors. The multiple target scene data is used to update the autonomous driving algorithm corresponding to the target driving scene.

[0138] In one embodiment, the feature extraction module 502 is specifically used to: determine the feature extraction algorithm corresponding to each example data according to the type of the example data of each modality in the example scene data; and perform feature extraction processing on each example data according to the corresponding feature extraction algorithm to obtain the multimodal feature vector.

[0139] In one embodiment, the example scene data includes example data of visual perception type and / or example data of driving planning and control type; the feature extraction module 502 is specifically used for: fusing the example data of each visual perception type to obtain fused data, and extracting features from the fused data according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain a first multimodal feature vector; or, extracting features from the example data of each visual perception type according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain multiple first candidate features, and fusing the first candidate features to obtain a first multimodal feature vector; extracting features from the example data of each driving planning and control type according to the feature extraction algorithm corresponding to the example data of the driving planning and control type to obtain multiple second candidate features, and fusing the second candidate features to obtain a second multimodal feature vector; and obtaining the multimodal feature vector based on the first multimodal feature vector and / or the second multimodal feature vector.

[0140] In one embodiment, if the example scene data includes example data of visual perception type and example data of driving planning and control type, the feature extraction module 502 is specifically used to: perform fusion processing on the first multimodal feature vector and the second multimodal feature vector to obtain the multimodal feature vector.

[0141] In one embodiment, the feature extraction module 502 is specifically used to: extract features from example data of each visual perception type based on a first deep neural network model.

[0142] In one embodiment, the feature extraction module 502 is specifically used to: extract features from each example data according to preset manual rules corresponding to example data of each driving planning and control type, or to extract features from the example data based on a second deep neural network model.

[0143] In one embodiment, example data of the visual perception type includes at least one of image data, lidar data, and millimeter-wave radar data; example data of the driving planning and control type includes at least one of vehicle control signals, vehicle planning signals, vehicle prediction signals, vehicle attitude signals, vehicle positioning signals, vehicle driving signals, and other vehicle behavior signals.

[0144] In one embodiment, the preset feature pool includes multiple reference multimodal feature vectors corresponding to different scenarios; the second acquisition module 503 is specifically used to: perform feature matching processing on the multimodal feature vector and each reference multimodal feature vector, and take the multiple reference multimodal feature vectors with the highest fit as the multiple target multimodal feature vectors.

[0145] In one embodiment, the preset feature pool includes multiple sub-feature pools corresponding to different scenarios, and each sub-feature pool includes multiple reference multimodal feature vectors; the second acquisition module 503 is specifically used to: determine a target sub-feature pool from the multiple sub-feature pools based on the multimodal feature vectors; perform feature matching processing between the multimodal feature vectors and each reference multimodal feature vector in the target sub-feature pool, and take the multiple reference multimodal feature vectors with the highest fit as the multiple target multimodal feature vectors.

[0146] For specific limitations regarding the data retrieval device, please refer to the limitations on the data retrieval method above, which will not be repeated here. Each module in the aforementioned data retrieval device can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in or independent of the processor in the electronic device in hardware form, or stored in the memory of the electronic device in software form, so that the processor can call and execute the operations corresponding to each module.

[0147] Figure 6 This is a block diagram illustrating a server 1400 according to an exemplary embodiment. (Refer to...) Figure 6 Server 1400 includes processing component 1420, which further includes one or more processors, and memory resources represented by memory 1422 for storing instructions or computer programs, such as application programs, that can be executed by processing component 1420. The application programs stored in memory 1422 may include one or more modules, each corresponding to a set of instructions. Furthermore, processing component 1420 is configured to execute instructions to perform the aforementioned data retrieval method.

[0148] Server 1400 may also include a power supply component 1424 configured to perform power management of device 1400, a wired or wireless network interface 1426 configured to connect device 1400 to a network, and an input / output (I / O) interface 1428. Server 1400 can operate on an operating system stored in memory 1422, such as Windows 14 Server™, Mac OS X™, Unix™, Linux™, FreeB14D™, or similar.

[0149] In an exemplary embodiment, a storage medium including instructions is also provided, such as a memory 1422 including instructions, which can be executed by the processor of server 1400 to perform the above-described method. The storage medium may be a non-transitory computer-readable storage medium, such as a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device.

[0150] In an exemplary embodiment, a computer program product is also provided, which, when executed by a processor, can implement the above-described methods. The computer program product includes one or more computer instructions. When these computer instructions are loaded and executed on a computer, some or all of the above-described methods can be implemented, wholly or partially, according to the processes or functions described in the embodiments of this disclosure.

[0151] It should be noted that any schemes described in this specification and embodiments that involve the processing of personal information will be carried out under the premise of having a legal basis (such as obtaining the consent of the personal information subject, or being necessary for the performance of a contract), and will only be processed within the scope stipulated or agreed upon. A user's refusal to process personal information beyond what is necessary for basic functions will not affect the user's use of basic functions.

[0152] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this disclosure can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.

[0153] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0154] The above-described embodiments are merely illustrative of several implementation methods of the present disclosure, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present disclosure embodiments, and these all fall within the protection scope of the present disclosure embodiments. Therefore, the protection scope of the patent for the embodiments of the present disclosure should be determined by the appended claims.

Claims

1. A data retrieval method, characterized by, The method comprises: obtaining example scene data of a target driving scene; the example scene data comprises example data of different modes related to an autonomous vehicle; performing feature extraction processing on the example scene data to obtain a multi-modal feature vector; comprising: determining a feature extraction algorithm corresponding to each example data in the example scene data according to the type of the example data of each mode; performing feature extraction processing on each example data according to the corresponding feature extraction algorithm to obtain the multi-modal feature vector; the example scene data comprises example data of a visual perception type and / or example data of a driving planning control type; the performing feature extraction processing on each example data according to the corresponding feature extraction algorithm to obtain the multi-modal feature vector comprises: performing fusion processing on each example data of the visual perception type to obtain fusion data, and performing feature extraction on the fusion data according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain a first multi-modal feature vector; or, performing feature extraction on each example data of the visual perception type according to the feature extraction algorithm corresponding to the example data of the visual perception type to obtain a plurality of first candidate features, and performing fusion processing on each first candidate feature to obtain a first multi-modal feature vector; performing feature extraction on each example data of the driving planning control type according to the feature extraction algorithm corresponding to the example data of the driving planning control type to obtain a plurality of second candidate features, and performing fusion processing on each second candidate feature to obtain a second multi-modal feature vector; obtaining the multi-modal feature vector according to the first multi-modal feature vector and / or the second multi-modal feature vector; obtaining a plurality of target multi-modal feature vectors from a preset feature pool according to the multi-modal feature vector; obtaining a plurality of target scene data corresponding to the target driving scene according to the plurality of target multi-modal feature vectors, the plurality of target scene data being used for updating an autonomous driving algorithm corresponding to the target driving scene.

2. The method of claim 1, wherein, If the example scene data comprises example data of the visual perception type and example data of the driving planning control type, the obtaining the multi-modal feature vector according to the first multi-modal feature vector and / or the second multi-modal feature vector comprises: performing fusion processing on the first multi-modal feature vector and the second multi-modal feature vector to obtain the multi-modal feature vector.

3. The method of claim 1, wherein, The performing feature extraction on each example data of the visual perception type according to the feature extraction algorithm corresponding to the example data of the visual perception type comprises: performing feature extraction on each example data of the visual perception type based on a first deep neural network model.

4. The method of claim 1, wherein, The performing feature extraction on each example data of the driving planning control type according to the feature extraction algorithm corresponding to the example data of the driving planning control type comprises: The preset artificial rule corresponding to the example data of each driving planning control type is used for feature extraction of each example data, or the second deep neural network model is used for feature extraction of the example data.

5. The method of claim 1, wherein, The example data of the visual perception type includes at least one of image data, laser radar data and millimeter wave radar data; and the example data of the driving planning control type includes at least one of vehicle control signals, vehicle planning signals, vehicle prediction signals, vehicle posture signals, vehicle positioning signals, vehicle driving signals and other vehicle behavior signals.

6. The method according to any one of claims 1 to 5, characterized in that, The preset feature pool includes a plurality of reference multi-modal feature vectors corresponding to different scenes respectively; and the plurality of target multi-modal feature vectors are obtained from the preset feature pool according to the multi-modal feature vector, including: The multi-modal feature vector is matched with each reference multi-modal feature vector for feature matching processing, and the plurality of reference multi-modal feature vectors with the highest adaptation degree are taken as the plurality of target multi-modal feature vectors.

7. The method according to any one of claims 1 to 5, characterized in that, The preset feature pool includes a plurality of sub-feature pools corresponding to different scenes respectively, and each sub-feature pool includes a plurality of reference multi-modal feature vectors; and the plurality of target multi-modal feature vectors are obtained from the preset feature pool according to the multi-modal feature vector, including: The target sub-feature pool is determined from the plurality of sub-feature pools according to the multi-modal feature vector; The multi-modal feature vector is matched with each reference multi-modal feature vector in the target sub-feature pool for feature matching processing, and the plurality of reference multi-modal feature vectors with the highest adaptation degree are taken as the plurality of target multi-modal feature vectors.

8. A data retrieval apparatus, characterized by comprising: The device includes: The first obtaining module is configured to obtain example scene data of a target driving scene; the example scene data includes a plurality of different modal example data related to an autonomous vehicle; The first obtaining module is configured to obtain example scene data of a target driving scene; the example scene data includes a plurality of different modal example data related to an autonomous vehicle; The feature extraction module is configured to: determine a feature extraction algorithm corresponding to each example data according to a type corresponding to the example data of each modality in the example scene data; and perform feature extraction processing on each example data according to the corresponding feature extraction algorithm to obtain the multi-modal feature vector. The example scene data includes example data of a visual perception type and / or example data of a driving planning control type. The feature extraction module is specifically configured to: perform fusion processing on each example data of the visual perception type to obtain fused data, and perform feature extraction on the fused data according to a feature extraction algorithm corresponding to the example data of the visual perception type to obtain a first multi-modal feature vector; or perform feature extraction on each example data of the visual perception type according to a feature extraction algorithm corresponding to the example data of the visual perception type to obtain a plurality of first candidate features, and perform fusion processing on each first candidate feature to obtain a first multi-modal feature vector; perform feature extraction on each example data of the driving planning control type according to a feature extraction algorithm corresponding to the example data of the driving planning control type to obtain a plurality of second candidate features, and perform fusion processing on each second candidate feature to obtain a second multi-modal feature vector; and obtain the multi-modal feature vector according to the first multi-modal feature vector and / or the second multi-modal feature vector. The second acquisition module is configured to acquire a plurality of target multi-modal feature vectors from a preset feature pool according to the multi-modal feature vector. The third acquisition module is configured to acquire a plurality of target scene data corresponding to the target driving scene according to the plurality of target multi-modal feature vectors, and the plurality of target scene data are used for updating processing of an automatic driving algorithm corresponding to the target driving scene.

9. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the computer program to implement the steps of the method in any one of claims 1 to 7.

10. A storage medium having stored thereon a computer program, characterized in that The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

11. A computer program product comprising a computer program, characterized in that, The computer program is executed by the processor to implement the steps of the method in any one of claims 1 to 7.

Images