Driving data processing method and device, computer equipment, storage medium and program product

Abstract

The invention relates to a driving data processing method and device, computer equipment, a storage medium and a program product. The method comprises the following steps: acquiring total data of a vehicle in a driving process, aligning a timestamp of the total data with a standard timestamp by taking a timestamp of a front view image acquired by a front view camera of the vehicle as the standard timestamp to obtain a plurality of front view data groups, and aiming at each front view data group, determining a front view image of the vehicle; and establishing a target database according to the forward-looking image in the forward-looking data set and the corresponding total data, obtaining query information input by a user according to the target database, and obtaining target data corresponding to the query information from the target database. By adopting the method, data screening and mining can be performed in a targeted manner, so that effective management of various data in the vehicle driving process is realized, and the data mining accuracy and convenience are improved.

Description

Technical Field

[0001] The present application relates to the field of intelligent driving technology, and in particular to a method, device, computer equipment, storage medium and program product for processing driving data. Background Art

[0002] In the development of autonomous driving technology, the iterative updates of intelligence and algorithm technology have increasingly rich and diversified demands for data. In order to improve the performance and safety of intelligent driving systems, efficient and accurate data mining methods are particularly important.

[0003] In the traditional method, multiple frames of sample images are first obtained, and the image feature vectors and text feature vectors of these images are extracted. Then, these feature vectors are used as training samples to train a cross-modal model so that the model can learn the correlation between the image feature vectors and the text feature vectors. After the model training is completed, by inputting the image feature vector of the driving scene image, the cross-modal model can output the corresponding text feature vector, and then obtain the scene description information. Based on this scene description information, an intelligent driving scene knowledge base is constructed to achieve more efficient and accurate data mining.

[0004] However, for other raw data recorded by the vehicle, such as geographic location, speed, acceleration, etc., since these data are not taken into account in traditional methods, some effective information may be ignored, thus affecting the accuracy of data mining. Summary of the invention

[0005] Based on this, it is necessary to provide a driving data processing method, device, computer equipment, storage medium and program product that can improve the accuracy of data mining in response to the above technical problems.

[0006] In a first aspect, the present application provides a method for processing driving data, comprising:

[0007] Obtain the full amount of data of the vehicle during driving; the full amount of data includes vehicle information, sensor information, sensor data, scene information and algorithm output data;

[0008] Using the timestamp of the front-view image captured by the front-view camera of the vehicle as the standard timestamp, aligning the timestamp of the full data with the standard timestamp to obtain multiple front-view data groups; the front-view data group includes a front-view image and the full data corresponding to the timestamp of the front-view image;

[0009] For each forward-looking data group, a target database is established according to the forward-looking image in the forward-looking data group and the corresponding full-volume data; the target database includes at least one of a label database, a vector database and a map database;

[0010] The query information input by the user according to the target database is obtained, and the target data corresponding to the query information is obtained from the target database.

[0011] In one embodiment, the process of establishing a tag database includes:

[0012] Determine scene annotation information corresponding to the front-view image;

[0013] A scene classification model is obtained by training based on scene annotation information, and scene information of the front-view image is obtained by recognizing the scene classification model;

[0014] A label database is established based on the scene information and the full data corresponding to the front-view image.

[0015] In one embodiment, the process of establishing a vector database includes:

[0016] Determine text description information corresponding to the front-view image;

[0017] An image-text cross-modal model is trained based on text description information and text mixed data, and a feature vector of the front-view image is obtained through the image-text cross-modal model;

[0018] A vector database is established based on the complete data corresponding to the feature vector and the front view image.

[0019] In one embodiment, the process of establishing a map database includes:

[0020] Obtaining position information corresponding to the forward-looking data group;

[0021] According to the position information, the forward-looking data group is mapped to the target map, and the map information corresponding to the forward-looking data group in the target map is obtained;

[0022] A map database is established based on the map information and the full amount of data corresponding to the front-view image.

[0023] In one embodiment, the query information includes a filter tag, semantic image information, and region information; and the step of acquiring target data corresponding to the query information from a target database includes:

[0024] Obtain target data corresponding to the scene information matching the filter tag from the target database;

[0025] The semantic image information is converted into a query vector, and the target data corresponding to the feature vector matching the query vector is obtained from the target database;

[0026] Target data corresponding to the map information matching the area information is obtained from the target database.

[0027] In one embodiment, the step of acquiring target data corresponding to a feature vector matching a query vector from a target database includes:

[0028] For each feature vector in the target database, obtain a first similarity between the query vector and the feature vector, and determine positive samples and negative samples from the feature vector according to all the first similarities;

[0029] Taking the positive sample as the positive anchor point and the negative sample as the secondary anchor point, the second similarities between the feature vector in the target database and the positive and secondary anchor points are obtained, and the target data corresponding to the feature vector matching the query vector is obtained according to all the second similarities.

[0030] In a second aspect, the present application also provides a driving data processing device, comprising:

[0031] The data acquisition module is used to obtain the full amount of data of the vehicle during driving; the full amount of data includes vehicle information, sensor information, sensor data, scene information and algorithm output data;

[0032] A timestamp alignment module is used to use the timestamp of the front-view image captured by the front-view camera of the vehicle as the standard timestamp, align the timestamp of the full data with the standard timestamp, and obtain multiple front-view data groups; the front-view data group includes a front-view image and the full data corresponding to the timestamp of the front-view image;

[0033] A database establishment module is used to establish a target database for each forward-looking data group according to the forward-looking image in the forward-looking data group and the corresponding full-volume data; the target database includes at least one of a label database, a vector database and a map database;

[0034] The data query module is used to obtain the query information input by the user according to the target database, and obtain the target data corresponding to the query information from the target database.

[0035] In a third aspect, the present application further provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method steps of any one of the first aspects are implemented.

[0036] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which implements any one of the method steps in the first aspect when the computer program is executed by a processor.

[0037] In a fifth aspect, the present application further provides a computer program product, comprising a computer program, which implements any one of the method steps in the first aspect when executed by a processor.

[0038] The above-mentioned driving data processing method, device, computer equipment, storage medium and program product obtain the full data of the vehicle during the driving process, use the timestamp of the front-view image collected by the vehicle's front-view camera as the standard timestamp, align the timestamp of the full data with the standard timestamp, and obtain multiple front-view data groups. For each front-view data group, a target database is established according to the front-view image and the corresponding full data in the front-view data group, and query information input by the user according to the target database is obtained, and target data corresponding to the query information is obtained from the target database. Since different types of data describe the driving conditions and environment of the vehicle from multiple angles, the content of the target database can be enriched. At the same time, by establishing multiple types of databases, data screening and mining can be carried out in a targeted manner, thereby realizing effective management of various types of data during vehicle driving and improving data mining accuracy and convenience. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] In order to more clearly illustrate the technical solutions in the embodiments of the present application or related technologies, the drawings required for use in the embodiments of the present application or related technical descriptions will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without paying creative work.

[0040] FIG1 is a diagram showing an application environment of a method for processing driving data in one embodiment;

[0041] FIG2 is a schematic flow chart of a method for processing driving data in one embodiment;

[0042] FIG3 is a schematic flow chart of a method for processing driving data in another embodiment;

[0043] FIG4 is a block diagram of a driving data processing device according to an embodiment;

[0044] FIG. 5 is a diagram showing the internal structure of a computer device in one embodiment. DETAILED DESCRIPTION

[0045] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0046] The method for processing driving data provided in the embodiment of the present application can be applied to the application environment shown in Figure 1. Among them, the terminal 102 communicates with the server 104 through the network. The data storage system can store the data that the server 104 needs to process. The data storage system can be integrated on the server 104, or it can be placed on the cloud or other network servers. Among them, the terminal 102 is used to obtain the full amount of data of the vehicle during driving through the server 104, and the timestamp of the front-view image collected by the front-view camera of the vehicle is used as the standard timestamp, and the timestamp of the full amount of data is aligned with the standard timestamp to obtain multiple front-view data groups. For each front-view data group, according to the front-view image in the front-view data group and the corresponding full amount of data, a target database is established, and the query information input by the user according to the target database is obtained, and the target data corresponding to the query information is obtained from the target database. Among them, the terminal 102 can be, but is not limited to, various personal computers, laptops, smart phones, tablet computers, Internet of Things devices and portable wearable devices, and the Internet of Things devices can be smart speakers, smart TVs, smart air conditioners, smart car-mounted devices, projection devices, etc. Portable wearable devices can be smart watches, smart bracelets, head-mounted devices, etc. The head mounted device may be a virtual reality (VR) device, an augmented reality (AR) device, smart glasses, etc. The server 104 may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0047] In an exemplary embodiment, as shown in FIG2 , a method for processing driving data is provided, which is described by taking the method applied to the terminal 102 in FIG1 as an example, and includes the following steps 202 to 208 . Among them:

[0048] S202: Obtaining full data of the vehicle during driving; the full data includes vehicle information, sensor information, sensor data, scene information and algorithm output data.

[0049] Optionally, during the driving process of the vehicle, the full amount of vehicle data is collected through various sensors and cameras installed on the vehicle, including vehicle information, sensor information, sensor data, scene information and algorithm output data, etc. Among them, the vehicle information covers the attributes of the vehicle itself, such as the vehicle number, model, vehicle version, speed, recording time, etc., which can help determine the basic characteristics of the vehicle. Sensor information includes relevant information of various sensors installed on the vehicle, such as the type and parameters of the sensor. Sensor data is the actual value collected by these sensors, such as RTK data, laser data, millimeter wave data and image data. Scene information includes the captured images captured by the camera, such as the front view image captured by the front view camera. Algorithm output data includes the results obtained after the vehicle intelligent driving assistance algorithm is calculated, such as the algorithm results of regulation, perception, and positioning. The comprehensive collection of these different aspects of data constitutes the full amount of data during the driving process.

[0050] S204: Using the timestamp of the front view image captured by the front view camera of the vehicle as the standard timestamp, aligning the timestamp of the full data with the standard timestamp to obtain multiple front view data groups; the front view data group includes a front view image and the full data corresponding to the timestamp of the front view image.

[0051] Optionally, the vehicle's forward-looking camera will continuously capture images in front of the vehicle. Each image has a corresponding timestamp, marking the specific time when the image was captured. The timestamp of the forward-looking image is selected as the standard timestamp, and the timestamps of each data in the previously collected full data are aligned according to this standard to ensure that all data corresponding to each moment (including the forward-looking image and other full data at the same moment) can be accurately matched together to form a complete information set to obtain a forward-looking data group. Each forward-looking data group contains a forward-looking image at a specific moment and full data with the same timestamp as the forward-looking image, so that subsequent data analysis can be based on complete information at the same moment.

[0052] S206: For each forward-looking data group, a target database is established according to the forward-looking image in the forward-looking data group and the corresponding full-volume data; the target database includes at least one of a label database, a vector database, and a map database.

[0053] Optionally, for each organized forward-looking data group, the forward-looking image and the corresponding full data are used to construct a target database, including at least one of a label database, a vector database, and a map database. The label database can be used to annotate the forward-looking image and the corresponding full data in detail. For example, for various objects (pedestrians, vehicles, traffic signs, etc.) in the forward-looking image, their specific category, location, status and other information can be annotated. For vehicle information, sensor data, etc. in the full data, their attributes related to specific scenes or events can be annotated. In this way, when a specific type of data needs to be found, the corresponding classification can be quickly located, thereby improving data retrieval efficiency.

[0054] Furthermore, the vector database will convert the front-view image and the corresponding full data into vector form. For the front-view image, it can be converted into a vector through a feature extraction algorithm. The vector can represent the key features of the image to a certain extent. For the sensor data, vehicle information, etc. in the full data, it can also be converted into a vector using a suitable mapping method. When it is necessary to find other data similar to a certain data, by calculating the similarity between vectors, the original data corresponding to other vectors with a high degree of similarity can be quickly located in the vector database. The map database can integrate the geographic information of the vehicle during driving. When it is necessary to find data in a specified area, the corresponding data can be quickly found by filtering through the geographic information.

[0055] S208: Acquire query information input by the user according to the target database, and acquire target data corresponding to the query information from the target database.

[0056] Optionally, when the user needs to query specified data, such as the user needs to query the front view image of the vehicle under certain weather conditions, or query the driving data of the vehicle on a certain road section, etc., by searching in the established target database, the target data with the highest matching degree with the query information can be quickly found, wherein the query information is information that matches the target database, and the user can quickly obtain data in the corresponding database by entering different filtering conditions when querying.

[0057] In the above-mentioned driving data processing method, by acquiring the full data of the vehicle during the driving process, the timestamp of the front-view image collected by the vehicle's front-view camera is used as the standard timestamp, and the timestamp of the full data is aligned with the standard timestamp to obtain multiple front-view data groups. For each front-view data group, a target database is established according to the front-view image and the corresponding full data in the front-view data group, and the query information input by the user according to the target database is obtained, and the target data corresponding to the query information is obtained from the target database. Since different types of data describe the driving conditions and environment of the vehicle from multiple angles, the content of the target database can be enriched. At the same time, by establishing multiple types of databases, data screening and mining can be carried out in a targeted manner, thereby realizing effective management of various types of data during vehicle driving and improving the accuracy and convenience of data mining.

[0058] In an exemplary embodiment, the process of establishing a label database includes: determining scene annotation information corresponding to a forward-view image; training a scene classification model based on the scene annotation information, and identifying scene information of the forward-view image through the scene classification model; and establishing a label database based on the scene information and the full amount of data corresponding to the forward-view image.

[0059] Optionally, first, the front-view image data is annotated for scene classification. In practical applications, the scene annotation information corresponding to the front-view image can be determined by manual annotation. For example, if the front-view image shows a rainy scene, it is annotated as "rainy day"; if the front-view image shows a city street with obvious urban elements such as high-rise buildings, it is annotated as "city"; if it is a scene on a highway, it is annotated as "highway", and so on. Through annotation, each front-view image is given preliminary scene annotation information, which can intuitively reflect the actual scene corresponding to the image. At the same time, open source visual scene mixed data is introduced. This is publicly available, labeled image data on the Internet. It comes from a wide range of sources and covers a variety of scene types. These two parts of data (own annotation data and open source data) are integrated together as a data set for training scene classification models. Afterwards, the model is trained using the integrated data set as input through a machine learning model or a deep learning algorithm (such as a convolutional neural network). During the training process, the model will learn the relationship between the various visual features of the image (such as the color, texture, shape, etc. of the image) and the corresponding scene annotation information (such as labels such as "rainy day", "city", "highway", etc.). By continuously adjusting the model's own parameters, it can accurately predict the corresponding scene labels based on the input image features, thereby training a vision-based scene classification model.

[0060] Furthermore, after the scene classification model is trained, the forward-looking image in the data group is input into the trained scene classification model. The model will analyze the input forward-looking image based on the knowledge learned during the training process, and then accurately determine the scene information corresponding to the forward-looking image. After obtaining the scene information of the forward-looking image, it is integrated with the corresponding forward-looking image and the full amount of data, and the integrated data is written into the database according to certain rules and formats to obtain a label database. In the label database, each record corresponds to a data group at a specific moment or specific situation, and each record carries rich scene information and label information of other related data. These label information can facilitate subsequent query, screening and analysis operations. For example, the vehicle driving data in a specific scene can be quickly located according to the scene label, or more detailed data analysis can be performed according to other data labels (such as vehicle model, sensor type, etc.).

[0061] In this embodiment, the full amount of basic data is integrated with the front-view image scene information obtained by identifying the scene classification model and written into the database to form a label database, so that different types of data are interrelated and complementary to each other. Moreover, when training the scene classification model, more accurate and diverse labels can be generated by combining the labeled data and open source visual scene mixed data.

[0062] In an exemplary embodiment, the process of establishing a vector database includes: determining the text description information corresponding to the front view image; training an image-text cross-modal model based on the text description information and text mixed data, and obtaining a feature vector of the front view image through the image-text cross-modal model; and establishing a vector database based on the feature vector and the full data corresponding to the front view image.

[0063] Optionally, during the driving process of the vehicle, the front-view image records the scene in front of the vehicle. For these front-view images, the corresponding text description information needs to be determined. These text description information can cover multiple aspects, such as the scene in the image (urban roads, highways, intersections, etc.), weather conditions (sunny, rainy, snowy, etc.), main objects in the picture (pedestrians, vehicles, traffic signs, etc.) and their relative positional relationships, etc., and through manual annotation and other methods, text description information that can accurately describe its content is generated for each front-view image. Afterwards, the front-view image and the corresponding text description information are integrated with the open source image-text mixed data as a data set for training the image-text cross-modal model, where the open source data is publicly available on the Internet and contains a large number of images and corresponding semantic description information.

[0064] Optionally, the image-text cross-modal model aims to learn the intrinsic relationship between images and text, that is, how to extract features corresponding to text descriptions from images, and understand semantic information related to image content from text. During the training process, the model continuously adjusts its own parameters to minimize the error between the predicted results and the true annotations. For example, when a front-view image and its corresponding text description are input, the model attempts to learn the correspondence between the visual features in the image (such as the shape, color, texture, etc. of the object) and the semantic features in the text description (such as scene category, object name, etc.), so that for a given image, a text description that matches its content can be accurately output, and vice versa.

[0065] Furthermore, after the image-text cross-modal model training is completed, all the front-view images in the label database are input into the model. The model will analyze and process each front-view image based on the knowledge learned during the training process, and extract the feature vector that can represent the key features of the image. This feature vector is a mathematical representation that condenses the important visual information of the image, such as the distribution pattern of objects in the image, the overall structure of the scene, etc. At the same time, the semantics corresponding to the front-view image and the open source image are input into the model, and the feature vector of the semantic text can also be obtained. It is a mathematical representation of the text semantic information, which can reflect the key semantic content in the text description, such as the type of scene, the category of the object, etc. After that, all the feature vectors (including image feature vectors and semantic text feature vectors) and the corresponding image information of the front-view data group and the open source data are stored in the vector database together. In the vector database, each record corresponds to a set of related data, including feature vectors, front-view images, and the full amount of data related to them.

[0066] In this embodiment, by training the image-text cross-modal model, a semantic association between the image and the text is established. The model learns the correspondence between the visual features of the image and the semantic features of the text, so that what is stored in the vector database is not just isolated image and text data, but a data set with inherent semantic connections. When the user wants to mine data for a specific scene, he only needs to enter a simple description, which can quickly obtain the required data, greatly improving the convenience of data mining.

[0067] In an exemplary embodiment, the process of establishing a map database includes: obtaining position information corresponding to a forward-looking data group; mapping the forward-looking data group to a target map based on the position information, and obtaining map information corresponding to the forward-looking data group in the target map; and establishing a map database based on the map information and the full amount of data corresponding to the forward-looking image.

[0068] Optionally, during the driving process of the vehicle, the location information corresponding to each forward-looking data group can be obtained through the positioning system and other related sensors equipped on the vehicle. These location information are usually presented in the form of coordinates such as longitude and latitude, accurately identifying the geographical location of the vehicle when collecting the forward-looking image and generating the corresponding full amount of data at a specific time. Afterwards, based on the location information, the corresponding forward-looking data group is mapped to the target map, wherein the target map can be a pre-selected electronic map suitable for vehicle driving scene analysis, such as a common city map, regional map, etc. Through this mapping operation, the data in the forward-looking data group is associated with the specific location on the map, so as to obtain the map information corresponding to the forward-looking data group in the target map. Finally, the obtained map information is integrated with the full amount of data corresponding to the forward-looking image to obtain a map database. In this database, each record corresponds to a forward-looking data group at a specific time and its related information on the map, forming a rich data set with geographic spatial attributes, which can be used for subsequent query, analysis and mining operations.

[0069] In this embodiment, by projecting the position information of the forward-looking data group onto a map and establishing a map database, vehicle driving data within a specific geographical area can be accurately located during data mining, thereby improving the accuracy and convenience of data mining.

[0070] In an exemplary embodiment, the query information includes a filter tag, semantic image information, and region information; target data corresponding to the query information is obtained from a target database, including: obtaining target data corresponding to scene information matching the filter tag from the target database; converting the semantic image information into a query vector, and obtaining target data corresponding to a feature vector matching the query vector from the target database; and obtaining target data corresponding to map information matching the region information from the target database.

[0071] Optionally, the query information includes filtering tags, semantic image information, and region information, wherein the filtering tags are conditions for filtering scene information in the tag database, for example, they may be specific tags such as "night", "city road", "large vehicle", etc. When querying from the tag database, the scene information matching these filtering tags will be found first, and then the corresponding target data will be obtained. These target data may cover vehicle information, sensor information, sensor data, and perception results, etc., such as the vehicle speed in the "city road at night" scene, the surrounding environment data detected by the sensor, etc.

[0072] For semantic image information, it is first converted into a query vector through a trained image-text cross-modal model. The model can learn the semantic relationship between image and text, thereby converting the semantic content expressed by the semantic image information into a vector form suitable for querying in the vector database. After that, the feature vectors matching the query vector are searched in the vector database, and these feature vectors correspond to the target data. For example, if an image of "a vehicle driving in the rain" is provided as semantic image information, after being converted into a query vector by the model, the target data corresponding to the feature vectors of similar scenes (such as other vehicles driving in similar rainy environments) can be found in the vector database, which may include the specific driving data of the vehicle in these scenes, the data collected by the sensor, etc.

[0073] Regional information is mainly used for data mining in geographic databases. By setting a specific geographic area, such as "ring highway", and a possible time range, such as "between x month and x month", the geographic database will filter out the map information in the corresponding area and time according to these conditions, and then obtain the corresponding target data. These target data can be related information such as the driving trajectory of vehicles in the area and time period, traffic flow, surrounding geographical environment, etc., which can be used to obtain the traffic conditions and vehicle driving conditions in a specific area within a specific time period.

[0074] It should be noted that, according to actual application requirements, users can choose to input at least one of the filter tags, semantic image information and area information for query. When the query information input by the user is received, the target data matching the query information can be quickly obtained through the target database.

[0075] Exemplarily, based on the constructed label database, data that meets specific scenarios can be mined by entering screening conditions. For example, to mine the front view image and lidar data in a complex scenario such as "Car No. 4 is on a city road at night, with at least 7 camera data, a large vehicle within 10m in front, and more than 6 bicycles within 50m around". First, enter these screening conditions into the label database, and the database will match and search for various labeled data based on these conditions to find those records that meet all the conditions, thereby obtaining the corresponding front view image and lidar data and other target data. This mining method mainly relies on the precise labeling and classification management of data in the label database, and can quickly locate data that meets the conditions of a specific scene combination.

[0076] Exemplarily, in a vector database, scene data that meets the requirements is mined by inputting semantic descriptions or image information. Specifically, the text describing the scene to be mined or the image corresponding to the scene is first input into the trained image-text cross-modal model to obtain the feature vector to be queried. Then, the feature vector to be queried is used to query the vector database, and the data with high cosine similarity to the feature vector to be queried is found by calculating the cosine similarity with each data in the database, and these data are the target data. For example, the text description of "the vehicle is driving slowly in the snow, and there are pedestrians in front" is input. After the feature vector to be queried is obtained through model processing, the data with high similarity is found in the vector database, and then the target data such as vehicle driving data and sensor data in the corresponding scene are obtained. This mining method utilizes the vector representation of data in the vector database and the semantic relationship established by the image-text cross-modal model, and can realize efficient data mining based on semantics.

[0077] In this embodiment, by providing query methods based on multiple query dimensions such as tags, semantic image information and area information, users can mine data from different angles, thereby improving the efficiency of data mining. Users can choose the most appropriate mining method based on the existing information to quickly obtain the required data.

[0078] In an exemplary embodiment, target data corresponding to a feature vector matching a query vector is obtained from a target database, including: for each feature vector in the target database, obtaining a first similarity between the query vector and the feature vector, and determining positive samples and negative samples from the feature vector according to all the first similarities; taking the positive sample as a positive anchor point and the negative sample as a secondary anchor point, obtaining a second similarity between the feature vector in the target database and the positive and secondary anchor points, and obtaining the target data corresponding to the feature vector matching the query vector according to all the second similarities.

[0079] Optionally, when obtaining target data corresponding to a feature vector matching the query vector from the target database, firstly, for each feature vector in the target database, a first similarity between the query vector and the feature vector is calculated. By calculating the first similarity, the similarity between the query vector and each feature vector in the target database can be measured. According to the values ​​of all the first similarities calculated, positive samples and negative samples are determined from the feature vectors in the target database. Generally speaking, positive samples are target data corresponding to feature vectors that have a high similarity to the query vector and meet the expected mining conditions, while negative samples are target data corresponding to feature vectors that have a low similarity to the query vector and do not meet the expected mining conditions. For example, if the query vector is generated based on a text such as "someone is crossing the road ahead on a rainy day", then the target data corresponding to feature vectors corresponding to similar scenes (rainy days and someone is crossing the road ahead) may be classified as positive samples, while the target data corresponding to feature vectors corresponding to other scenes (such as sunny days and no pedestrians on the road) may be classified as negative samples.

[0080] Furthermore, after determining the positive sample and the negative sample, the positive sample is used as the positive anchor point and the negative sample is used as the negative anchor point. Then, the second similarity between other feature vectors in the target database and these positive and negative anchor points is obtained. According to the values ​​of all the second similarities calculated, the target data corresponding to the feature vector matching the query vector is obtained. By comprehensively considering the direct similarity between the query vector and the feature vector (the first similarity) and the similarity between the feature vector and the positive and negative anchor points (the second similarity), the target data corresponding to the feature vectors that are similar to the query vector itself and meet the expected mining conditions in terms of overall characteristics are more accurately screened. By repeatedly setting anchor points, calculating similarities and screening data, each iteration will further optimize the screening conditions according to the new positive and negative anchor points, so that the screened data is more in line with the expected mining conditions, thereby obtaining high-quality mining results. Among them, the first similarity is the pre-similarity between the query vector and the feature vector, and the calculation formula of the second similarity is expressed as: similarity = (the sum of the similarities between the positive anchor vector and the searched vector - the sum of the similarities between the negative anchor vector and the searched vector) / (the total number of positive and negative anchor points).

[0081] In this embodiment, by successively calculating the first similarity between the query vector and the feature vector and the second similarity between the feature vector and the primary and secondary anchor points, and performing multiple iterations, data is screened from multiple dimensions, thereby improving the accuracy of data mining.

[0082] In an exemplary embodiment, as shown in FIG3 , a method for processing driving data is provided, the method comprising the following steps:

[0083] Obtain full data of the vehicle during driving; full data includes vehicle information, sensor information, sensor data, scene information and algorithm output data.

[0084] The timestamp of the front-view image captured by the front-view camera of the vehicle is used as the standard timestamp, and the timestamp of the full data is aligned with the standard timestamp to obtain multiple front-view data groups; the front-view data group includes a front-view image and the full data corresponding to the timestamp of the front-view image.

[0085] For each forward-looking data group, a target database is established according to the forward-looking images in the forward-looking data group and the corresponding full-volume data; the target database includes at least one of a label database, a vector database and a map database.

[0086] The process of establishing the label database includes: determining the scene annotation information corresponding to the front view image; training a scene classification model based on the scene annotation information, and identifying the scene information of the front view image through the scene classification model; and establishing the label database based on the scene information and the full amount of data corresponding to the front view image.

[0087] The process of establishing a vector database includes: determining the text description information corresponding to the front view image; training an image-text cross-modal model based on the text description information and text mixed data, and obtaining the feature vector of the front view image through the image-text cross-modal model; and establishing a vector database based on the feature vector and the full amount of data corresponding to the front view image.

[0088] The process of establishing a map database includes: obtaining position information corresponding to the forward-looking data group; mapping the forward-looking data group to a target map according to the position information, and obtaining map information corresponding to the forward-looking data group in the target map; and establishing a map database according to the map information and the full amount of data corresponding to the forward-looking image.

[0089] The query information input by the user according to the target database is obtained, wherein the query information includes filter tags, semantic image information and region information.

[0090] The target data corresponding to the scene information matching the filter tag is obtained from the target database.

[0091] For each feature vector in the target database, the first similarity between the query vector and the feature vector is obtained, and the positive sample and the negative sample are determined from the feature vector according to all the first similarities; with the positive sample as the positive anchor point and the negative sample as the secondary anchor point, the second similarity between the feature vector in the target database and the positive and secondary anchor points is obtained, and the target data corresponding to the feature vector matching the query vector is obtained according to all the second similarities.

[0092] Target data corresponding to the map information matching the area information is obtained from the target database.

[0093] In this embodiment, by acquiring the full data of the vehicle during the driving process, taking the timestamp of the front-view image captured by the vehicle's front-view camera as the standard timestamp, the timestamp of the full data is aligned with the standard timestamp to obtain multiple front-view data groups. For each front-view data group, a target database is established based on the front-view image and the corresponding full data in the front-view data group, the query information input by the user according to the target database is obtained, and the target data corresponding to the query information is obtained from the target database. Since different types of data describe the driving conditions and environment of the vehicle from multiple angles, the content of the target database can be enriched. At the same time, by establishing multiple types of databases, data screening and mining can be carried out in a targeted manner, thereby realizing effective management of various types of data during vehicle driving and improving the accuracy and convenience of data mining.

[0094] It should be understood that, although the steps in the flowcharts involved in the above embodiments are displayed in sequence according to the indication of the arrows, these steps are not necessarily executed in sequence according to the order indicated by the arrows. Unless there is a clear explanation in this article, the execution of these steps is not strictly limited in order, and these steps can be executed in other orders. Moreover, at least a part of the steps in the flowcharts involved in the above embodiments may include multiple steps or multiple stages, and these steps or stages are not necessarily executed at the same time, but can be executed at different times, and the execution order of these steps or stages is not necessarily carried out in sequence, but can be executed in turn or alternately with other steps or at least a part of the steps or stages in other steps.

[0095] Based on the same inventive concept, the embodiment of the present application also provides a driving data processing device for implementing the driving data processing method involved above. The implementation solution provided by the device to solve the problem is similar to the implementation solution recorded in the above method, so the specific limitations in the one or more driving data processing device embodiments provided below can refer to the limitations of the driving data processing method above, and will not be repeated here.

[0096] In an exemplary embodiment, as shown in FIG4 , a driving data processing device is provided, comprising: a data acquisition module 10, a timestamp alignment module 20, a database establishment module 30 and a data query module 40, wherein:

[0097] The data acquisition module 10 is used to acquire the full amount of data of the vehicle during driving; the full amount of data includes vehicle information, sensor information, sensor data, scene information and algorithm output data.

[0098] The timestamp alignment module 20 is used to align the timestamp of the full data with the standard timestamp using the timestamp of the front view image captured by the front view camera of the vehicle as the standard timestamp to obtain multiple front view data groups; the front view data group includes a front view image and the full data corresponding to the timestamp of the front view image.

[0099] The database establishment module 30 is used to establish a target database for each forward-looking data group according to the forward-looking image in the forward-looking data group and the corresponding full-volume data; the target database includes at least one of a label database, a vector database and a map database.

[0100] The data query module 40 is used to obtain query information input by the user according to the target database, and obtain target data corresponding to the query information from the target database.

[0101] In an exemplary embodiment, the database establishment module 30 is also used to determine the scene annotation information corresponding to the forward-view image; train a scene classification model based on the scene annotation information, and identify the scene information of the forward-view image through the scene classification model; and establish a label database based on the scene information and the full amount of data corresponding to the forward-view image.

[0102] In an exemplary embodiment, the database establishment module 30 is also used to determine the text description information corresponding to the foreground image; train an image-text cross-modal model based on the text description information and text mixed data, and obtain the feature vector of the foreground image through the image-text cross-modal model; and establish a vector database based on the feature vector and the full data corresponding to the foreground image.

[0103] In an exemplary embodiment, the database establishment module 30 is also used to obtain the position information corresponding to the forward-looking data group; based on the position information, the forward-looking data group is mapped to the target map to obtain the map information corresponding to the forward-looking data group in the target map; and a map database is established based on the map information and the full amount of data corresponding to the forward-looking image.

[0104] In an exemplary embodiment, the query information includes a filter tag, semantic image information, and area information; the data query module 40 is also used to obtain target data corresponding to scene information matching the filter tag from a target database; convert the semantic image information into a query vector, and obtain target data corresponding to a feature vector matching the query vector from the target database; and obtain target data corresponding to map information matching the area information from the target database.

[0105] In an exemplary embodiment, the data query module 40 is also used to obtain, for each feature vector in the target database, a first similarity between the query vector and the feature vector, and determine positive samples and negative samples from the feature vector based on all the first similarities; using the positive sample as the positive anchor point and the negative sample as the secondary anchor point, obtain the second similarity between the feature vector in the target database and the positive and secondary anchor points, and obtain the target data corresponding to the feature vector that matches the query vector based on all the second similarities.

[0106] Each module in the above-mentioned driving data processing device can be implemented in whole or in part by software, hardware or a combination thereof. Each of the above-mentioned modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute operations corresponding to each of the above modules.

[0107] In an exemplary embodiment, a computer device is provided, which may be a terminal, and its internal structure diagram may be shown in FIG5. The computer device includes a processor, a memory, an input / output interface, a communication interface, a display unit, and an input device. The processor, the memory, and the input / output interface are connected via a system bus, and the communication interface, the display unit, and the input device are connected to the system bus via the input / output interface. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The input / output interface of the computer device is used to exchange information between the processor and an external device. The communication interface of the computer device is used to communicate with an external terminal in a wired or wireless manner, and the wireless manner may be implemented through WIFI, a mobile cellular network, near field communication (NFC) or other technologies. When the computer program is executed by the processor, a method for processing driving data is implemented. The display unit of the computer device is used to form a visually visible picture, which may be a display screen, a projection device, or a virtual reality imaging device. The display screen can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer covering the display screen, or a button, trackball or touchpad set on the computer device shell, or an external keyboard, touchpad or mouse.

[0108] Those skilled in the art will understand that the structure shown in FIG. 5 is merely a block diagram of a partial structure related to the solution of the present application, and does not constitute a limitation on the computer device to which the solution of the present application is applied. The specific computer device may include more or fewer components than shown in the figure, or combine certain components, or have a different arrangement of components.

[0109] In an exemplary embodiment, a computer device is provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor implements the following steps when executing the computer program: obtaining full data of a vehicle during driving; the full data includes vehicle information, sensor information, sensor data, scene information and algorithm output data; taking the timestamp of a front view image captured by a front view camera of the vehicle as a standard timestamp, aligning the timestamp of the full data with the standard timestamp to obtain multiple front view data groups; the front view data group includes a front view image and full data corresponding to the timestamp of the front view image; for each front view data group, establishing a target database according to the front view image in the front view data group and the corresponding full data; the target database includes at least one of a label database, a vector database and a map database; obtaining query information input by a user according to the target database, and obtaining target data corresponding to the query information from the target database.

[0110] In one embodiment, the process of establishing a label database involved in executing a computer program by a processor includes: determining scene annotation information corresponding to a forward-view image; training a scene classification model based on the scene annotation information, and identifying scene information of the forward-view image through the scene classification model; and establishing a label database based on the scene information and the full amount of data corresponding to the forward-view image.

[0111] In one embodiment, the process of establishing a vector database involved in executing a computer program by a processor includes: determining text description information corresponding to a front-view image; training an image-text cross-modal model based on the text description information and text mixed data, and obtaining a feature vector of the front-view image through the image-text cross-modal model; and establishing a vector database based on the feature vector and the full amount of data corresponding to the front-view image.

[0112] In one embodiment, the process of establishing a map database involved in the execution of a computer program by a processor includes: obtaining position information corresponding to a forward-looking data group; mapping the forward-looking data group to a target map based on the position information, and obtaining map information corresponding to the forward-looking data group in the target map; and establishing a map database based on the map information and the full amount of data corresponding to the forward-looking image.

[0113] In one embodiment, the query information includes a filter tag, semantic image information, and region information; the acquisition of target data corresponding to the query information from a target database involved when the processor executes a computer program includes: acquiring target data corresponding to scene information matching the filter tag from the target database; converting the semantic image information into a query vector, and acquiring target data corresponding to a feature vector matching the query vector from the target database; and acquiring target data corresponding to map information matching the region information from the target database.

[0114] In one embodiment, the processor executes a computer program involving obtaining target data corresponding to a feature vector matching a query vector from a target database, including: for each feature vector in the target database, obtaining a first similarity between the query vector and the feature vector, and determining positive samples and negative samples from the feature vector according to all first similarities; taking the positive sample as a positive anchor point and the negative sample as a secondary anchor point, obtaining a second similarity between the feature vector in the target database and the positive and secondary anchor points, and obtaining the target data corresponding to the feature vector matching the query vector according to all second similarities.

[0115] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, and when the computer program is executed by a processor, the following steps are implemented: obtaining full data of a vehicle during driving; the full data includes vehicle information, sensor information, sensor data, scene information and algorithm output data; taking the timestamp of a front view image captured by a front view camera of the vehicle as a standard timestamp, aligning the timestamp of the full data with the standard timestamp to obtain multiple front view data groups; the front view data group includes a front view image and full data corresponding to the timestamp of the front view image; for each front view data group, establishing a target database according to the front view image in the front view data group and the corresponding full data; the target database includes at least one of a label database, a vector database and a map database; obtaining query information input by a user according to the target database, and obtaining target data corresponding to the query information from the target database.

[0116] In one embodiment, the process of establishing a label database involved when a computer program is executed by a processor includes: determining scene annotation information corresponding to a forward-view image; training a scene classification model based on the scene annotation information, and identifying scene information of the forward-view image through the scene classification model; and establishing a label database based on the scene information and the full amount of data corresponding to the forward-view image.

[0117] In one embodiment, the process of establishing a vector database involved when a computer program is executed by a processor includes: determining text description information corresponding to a front-view image; training an image-text cross-modal model based on the text description information and text mixed data, and obtaining a feature vector of the front-view image through the image-text cross-modal model; and establishing a vector database based on the feature vector and the full amount of data corresponding to the front-view image.

[0118] In one embodiment, the process of establishing a map database involved when a computer program is executed by a processor includes: obtaining position information corresponding to a forward-looking data group; mapping the forward-looking data group to a target map based on the position information, and obtaining map information corresponding to the forward-looking data group in the target map; and establishing a map database based on the map information and the full amount of data corresponding to the forward-looking image.

[0119] In one embodiment, the query information includes a filter tag, semantic image information, and region information; the computer program executed by the processor involves obtaining target data corresponding to the query information from a target database, including: obtaining target data corresponding to scene information matching the filter tag from the target database; converting the semantic image information into a query vector, and obtaining target data corresponding to a feature vector matching the query vector from the target database; and obtaining target data corresponding to map information matching the region information from the target database.

[0120] In one embodiment, the computer program executed by a processor involves obtaining target data corresponding to a feature vector matching a query vector from a target database, including: for each feature vector in the target database, obtaining a first similarity between the query vector and the feature vector, and determining positive samples and negative samples from the feature vector based on all first similarities; using the positive sample as a positive anchor point and the negative sample as a secondary anchor point, obtaining a second similarity between the feature vector in the target database and the positive and secondary anchor points, and obtaining the target data corresponding to the feature vector matching the query vector based on all second similarities.

[0121] In one embodiment, a computer program product is provided, including a computer program, which implements the following steps when executed by a processor: obtaining full data of a vehicle during driving; the full data includes vehicle information, sensor information, sensor data, scene information and algorithm output data; taking the timestamp of a front view image captured by a front view camera of the vehicle as a standard timestamp, aligning the timestamp of the full data with the standard timestamp to obtain multiple front view data groups; the front view data group includes a front view image and full data corresponding to the timestamp of the front view image; for each front view data group, establishing a target database according to the front view image in the front view data group and the corresponding full data; the target database includes at least one of a label database, a vector database and a map database; obtaining query information input by a user according to the target database, and obtaining target data corresponding to the query information from the target database.

[0122] In one embodiment, the process of establishing a label database involved when a computer program is executed by a processor includes: determining scene annotation information corresponding to a forward-view image; training a scene classification model based on the scene annotation information, and identifying scene information of the forward-view image through the scene classification model; and establishing a label database based on the scene information and the full amount of data corresponding to the forward-view image.

[0123] In one embodiment, the process of establishing a vector database involved when a computer program is executed by a processor includes: determining text description information corresponding to a front-view image; training an image-text cross-modal model based on the text description information and text mixed data, and obtaining a feature vector of the front-view image through the image-text cross-modal model; and establishing a vector database based on the feature vector and the full amount of data corresponding to the front-view image.

[0124] In one embodiment, the process of establishing a map database involved when a computer program is executed by a processor includes: obtaining position information corresponding to a forward-looking data group; mapping the forward-looking data group to a target map based on the position information, and obtaining map information corresponding to the forward-looking data group in the target map; and establishing a map database based on the map information and the full amount of data corresponding to the forward-looking image.

[0125] In one embodiment, the query information includes a filter tag, semantic image information, and region information; the computer program executed by the processor involves obtaining target data corresponding to the query information from a target database, including: obtaining target data corresponding to scene information matching the filter tag from the target database; converting the semantic image information into a query vector, and obtaining target data corresponding to a feature vector matching the query vector from the target database; and obtaining target data corresponding to map information matching the region information from the target database.

[0126] In one embodiment, the computer program executed by a processor involves obtaining target data corresponding to a feature vector matching a query vector from a target database, including: for each feature vector in the target database, obtaining a first similarity between the query vector and the feature vector, and determining positive samples and negative samples from the feature vector based on all first similarities; using the positive sample as a positive anchor point and the negative sample as a secondary anchor point, obtaining a second similarity between the feature vector in the target database and the positive and secondary anchor points, and obtaining the target data corresponding to the feature vector matching the query vector based on all second similarities.

[0127] A person of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment method can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to the memory, database or other medium used in the embodiments provided in the present application can include at least one of non-volatile memory and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, optical memory, high-density embedded non-volatile memory, resistive random access memory (ReRAM), magnetic random access memory (MRAM), ferroelectric random access memory (FRAM), phase change memory (PCM), graphene memory, etc. Volatile memory can include random access memory (RAM) or external cache memory, etc. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM). The database involved in each embodiment provided in this application may include at least one of a relational database and a non-relational database. Non-relational databases may include distributed databases based on blockchains, etc., but are not limited to this. The processor involved in each embodiment provided in this application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic device, a data processing logic device based on quantum computing, an artificial intelligence (AI) processor, etc., but are not limited to this.

[0128] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this application.

[0129] The above-described embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the attached claims.

Claims

1. A method for processing driving data, characterized in that: The method comprises: Acquire the full amount of data of the vehicle during driving; the full amount of data includes vehicle information, sensor information, sensor data, scene information and algorithm output data; Using the timestamp of the front-view image captured by the front-view camera of the vehicle as the standard timestamp, aligning the timestamp of the full data with the standard timestamp to obtain a plurality of front-view data groups; the front-view data groups include a front-view image and the full data corresponding to the timestamp of the front-view image; For each forward-looking data group, a target database is established according to the forward-looking images in the forward-looking data group and the corresponding full-volume data; the target database includes at least one of a label database, a vector database and a map database; Acquire query information input by a user according to the target database, and acquire target data corresponding to the query information from the target database.

2. The method according to claim 1, characterized in that: The process of establishing the label database includes: Determining scene annotation information corresponding to the front-view image; A scene classification model is obtained by training based on the scene annotation information, and scene information of the front-view image is obtained by identifying the scene classification model; A label database is established based on the scene information and the full amount of data corresponding to the front-view image.

3. The method according to claim 1, characterized in that The process of establishing the vector database includes: Determining text description information corresponding to the front-view image; An image-text cross-modal model is obtained by training based on the text description information and the text mixed data, and a feature vector of the front-view image is obtained through the image-text cross-modal model; A vector database is established based on the feature vector and the full amount of data corresponding to the front-view image.

4. The method according to claim 1, characterized in that: The process of establishing the map database includes: Acquire position information corresponding to the forward-looking data group; According to the position information, mapping the forward-looking data group to a target map, and obtaining map information corresponding to the forward-looking data group in the target map; A map database is established based on the map information and the full amount of data corresponding to the front-view image.

5. The method according to claims 2 to 4, characterized in that The query information includes a filter tag, semantic image information, and region information; and obtaining target data corresponding to the query information from the target database includes: Acquire target data corresponding to the scene information matching the screening tag from the target database; Converting the semantic image information into a query vector, and acquiring target data corresponding to a feature vector matching the query vector from the target database; Target data corresponding to the map information matching the area information is acquired from the target database.

6. The method according to claim 5, characterized in that The acquiring, from the target database, target data corresponding to the feature vector matching the query vector comprises: For each feature vector in the target database, obtaining a first similarity between the query vector and the feature vector, and determining positive samples and negative samples from the feature vector according to all the first similarities; The positive sample is used as a positive anchor point and the negative sample is used as a secondary anchor point to obtain the second similarities between the feature vector in the target database and the positive and secondary anchor points, and the target data corresponding to the feature vector matching the query vector is obtained according to all the second similarities.

7. A driving data processing device, characterized in that: The device comprises: A data acquisition module is used to acquire the full amount of data of the vehicle during driving; the full amount of data includes vehicle information, sensor information, sensor data, scene information and algorithm output data; A timestamp alignment module, configured to use the timestamp of the front-view image captured by the front-view camera of the vehicle as a standard timestamp, align the timestamp of the full data with the standard timestamp, and obtain a plurality of front-view data groups; the front-view data groups include a front-view image and the full data corresponding to the timestamp of the front-view image; A database establishment module, for establishing a target database for each forward-looking data group according to the forward-looking images in the forward-looking data group and the corresponding full-volume data; the target database includes at least one of a label database, a vector database and a map database; The data query module is used to obtain query information input by the user according to the target database, and obtain target data corresponding to the query information from the target database.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images