Training data extraction method, system, device and medium for vehicle-road cooperation

Abstract

The application discloses a kind of training data extraction methods for vehicle-road cooperation, comprising the following steps: obtaining the video data collected;The collected video data is identified and the object that cannot be identified is determined;Abnormal identification event is generated based on the object that cannot be identified;According to the abnormal identification event, the corresponding video interval is extracted from the video data and used as the training data of the training engine.The application also discloses a kind of system, computer equipment and readable storage medium.The scheme presented in the application identifies the video data collected, thereby obtaining the object that cannot be identified, extracts the corresponding video interval based on the object that cannot be identified to be used as training data, effectively improves the extraction efficiency of real road terminal test data of vehicle-road cooperation simulation test engine.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of vehicle-road cooperation, and in particular to a training data extraction method, system, device and medium for vehicle-road cooperation. BACKGROUND

[0002] In the vehicle-road cooperation scenario of automatic driving, model training of a simulation test engine is very important, and the simulation test engine may need tens of thousands or even hundreds of thousands of different scene data for training to achieve a certain effect. The data input of these training models mainly depends on the original video and point cloud data of the real road end. Therefore, many cameras and radars need to be installed to monitor the real-time road conditions, and then the sensor data is recorded and stored through a recording service. The original data obtained by a large number of sensors recording is very large, and the data volume per day can reach TB or PB level, but the proportion of scene data effective for the simulation test engine in these massive data is low. How to store the recorded data at low cost and efficiently extract and use these valuable monitoring data is very important in the field of autonomous driving. The usual way is to upload all the full amount of monitoring data to the central cloud for storage, and then manually extract and use according to some conditions. This way of extracting scene data for training engine is very inefficient. SUMMARY

[0003] Therefore, in order to overcome at least one aspect of the above problems, the embodiments of the present application propose a training data extraction method for vehicle-road cooperation, comprising the following steps:

[0004] acquiring collected video data;

[0005] identifying the collected video data and determining unidentifiable objects;

[0006] generating an abnormal identification event based on the unidentifiable objects;

[0007] extracting a corresponding video interval from the video data according to the abnormal identification event and taking it as training data for a training engine.

[0008] In some embodiments, identifying the collected video data and determining unidentifiable objects further comprises:

[0009] setting a time sliding window;

[0010] dividing the video data into a plurality of video segments based on the time sliding window;

[0011] dividing each of the video segments into a plurality of image frames and identifying each of the image frames and determining unidentifiable objects.

[0012] In some embodiments, dividing each of the video clips into a plurality of image frames and identifying each of the image frames and determining unrecognizable objects further comprises:

[0013] performing a first detection on the image frames using a first recognition model and labeling vehicles, pedestrians, and cyclists;

[0014] performing a second detection on the image frames that have been detected and labeled by the first recognition model using a second recognition model and corresponding labeling of recognizable objects, and labeling remaining objects as unrecognizable.

[0015] In some embodiments, generating an abnormal recognition event based on the unrecognizable objects further comprises:

[0016] In response to detecting the presence of unrecognizable objects or determining that the number of unrecognizable objects reaches a threshold, generating an abnormal recognition event, wherein the abnormal recognition event includes an event ID and a time interval corresponding to the unrecognizable objects.

[0017] In some embodiments, obtaining the collected video further comprises:

[0018] collecting video data and radar point cloud data using an edge node;

[0019] storing the video data and the radar point cloud data to the edge node;

[0020] obtaining the video data from the edge node.

[0021] In some embodiments, extracting a corresponding video interval from the video data according to the abnormal recognition event and using it as training data for a training engine further comprises:

[0022] extracting radar point cloud data and video data of a corresponding time range from the edge node according to the time interval corresponding to the unrecognizable objects and storing them to a preset storage location;

[0023] The training engine generates a download address according to the event ID and obtains radar point cloud data and video data of a corresponding time range from the preset storage location according to the download address.

[0024] In some embodiments, further comprising:

[0025] converting the video data of the corresponding time range into ROS format point cloud data.

[0026] Based on the same inventive concept, according to another aspect of the present application, embodiments of the present application also provide a training data extraction system for vehicle-road cooperation, comprising:

[0027] an acquisition module configured to acquire the collected video data;

[0028] an identification module configured to identify the collected video data and determine an object that cannot be identified;

[0029] a generation module configured to generate an abnormal identification event based on the object that cannot be identified;

[0030] an extraction module configured to extract a corresponding video interval from the video data according to the abnormal identification event and as training data of a training engine.

[0031] Based on the same inventive concept, according to another aspect of the present application, embodiments of the present application further provide a computer device, comprising:

[0032] at least one processor; and

[0033] a memory storing a computer program capable of running on the processor, wherein the processor executes the program to perform the steps of any one of the above-mentioned methods for extracting training data for vehicle-road cooperation.

[0034] Based on the same inventive concept, according to another aspect of the present application, embodiments of the present application further provide a computer readable storage medium storing a computer program, wherein the computer program is executed by a processor to perform the steps of any one of the above-mentioned methods for extracting training data for vehicle-road cooperation.

[0035] The present application has one of the following beneficial technical effects: The present application extracts a corresponding video interval based on an object that cannot be identified, which is obtained by identifying collected video data, to serve as training data, thereby effectively improving the extraction efficiency of real road test data of a vehicle-road cooperation simulation test engine. BRIEF DESCRIPTION OF DRAWINGS

[0036] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other embodiments can be obtained from these drawings without creative labor.

[0037] Figure 1 a flowchart of the method for extracting training data for vehicle-road cooperation provided by the embodiments of the present application;

[0038] Figure 2 a flowchart of the method for extracting training data for vehicle-road cooperation provided by the embodiments of the present application;

[0039] Figure 3 A structural schematic diagram of a training data extraction system for vehicle-road cooperation provided for an embodiment of the present application is shown in the figure;

[0040] Figure 4 A structural schematic diagram of a computer device provided for an embodiment of the present application is shown in the figure;

[0041] Figure 5 A structural schematic diagram of a computer readable storage medium provided for an embodiment of the present application is shown in the figure. DETAILED DESCRIPTION

[0042] To make the objectives, technical solutions and advantages of the present application clearer, the following further describes embodiments of the present application in detail with reference to the accompanying drawings.

[0043] It should be noted that all the expressions of "first" and "second" in the embodiments of the present application are used to distinguish two same-named different entities or different parameters, and "first" and "second" are only used for the convenience of description, and should not be understood as a limitation of the embodiments of the present application. The subsequent embodiments will not be described one by one.

[0044] According to one aspect of the present application, an embodiment of the present application proposes a training data extraction method for vehicle-road cooperation, as shown in the figure, which can include the following steps: Figure 1

[0045] S1, acquiring collected video data;

[0046] S2, identifying the collected video data and determining unidentifiable objects;

[0047] S3, generating an abnormal identification event based on the unidentifiable objects;

[0048] S4, extracting a corresponding video interval from the video data according to the abnormal identification event and taking it as training data of a training engine.

[0049] The scheme proposed by the present application identifies the collected video data to obtain unidentifiable objects, extracts a corresponding video interval based on the unidentifiable objects as training data, and effectively improves the extraction efficiency of real road end test data of a vehicle-road cooperation simulation test engine.

[0050] In some embodiments, S1, acquiring the collected video, further includes:

[0051] Collecting video data and radar point cloud data by using an edge node;

[0052] Saving the video data and the radar point cloud data to the edge node;​

[0053] acquiring the video data from the edge node.

[0054] Specifically, as shown in Figure 2 The sensor data recording service can be started on the edge storage node, and a plurality of roadside cameras and sensors such as radars are matched to each recording service through global scheduling coordination, so as to collect the generated video data and point cloud data. However, such recording is generally performed all day long, and the data volume recorded by a single machine per day can reach tens of TB. The recorded data can be compressed and encoded and stored on the edge storage node, and the disk occupied by the compressed data is greatly reduced. At the same time, some metadata information of the recorded data is reported to the recording management center.

[0055] In some embodiments, S2, identifying the collected video data and determining the unidentifiable object further comprises:

[0056] S21, setting a time sliding window;

[0057] S22, dividing the video data into a plurality of video clips based on the time sliding window;

[0058] S23, dividing each of the video clips into a plurality of image frames and identifying each of the image frames and determining the unidentifiable object.

[0059] Specifically, as shown in Figure 2 The video data can come from the edge storage node. When the edge storage node collects the video data, it is reported to the data processing and anomaly detection center of the vehicle-road cooperation for real-time processing and anomaly detection. When processing the video data, a time sliding window can be set, and each time a video clip corresponding to the size of the time sliding window can be loaded, so that the video data is divided into a plurality of video clips for processing in turn. When processing each video clip, the video clip can be divided into a plurality of image frames, and each of the image frames can be identified and labeled as an unidentifiable object.

[0060] In some embodiments, S23, dividing each of the video clips into a plurality of image frames and identifying each of the image frames and determining the unidentifiable object further comprises:

[0061] using a first identification model to perform first detection on the image frames and label vehicles, pedestrians and cyclists;

[0062] using a second identification model to perform second detection on the image frames detected and labeled by the first identification model and corresponding labeling on the identifiable objects, and labeling the remaining objects as unidentifiable.

[0063] Specifically, when detecting each image frame, the image frame can be first processed in grayscale, and then a standard ONNX format vehicle and pedestrian inspection model (first recognition model) trained is used to recognize the image frame, and pedestrians, vehicles and cyclists that can be normally recognized are excluded. Then, a second recognition model is used for abnormal object detection, and the objects that cannot be recognized are labeled and saved. In the entire time sliding window, if no object that cannot be recognized appears in a certain position of the image in the early stage of the window, but an object that cannot be recognized appears in the later stage, it is judged that an object that cannot be recognized is found.

[0064] In some embodiments, S3, generating an abnormal recognition event based on the object that cannot be recognized, further comprises:

[0065] In response to detecting the presence of the object that cannot be recognized or determining that the number of objects that cannot be recognized reaches a threshold, an abnormal recognition event is generated, wherein the abnormal recognition event includes an event ID, and a time interval corresponding to the object that cannot be recognized.

[0066] Specifically, as shown in Figure 2 As long as the data processing and anomaly detection center detects an object that cannot be recognized or the data processing and anomaly detection center detects that the number of objects that cannot be recognized reaches a threshold, an abnormal scene event is actively thrown to the event distribution center, and the event object includes an event ID, a location of an edge storage node or a location coordinate of a corresponding sensor, an ID and a time interval to which the object that cannot be recognized is located t0-t1.

[0067] It should be noted that the time interval of the object that cannot be recognized and the size of the time sliding window can be inconsistent.

[0068] In some embodiments, S4, according to the abnormal recognition event, a corresponding video interval is extracted from the video data and used as training data of a training engine, further comprising:

[0069] According to the time interval corresponding to the object that cannot be recognized, radar point cloud data and video data of the corresponding time range are extracted from the edge node and saved to a preset storage location;

[0070] The training engine generates a download address according to the event ID, and acquires radar point cloud data and video data of the corresponding time range from the preset storage location according to the download address.

[0071] In some embodiments, further comprising:

[0072] The video data of the corresponding time range is converted into ROS format point cloud data.

[0073] Specifically, as shown in Figure 2As shown, after the data processing and anomaly detection center throws an abnormal scenario event to the event distribution center, the event distribution center forwards the abnormal scenario event to the subscription end, such as the automatic recording management center, a plurality of training engines (for example Figure 2 The simulation training engine and the AI engine) according to the event publishing end, and different subscription ends make different responses after receiving the event.

[0074] The automatic recording management center finds the edge node to which the related sensor set belongs according to the position of the edge storage node or the position coordinates, ID, etc. of the corresponding sensor and the metadata stored in the database, and sends a data extraction command to the data extraction service running on the edge storage node, and then the service packs and sends the video data and the point cloud data in the corresponding time interval to the cloud storage platform.

[0075] The training engine (for example, the simulation training engine and the AI engine) receives the event, uses the event ID, splices the download address according to the pre-prepared splicing rule, and downloads the video and the point cloud data extracted by the automatic recording management center and uploaded to the cloud storage platform. Then, the engine application converts the small amount of compressed video data into ROS format point cloud data through a specific data conversion script, and then performs corresponding data playback. Because the size of the compressed video data is much smaller than that of the ROS format data, through this conversion method, the disk and bandwidth occupied by the recording and extraction data are greatly reduced, and the training simulation engine can test the training data model through the ROS playback method without affecting the training simulation engine.

[0076] In some embodiments, in addition to the data processing and anomaly detection center automatically triggering the extraction of the training data, the vehicle end can also actively extract the sensor data of a certain period of time through the reporting of the geographical position and other information or other system active interface calling modes.

[0077] The scheme provided by the present application can identify the collected video data, obtain the unidentifiable object, extract the corresponding video interval based on the unidentifiable object as the training data, and effectively improve the extraction efficiency of the real road end test data of the vehicle-road cooperation simulation test engine.

[0078] Based on the same inventive concept, according to another aspect of the present application, embodiments of the present application also provide a training data extraction system 400 for vehicle-road cooperation, as shown in Figure 3 The training data extraction system 400 comprises:

[0079] An acquisition module 401 configured to acquire the collected video data;

[0080] An identification module 402 configured to identify the collected video data and determine the unidentifiable object.

[0081] The generating module 403 is configured to generate an abnormal identification event based on the unidentifiable object;

[0082] The extracting module 404 is configured to extract a corresponding video interval from the video data according to the abnormal identification event as training data of the training engine.

[0083] The scheme provided by the present application can effectively improve the extraction efficiency of real road test data of the vehicle-road cooperative simulation test engine by identifying the collected video data to obtain unidentifiable objects and extracting corresponding video intervals based on the unidentifiable objects as training data.

[0084] In some embodiments, the identification module 402 is further configured to:

[0085] set a time sliding window;

[0086] divide the video data into a plurality of video clips based on the time sliding window;

[0087] divide each of the video clips into a plurality of image frames and identify each of the image frames to determine unidentifiable objects.

[0088] In some embodiments, the identification module 402 is further configured to:

[0089] use a first identification model to perform first detection on the image frames and label vehicles, pedestrians and cyclists;

[0090] use a second identification model to perform second detection on the image frames detected and labeled by the first identification model and make corresponding labels for identifiable objects, and label the remaining objects as unidentifiable.

[0091] In some embodiments, the generating module 403 is further configured to:

[0092] in response to detecting the presence of unidentifiable objects or determining that the number of unidentifiable objects reaches a threshold value, generating an abnormal identification event, wherein the abnormal identification event includes an event ID and a time interval corresponding to the unidentifiable objects.

[0093] In some embodiments, the obtaining module 401 is further configured to:

[0094] collect video data and radar point cloud data using an edge node;

[0095] save the video data and the radar point cloud data to the edge node;

[0096] obtain the video data from the edge node.

[0097] In some embodiments, the extraction module 404 is further configured to:

[0098] extracting, from the edge node, radar point cloud data and video data of a corresponding time range according to the time interval corresponding to the unidentified object and saving to a preset storage location;

[0099] The training engine generates a download address according to the event ID and acquires radar point cloud data and video data of a corresponding time range from the preset storage location according to the download address.

[0100] In some embodiments, the extraction module 404 is further configured to:

[0101] convert the video data of the corresponding time range into point cloud data in ROS format.

[0102] Based on the same inventive concept, according to another aspect of the present application, as shown in Figure 4 The embodiments of the present application also provide a computer device 501, which comprises:

[0103] at least one processor 520; and

[0104] a memory 510, the memory 510 storing a computer program 511 executable on the processor, and the processor 520 executes the steps of any one of the above-mentioned methods for extracting training data for vehicle-road cooperation when executing the program.

[0105] Based on the same inventive concept, according to another aspect of the present application, as shown in Figure 5 The embodiments of the present application also provide a computer readable storage medium 601, which stores a computer program 610, and the computer program 610 is executable by a processor to execute the steps of any one of the above-mentioned methods for extracting training data for vehicle-road cooperation.

[0106] Finally, it needs to be noted that those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by a computer program, and the program can be stored in a computer readable storage medium, and when the program is executed, the processes of the above-mentioned embodiments can be included.

[0107] In addition, it should be understood that the computer readable storage medium (for example, the memory) herein can be a volatile memory or a non-volatile memory, or can include both volatile memory and non-volatile memory.

[0108] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the disclosure herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present embodiments.

[0109] The above are exemplary embodiments disclosed by the present embodiments, but it should be noted that various changes and modifications can be made without departing from the scope of the present embodiments disclosed by the claims. The functions, steps and / or actions of the method claims described herein need not be performed in any particular order. Furthermore, although elements of the present embodiments disclosed can be described or claimed in individual forms, unless expressly limited to the singular, these elements can also be understood to be implemented in a plurality of forms. The use of "including," "comprising," or "having" and variations thereof herein is meant to encompass the items listed thereafter and equivalents thereof as well as additional items.

[0110] It should be understood that, as used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. It will be further understood that the terms "comprises" and / or "comprising," when used in this specification, specify the presence of stated features, integers, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups thereof.

[0111] The above-mentioned embodiment number of the present embodiments is only for description, not representing the advantages and disadvantages of the embodiments.

[0112] Those of ordinary skill in the art can understand that all or part of the steps of the above-mentioned embodiments can be completed by hardware, or can be instructed to relevant hardware by a program, and the program can be stored in a computer readable storage medium. The storage medium mentioned above can be a read-only memory, a magnetic disk or an optical disk, etc.

[0113] Those of ordinary skill in the art will understand that the above discussion of various embodiments is merely exemplary and is not intended to suggest that the scope of the present embodiments disclosed (including the claims) is limited to these examples; under the concept of the present embodiments, the above embodiments or technical features in different embodiments can also be combined, and there are many other changes of the above aspects of the present embodiments. In order to be brief, they are not provided in detail. Therefore, any omission, modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present embodiments should be included in the protection scope of the present embodiments.

Claims

1. A training data extraction method for vehicle infrastructure integration, characterized in that, The method comprises the following steps: acquiring collected video data; identifying the collected video data and determining unrecognizable objects; generating an abnormal recognition event based on the unrecognizable objects; extracting a corresponding video interval from the video data according to the abnormal recognition event and taking the video interval as training data of a training engine; the step of generating an abnormal recognition event based on the unrecognizable objects further comprises: in response to detecting the presence of unrecognizable objects or determining that the number of unrecognizable objects reaches a threshold, generating an abnormal recognition event, wherein the abnormal recognition event comprises an event ID and a time interval corresponding to the unrecognizable objects; the step of extracting a corresponding video interval from the video data according to the abnormal recognition event and taking the video interval as training data of a training engine further comprises: extracting radar point cloud data and video data of a corresponding time range from an edge node according to the time interval corresponding to the unrecognizable objects and saving the radar point cloud data and the video data to a preset storage location; the training engine generates a download address according to the event ID and acquires the radar point cloud data and the video data of the corresponding time range from the preset storage location according to the download address.

2. The method of claim 1, wherein, The step of identifying the collected video data and determining unrecognizable objects further comprises: setting a time sliding window; dividing the video data into multiple video segments based on the time sliding window; dividing each of the video segments into multiple image frames and identifying each of the image frames and determining unrecognizable objects.

3. The method of claim 2, wherein, The step of dividing each of the video segments into multiple image frames and identifying each of the image frames and determining unrecognizable objects further comprises: detecting and labeling vehicles, pedestrians and cyclists in the image frames for the first time by using a first recognition model; detecting and labeling the image frames that have been detected and labeled by the first recognition model for the second time by using a second recognition model, labeling recognizable objects accordingly, and labeling remaining objects as unrecognizable.

4. The method of claim 1, wherein, The step of acquiring collected video further comprises: collecting video data and radar point cloud data by using an edge node; saving the video data and the radar point cloud data to the edge node; acquiring the video data from the edge node.

5. The method of claim 1, wherein, The method further comprises: converting the video data of the corresponding time range into point cloud data in ROS format. 6.A training data extraction system for vehicle infrastructure integration, characterized in that, The method comprises: an acquisition module configured to acquire collected video data; an identification module configured to identify the collected video data and determine unrecognizable objects; a generation module configured to generate an abnormal recognition event based on the unrecognizable objects; an extraction module configured to extract a corresponding video interval from the video data according to the abnormal recognition event and take the video interval as training data of a training engine; the generation module is further configured to: in response to detecting the presence of unrecognizable objects or determining that the number of unrecognizable objects reaches a threshold, generate an abnormal recognition event, wherein the abnormal recognition event comprises an event ID and a time interval corresponding to the unrecognizable objects; the extraction module is further configured to: extract radar point cloud data and video data of a corresponding time range from an edge node according to the time interval corresponding to the unrecognizable objects and save the radar point cloud data and the video data to a preset storage location. extracting and saving radar point cloud data and video data of the corresponding time range from the edge node according to the time interval corresponding to the unidentified object to a preset storage location; the training engine generates a download address according to the event ID and acquires radar point cloud data and video data of the corresponding time range from the preset storage location according to the download address. 7.A computer device, comprising: at least one processor; and a memory storing a computer program capable of running on the processor, wherein the processor executes the program to perform the steps of the method according to any one of claims 1-5.

8. A computer-readable storage medium storing a computer program, the computer-readable storage medium comprising: the computer program is executed by the processor to perform the steps of the method according to any one of claims 1-5.

Images