A data set construction method and device, computer equipment and storage medium

Abstract

The present disclosure provides a dataset construction method and device, computer equipment and a storage medium, wherein the method comprises: obtaining at least one sample video; respectively performing sampling processing on the sample video to obtain a sample image group; wherein the sampling processing includes at least one of first sampling processing based on driving distance, second sampling processing based on steering information, and third sampling processing based on position information; and based on the sample video frames contained in the sample image group, a dataset corresponding to the sample video is constructed.

Description

Technical Field

[0001] The present disclosure relates to the field of autonomous driving technology, and more specifically, to a data set construction method, apparatus, computer equipment, and storage medium. Background Art

[0002] In the field of autonomous driving, various networks are needed to control the driving of autonomous vehicles, such as lane detection networks. Therefore, in the field of autonomous driving, the construction of data sets is a particularly important step.

[0003] In related technologies, when constructing a dataset, a test vehicle is typically used to capture road video, which is then sampled at a fixed frame rate. The sampled video frames form the dataset. However, due to the varying importance of the video frames captured in different driving scenarios—for example, in high-speed driving scenarios, the image features of the sampled frames are less diverse—sampling at a fixed frame rate results in a dataset containing many similar video frames, which in turn reduces the dataset's quality and impacts neural network training. Summary of the Invention

[0004] The embodiments of the present disclosure at least provide a data set construction method, apparatus, computer equipment, and storage medium.

[0005] In a first aspect, an embodiment of the present disclosure provides a method for constructing a dataset, comprising:

[0006] Obtain at least one sample video;

[0007] Performing sampling processing on the sample videos respectively to obtain a sample image group; wherein the sampling processing includes at least one of a first sampling processing based on driving distance, a second sampling processing based on steering information, and a third sampling processing based on position information;

[0008] Based on the sample video frames included in the sample image group, a data set corresponding to the sample video is constructed.

[0009] When sampling sample videos, the sample videos are sampled through a variety of sampling methods, enriching the sampling methods of the sample videos; in addition, when sampling, the different driving scenarios of the sample videos are combined, and the sample videos are sampled and processed based on the driving distance, steering information, and position information respectively, ensuring the richness of the sample data in different driving scenarios, thereby improving the training effect of the neural network.

[0010] In a possible implementation, obtaining at least one sample video includes:

[0011] Acquire at least one sample video and sensor data corresponding to the acquisition of the sample video;

[0012] The sampling process of the sample videos respectively includes:

[0013] The sample video is sampled based on the sensor data corresponding to the sample video.

[0014] In this way, when sampling and processing the sample video, sensor data is combined to facilitate judgment of various driving scenarios corresponding to the sample video, and thus facilitate different sampling and processing of the sample video.

[0015] In a possible implementation, the sensor data includes speed information corresponding to each sample video frame of the sample video;

[0016] The sampling process of the sample video based on the sensor data corresponding to the sample video includes:

[0017] Starting from the first frame of the sample video frames, based on the speed information corresponding to each sample video frame and the time interval between each sample video frame, determining a first target video frame whose travel distance from the first frame is not less than a first preset distance;

[0018] The next video frame of the target video frame is re-determined as the first video frame, and the step of determining the target video frame is returned to be executed; wherein the target video frame is the video frame of the sampled image group obtained after the first sampling process.

[0019] In this way, the sample videos are sampled and processed at a fixed distance, which can cover most of the normal driving scenes.

[0020] In a possible implementation, the target video frame includes a high-speed video frame and a low-speed video frame;

[0021] The method further comprises:

[0022] determining a target video frame whose corresponding speed information is less than a preset speed threshold as the low-speed video frame, and determining a target video frame whose corresponding speed information is not less than the preset speed threshold as the high-speed video frame;

[0023] The step of constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group includes:

[0024] Based on the first construction ratio corresponding to the high-speed video frame and the second construction ratio corresponding to the low-speed video frame, a first video frame is screened out from the target video frame, and a data set corresponding to the sample video is constructed based on the first video frame.

[0025] In this way, by determining the high-speed video frames and low-speed video frames in the target video frames, the target video frames can be targeted and screened when constructing the dataset, effectively reducing data redundancy.

[0026] In a possible implementation, the sensor data includes a heading angle corresponding to each sample video frame of the sample video;

[0027] The sampling process of the sample video based on the sensor data corresponding to the sample video includes:

[0028] Determine a turning video segment whose corresponding heading angle is greater than a preset heading angle threshold;

[0029] The steering video segment is sampled according to a preset first sampling interval.

[0030] In this way, the turning video segments in the turning state in the sample video can be screened out and sampled, thereby increasing the richness of the sample data in the turning state.

[0031] In a possible implementation, the sensor data further includes speed information corresponding to each sample video frame of the sample video;

[0032] The sampling process of the steering video segment according to a preset sampling interval includes:

[0033] determining a turning movement distance corresponding to each turning video segment based on speed information corresponding to the turning video frames and a time interval between the turning video frames;

[0034] When the turning movement distance exceeds a second preset distance, sampling processing is performed on each of the turning video segments according to a preset first sampling interval.

[0035] In this way, the driving distance corresponding to the turning video segment is used to determine whether the sample collection vehicle corresponding to each turning video segment is in a turning state, and the sample video frames in which the sample collection vehicle is in a state of changing lanes, avoiding vehicles, etc. in the turning video segment can be removed.

[0036] In a possible implementation, the sensor data includes position information corresponding to each sample video frame of the sample video;

[0037] The sampling process of the sample video based on the sensor data corresponding to the sample video includes:

[0038] Determining at least one sample video segment shot in the target area based on the position information corresponding to each sample video frame;

[0039] Sampling is performed on each sample video segment according to a preset second sampling interval.

[0040] In this way, by sampling the intersection area, we can collect data in more complex scenarios, making the dataset cover more comprehensive scenarios.

[0041] In a possible implementation, the target area includes an intersection area;

[0042] After sampling each sample video segment according to the preset second sampling interval, the method further includes:

[0043] Identify the sampled regional video frames and determine the region type corresponding to each regional video frame based on the identification result;

[0044] The step of constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group includes:

[0045] Based on the construction ratios corresponding to the respective region types, a second video frame is screened out from the region video frame, and a data set corresponding to the sample video is constructed based on the second video frame.

[0046] In this way, by distinguishing the area types of intersection areas, the data similarity in the dataset can be reduced and the data quality can be improved.

[0047] In a possible implementation, constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group includes:

[0048] Determining the embedded representation vectors corresponding to the sample video frames included in the sample image group;

[0049] Based on the embedded representation vectors corresponding to the sample video frames, the sample video frames in the sample image group are deduplicated, and the sample image group after deduplication is used as the data set corresponding to the sample video.

[0050] In this way, the sampled image group is deduplicated using the embedded representation vector of each sample video frame in the sampled image group, thereby removing duplicate images of sampled vehicles at rest at intersections, thereby reducing data redundancy.

[0051] In a possible implementation manner, the method further includes:

[0052] The neural network to be trained is trained based on the data set corresponding to the sample video.

[0053] In a second aspect, an embodiment of the present disclosure further provides a data set construction device, including:

[0054] An acquisition module, configured to acquire at least one sample video;

[0055] a sampling module, configured to perform sampling processing on the sample videos to obtain a sample image group; wherein the sampling processing includes at least one of a first sampling processing based on driving distance, a second sampling processing based on steering information, and a third sampling processing based on position information;

[0056] A construction module is used to construct a data set corresponding to the sample video based on the sample video frames included in the sample image group.

[0057] In a possible implementation, the acquisition module, when acquiring at least one sample video, is configured to:

[0058] Acquire at least one sample video and sensor data corresponding to the acquisition of the sample video;

[0059] The sampling module is used to:

[0060] The sample video is sampled based on the sensor data corresponding to the sample video.

[0061] In a possible implementation, the sensor data includes speed information corresponding to each sample video frame of the sample video;

[0062] The sampling module, when performing sampling processing on the sample video based on the sensor data corresponding to the sample video, is used to:

[0063] Starting from the first frame of the sample video frames, based on the speed information corresponding to each sample video frame and the time interval between each sample video frame, determining a first target video frame whose travel distance from the first frame is not less than a first preset distance;

[0064] The next video frame of the target video frame is re-determined as the first video frame, and the step of determining the target video frame is returned to be executed; wherein the target video frame is the video frame of the sampled image group obtained after the first sampling process.

[0065] In a possible implementation, the target video frame includes a high-speed video frame and a low-speed video frame;

[0066] The sampling module is further used for:

[0067] determining a target video frame whose corresponding speed information is less than a preset speed threshold as the low-speed video frame, and determining a target video frame whose corresponding speed information is not less than the preset speed threshold as the high-speed video frame;

[0068] The construction module, when constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group, is used to:

[0069] Based on the first construction ratio corresponding to the high-speed video frame and the second construction ratio corresponding to the low-speed video frame, a first video frame is screened out from the target video frame, and a data set corresponding to the sample video is constructed based on the first video frame.

[0070] In a possible implementation, the sensor data includes a heading angle corresponding to each sample video frame of the sample video;

[0071] The sampling module, when performing sampling processing on the sample video based on the sensor data corresponding to the sample video, is used to:

[0072] Determine a turning video segment whose corresponding heading angle is greater than a preset heading angle threshold;

[0073] The steering video segment is sampled according to a preset first sampling interval.

[0074] In a possible implementation, the sensor data further includes speed information corresponding to each sample video frame of the sample video;

[0075] The sampling module is configured to:

[0076] determining a turning movement distance corresponding to each turning video segment based on speed information corresponding to the turning video frames and a time interval between the turning video frames;

[0077] When the turning movement distance exceeds a second preset distance, sampling processing is performed on each of the turning video segments according to a preset first sampling interval.

[0078] In a possible implementation, the sensor data includes position information corresponding to each sample video frame of the sample video;

[0079] The sampling module, when performing sampling processing on the sample video based on the sensor data corresponding to the sample video, is used to:

[0080] Determining at least one sample video segment shot in the target area based on the position information corresponding to each sample video frame;

[0081] Sampling is performed on each sample video segment according to a preset second sampling interval.

[0082] In a possible implementation, the target area includes an intersection area;

[0083] The sampling module is further configured to: after sampling each sample video segment according to a preset second sampling interval:

[0084] Identify the sampled regional video frames and determine the region type corresponding to each regional video frame based on the identification result;

[0085] The construction module, when constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group, is used to:

[0086] Based on the construction ratios corresponding to the respective region types, a second video frame is screened out from the region video frame, and a data set corresponding to the sample video is constructed based on the second video frame.

[0087] In one possible implementation, the construction module, when constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group, is configured to:

[0088] Determining the embedded representation vectors corresponding to the sample video frames included in the sample image group;

[0089] Based on the embedded representation vectors corresponding to the sample video frames, the sample video frames in the sample image group are deduplicated, and the sample image group after deduplication is used as the data set corresponding to the sample video.

[0090] In a possible implementation, the device further includes a training module, configured to:

[0091] The neural network to be trained is trained based on the data set corresponding to the sample video.

[0092] In a third aspect, an embodiment of the present disclosure further provides a computer device comprising: a processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are performed.

[0093] In a fourth aspect, an embodiment of the present disclosure further provides a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps of the above-mentioned first aspect or any possible implementation of the first aspect are executed.

[0094] For a description of the effects of the above-mentioned dataset construction apparatus, computer equipment, and computer-readable storage medium, please refer to the description of the above-mentioned dataset construction method, which will not be repeated here.

[0095] In order to make the above-mentioned objectives, features and advantages of the present disclosure more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0096] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the following briefly introduces the drawings required for use in the embodiments. The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and, together with the specification, are used to illustrate the technical solutions of the present disclosure. It should be understood that the following drawings only illustrate certain embodiments of the present disclosure and should not be regarded as limiting the scope. For those of ordinary skill in the art, other relevant drawings can be obtained based on these drawings without inventive effort.

[0097] Figure 1 A flowchart of a data set construction method provided by an embodiment of the present disclosure is shown;

[0098] Figure 2 A flowchart of a first sampling processing method based on driving distance in the method for constructing a data set provided by an embodiment of the present disclosure is shown;

[0099] Figure 3 A flowchart of a second sampling processing method based on steering information in the method for constructing a data set provided by an embodiment of the present disclosure is shown;

[0100] Figure 4 A flowchart of a third sampling processing method based on location information in the method for constructing a data set provided by an embodiment of the present disclosure is shown;

[0101] Figure 5 A flowchart showing a specific method for constructing a dataset corresponding to the sample video in the dataset construction method provided by an embodiment of the present disclosure is shown;

[0102] Figure 6 A schematic diagram of the architecture of a data set construction device provided by an embodiment of the present disclosure is shown;

[0103] Figure 7 A schematic structural diagram of a computer device provided by an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0104] In order to make the purpose, technical solutions and advantages of the embodiments of the present disclosure clearer, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. The components of the embodiments of the present disclosure generally described and shown in the drawings herein can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present disclosure provided in the drawings is not intended to limit the scope of the disclosure for which protection is sought, but merely represents selected embodiments of the present disclosure. Based on the embodiments of the present disclosure, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of the present disclosure.

[0105] After research, it was found that in related technologies, when constructing a data set, road videos are generally collected by test vehicles, and then the road videos are sampled at a fixed frame rate, and the sampled video frames constitute the data set. However, due to the different importance of video frames collected in different driving scenarios, for example, in high-speed driving scenarios, the differences between the image features of the sampled video frames are small. The fixed frame rate sampling method will cause the data set to contain more similar video frames, which will result in lower quality of the data set and affect the training effect of the neural network. At the same time, such a sampling method will result in too few video frames corresponding to important sections with a relatively small time share (such as curves, intersections, traffic light information, pedestrians, zebra crossings, etc.), which will result in lower richness of the data set.

[0106] Based on the above research, the present disclosure provides a dataset construction method, apparatus, computer equipment and storage medium. When sampling sample videos, the sample videos are sampled through multiple sampling methods, thereby enriching the sampling methods of the sample videos. In addition, when sampling, the different driving scenarios of the sample videos are combined, and the sample videos are sampled and processed based on the driving distance, steering information and position information respectively, thereby ensuring the richness of the sample data in different driving scenarios, thereby improving the training effect of the neural network.

[0107] The defects in the above solutions are the results obtained by the inventors after practice and careful research. Therefore, the process of discovering the above problems and the solutions proposed by this disclosure for the above problems below should be the contributions made by the inventors to this disclosure during the disclosure process.

[0108] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined or explained in subsequent drawings.

[0109] To facilitate understanding of this embodiment, a data set construction method disclosed in an embodiment of the present disclosure is first introduced in detail. The executor of the data set construction method provided in the embodiment of the present disclosure is generally a computer device with certain computing capabilities, and the computer device includes, for example: a server or other processing device. In some possible implementation methods, the data set construction method can be implemented by a processor calling computer-readable instructions stored in a memory.

[0110] See also Figure 1 FIG. 1 is a flowchart of a method for constructing a data set according to an embodiment of the present disclosure, wherein the method includes steps 101 to 103, wherein:

[0111] Step 101: Obtain at least one sample video.

[0112] Step 102: perform sampling processing on the sample videos to obtain a sample image group; wherein the sampling processing includes at least one of a first sampling processing based on driving distance, a second sampling processing based on steering information, and a third sampling processing based on position information.

[0113] Step 103: construct a data set corresponding to the sample video based on the sample video frames included in the sample image group.

[0114] The following are detailed instructions for the above steps:

[0115] For step 101,

[0116] In one possible implementation, obtaining at least one sample video may involve the server acquiring a sample video automatically uploaded by a video capture device (such as a dashcam); or, after completing video capture, a sample video manually input by a technician. The at least one sample video may be a sample video captured based on at least one driving route; the at least one driving route may include at least one road scene or at least one driving condition. For example, the road scene may include a highway, a city road, or near a school; and the different driving conditions may include driving straight or turning.

[0117] Specifically, the driving route may be a pre-set fixed route, or a route randomly driven by the sample collection vehicle.

[0118] For steps 102 and 103,

[0119] In one possible implementation, when acquiring at least one sample video, the sensor data corresponding to the sample video can also be synchronously acquired. Furthermore, when sampling and processing the sample videos respectively, the sample videos can be sampled and processed based on the sensor data corresponding to the sample videos.

[0120] The sensor data may be detected by a sensor installed on the sample collection vehicle. For example, the sensor data may include speed information, heading angle, and position information corresponding to each sample video frame of the sample video.

[0121] In one possible scenario, the speed information may be detected by a speed sensor on the sample collection vehicle. Since the shooting frame rate of the sample video may be relatively high, for example, 60 frames per second, and the detection frequency of the speed sensor may be relatively low, for example, once per second, the speed information of all video frames shot at the Nth second in the sample video may be the speed information detected by the speed sensor at the Nth second, that is, the speed information of all video frames shot at the Nth second in the sample video is the same.

[0122] The method for determining the heading angle and the position information is similar to the method for determining the speed information, and will not be described in detail here.

[0123] The following describes in detail the three sampling processes described in step 102:

[0124] 1. First sampling process based on driving distance.

[0125] The driving distance may refer to the actual distance traveled by the sample collection vehicle during driving, and the first sampling process refers to sampling the sample video at a fixed driving distance as a sampling interval.

[0126] Here, the driving distance can be calculated by collecting the driving speed and driving time of the sample vehicle. Therefore, when performing the first sampling processing, it is necessary to sample the sample video based on the speed information corresponding to each sample video frame of the sample video (that is, the above-mentioned sensor data).

[0127] In a possible implementation, when the first sampling process is performed on the sample video based on the sensor data corresponding to the sample video, the following can be done: Figure 2 As shown, it includes the following steps:

[0128] Step 201: Starting from the first frame of the sample video frames, based on the speed information corresponding to each sample video frame and the time interval between each sample video frame, determine the first target video frame whose travel distance from the first frame is not less than a first preset distance.

[0129] Step 202: re-determine the next video frame of the target video frame as the first video frame, and return to the step of determining the target video frame; wherein the target video frame is the video frame of the sampled image group obtained after the first sampling process.

[0130] In one possible implementation, when determining the first target video frame whose driving distance from the first video frame is not less than a first preset distance, starting from the first frame of the sample video frame, the driving distance corresponding to each adjacent sample video frame can be calculated based on the speed information corresponding to each adjacent sample video frame and the time interval between each adjacent sample video frame; then, the driving distances corresponding to the adjacent sample video frames are accumulated in sequence, and the accumulated result is compared with the first preset distance to determine the target video frame.

[0131] Specifically, when calculating the driving distance corresponding to each adjacent sample video frame, the following formula can be used:

[0132]

[0133] Wherein, v represents the speed information corresponding to each sample video frame; i represents the number of each sample video frame; and t represents the time corresponding to each sample video frame, in ms.

[0134] In a possible implementation, the driving distance between each video frame in the sample video and the first video frame can be directly determined, and then the driving distance from the first video frame is determined to be the first preset distance, or the first target video frame that exceeds the first preset distance.

[0135] Alternatively, in another possible implementation, the driving distance between each video frame and the first video frame can be determined in sequence according to the order of each video frame in the sample video frame, and when the driving distance is less than the first preset distance, the driving distance between the next video frame and the first video frame can be determined, and so on, until the target video frame is determined.

[0136] In one possible implementation, the first preset distance may be different for different driving speeds. For high-speed sections, due to the higher speed, if the first preset distance is set small, frequent sampling is required, the similarity between the collected images is high, and the sample quality is poor. For low-speed sections, due to the slower speed, if the first preset distance is set large, the sampling will be coarse, affecting the richness of the sample data.

[0137] Therefore, when sampling the sample video based on the first sampling process, the sample video can be first divided into high-speed driving area video and low-speed driving area video based on the speed information of each sample video frame and the preset speed threshold, and then the high-speed driving area video is sampled according to the first sampling distance, and the low-speed driving area video is sampled according to the second sampling distance, wherein the first sampling distance and the second sampling distance are both the first preset distance.

[0138] Exemplarily, the first sampling distance may be 13 meters, and the second sampling distance may be 7 meters.

[0139] In one possible implementation, the target video frame includes a high-speed video frame and a low-speed video frame. After performing a first sampling process on the first sample video, the speed information of the target video frame can also be determined, and the target video frame whose corresponding speed information is less than a preset speed threshold is determined as the low-speed video frame, and the target video frame whose corresponding speed information is not less than the preset speed threshold is determined as the high-speed video frame.

[0140] Since the image similarity of high-speed video frames is higher and the image similarity of low-speed video frames is lower, when constructing a data set, the first video frame can be filtered out from the target video frame based on the first construction ratio corresponding to the high-speed video frame and the second construction ratio corresponding to the low-speed video frame, and a data set corresponding to the sample video can be constructed based on the first video frame.

[0141] Exemplarily, the dataset may be constructed based on 30% of high-speed video frames and 50% of low-speed video frames in the target video frames.

[0142] In addition to pre-dividing the sample video into high-speed driving area videos and low-speed driving area videos and then performing sampling processing, the determination can also be performed after determining the target video frame.

[0143] Specifically, the target video frame can be determined according to the lower second sampling distance, and then after the target video frame is determined, it is judged whether the speed information of the target video frame is less than a preset speed threshold. If it is less than, the target video frame is directly determined as a low-speed video frame; if it is not less than, the target video frame can be re-determined according to the first sampling distance based on the current one, and then it is judged whether the speed information of the re-determined target video frame is less than the preset speed threshold. If it is not less than, the re-determined target video frame is determined as a high-speed video frame.

[0144] Since the vehicle speed changes smoothly and the difference between the first sampling distance and the second sampling distance is small, it is generally unlikely that the speed information of the re-determined target video frame is less than the preset speed threshold. If the speed information of the re-determined target video frame is less than the preset speed threshold, the re-determined target video frame can be directly used as a low-speed video frame.

[0145] Exemplarily, the first sampling distance is 13 meters, the second sampling distance is 7 meters, and the preset speed threshold is 60 meters / second. Then, sampling can be performed at intervals of 7 meters first. After sampling to obtain video frame 1, if the speed information of video frame 1 is 50 meters / second, it can be determined that video frame 1 is a low-speed video frame, and then sampling can continue at intervals of 7 meters. After sampling to obtain video frame 2, if the speed information of video frame 2 is 65 meters / second, sampling can continue at a sampling interval of 13 meters on the current basis, that is, a video frame is determined to be 6 meters away from video frame 2. If sampling continues at a sampling interval of 13 meters, video frame 3 is obtained. If the speed information of video frame 3 is 70 meters / second, it can be determined that video frame 3 is a high-speed video frame.

[0146] In this way, fixed-distance sampling can cover most normal driving scenarios. At the same time, using different distance thresholds for high-speed and low-speed scenarios can effectively reduce data redundancy.

[0147] 2. Second Sampling Processing Based on Steering Information

[0148] The steering information may be information indicating whether the sample collection vehicle is turning, and the second sampling process may refer to sampling the sample video frames in the sample video that are in a turning state.

[0149] Whether the vehicle is turning can be judged by the corresponding heading angle during the vehicle's driving process. Therefore, when performing the second sampling processing, the sample video can be sampled and processed based on the heading angle corresponding to each sample video frame of the sample video (that is, the above-mentioned sensor data).

[0150] In a possible implementation, when the second sampling process is performed on the sample video based on the sensor data corresponding to the sample video, the following can be done: Figure 3 As shown, it includes the following steps:

[0151] Step 301: Determine a turning video segment whose corresponding heading angle is greater than a preset heading angle threshold.

[0152] Exemplarily, when determining a turn video segment whose corresponding heading angle is greater than a preset heading angle threshold, the heading angles corresponding to the sample video frames are sequentially compared with the preset heading angle threshold. When the heading angle corresponding to any sample video frame is greater than the preset heading angle threshold, the sample video frame is determined to be the start turn video frame. The end turn video frame is then determined based on the heading angle corresponding to the sample video frame following the start turn video frame. Furthermore, the turn video segment is a video segment consisting of the sample video frames between the start turn video frame and the end turn video frame. Exemplarily, the preset heading angle threshold can be set to 0.2.

[0153] Exemplarily, the heading angles corresponding to the sample video frames are 0.1, 0.24, 0.25, 0.3, 0.5, 0.45, 0.35, 0.26, 0.19, and 0.15, respectively, wherein the second sample video frame is the start turning video frame (0.24>0.2), and the ninth sample video frame is the end turning video frame (i.e., the ninth sample video frame is the sample video frame after the second sample video frame, and the first frame heading angle is less than the preset heading angle threshold), and the turning video segment is the sample video segment corresponding to the second sample video frame to the ninth sample video frame.

[0154] Step 302: Sampling the turn video segment according to a preset first sampling interval.

[0155] The sample collection vehicle may also cause the heading angle to be greater than the preset heading angle threshold when performing operations such as changing lanes or avoiding vehicles during driving. In order to avoid including such sample video frames in the sampled image group, when sampling and processing the turning video segment according to the preset first sampling interval, it is also necessary to determine whether the sample collection vehicle corresponding to the turning video segment being sampled is in a turning state.

[0156] In one possible implementation, when sampling the turn video segments at a preset first sampling interval, the turn movement distance corresponding to each turn video segment can be determined based on the speed information corresponding to the turn video frames and the time interval between the turn video frames. Then, if the turn movement distance exceeds a second preset distance, the turn video segments are sampled at the preset first sampling interval. For example, the second preset distance can be set to 7 meters, and the first sampling interval can be 5.

[0157] Specifically, when calculating the turning distance corresponding to each turning video segment, the turning distance of each adjacent turning video frame can be first determined based on the speed information corresponding to the adjacent turning video frames and the time interval between the adjacent turning video frames; then the turning distances corresponding to the adjacent turning video frames are summed to obtain the turning distance corresponding to each turning video segment.

[0158] For example, the turning distances corresponding to the adjacent turning video frames in the turning video segment are 0.1m, 0.15m, 0.19m, 0.21m, 0.3m, 0.5m, 1m, 3m, 2.6m, and 2m, respectively. If the turning distance corresponding to the turning video segment is 10.05m>7m (i.e., the second preset distance), then the sample collection vehicle corresponding to the turning video segment is in a turning state. Then, the turning video segment is sampled and processed at a sampling interval of every 5 video frames, and the first turning video frame, the sixth turning video frame, and the ninth turning video frame can be sampled respectively. In order to ensure that the constructed data set contains turning data from more perspectives, when sampling and processing each turning video segment, the first turning video frame (i.e., the video frame at the start of the turn) and the last turning video frame (i.e., the video frame at the end of the turn) in the turning video segment can be simultaneously used as sample video frames in the data set.

[0159] 3. Third Sampling Processing Based on Position Information

[0160] The position information may refer to the longitude and latitude of the corresponding position of the sample collection vehicle during driving, and the third sampling process may refer to sampling the sample video frames collected in the target area in the sample video.

[0161] When performing the third sampling process, it is possible to determine whether each sample video frame appears in a target area based on the position information corresponding to each sample video frame of the sample video (ie, the above-mentioned sensor data).

[0162] In a possible implementation, when the third sampling process is performed on the sample video based on the sensor data corresponding to the sample video, the following can be done: Figure 4 As shown, it includes the following steps:

[0163] Step 401: Determine at least one sample video segment shot in a target area based on the position information corresponding to each sample video frame.

[0164] Here, the target area may refer to an area containing rich traffic information, such as an intersection, a T-junction, a zebra crossing area, etc. The following steps are described below by taking the target area as an intersection as an example.

[0165] In one possible implementation, when determining at least one sample video shot in a target area, we may first start with the first sample video frame and determine the first sample video frame entering the target area based on the position information corresponding to each sample video frame; then determine the second sample video frame that leaves the target area and is located after the first sample video frame; and finally determine the video segment between the adjacent first sample video frame and the second sample video frame as the sample video segment shot in the target area.

[0166] In one possible implementation, when determining the first sample video frame that enters the target area, starting with the first sample video frame, a first position distance between the position information corresponding to each sample video frame and the target area can be determined sequentially; and the first position distance can be compared with a preset position distance. If the first position distance is less than the preset position distance, the sample video frame corresponding to the first position distance is determined to be the first sample video frame.

[0167] In one possible implementation, when determining the second sample video frame that leaves the target area, starting with the first sample video frame, the second position distance between the position information corresponding to each sample video frame and the target area can be determined sequentially; the second position distance is then compared with a preset position distance. If the second position distance is greater than the preset position distance, the sample video frame corresponding to the second position distance is determined as the second sample video frame; the video frame following the second sample video frame is then used as the first sample video frame, and the first sample video frame is re-determined.

[0168] For example, when determining the location information of the target area, an online map can be used for determination. After the longitude and latitude corresponding to the sample video frame are input into the online map, the online map can automatically determine the location information of the target area closest to the current longitude and latitude. The online map can be a self-built OSM (OpenStreetMap) map.

[0169] Step 402: Sampling each sample video segment according to a preset second sampling interval.

[0170] Since the sample collection vehicle will encounter a variety of intersections during driving, some intersections are located on busy streets and contain multiple traffic lights, crosswalks, etc.; some intersections are located at the entrance of a residential area, with simple road conditions and no traffic lights. Intersections on busy streets can often contain more pedestrians and vehicles. Therefore, after sampling and processing each sample video segment according to the preset second sampling interval, the sampled regional video frames must be identified, and the regional type corresponding to each regional video frame is determined based on the identification results. Among them, the regional type can be a large intersection area (i.e., an intersection containing traffic lights) and a small intersection area (i.e., an intersection that does not contain traffic lights).

[0171] In a possible implementation, when identifying the sampled regional video frame, the online map may be used to determine whether the regional video frame includes a traffic light.

[0172] In another possible implementation, when identifying the sampled regional video frame, traffic lights may be identified in the regional video frame based on a pre-trained road recognition network.

[0173] After determining the region type corresponding to each regional video frame, when constructing a data set corresponding to the sample video based on the sample video frames contained in the sampling image group, it is also possible to filter out a second video frame from the regional video frame based on the construction ratio corresponding to each region type, and construct a data set corresponding to the sample video based on the second video frame.

[0174] For example, a data set may be constructed based on 80% of regional video frames of large intersection areas and 20% of regional video frames of small intersection areas.

[0175] In this way, by sampling the intersection area, we can collect data from more complex scenarios, making the dataset more comprehensive. At the same time, by distinguishing the area types of the intersection area, we can reduce the data similarity in the dataset and improve the data quality.

[0176] It should be noted that a driving route may include high-speed sections, low-speed sections, sections that require turning, and sections at intersections. Therefore, when sampling and processing any of the sample videos, the sample video can be subjected to the first sampling processing, the second sampling processing, and the third sampling processing in sequence or simultaneously.

[0177] In a possible implementation, when constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group, the specific process may be as follows: Figure 5 As shown, it includes the following steps:

[0178] Step 501: Determine the embedding representation vectors corresponding to the sample video frames included in the sample image group.

[0179] Specifically, when determining the embedded representation vector, the following formula may be used to calculate the embedded representation vector corresponding to each sample video frame in the sampled image group.

[0180] v i =G(x i )

[0181] Among them, G represents the convolutional neural network using metric learning and training, and i represents the sample video frame number currently calculated.

[0182] Step 502: Deduplication processing is performed on the sample video frames in the sample image group based on the embedded representation vectors corresponding to the sample video frames, and the sample image group after deduplication processing is used as the data set corresponding to the sample video.

[0183] In one possible implementation, different sampling methods can obtain different groups of sampled image groups. When deduplication processing is performed on the sample video frames in the sampled image groups based on the embedded representation vectors corresponding to each sample video frame, the sample video frames in the same group can be deduplicated first based on the embedded representation vectors corresponding to each sample video frame in the same group, and then deduplication processing can be performed on any two groups of sample video frames based on the deduplicated sample video frames.

[0184] Specifically, when performing deduplication processing, the cosine distance between the embedded representation vectors of each two sample video frames can be determined first; then the cosine distance is compared with a preset similarity threshold. If it is greater than the preset similarity threshold, any one of the two sample video frames is eliminated, or the sample video frame with lower quality is eliminated (such as eliminating the sample video frame with lower clarity), or the sample video frame that is ranked later in the sample video is eliminated.

[0185] Specifically, the calculation formula of the cosine distance is as follows:

[0186]

[0187] Where i represents any sample video frame in the sampled image group, and j represents a sample video frame within a certain range from the i video frame. For example, if i is the 4th sample video frame in the sampled image group, then j can be any sample video frame between the 4th and 14th.

[0188] In a possible implementation, the sample image group after deduplication processing is visualized so that technicians can verify each sample image in the sample image group, thereby further reducing redundancy between each sample image in the sample image group.

[0189] After obtaining the data set corresponding to the sample video, the neural network to be trained can be trained based on the data set corresponding to the sample video.

[0190] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0191] Based on the same inventive concept, the embodiments of the present disclosure also provide a dataset construction device corresponding to the dataset construction method. Since the principle of solving the problem by the device in the embodiments of the present disclosure is similar to the above-mentioned dataset construction method in the embodiments of the present disclosure, the implementation of the device can refer to the implementation of the method, and the repeated parts will not be repeated.

[0192] Reference Figure 6 FIG. 6 is a schematic diagram of the architecture of a data set construction device provided by an embodiment of the present disclosure, wherein the device includes: an acquisition module 601, a sampling module 602, a construction module 603, and a training module 604; wherein,

[0193] An acquisition module 601 is configured to acquire at least one sample video;

[0194] a sampling module 602 for performing sampling processing on the sample videos to obtain a sample image group; wherein the sampling processing includes at least one of a first sampling processing based on driving distance, a second sampling processing based on turning information, and a third sampling processing based on position information;

[0195] The construction module 603 is configured to construct a data set corresponding to the sample video based on the sample video frames included in the sample image group.

[0196] In a possible implementation, the acquisition module 601, when acquiring at least one sample video, is configured to:

[0197] Acquire at least one sample video and sensor data corresponding to the acquisition of the sample video;

[0198] The sampling module 602 is used to:

[0199] The sample video is sampled based on the sensor data corresponding to the sample video.

[0200] In a possible implementation, the sensor data includes speed information corresponding to each sample video frame of the sample video;

[0201] The sampling module 602, when performing sampling processing on the sample video based on the sensor data corresponding to the sample video, is configured to:

[0202] Starting from the first frame of the sample video frames, based on the speed information corresponding to each sample video frame and the time interval between each sample video frame, determining a first target video frame whose travel distance from the first frame is not less than a first preset distance;

[0203] The next video frame of the target video frame is re-determined as the first video frame, and the step of determining the target video frame is returned to be executed; wherein the target video frame is the video frame of the sampled image group obtained after the first sampling process.

[0204] In a possible implementation, the target video frame includes a high-speed video frame and a low-speed video frame;

[0205] The sampling module 602 is further configured to:

[0206] determining a target video frame whose corresponding speed information is less than a preset speed threshold as the low-speed video frame, and determining a target video frame whose corresponding speed information is not less than the preset speed threshold as the high-speed video frame;

[0207] The construction module 603, when constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group, is used to:

[0208] Based on the first construction ratio corresponding to the high-speed video frame and the second construction ratio corresponding to the low-speed video frame, a first video frame is screened out from the target video frame, and a data set corresponding to the sample video is constructed based on the first video frame.

[0209] In a possible implementation, the sensor data includes a heading angle corresponding to each sample video frame of the sample video;

[0210] The sampling module 602, when performing sampling processing on the sample video based on the sensor data corresponding to the sample video, is configured to:

[0211] Determine a turning video segment whose corresponding heading angle is greater than a preset heading angle threshold;

[0212] The steering video segment is sampled according to a preset first sampling interval.

[0213] In a possible implementation, the sensor data further includes speed information corresponding to each sample video frame of the sample video;

[0214] The sampling module 602, when sampling the turn video segment according to a preset sampling interval, is used to:

[0215] determining a turning movement distance corresponding to each turning video segment based on speed information corresponding to the turning video frames and a time interval between the turning video frames;

[0216] When the turning movement distance exceeds a second preset distance, sampling processing is performed on each of the turning video segments according to a preset first sampling interval.

[0217] In a possible implementation, the sensor data includes position information corresponding to each sample video frame of the sample video;

[0218] The sampling module 602, when performing sampling processing on the sample video based on the sensor data corresponding to the sample video, is configured to:

[0219] Determining at least one sample video segment shot in the target area based on the position information corresponding to each sample video frame;

[0220] Sampling is performed on each sample video segment according to a preset second sampling interval.

[0221] In a possible implementation, the target area includes an intersection area;

[0222] The sampling module 602 is further configured to: after sampling each sample video segment according to a preset second sampling interval:

[0223] Identify the sampled regional video frames and determine the region type corresponding to each regional video frame based on the identification result;

[0224] The construction module 603, when constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group, is used to:

[0225] Based on the construction ratios corresponding to the respective region types, a second video frame is screened out from the region video frame, and a data set corresponding to the sample video is constructed based on the second video frame.

[0226] In a possible implementation, the construction module 603, when constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group, is configured to:

[0227] Determining the embedded representation vectors corresponding to the sample video frames included in the sample image group;

[0228] Based on the embedded representation vectors corresponding to the sample video frames, the sample video frames in the sample image group are deduplicated, and the sample image group after deduplication is used as the data set corresponding to the sample video.

[0229] In a possible implementation, the apparatus further includes a training module 604 configured to:

[0230] The neural network to be trained is trained based on the data set corresponding to the sample video.

[0231] For descriptions of the processing flow of each module in the device and the interaction flow between each module, reference can be made to the relevant descriptions in the above method embodiment, which will not be described in detail here.

[0232] Based on the same technical concept, the embodiment of the present disclosure also provides a computer device. Figure 7 , which is a schematic diagram of the structure of a computer device 700 provided in an embodiment of the present disclosure, including a processor 701, a memory 702, and a bus 703. The memory 702 is used to store execution instructions and includes a memory 7021 and an external memory 7022. The memory 7021 is also referred to as internal memory and is used to temporarily store operation data in the processor 701 and data exchanged with an external memory 7022 such as a hard disk. The processor 701 exchanges data with the external memory 7022 via the memory 7021. When the computer device 700 is running, the processor 701 communicates with the memory 702 via the bus 703, so that the processor 701 executes the following instructions:

[0233] Obtain at least one sample video;

[0234] Performing sampling processing on the sample videos respectively to obtain a sample image group; wherein the sampling processing includes at least one of a first sampling processing based on driving distance, a second sampling processing based on steering information, and a third sampling processing based on position information;

[0235] Based on the sample video frames included in the sample image group, a data set corresponding to the sample video is constructed.

[0236] The present disclosure also provides a computer-readable storage medium having a computer program stored thereon. When executed by a processor, the computer program executes the steps of the dataset construction method described in the above method embodiment. The storage medium may be a volatile or non-volatile computer-readable storage medium.

[0237] The present disclosure also provides a computer program product that carries program code. The program code includes instructions that can be used to execute the steps of the dataset construction method described in the above method embodiment. For details, please refer to the above method embodiment and will not be repeated here.

[0238] The computer program product may be implemented in hardware, software, or a combination thereof. In one embodiment, the computer program product is implemented as a computer storage medium. In another embodiment, the computer program product is implemented as a software product, such as a software development kit (SDK).

[0239] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems and devices described above can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here. In the several embodiments provided in the present disclosure, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. The device embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0240] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0241] In addition, each functional unit in each embodiment of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0242] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a non-volatile computer-readable storage medium that is executable by a processor. Based on this understanding, the technical solution of the present disclosure, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present disclosure. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0243] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present disclosure, which are used to illustrate the technical solutions of the present disclosure, rather than to limit them. The scope of protection of the present disclosure is not limited thereto. Although the present disclosure has been described in detail with reference to the above-described embodiments, those skilled in the art should understand that any person skilled in the art can modify or easily conceive of changes to the technical solutions described in the above-described embodiments within the technical scope disclosed in the present disclosure, or replace some of the technical features therein with equivalents. Such modifications, changes, or replacements do not deviate from the spirit and scope of the technical solutions of the embodiments of the present disclosure, and should be included in the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure shall be subject to the scope of protection of the claims.

Claims

1. A method for constructing a data set, characterized in that: include: Acquire at least one sample video and sensor data corresponding to the acquisition of the sample video; Based on the sensor data corresponding to the sample videos, sampling and processing are performed on the sample videos to obtain a sample image group; wherein the sampling and processing includes at least one of a first sampling and processing based on driving distance, a second sampling and processing based on steering information, and a third sampling and processing based on position information; Constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group; The sensor data includes the heading angle corresponding to each sample video frame of the sample video and the speed information corresponding to each sample video frame of the sample video; the sampling and processing of the sample video based on the sensor data corresponding to the sample video includes: determining the turning video segments whose corresponding heading angles are greater than a preset heading angle threshold; determining the turning movement distance corresponding to each turning video segment based on the speed information corresponding to the turning video frames and the time interval between the turning video frames; and when the turning movement distance exceeds a second preset distance, sampling and processing the each turning video segment according to a preset first sampling interval.

2. The method according to claim 1, characterized in that The sensor data includes speed information corresponding to each sample video frame of the sample video; The sampling process of the sample video based on the sensor data corresponding to the sample video includes: Starting from the first frame of the sample video frames, based on the speed information corresponding to each sample video frame and the time interval between each sample video frame, determining a first target video frame whose travel distance from the first frame is not less than a first preset distance; The next video frame of the target video frame is re-determined as the first video frame, and the step of determining the target video frame is returned to be executed; wherein the target video frame is the video frame of the sampled image group obtained after the first sampling process.

3. The method according to claim 2, characterized in that The target video frame includes a high-speed video frame and a low-speed video frame; The method further comprises: determining a target video frame whose corresponding speed information is less than a preset speed threshold as the low-speed video frame, and determining a target video frame whose corresponding speed information is not less than the preset speed threshold as the high-speed video frame; The step of constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group includes: Based on the first construction ratio corresponding to the high-speed video frame and the second construction ratio corresponding to the low-speed video frame, a first video frame is screened out from the target video frame, and a data set corresponding to the sample video is constructed based on the first video frame.

4. The method according to claim 2, characterized in that The sensor data includes position information corresponding to each sample video frame of the sample video; The sampling process of the sample video based on the sensor data corresponding to the sample video includes: Determining at least one sample video segment shot in the target area based on the position information corresponding to each sample video frame; Sampling is performed on each sample video segment according to a preset second sampling interval.

5. The method according to claim 4, characterized in that The target area includes an intersection area; After sampling each sample video segment according to the preset second sampling interval, the method further includes: Identify the sampled regional video frames and determine the region type corresponding to each regional video frame based on the identification result; The step of constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group includes: Based on the construction ratios corresponding to the respective region types, a second video frame is screened out from the region video frame, and a data set corresponding to the sample video is constructed based on the second video frame.

6. The method according to any one of claims 1 to 5, characterized in that: The step of constructing a data set corresponding to the sample video based on the sample video frames included in the sample image group includes: Determining the embedded representation vectors corresponding to the sample video frames included in the sample image group; Based on the embedded representation vectors corresponding to the sample video frames, the sample video frames in the sample image group are deduplicated, and the sample image group after deduplication is used as the data set corresponding to the sample video.

7. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The neural network to be trained is trained based on the data set corresponding to the sample video.

8. A data set construction device, characterized in that: include: An acquisition module, configured to acquire at least one sample video and corresponding sensor data when the sample video is acquired; a sampling module configured to perform sampling processing on the sample videos based on sensor data corresponding to the sample videos to obtain a sample image group; wherein the sampling processing includes at least one of a first sampling processing based on driving distance, a second sampling processing based on steering information, and a third sampling processing based on position information; A construction module, configured to construct a data set corresponding to the sample video based on the sample video frames included in the sample image group; The sensor data includes the heading angle corresponding to each sample video frame of the sample video and the speed information corresponding to each sample video frame of the sample video; the sampling module is specifically used to: determine the turning video segments whose corresponding heading angles are greater than a preset heading angle threshold; determine the turning movement distance corresponding to each turning video segment based on the speed information corresponding to the turning video frames and the time interval between the turning video frames; when the turning movement distance exceeds a second preset distance, sample and process the turning video segments according to a preset first sampling interval.

9. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor. When the computer device is running, the processor and the memory communicate via the bus. When the machine-readable instructions are executed by the processor, the steps of the data set construction method according to any one of claims 1 to 7 are performed.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, which, when executed by a processor, executes the steps of the data set construction method according to any one of claims 1 to 7.

Images