Scene data set construction method and device, equipment and storage medium

Abstract

The invention relates to the technical field of data processing, and particularly provides a scene data set construction method and device, equipment and a storage medium, and the method comprises the steps: obtaining at least two environment description parameters which are used for describing information of corresponding dimensions in an environment scene; screening out a target video frame set matched with the at least two target environment description parameters from a pre-stored video frame database; and based on a target video frame identifier in the target video frame data set, aggregating video frames corresponding to the target video frame identifier to obtain a target scene data set, the target scene data set comprising at least one target video clip, and the target video clip being obtained by aggregating the target video frames according to a timestamp. The accuracy of real scene data is improved, and the accuracy of a subsequent simulated data scene based on a real scene is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a method, device, equipment and storage medium for constructing a scene data set. Background Art

[0002] With the rapid development of intelligence, the Internet of Vehicles is becoming the focus of the automotive industry. Autonomous driving is the core of the Internet of Vehicles. Autonomous driving mainly includes technologies such as artificial intelligence, computer vision, lidar, millimeter-wave radar, and high-precision maps. By sensing the surrounding environment, driving decisions can be made in real time to achieve the purpose of autonomous driving. The development and implementation of intelligent driving requires a lot of testing and verification. It is necessary to evaluate the performance and safety of intelligent driving under various interference conditions, including roads, traffic participants, weather, lighting and other complex factors. As a result, the actual road test has high time cost and low efficiency. There are some dangers in some complex weather or extreme scenarios. The scene data in the scene library can easily and efficiently generate simulation scenes in a variety of ways. Through real scenes, building virtual scenes can realize closed-loop experiments of autonomous driving perception, decision planning and control algorithms, thereby meeting the needs of autonomous driving testing.

[0003] Most of the scene data in the scene library comes from real data, simulated data, and synthetic data. Real data refers to the feature extraction and annotation of the collected sensor data to identify static or dynamic behaviors. Simulated data refers to the data simulated by the constructed virtual environment. Synthetic data refers to the establishment of various scene data by scene transformation and scene generation from real data and simulated data. The main process of synthetic data includes: labeling environmental parameters from the collected real data, and classifying and organizing the data according to different scene characteristics through labeling. Labels can include: environmental parameters of multiple dimensions such as weather, roads, traffic, lane lines, and lighting. Environmental parameters are usually generated by manual labeling and AI labeling. For example, the label for the time period from the start time 1724815690 to 1724815740 is rainy day. Each data packet will have multiple environmental parameters. If a scene video of lane lines on a rainy day is needed, after retrieving all labels containing lane lines and rainy days in all scene labels, the video clips corresponding to the two labels are aggregated to obtain a scene video. After the above aggregation, a specific scene set such as lane lines on a rainy day is generated.

[0004] Due to the labeling of time intervals, the scene video obtained after aggregating video clips may be intermittent, etc. Although the scene video contains the rainy lane line scene, it does not satisfy that every frame is rainy and lane lines. In the case of multiple scenes or more complex scene fusion, it is impossible to obtain an intersection segment of all scenes. The scene video is not accurate enough, resulting in the complex scene data after synthesis not achieving the expected effect and failing to meet the test requirements. Summary of the invention

[0005] In order to solve the above technical problems, the present application provides a method, device, equipment and storage medium for constructing a scene data set, which improves the accuracy of real scene data and improves the accuracy of subsequent data scenes simulated based on real scenes.

[0006] In a first aspect, the present application provides a method for constructing a scene dataset, the method comprising: obtaining at least two environment description parameters, the environment description parameters being used to describe information of corresponding dimensions in an environment scene; screening out a set of target video frames that match at least two target environment description parameters from a pre-stored video frame database, the video frame database comprising a correspondence between multiple sets of video frame identifiers and environment parameters, the target video frame data set comprising multiple target video frame identifiers, the environment parameters corresponding to the target video frame identifiers being the same as at least two target environment description parameters; based on the target video frame identifiers in the target video frame data set, aggregating the video frames corresponding to the target video frame identifiers to obtain a target scene dataset, the target scene dataset comprising at least one target video segment, the target video segment being obtained by aggregating the target video frames according to timestamps.

[0007] In a second aspect, the present application provides a device for constructing a scene data set, the device comprising: an acquisition module, used to acquire at least two environment description parameters, wherein the environment description parameters are used to describe information of corresponding dimensions in an environment scene; a screening module, used to screen out a target video frame set that matches the at least two target environment description parameters from a pre-stored video frame database, wherein the video frame database includes a correspondence between multiple groups of video frame identifiers and environment parameters, and the target video frame data set includes multiple target video frame identifiers, and the environment parameters corresponding to the target video frame identifiers are the same as the at least two target environment description parameters; a construction module, used to aggregate the video frames corresponding to the target video frame identifiers based on the target video frame identifiers in the target video frame data set to obtain a target scene data set, wherein the target scene data set includes at least one video clip, and the video clip is obtained by aggregating the target video frames according to timestamps.

[0008] In a third aspect, the present application provides a device for constructing a scene dataset, the device comprising: one or more processors; a storage device for storing one or more programs; when the one or more programs are executed by one or more processors, the one or more processors implement the method for constructing a scene dataset as described in the first aspect above.

[0009] In a fourth aspect, the present application provides a storage medium, which may be a computer-readable storage medium, storing a computer program thereon, which, when executed by a processor, implements the method for constructing a scene data set as in the first aspect above.

[0010] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program or instructions, and when the computer program or instructions are executed by a processor, the method for constructing a scene data set as described in any one of the first aspects above is implemented.

[0011] Compared with the prior art, the technical solution provided by the embodiments of the present application has the following advantages:

[0012] The embodiment of the present application provides a method, device, equipment and storage medium for constructing a scene data set, the method comprising: obtaining at least two environment description parameters, the environment description parameters are used to describe the information of the corresponding dimension in the environment scene; screening out multiple target video frame sets that match at least two target environment description parameters from a pre-stored video frame database, the video frame database includes a correspondence between multiple sets of video frame identifiers and environment parameters, the target video frame data set includes multiple target video frame identifiers, and the environment parameters corresponding to the target video frame identifiers are the same as the at least two target environment description parameters; based on the target video frame identifiers in the target video frame data set, the video frames corresponding to the target video frame identifiers are aggregated to obtain a target scene data set, the target scene data set includes at least one target video clip, and the target video clip is obtained by aggregating the target video frames according to the timestamp. In the embodiment of the present application, by matching the video frames that match the environment description parameters, the matching video frames are aggregated to obtain the basic video clips of the test scene, avoiding the problems of intermittent and inaccurate video clips in the related art, improving the accuracy of the real scene data, and improving the accuracy of the data scene subsequently simulated based on the real scene. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present application and, together with the description, serve to explain the principles of the present application.

[0014] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0015] Figure 1 A schematic diagram of a flow chart of a method for constructing a scene data set provided in an embodiment of the present application;

[0016] Figure 2 A flowchart of a method for determining a target video frame identifier provided in an embodiment of the present application;

[0017] Figure 3 A flowchart of another method for determining a target video frame identifier provided in an embodiment of the present application;

[0018] Figure 4 A flowchart of a method for constructing a video database provided in an embodiment of the present application;

[0019] Figure 5 A schematic diagram of the structure of a device for constructing a scene data set provided in an embodiment of the present application;

[0020] Figure 6 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to more clearly understand the above-mentioned purposes, features and advantages of the present application, the scheme of the present application will be further described below. It should be noted that the embodiments of the present application and the features in the embodiments can be combined with each other without conflict.

[0022] In the following description, many specific details are set forth to facilitate a full understanding of the present application, but the present application may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only part of the embodiments of the present application, rather than all of the embodiments.

[0023] The term "including" and its variations used herein are open inclusions, i.e., "including but not limited to". The term "based on" means "based at least in part on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". The relevant definitions of other terms will be given in the following description.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in this application are only used to distinguish different devices, modules or units, and are not used to limit the order or interdependence of the functions performed by these devices, modules or units.

[0025] It should be noted that the modifications of "one" and "plurality" mentioned in the present application are illustrative rather than restrictive, and those skilled in the art should understand that unless otherwise clearly indicated in the context, it should be understood as "one or more".

[0026] The following is a detailed description of the method for constructing the scene data set provided by the present application in conjunction with the accompanying drawings and specific implementation methods.

[0027] Figure 1 This is a flowchart of a method for constructing a scene data set in an embodiment of the present application. This embodiment is applicable to the case of constructing a scene database. The method can be executed by a scene data set construction device. The scene data set construction device can be implemented in software and / or hardware. The scene data set construction device can be configured in an electronic device.

[0028] like Figure 1 As shown, the method for constructing a scene data set provided in the embodiment of the present application mainly includes steps S101-S103.

[0029] S101. Obtain at least two environment description parameters, where the environment description parameters are used to describe information of corresponding dimensions in an environment scene.

[0030] Among them, the scene data in the scene data set constructed by the embodiment of the present application can generate a test scene, which can generally be used to test an intelligent driving algorithm, and the intelligent driving algorithm can be installed on mobile phones, computers, intelligent voice interaction devices, smart home appliances, vehicle-mounted terminals, etc.

[0031] The environment description parameters are used to describe the environment scene. Taking the construction of a scene data set for simulating a simulation environment as an example, the scene data set can be used to build a simulation environment for testing the algorithms installed in autonomous driving vehicles, and the environment description parameters are used to describe the above simulation environment.

[0032] In one embodiment, the above-mentioned environment description parameters include at least one of the following: road topology description parameters, road facilities and rule parameters, temporary control and event parameters, environmental condition parameters, and digital information parameters. The acquisition process of the environment description parameters reflects the modeling results of the environment scene, that is, by setting the road topology description parameters, road facilities and rule parameters, temporary control and event parameters, environmental condition parameters, and digital information parameters, an environment scene that meets the user's needs can be obtained.

[0033] Specifically, the road topology description parameters include at least one of the following: a road curvature type parameter and a lane number parameter. The road curvature type can describe the curvature of the road, such as whether the road is a straight road, a curve, or a roundabout. The lane number parameter is used to describe the number of lanes in the road, such as four lanes, six lanes, or eight lanes.

[0034] The above-mentioned road facility rule parameters include at least one of the following: lane line parameters, toll station parameters, and bus stop parameters. In the embodiment of the present application, the lane line parameters are used to describe the shape and / or color of the lane line, for example, whether it is a single line, a double line, a dashed line or a solid line, whether the lane line is white or yellow, etc. In the embodiment of the present application, the toll station parameters can be used to describe the number and / or location of toll stations, and the above-mentioned bus stop parameters can be used to describe the number and / or location of bus stops.

[0035] The above temporary control and event parameters include at least one of the following: traffic control parameters, traffic accident parameters, such as temporary traffic control on a certain road section, a traffic accident in a certain place, etc.

[0036] The above-mentioned environmental condition parameters include at least one of the following: time parameters, weather parameters, and lighting parameters. For example, the time parameter characterizes the time in the test scene, such as whether it is day or night, etc., the weather parameter characterizes the weather in the test scene, such as whether it is sunny, rainy, foggy, snowy or cloudy, etc., and the lighting parameter is used to characterize the light intensity in the test scene. The above-mentioned digital information parameters include at least one of the following second-layer environmental parameters: roadside unit location parameters, roadside unit information description parameters. In the embodiment of the present application, the digital information parameters mainly describe the information sent by the V2X device, such as OBU, RSU and other devices. Specifically, this information may include the location of the V2X or other information sent by the V2X device.

[0037] The at least two environment description parameters may be two or more description parameters of the above environment parameters.

[0038] Acquiring at least two environment description parameters includes: displaying the configurable environment description parameters in the terminal interface, and in response to the user's configuration operation on the environment description parameters, acquiring at least two environment description parameters corresponding to the configuration operation. The at least two environment description parameters in the embodiment of the present application can be determined by the user's configuration, and are not specifically limited in the embodiment of the present application.

[0039] S102. Filter out a plurality of target video frame sets that match at least two target environment description parameters from a pre-stored video frame database, wherein the video frame database includes a correspondence between a plurality of sets of video frame identifiers and environment parameters, and the target video frame data set includes a plurality of target video frame identifiers, and the environment parameters corresponding to the target video frame identifiers are the same as at least two target environment description parameters.

[0040] Among them, the video frame can be understood as a video frame obtained by extracting frames from the actual captured video. Optionally, the above-mentioned video frame may include a key frame. The video frame identifier can be understood as data that uniquely identifies the identity of the video frame. Exemplarily, the video frame identifier can be the timestamp of the video frame. Environmental parameters can be understood as information in multiple dimensions of the scene displayed in the video frame. The corresponding relationship between the video frame identifier and the environmental parameter refers to the relationship between the video frame identifier and the scene parameters of the scene displayed in the video frame it indicates.

[0041] Table 1 shows the correspondence between two sets of video frame identifiers and environmental parameters. As shown in Table 1, both sets of correspondence are collected by the vehicle with license plate number 590. The scene parameters corresponding to the video frame identifier 1723186679 are cloudy, curved, dim, [cone barrel], and solid line. The scene parameters corresponding to the video frame identifier 1723186680 are cloudy, curved, dim, [cone barrel], and solid line. It should be noted that the two sets of correspondence are used as an example in the embodiment of the present application, and there are multiple sets of correspondence in the video frame database.

[0042] Table 1

[0043] id Vehicle number Video frame identification weather the way light obstacle Lane lines 1 590 1723186679 cloudy day Bend Dark [Cone barrel] Solid Line 2 590 1723186680 cloudy day Bend Dark [Cone barrel] Solid Line

[0044] A search is performed in a video frame database based on at least two target environment description parameters, a scene parameter identical to the at least two target environment description parameters is searched in the video frame database, and a video frame corresponding to the scene parameter is used as a target video frame.

[0045] For example, if at least two environmental description parameters include cloudy weather, curved roads, and dim lights, then the video frame database is searched for scene parameters that are the same as cloudy weather, curved roads, and dim lights, and the video frames corresponding to the scene parameters of cloudy, curved roads, and dim lights are used as target video frames. All target video frame identifiers retrieved from the video frame database are combined together to form a target video frame set.

[0046] Furthermore, the target video frame set may also include a target environment description parameter that corresponds to the target video frame identifier.

[0047] S103. Based on the target video frame identifier in the target video frame data set, the video frames corresponding to the target video frame identifier are aggregated to obtain a target scene data set, wherein the target scene data set includes at least one target video segment, and the target video segment is obtained by aggregating the target video frames according to timestamps.

[0048] Video frame aggregation refers to combining multiple video frames arranged in a certain order into a video clip. The video frame identifier can be a timestamp of the video frame, and aggregating the video frames corresponding to the target video frame identifier includes: aggregating the video frames corresponding to the target video frame identifier in the order of the timestamps.

[0049] The target scene data set includes at least one target video clip, and the scene displayed by the target video clip is a scene described by at least two environment description parameters.

[0050] In one possible implementation, based on the target video frame identifier in the target video frame data set, the video frames corresponding to the target video frame identifier are aggregated to obtain a target scene data set, including: determining the vehicle identifier to which the target video frame identifier in the target video frame data set belongs; dividing the target video frame identifiers with the same vehicle identifier into a group; and for each group of target video frame identifiers, aggregating the video frames corresponding to the target video frame identifier to obtain a target scene data set.

[0051] The vehicle identification may be understood as information identifying the vehicle, which may be defined by the user, or may use other identity information of the vehicle, such as the vehicle identification code, the engine number of the vehicle, the vehicle license plate number, and the like.

[0052] The same vehicle identification indicates that they are the same vehicle. The target video frame identifications with the same vehicle identification are grouped together, and it can be understood that all the video frames corresponding to the target video frame identifications in the group are collected by the same vehicle.

[0053] Specifically, the vehicle identification to which each video frame belongs is pre-stored in the video frame database. After the target video frame identification is determined, the vehicle identification to which the target video frame identification belongs is directly read from the video frame database.

[0054] In one possible implementation, Figure 2 As shown in the figure, “aggregating the video frames corresponding to the target video frame identifier to obtain the target scene dataset” can be optimized to include the following steps:

[0055] S201 , sorting the timestamps of target video frames in ascending order to form a time sequence.

[0056] In the embodiment of the present application, the video frame identifier is the timestamp of the video frame. Sorting the timestamps in ascending order can be understood as sorting the timestamps in ascending order, and a time sequence composed of the timestamps can be obtained.

[0057] In a possible implementation, after the target video frames in the target video frame data set are grouped according to the vehicle identifications to which they belong, the vehicle identifications to which the target video frames included in each group belong are the same. According to the groups, the timestamps of the target video frames in each group are sorted in ascending order, and a time series composed of timestamps can be obtained.

[0058] In other words, each vehicle identification has a corresponding time series, and the time series is composed of the time stamps of the target video frames corresponding to the vehicle identification. The process of S202-S204 is executed for the time series corresponding to each vehicle identification.

[0059] S202: Select a subsequence that meets a set condition from the time sequence, where the subsequence that meets the set condition includes: a difference between an end timestamp of the subsequence and a start timestamp of the subsequence is greater than a first set threshold.

[0060] As can be seen from the above, a time series is composed of a series of timestamps. The first timestamp in a time series is called the start timestamp, and the last timestamp in a time series is called the end timestamp. A subsequence is a sequence of timestamps that meet the set conditions intercepted from a time series. From a time series, one subsequence that meets the set conditions can be intercepted, or multiple subsequences that meet the set conditions can be intercepted.

[0061] The first set threshold value can be understood as a value measuring the length of the subsequence. In other words, the first set threshold value can be understood as a value measuring the number of timestamps included in the subsequence. The difference between the end timestamp of the subsequence and the start timestamp of the subsequence is greater than the first set threshold value, which means that the subsequence includes at least the first set threshold number of video frames.

[0062] In the embodiment of the present application, the number of video frames included in the subsequence is limited, which can ensure the length of the target video segment and improve the accuracy of the scene video.

[0063] In a possible implementation, the subsequence of the set condition further includes: a difference between any two adjacent timestamps in the subsequence is less than a second set threshold, and the first set threshold is greater than the second set threshold.

[0064] The second set threshold can be understood as data that measures the difference between two adjacent timestamps, where the latter timestamp of the two adjacent timestamps increases by 1 compared to the former timestamp. In other words, the second set threshold is a value used to measure the number of frame drops between video frames. The second set threshold can be set by the staff according to actual conditions. For example: in a video clip, the number of frame drops allowed is 2, and the latter timestamp of the two adjacent timestamps increases by 1 compared to the former timestamp. Then, the second set threshold is 3. If the difference between the latter timestamp and the former timestamp in the adjacent video frames is 3, it means that 2 video frames are lost between the latter timestamp and the former timestamp. At this time, the pre-set frame drop setting is met and the set conditions are satisfied.

[0065] In a possible implementation, the first set threshold value can be adjusted according to the number of lost frames in the subsequence. For example, if the subsequence is pre-set to include 5 video frames, and there is no frame loss in the timestamps included in the subsequence, the first set threshold value can be 5. If frame loss is detected in the subsequence during the detection process, the first set threshold value is the sum of the pre-set number of video frames and the number of lost frames. If the number of lost frames in the subsequence is 2, the first set number is dynamically adjusted to 7.

[0066] S203: Aggregate the video frames corresponding to the timestamps included in the subsequence to obtain a target video segment.

[0067] The subsequence includes multiple timestamps arranged in ascending order, and the video frames corresponding to the timestamps are aggregated according to the order of the timestamps to obtain a target video segment.

[0068] S204: Add the target video clip to the target scene dataset.

[0069] The target video clip can be used as a basic scene data. The target video clip can be added to the target scene data set to provide data support for the subsequent synthesis of complex scene data.

[0070] For example, when the scene description parameters are rainy days and lane lines, a specific rainy day lane line scene set is generated through the above aggregation. Based on the rainy day lane line scene dataset, multiple scene sets such as rainy day lane lines at night, rainy day lane lines during the day, and cloudy day lane lines can also be generated. Finally, a variety of complex and difficult datasets are provided for annotation, training, simulation and other links.

[0071] In one possible implementation, Figure 3 As shown in the figure, “selecting a subsequence that meets the set conditions from a time series” can be optimized to include the following steps:

[0072] S301 , sorting the timestamps of target video frames in ascending order to form a time sequence.

[0073] The process of S301 provided in the embodiment of the present application is the same as that of S201 provided in the above embodiment. For details, please refer to the description in the above embodiment, which will not be repeated in the embodiment of the present application.

[0074] S302: Use the start timestamp in the time series as the first timestamp.

[0075] The time series is t1, t2, t3, ..., tn, and there may be time discontinuities in the time series. Among them, t1 is the start timestamp in the time series, and tn is the end timestamp in the time series.

[0076] In the embodiment of the present application, the start timestamp t1 in the time series is used as the first timestamp tj.

[0077] S303: Starting from the first timestamp, calculate the differences between adjacent timestamps in sequence.

[0078] The first timestamp tj is taken as the previous timestamp in the adjacent timestamps, the next timestamp after the previous timestamp tj is ti, and the difference Δt1 between the next timestamp ti and the previous timestamp tj is calculated, Δt1 = ti-tj.

[0079] S304: Determine whether the difference between adjacent timestamps is greater than a second set threshold, if not, execute S305, if yes, execute S309.

[0080] It is determined whether the difference Δt1 between adjacent timestamps is greater than a second set threshold. If the difference Δt1 between adjacent timestamps is less than or equal to the second set threshold, it indicates that the preset allowed frame loss number is satisfied between adjacent timestamps, and then S305 is executed.

[0081] If the difference Δt1 between adjacent timestamps is greater than the second set threshold, it indicates that the pre-set number of allowed frame losses is not met between adjacent timestamps. At this time, it is necessary to use the next timestamp as the first timestamp, that is, the next timestamp as the starting timestamp of the subsequence, and restart the subsequence selection step.

[0082] S305: If the difference between the adjacent timestamps is less than or equal to the second set threshold, calculate the difference between the latter timestamp and the first timestamp in the adjacent timestamps.

[0083] If the difference Δt1 between adjacent timestamps is less than or equal to the second set threshold, it indicates that the preset number of allowed frame losses is satisfied between adjacent timestamps. In this case, the difference between the latter timestamp and the first timestamp is calculated.

[0084] Assuming that the next timestamp is tk, the difference Δt2 between the next timestamp tk and the first timestamp tj is calculated, Δt2=tk-tj.

[0085] S306: Determine whether the difference between the latter timestamp and the first timestamp in adjacent timestamps is equal to or greater than a first set threshold. If yes, execute S307; if not, return to the step of calculating the difference between adjacent timestamps.

[0086] It is determined whether the difference Δt2 between the subsequent timestamp and the first timestamp is equal to or greater than the first set threshold. If the difference Δt2 between the subsequent timestamp and the first timestamp is equal to or greater than the first set threshold, it indicates that the interval between the subsequent timestamp and the first timestamp meets the requirement for the number of video frames in the subsequence.

[0087] S307: If the difference between the subsequent timestamp and the first timestamp is equal to or greater than a first set threshold, multiple timestamps included between the first timestamp and the subsequent timestamp are taken as a subsequence.

[0088] If the difference Δt2 between the subsequent timestamp and the first timestamp is equal to or greater than the first set threshold, it indicates that the interval between the subsequent timestamp and the first timestamp meets the requirement of the number of video frames in the subsequence. The multiple timestamps included between the first timestamp tj and the subsequent timestamp tk are taken as the subsequence [tj, tk].

[0089] S308: Take the timestamp sorted after the next timestamp as the first timestamp, and return to execute S303.

[0090] The timestamp after the next timestamp is taken as the first timestamp, that is, the timestamp after the next timestamp is taken as the start timestamp of the subsequence, and the subsequence selection step is restarted.

[0091] S309: If the difference between the adjacent timestamps is greater than the second set threshold, use the later timestamp among the adjacent timestamps as the first timestamp, and return to S303.

[0092] If the difference Δt1 between adjacent timestamps is greater than the second set threshold, it indicates that the pre-set number of allowed frame loss between adjacent timestamps is not met. If the difference Δt1 between adjacent timestamps is greater than the second set threshold, it indicates that the pre-set number of allowed frame loss between adjacent timestamps is not met. At this time, it is necessary to use the latter timestamp as the first timestamp, that is, use the latter timestamp as the starting timestamp of the subsequence, and restart the subsequence selection step.

[0093] S310: Aggregate video frames corresponding to timestamps included in the subsequence to obtain a target video segment.

[0094] S311, adding the target video clip to the target scene dataset.

[0095] The process of steps S310-S311 in the embodiment of the present application is the same as that of steps S203-S204 in the above embodiment. For details, please refer to the description in the above embodiment, and the embodiment of the present application will not be repeated.

[0096] The embodiment of the present application provides a method for constructing a scene data set, the method comprising: obtaining at least two environment description parameters, the environment description parameters are used to describe the information of the corresponding dimension in the environment scene; screening out a plurality of target video frame sets matching at least two target environment description parameters from a pre-stored video frame database, the video frame database including a correspondence between multiple sets of video frame identifiers and environment parameters, the target video frame data set including multiple target video frame identifiers, the environment parameters corresponding to the target video frame identifiers are the same as the at least two target environment description parameters; based on the target video frame identifiers in the target video frame data set, aggregating the video frames corresponding to the target video frame identifiers to obtain a target scene data set, the target scene data set including at least one target video clip, the target video clip being obtained by aggregating the target video frames according to timestamps. In the embodiment of the present application, by using video frames matching the environment description parameters, the matching video frames are aggregated to obtain the basic video clips of the test scene, thereby avoiding the problems of overlapping, intermittent, and inaccurate video clips in the related art, improving the accuracy of the real scene data, and improving the accuracy of the subsequent data scenes simulated based on the real scene.

[0097] In an exemplary implementation, taking the construction of a dataset of scenes with dim lighting on a cloudy day as an example, the duration of each target video segment is 5, and the data can be interrupted for 3 seconds.

[0098] The extracted target video frame set is shown in Table 2, including video frames with timestamps of 17231896679, 17231896680, 17231896681, 17231896682, 17231896685, 17231896686, 17231896687, and 17231896688. The scene parameters corresponding to each video frame are all cloudy and dim.

[0099] Table 2

[0100] 17231896679 Cloudy day with dim lights 17231896680 Cloudy day with dim lights 17231896681 Cloudy day with dim lights 17231896682 Cloudy day with dim lights 17231896685 Cloudy day with dim lights 17231896686 Cloudy day with dim lights 17231896687 Cloudy day with dim lights 17231896688 Cloudy day with dim lights

[0101] As shown in Table 2, a 2-second frame is lost after the timestamp 1723186682 in this set. Start from the initial 1723186679 corresponding to t1 in the above formula, and search backwards in sequence. When there is time discontinuity, calculate the difference time frame according to the above formula Δt1=tj-ti. When the difference time frame is less than the set allowed frame break of 3 seconds, continue to search backwards in sequence. Stop when this time segment meets 5 seconds, and then start based on the stop time as t1 and so on. When the remaining time does not meet the segment length, it is discarded. The result is scene data {[1723186679, 1723186684]}.

[0102] In the embodiment of the present application, the aggregated target video clips are more accurate and focused. Each frame contains all scenes. Users can also freely set composition rules. The new data set data generated based on the above scene data is more accurate, which improves the efficiency of annotation, training, and simulation.

[0103] Based on the above embodiments, the present application provides a method for constructing a video frame database, such as Figure 4 As shown, the method for constructing a video frame database provided in the embodiment of the present application mainly includes S401-S404.

[0104] S401. Acquire real scene data, where the real scene data includes a real scene video, at least one environmental parameter, and a time interval corresponding to the environmental parameter.

[0105] The real scene video may refer to a scene video collected in an actual driving environment. The environment parameter refers to a scene parameter corresponding to the scene displayed by the real scene video.

[0106] Because the scene parameters of the video clips corresponding to different time intervals in a real scene video may be different. For example: when the vehicle changes from straight driving to curved driving, the road parameters in the corresponding scene parameters are different. Another example: when the vehicle is driving, the weather changes from sunny to rainy, and the weather parameters in the corresponding scene parameters are different. The scene parameters in the real scene video are marked according to the time interval. For example: 10-60 seconds is straight driving, and 60-90 seconds is curved driving. Therefore, after obtaining the environmental parameters, it is also necessary to obtain which interval in the real scene video the environmental parameters are for.

[0107] S402: Perform frame extraction processing on the real scene video to obtain at least one video frame.

[0108] Set the frame extraction frequency according to the requirements. For example, extract one frame per second (FPS is 1), or extract one frame every few frames. Traverse each frame of the real scene video and decide whether to save the current video frame according to the set frame extraction frequency. If the set frame extraction frequency is met, save the current video frame.

[0109] In a possible implementation, key frames in real scene videos are extracted as the final extracted video frames. Specifically, all video frames are regarded as points in a multidimensional space, and all video frames are divided into several groups through a clustering algorithm, and then the central frame of each group is selected as a key frame. Alternatively, the differences between adjacent frames are compared, and those frames with large changes are selected as key frames, or sparse coding technology is used to find a small part of frames that can best represent the entire video as key frames.

[0110] S403: For each video frame, obtain the time interval to which the video frame belongs and the environmental parameters corresponding to the time interval.

[0111] S404: Establish a correspondence between the frame identifier of the video frame and the environmental parameter, and store the correspondence in a video frame database.

[0112] The real scene data includes the time interval corresponding to each scene parameter. After extracting the real scene video to obtain a video frame, the time interval to which the video frame belongs is determined, and the scene parameter marked in the time interval is used as the scene parameter of the video frame.

[0113] For example, the scene parameter of 10-60 seconds is a rainy straight line, and the video frame after extraction is the 35th second. Then 35 seconds is between 10-60 seconds. The scene parameter of 10-60 seconds is obtained, and the scene parameter of the rainy straight line is used as the scene parameter of the 35th second video frame.

[0114] In an embodiment of the present application, a video frame database is constructed by extracting frames from a real scene video and annotating the extracted video frames using their corresponding scene parameters, thereby providing basic data for subsequent construction of a scene data set.

[0115] Figure 5 is a schematic diagram of the structure of a scene data set construction device in an embodiment of the present application, such as Figure 5 As shown, the scene data set construction device 50 provided in the embodiment of the present application mainly includes: an acquisition module 51, a screening module 52 and a construction module 53.

[0116] Among them, the acquisition module 51 is used to obtain at least two environment description parameters, which are used to describe the information of corresponding dimensions in the environment scene; the screening module 52 is used to screen out multiple target video frame sets that match at least two target environment description parameters from a pre-stored video frame database, the video frame database includes multiple sets of correspondences between video frame identifiers and environment parameters, the target video frame data set includes multiple target video frame identifiers, and the environment parameters corresponding to the target video frame identifiers are the same as at least two target environment description parameters; the construction module 53 is used to aggregate the video frames corresponding to the target video frame identifiers based on the target video frame identifiers in the target video frame data set to obtain a target scene data set, the target scene data set includes at least one target video segment, and the target video segment is obtained by aggregating the target video frames according to timestamps.

[0117] The embodiment of the present application provides a device for constructing a scene data set, which is used to execute the following process: obtaining at least two environment description parameters, the environment description parameters are used to describe the information of the corresponding dimension in the environment scene; screening out multiple target video frame sets that match at least two target environment description parameters from a pre-stored video frame database, the video frame database includes a correspondence between multiple sets of video frame identifiers and environment parameters, the target video frame data set includes multiple target video frame identifiers, and the environment parameters corresponding to the target video frame identifiers are the same as at least two target environment description parameters; based on the target video frame identifiers in the target video frame data set, the video frames corresponding to the target video frame identifiers are aggregated to obtain a target scene data set, the target scene data set includes at least one target video clip, and the target video clip is obtained by aggregating the target video frames according to the timestamp. In the embodiment of the present application, the basic video clips of the test scene are obtained by aggregating the matched video frames through the video frames that match the environment description parameters, thereby avoiding the problems of overlapping, intermittent, and inaccurate video clips in the related art, improving the accuracy of the real scene data, and improving the accuracy of the data scene subsequently simulated based on the real scene.

[0118] In a possible implementation, the device also includes: a video frame database construction module, which is used to obtain real scene data, the real scene data includes real scene video, at least one environmental parameter and a time interval corresponding to the environmental parameter; frame extraction processing is performed on the real scene video to obtain at least one video frame; for each video frame, the time interval to which the video frame belongs, and the environmental parameter corresponding to the time interval are obtained; a correspondence between the frame identifier of the video frame and the environmental parameter is established, and the correspondence is stored in the video frame database.

[0119] In one possible implementation, based on the target video frame identifier in the target video frame data set, a construction module 53 is specifically used to determine the vehicle identifier to which the target video frame identifier in the target video frame data set belongs; the target video frame identifiers with the same vehicle identifier are divided into a group; for each group of target video frame identifiers, the video frames corresponding to the target video frame identifiers are aggregated to obtain a target scene data set.

[0120] In one possible implementation, the target video frame identifier includes the timestamp of the target video frame; a construction module 53 is specifically used to sort the timestamps of the target video frames in ascending order to form a time series; a subsequence that meets a set condition is selected from the time series, and the subsequence with the set condition includes: the difference between the end timestamp of the subsequence and the start timestamp of the subsequence is greater than a first set threshold; the video frames corresponding to the timestamps included in the subsequence are aggregated to obtain a target video segment; and the target video segment is added to the target scene data set.

[0121] In a possible implementation, the subsequence of the set condition further includes: a difference between any two adjacent timestamps in the subsequence is less than a second set threshold, and the first set threshold is greater than the second set threshold.

[0122] In one possible implementation, the construction module 53 is used to select a subsequence that meets the set conditions from the time series, and is specifically used to take the start timestamp in the time series as the first timestamp; starting from the first timestamp, the difference between adjacent timestamps is calculated in sequence; if the difference between adjacent timestamps is less than or equal to the second set threshold, the difference between the latter timestamp and the first timestamp is calculated; if the difference between the latter timestamp and the first timestamp reaches the first set threshold, the multiple timestamps included between the first timestamp and the latter timestamp are taken as a subsequence; the timestamp sorted after the latter timestamp is taken as the first timestamp, and the step of calculating the difference between adjacent timestamps in sequence starting from the first timestamp is returned to be executed.

[0123] In one possible implementation, the construction module 53 is used to select a subsequence that meets the set conditions from the time series, and is also used to, if the difference between adjacent timestamps is greater than a second set threshold, use the timestamp that is sorted after the next timestamp in the adjacent timestamps as the first timestamp, and return to execute the step of calculating the difference between adjacent timestamps in sequence starting from the first timestamp.

[0124] The scene data set construction device provided in the embodiment of the present application can execute the scene data set construction method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0125] Figure 6is a schematic diagram of the structure of an electronic device provided in this embodiment. The electronic device may include a scene data set construction device, such as Figure 6 As shown, the electronic device 600 includes a processor 610, a memory 620, an input device 630, and an output device 640; the number of the processor 610 in the electronic device can be one or more. Figure 6 A processor 610 is taken as an example; the processor 610, the memory 620, the input device 630 and the output device 640 in the electronic device can be connected via a bus or other means. Figure 6 The example of connecting through bus is taken in the following.

[0126] The memory 620, as a computer-readable storage medium, can be used to store software programs, computer executable programs and modules, such as program instructions / modules corresponding to the method for constructing a scene data set in the embodiment of the present invention. The processor 610 executes various functional applications and data processing of the electronic device by running the software programs, instructions and modules stored in the memory 620, that is, implements the method for constructing a scene data set provided in the embodiment of the present invention.

[0127] The memory 620 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system and at least one application required for a function; the data storage area may store data created according to the use of the terminal, etc. In addition, the memory 620 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some instances, the memory 620 may further include a memory remotely arranged relative to the processor 610, and these remote memories may be connected to the electronic device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0128] The input device 630 may be used to receive input digital or character information and generate key signal input related to user settings and function control of the electronic device, and may include a keyboard, a mouse, etc. The output device 640 may include a display device such as a display screen.

[0129] This embodiment also provides a storage medium containing computer executable instructions, which, when executed by a computer processor, are used to implement the method for constructing a scene data set provided in the embodiment of the present invention.

[0130] Of course, the computer executable instructions of a storage medium containing computer executable instructions provided in an embodiment of the present invention are not limited to the operations of the method described above, but can also execute related operations in the method for constructing a scene data set provided in any embodiment of the present invention.

[0131] Through the above description of the implementation methods, the technicians in the relevant field can clearly understand that the present invention can be implemented by means of software and necessary general hardware, and of course it can also be implemented by hardware, but in many cases the former is a better implementation method. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as a computer floppy disk, read-only memory (ROM), random access memory (RAM), flash memory (FLASH), hard disk or optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, server, or network device, etc.) to execute the methods described in each embodiment of the present invention.

[0132] It is worth noting that in the embodiment of the above-mentioned scene data set construction device, the various units and modules included are only divided according to functional logic, but are not limited to the above-mentioned division, as long as the corresponding functions can be achieved; in addition, the specific name of each functional unit is only for the convenience of distinguishing each other, and is not used to limit the scope of protection of the present invention.

[0133] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0134] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments described herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method for constructing a scene dataset, characterized in that: The method comprises: Acquire at least two environment description parameters, where the environment description parameters are used to describe information of corresponding dimensions in an environment scene; A target video frame set matching the at least two target environment description parameters is selected from a pre-stored video frame database, wherein the video frame database includes a correspondence relationship between multiple sets of video frame identifiers and environment parameters, the target video frame data set includes multiple target video frame identifiers, and the environment parameters corresponding to the target video frame identifiers are the same as the at least two target environment description parameters; Based on the target video frame identifier in the target video frame data set, the video frames corresponding to the target video frame identifier are aggregated to obtain a target scene data set, wherein the target scene data set includes at least one target video segment, and the target video segment is obtained by aggregating the target video frames according to timestamps.

2. The method according to claim 1, characterized in that The method further comprises: Acquire real scene data, where the real scene data includes a real scene video, at least one environmental parameter, and a time interval corresponding to the environmental parameter; Performing frame extraction processing on the real scene video to obtain at least one video frame; For each of the video frames, obtaining the time interval to which the video frame belongs and the environmental parameters corresponding to the time interval; A correspondence between the video frame identifier of the video frame and the environmental parameter corresponding to the time interval is established, and the correspondence is stored in the video frame database.

3. The method according to claim 1, characterized in that Based on the target video frame identifier in the target video frame data set, the video frames corresponding to the target video frame identifier are aggregated to obtain a target scene data set, including: Determine the vehicle identification to which the target video frame identification in the target video frame data set belongs; Divide the target video frame identifiers having the same vehicle identifier into a group; For each group of target video frame identifiers, the video frames corresponding to the target video frame identifiers are aggregated to obtain a target scene data set.

4. The method according to claim 3, characterized in that The target video frame identifier includes a timestamp of the target video frame; Aggregating the video frames corresponding to the target video frame identifier to obtain a target scene dataset includes: Sorting the timestamps of the target video frames in ascending order to form a time sequence; Selecting a subsequence satisfying a set condition from the time sequence, wherein the subsequence satisfying the set condition includes that a difference between an end timestamp of the subsequence and a start timestamp of the subsequence is greater than a first set threshold; Aggregating video frames corresponding to timestamps included in the subsequence to obtain a target video segment; The target video segment is added to a target scene dataset.

5. The method according to claim 4, characterized in that The subsequence of the set conditions also includes: a difference between any two adjacent timestamps in the subsequence is less than a second set threshold, and the first set threshold is greater than the second set threshold.

6. The method according to claim 5, characterized in that Selecting a subsequence that meets a set condition from the time series includes: Taking the start timestamp in the time series as the first timestamp; Starting from the first timestamp, the differences between adjacent timestamps are calculated in sequence; If the difference between the adjacent timestamps is less than or equal to the second set threshold, calculating the difference between the latter timestamp in the adjacent timestamps and the first timestamp; If the difference between the subsequent timestamp and the first timestamp reaches the first set threshold, taking multiple timestamps included between the first timestamp and the subsequent timestamp as a subsequence; The timestamp sorted after the latter timestamp is taken as the first timestamp, and the step of calculating the difference between adjacent timestamps in sequence starting from the first timestamp is returned to be executed.

7. The method according to claim 6, characterized in that Also includes: If the difference between adjacent timestamps is greater than the second set threshold, the latter timestamp among the adjacent timestamps is used as the first timestamp, and the process returns to the step of calculating the difference between adjacent timestamps in sequence starting from the first timestamp.

8. A device for constructing a scene data set, characterized in that: The device comprises: An acquisition module, used to acquire at least two environment description parameters, where the environment description parameters are used to describe information of corresponding dimensions in an environment scene; A screening module is used to screen out a target video frame set that matches the at least two target environment description parameters from a pre-stored video frame database, wherein the video frame database includes a correspondence relationship between multiple sets of video frame identifiers and environment parameters, and the target video frame data set includes multiple target video frame identifiers, and the environment parameters corresponding to the target video frame identifiers are the same as the at least two target environment description parameters; A construction module is used to aggregate the video frames corresponding to the target video frame identifier in the target video frame data set based on the target video frame identifier in the target video frame data set to obtain a target scene data set, wherein the target scene data set includes at least one target video segment, and the target video segment is obtained by aggregating the target video frames according to timestamps.

9. A device for constructing a scene data set, characterized in that: The device comprises: one or more processors; A storage device for storing one or more programs; When the one or more programs are executed by the one or more processors, the one or more processors implement the method for constructing a scene dataset as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method for constructing a scene data set as described in any one of claims 1 to 7 is implemented.

Images