Systems, methods, devices, and storage media for determining keyframes

Abstract

This invention discloses a system, method, device, and storage medium for determining keyframes. The system includes: an acquisition module, a data correction module, a first determination module, and a second determination module; wherein the acquisition module, data correction module, and first determination module are sequentially and communicatively connected; the acquisition module and the second determination module are communicatively connected; the acquisition module is used to acquire an original frame sequence and a historical frame sequence; the data correction module is used to correct the original frame sequence to obtain a target frame sequence; the first determination module is used to filter the target frame sequence based on the comparison result of each information entropy with a threshold of the information entropy, and determine a first target keyframe set; the second determination module is used to filter the historical frame sequence based on the target risk level, and determine a second target keyframe set. The technical solution of this invention improves the accuracy of data storage and the accuracy of autonomous driving.

Description

Technical Field

[0001] This invention belongs to the field of autonomous driving technology, and in particular relates to a system, method, device and storage medium for determining key frames. Background Technology

[0002] Currently, autonomous driving storage systems at Level 3 and above require the accumulation of a large amount of data during the early development and later mass production processes. Therefore, it is crucial to effectively utilize storage resources and store different data for different functions.

[0003] However, current autonomous driving storage systems directly store acquired data without correction or filtering, leading to the neglect of data errors and resulting in low reliability of decisions. Furthermore, a large amount of data cannot be stored, and high-quality data cannot be maximized, wasting storage and computing resources. This further reduces the safety of autonomous driving. In addition, current solutions mostly rely on post-hoc methods for addressing potentially hazardous behaviors, failing to predict risks and reducing the iteration efficiency, safety, and comfort of autonomous driving systems. Summary of the Invention

[0004] This invention aims to at least partially address one of the technical problems in related technologies. Therefore, one object of this invention is to provide a system, method, apparatus, and storage medium for determining keyframes.

[0005] To address the aforementioned technical problems, embodiments of the present invention provide the following technical solutions:

[0006] A system for determining keyframes includes:

[0007] The system comprises an acquisition module, a data correction module, a first determination module, and a second determination module; wherein the acquisition module, the data correction module, and the first determination module are sequentially and communicatively connected; and the acquisition module is communicatively connected to the second determination module.

[0008] The acquisition module is used to acquire the original frame sequence and the historical frame sequence;

[0009] The data correction module is used to correct the original frame sequence to obtain the target frame sequence;

[0010] The first determining module is used to filter the target frame sequence based on the comparison result of each information entropy with the threshold of the information entropy, and determine the first target key frame set;

[0011] The second determining module is used to filter the historical frame sequence based on the target risk level to determine the second target key frame set.

[0012] Optionally, the data correction module includes a labeling unit;

[0013] The annotation unit acquires the original frame sequence;

[0014] The annotation unit filters the original frame sequence to obtain the frame sequence to be corrected;

[0015] The labeling unit determines the target label;

[0016] The annotation unit annotates the frame sequence to be corrected based on the target label to obtain the corrected frame sequence;

[0017] The annotation unit obtains the target frame sequence based on the corrected frame sequence.

[0018] Optionally, the first determining module includes an update distribution unit;

[0019] The update distribution unit obtains the original distribution result of the target frame sequence based on the target frame sequence;

[0020] The updated distribution unit extracts the original distribution results to obtain the target frame to be determined and the remaining distribution results;

[0021] The update distribution unit extracts the target frame sequence to obtain the target frame to be determined and the remaining target frame sequence;

[0022] The update distribution unit traverses each of the remaining target frame sequences to obtain the update distribution result.

[0023] Optionally, the update distribution unit obtains the original distribution result, extracts the i-th target frame to be determined from the original distribution result, and obtains the i-th remaining distribution result;

[0024] The update distribution unit extracts the i-th target frame to be determined from the target frame sequence, wherein the target frame sequence includes L target frames; where L≥i≥1, and L and i are both integers;

[0025] The update distribution unit determines the remaining L-1 target frames as the i-th remaining target frame sequence;

[0026] The update distribution unit traverses the i-th remaining target frame sequence to obtain the i-th update distribution result corresponding to the i-th target frame to be determined.

[0027] Optionally, the first determining module further includes a calculation unit;

[0028] The calculation unit obtains the remaining distribution result and the updated distribution result for each of the target frames to be determined;

[0029] The calculation unit calculates the remaining distribution results and the updated distribution results to obtain the information entropy of each target frame to be determined;

[0030] The calculation unit compares each information entropy with a threshold of the information entropy; if the information entropy is greater than the threshold of the information entropy, the target frame to be determined that matches the information entropy is determined as the first target key frame.

[0031] Optionally, the computing unit obtains the i-th updated distribution result and the i-th residual distribution result;

[0032] The calculation unit calculates the i-th updated distribution result and the i-th remaining distribution result to obtain the i-th information entropy; if the i-th information entropy is greater than the threshold of the information entropy, then the i-th target frame to be determined corresponding to the i-th information entropy is determined as the first target key frame.

[0033] Optionally, the second determining module includes a classification unit;

[0034] The classification unit acquires multiple historical frame sequences;

[0035] The classification unit extracts features from multiple historical frame sequences to obtain feature data;

[0036] The classification unit clusters the feature data to obtain the target frame sequence;

[0037] The classification unit extracts features from the target frame sequence to obtain multiple target feature frame sequences;

[0038] The classification unit clusters multiple target feature frame sequences to determine the risk level of each target feature frame sequence.

[0039] The classification unit filters the target feature frame sequence based on the target risk level to determine the second target key frame set.

[0040] Embodiments of the present invention also provide a method for determining keyframes, comprising:

[0041] Obtain the original frame sequence and the historical frame sequence;

[0042] The original frame sequence is corrected to obtain the target frame sequence;

[0043] Based on the comparison results of each information entropy with the threshold of the information entropy, the target frame sequence is filtered, and the first target key frame set is determined;

[0044] The historical frame sequence is filtered based on the target risk level to determine the second target key frame set.

[0045] Embodiments of the present invention also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the system described above.

[0046] Embodiments of the present invention also provide a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed, controls the device containing the computer-readable storage medium to execute the system as described above.

[0047] The embodiments of the present invention have the following technical effects:

[0048] The above-mentioned technical solution of the present invention 1) performs full or selective correction on the original frame sequence based on the data correction module to obtain the target frame sequence, thereby improving the accuracy of data storage; in addition, the embodiments of the present invention, based on the first determination module, filters each target frame according to the information entropy and the information entropy threshold to determine the first target key frame set, and can label each first target key frame manually or automatically, further improving the accuracy of data storage, reducing the effective storage resource occupation of data, and improving data quality and the accuracy of the autonomous driving system sub-module.

[0049] 2) Based on the second determination module, the second target key frame set is obtained, which supplements the first target key frame set. When the second target key frame set is sent to the perception device or called by other subsequent algorithms, the calculation accuracy of the perception device is improved, the computing power consumption of the system is reduced, and thus the accuracy, safety and comfort of autonomous driving are improved.

[0050] 3) Based on the comparison module, a third target key frame set is obtained, which supplements the first target key frame set and the second target key frame set, further improving the accuracy of subsequent algorithms called by perception devices, thereby greatly improving the accuracy and safety of autonomous driving.

[0051] Additional aspects and advantages of the invention will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of the invention. Attached Figure Description

[0052] Figure 1 This is a schematic diagram of the structure of a system for determining keyframes provided in an embodiment of the present invention;

[0053] Figure 2 This is an example of the structure of a system for determining keyframes provided in an embodiment of the present invention;

[0054] Figure 3 This is a flowchart illustrating a method for determining keyframes provided in an embodiment of the present invention. Detailed Implementation

[0055] Embodiments of the present invention are described in detail below, examples of which are illustrated in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain the present invention, and should not be construed as limiting the present invention.

[0056] To facilitate understanding of the embodiments by those skilled in the art, some terms are explained in detail:

[0057] (1) CNN: Convolutional Neural Networks.

[0058] (2) Transformer: A transformation model that computes the representation of input and output based on self-attention.

[0059] (3) YOLO: You Only Look Once, an object recognition and localization algorithm based on deep neural networks.

[0060] (4) KL divergence: Kullback-Leibler divergence is an asymmetric measure of the difference between two probability distributions.

[0061] like Figure 1 As shown, an embodiment of the present invention provides a system for determining keyframes, comprising:

[0062] The system comprises an acquisition module, a data correction module, a first determination module, and a second determination module; wherein the acquisition module, the data correction module, and the first determination module are sequentially and communicatively connected; and the acquisition module is communicatively connected to the second determination module.

[0063] The acquisition module is used to acquire the original frame sequence and the historical frame sequence;

[0064] The data correction module is used to correct the original frame sequence to obtain the target frame sequence;

[0065] The first determining module is used to filter the target frame sequence based on the comparison result of each information entropy with the threshold of the information entropy, and determine the first target key frame set;

[0066] The second determining module is used to filter the historical frame sequence based on the target risk level to determine the second target key frame set.

[0067] In embodiments of the present invention, the system, as a subsystem of an autonomous driving system, interacts with other subsystems or devices in other autonomous driving systems via a network; for example, in embodiments of the present invention, the system can interact with perception devices in an autonomous driving system via a network.

[0068] In embodiments of the present invention, the acquisition module and the sensing device are connected via a network. The acquisition module can acquire data monitored by the sensing device in real time and use it to generate an original frame sequence. In addition, in embodiments of the present invention, the acquisition module is also used to obtain historical data of the target vehicle, such as historical trajectory data, and specifically, a historical frame sequence can be generated based on the historical trajectory data.

[0069] The original frame sequence includes, but is not limited to, image format data captured by the camera, matrix data captured by the radar sensor, and driving behavior data generated during the target vehicle's movement.

[0070] Furthermore, the acquisition module includes scene units, which are used to determine the current scene of the target vehicle, including scenes such as bridges, hillsides, rainy days, snowy days, large curvatures, and other vehicles entering the scene, in order to increase the information content of the original frame. The scene units can be implemented based on open-source scene annotation deep learning models, such as YOLO.

[0071] In embodiments of the present invention, both the first target keyframe and the second target keyframe are risk frame data, used to characterize risk data during the autonomous driving process;

[0072] In practical applications, after obtaining the first target keyframe and the second target keyframe, each first target keyframe or second target keyframe is manually or automatically annotated. Then, the set of first target keyframes or the set of second target keyframes formed based on these risk data are stored in the autonomous driving storage system for subsequent algorithm calls, thereby improving the accuracy of subsequent algorithms and the safety of autonomous driving.

[0073] In this embodiment of the invention, the storage system can cover the data of the early test vehicles, as well as the data of the target vehicles and the data generated by the shadow mode after mass production.

[0074] For example, in an embodiment of the present invention, after storing the first target keyframe set and the second target keyframe set, the storage system of the autonomous driving system can be called by the autonomous driving perception device or other subsequent algorithms to provide users with better and safer driving strategies.

[0075] In an optional embodiment of the present invention, the data correction module includes a labeling unit;

[0076] The annotation unit acquires the original frame sequence;

[0077] The annotation unit filters the original frame sequence to obtain the frame sequence to be corrected;

[0078] The labeling unit determines the target label;

[0079] The annotation unit annotates the frame sequence to be corrected based on the target label to obtain the corrected frame sequence;

[0080] The annotation unit obtains the target frame sequence based on the corrected frame sequence.

[0081] In embodiments of the present invention, the annotation unit processes numerical data, image data, and video data respectively;

[0082] Specifically, the first step is to obtain the target model, which includes acquiring a training sample set and training the initial model multiple times based on the training sample set to obtain the target model.

[0083] For obtaining the training sample set, it can be based on the historical data of the target vehicle. The historical data can be scene data within a time period (5 minutes, etc.) or a road segment of a certain length (100 meters, etc.).

[0084] Then, these scene data (roads, bridges, weather, etc.) are segmented into multiple frame sequences, each consisting of 30 frames; these frame sequences are then sequentially input into the initial model, and the annotation results are output sequentially.

[0085] Repeat the above process to train the initial model multiple times, thereby obtaining the target model.

[0086] In practical application scenarios, the annotation unit obtains a preset number of raw frames (e.g., 30, 40 or 50 frames), and then obtains a unique target label based on the target model to uniformly re-annotate these preset number of raw frames, outputting a corresponding corrected frame sequence.

[0087] Therefore, it is possible to re-annotate a portion of the original frame sequence to obtain a corrected frame sequence;

[0088] Similarly, as needed, all original frame sequences can be re-annotated to correct them and obtain corrected frame sequences.

[0089] That is, in the embodiments of the present invention, a portion of the original frame sequence can be determined as the frame sequence to be corrected, while the remaining portion is not processed. For example, a sub-original frame sequence corresponding to a 100-meter road segment can be selected as the frame sequence to be corrected. Then, based on the processing capability of the target model (the number of frames to be corrected labeled each time, such as 30 frames, 40 frames, etc.), the sub-original frame sequence is segmented into segments of 30 frames each, and then input into the target model in chronological order. The target model re-labels each segment of the frame sequence to be corrected based on the unique target label corresponding to each segment. This process is repeated to achieve the correction of the entire road segment's corresponding frame sequence.

[0090] After re-annotating and correcting the frame sequence to be corrected based on the target model, the corrected frame sequence is obtained.

[0091] Based on the corrected frame sequence and the remaining parts mentioned above, the target frame sequence is obtained.

[0092] For example, the initial model mentioned above can be a CNN+transformer model. By training the CNN+transformer model multiple times based on the above steps, the target model can be obtained.

[0093] It should be noted that the specific algorithm for obtaining the target label is not within the scope of protection of this invention and will not be described further.

[0094] In an optional embodiment of the present invention, the first determining module includes an update distribution unit;

[0095] The first determining module includes an update distribution unit;

[0096] The update distribution unit obtains the original distribution result of the target frame sequence based on the target frame sequence;

[0097] The updated distribution unit extracts the original distribution results to obtain the target frame to be determined and the remaining distribution results;

[0098] The update distribution unit extracts the target frame sequence to obtain the target frame to be determined and the remaining target frame sequence;

[0099] The update distribution unit traverses each of the remaining target frame sequences to obtain the update distribution result.

[0100] In embodiments of the present invention, the amount of information contained in each target frame and its impact on other target frames in the entire target frame sequence are determined based on the information entropy of each target frame.

[0101] Therefore, embodiments of the present invention calculate the information entropy of each target frame:

[0102] Specifically, during the calculation process, each target frame in the target frame sequence is identified as the target frame to be determined.

[0103] Then, the target frame to be determined is extracted from the target frame sequence, the distribution unit is updated, and all the remaining target frames are traversed to regenerate a distribution result, specifically by updating the distribution result;

[0104] In addition, in the embodiments of the present invention, the original distribution result is obtained based on the target frame sequence. After extracting the target frame to be determined from the original distribution result, the remaining distribution result is obtained without any other processing of the distribution result of the remaining target frames.

[0105] By following this process, we can obtain the updated distribution result and the remaining distribution result for each target frame to be determined in the target frame sequence, which can then be used in subsequent algorithms.

[0106] In an optional embodiment of the present invention, the update distribution unit obtains the original distribution result, extracts the i-th target frame to be determined from the original distribution result, and obtains the i-th remaining distribution result;

[0107] The update distribution unit extracts the i-th target frame to be determined from the target frame sequence, wherein the target frame sequence includes L target frames; where L≥i≥1, and L and i are both integers;

[0108] The update distribution unit determines the remaining L-1 target frames as the i-th remaining target frame sequence;

[0109] The update distribution unit traverses the i-th remaining target frame sequence to obtain the i-th update distribution result corresponding to the i-th target frame to be determined.

[0110] In an embodiment of the present invention, after the i-th target frame to be determined is extracted from the target frame sequence, a target frame to be determined is obtained. ;

[0111] The update distribution unit iterates through the remaining L-1 target frames and regenerates a corresponding update distribution result. ;

[0112] After extracting the i-th target frame to be determined from the original distribution result, the i-th residual distribution result is obtained. .

[0113] By analogy, we can obtain the updated distribution results and the remaining distribution results for each target frame to be determined, corresponding to the following two-dimensional array:

[0114] ( , ), ( , ), ( , )……( , ).

[0115] In an optional embodiment of the present invention, the first determining module further includes a calculation unit;

[0116] The calculation unit obtains the remaining distribution result and the updated distribution result for each of the target frames to be determined;

[0117] The calculation unit calculates the remaining distribution results and the updated distribution results to obtain the information entropy of each target frame to be determined;

[0118] The calculation unit compares each information entropy with a threshold of the information entropy; if the information entropy is greater than the threshold of the information entropy, the target frame to be determined that matches the information entropy is determined as the first target key frame.

[0119] In embodiments of the present invention, a threshold for information entropy is first preset according to actual needs (this value can be adjusted manually or automatically by the autonomous driving system). The actual value of the threshold for information entropy is not limited in embodiments of the present invention.

[0120] Specifically, in embodiments of the present invention, the information entropy of each target frame to be determined can be calculated based on the following formula:

[0121]

[0122] in, as well as It can be calculated based on the KL divergence principle, that is, by recalculating the KL divergence of the remaining target frame sequence, the corresponding updated distribution result can be obtained;

[0123] By calculating the KL divergence of the target frame sequence, the original distribution result of the target frame sequence can be obtained.

[0124] In an optional embodiment of the present invention, the computing unit obtains the i-th updated distribution result and the i-th residual distribution result;

[0125] The calculation unit calculates the i-th updated distribution result and the i-th remaining distribution result to obtain the i-th information entropy; if the i-th information entropy is greater than the threshold of the information entropy, then the i-th target frame to be determined corresponding to the i-th information entropy is determined as the first target key frame.

[0126] In an embodiment of the present invention, the information entropy of each target frame to be determined in the target frame sequence is calculated, and the following calculation results can be obtained:

[0127] The first information entropy = ;

[0128] The second information entropy = ;

[0129] The third information entropy = ;

[0130] ...

[0131] The entropy of the Lth information = .

[0132] After calculating and obtaining L information entropies, each of the L information entropies is compared with the information entropy threshold. If an information entropy is greater than the information entropy threshold, the target frame to be determined corresponding to that information entropy is determined as the first target key frame.

[0133] By following this logic, by filtering L information entropies based on the information entropy threshold, multiple first target keyframes can be obtained, thus obtaining the first target keyframe set.

[0134] In embodiments of the present invention, the original frame sequence is fully or selectively corrected based on the data correction module to obtain the target frame sequence, thereby improving the accuracy of data storage. In addition, in embodiments of the present invention, each target frame is screened based on the information entropy and the information entropy threshold by the first determination module to determine the first target key frame set, further improving the accuracy of data storage.

[0135] In an optional embodiment of the present invention, the second determining module includes a classification unit;

[0136] The classification unit acquires multiple historical frame sequences;

[0137] The classification unit extracts features from multiple historical frame sequences to obtain feature data;

[0138] The classification unit clusters the feature data to obtain the target frame sequence;

[0139] The classification unit extracts features from the target frame sequence to obtain multiple target feature frame sequences;

[0140] The classification unit clusters multiple target feature frame sequences to determine the risk level of each target feature frame sequence.

[0141] The classification unit filters the target feature frame sequence based on the target risk level to determine the second target key frame set.

[0142] In an embodiment of the present invention, firstly, for a certain road segment, such as a 100-meter road segment, some historical data of the road segment can be obtained. Specifically, it can be the historical trajectory data of the road segment within multiple time periods such as 10 days or 20 days. The historical data includes multiple historical frame sequences.

[0143] Then, feature extraction is performed on these historical frame sequences. Feature extraction can be performed on these historical trajectory data based on multiple dimensions to obtain feature data, such as: the number of times the turning angle is greater than 30 degrees, the number of times the turning angle is greater than 60 degrees, the average speed, the frequency of vehicle head angle changes, or the angle with the road edge, etc.

[0144] For example, feature extraction is performed on multiple historical frame sequences based on the number of times the turning angle is greater than 30 degrees.

[0145] Then, the feature data (number of times the turning angle is greater than 30 degrees) corresponding to each historical frame sequence is obtained in sequence: 1, 2, 3, 5, 6, 1, 2, 3, ... 1.

[0146] The feature data obtained from the above feature extraction is input into the clustering model, and then, based on preset parameters, the following is output:

[0147] Category 1 (number of times the turning angle is greater than 30 degrees ≤ 2 times): 20.

[0148] Category 2 (2 times < number of times with a turning angle greater than 30 degrees ≤ 4 times): 100.

[0149] Category 2 (4 times < number of times with a turning angle greater than 30 degrees ≤ 6 times): 50.

[0150] Therefore, the number of times the turning angle of this road segment is greater than 30 degrees is 3;

[0151] Similarly, repeat the above steps to cluster other features and obtain the target value for each feature. For example, the average speed is 'a', the number of times the turning angle is greater than 60 degrees is 4, the frequency of the vehicle's front angle change is 2, and the angle with the road edge is 30 degrees.

[0152] Based on the target value of each feature, the target trajectory corresponding to that road segment can be obtained.

[0153] Then, feature extraction is performed again on the target trajectory sequence based on a certain preset feature to obtain multiple target feature frame sequences;

[0154] These target feature frame sequences are sequentially input into the clustering model, and the risk level corresponding to each target feature frame sequence is output based on preset parameters. Among them, the risk levels can be preset, including the first risk level, the second risk level, the third risk level, and the fourth risk level, and the screening conditions corresponding to each risk level can be preset.

[0155] Based on the aforementioned preset screening criteria, each target feature frame sequence is screened to determine the risk level of each target feature frame sequence.

[0156] For example, if a sequence of 100 target feature frames is input, the following output result can be obtained:

[0157] First risk level: the first target feature frame sequence... the thirty-sixth target feature frame sequence and 20 other target feature frame sequences.

[0158] Second risk level: the third target feature frame sequence... the seventieth target feature frame sequence and 40 other target feature frame sequences.

[0159] Third risk level: 20 target feature frame sequences, including the sixth target feature frame sequence, the hundredth target feature frame sequence, etc.

[0160] Fourth risk level: 20 target feature frame sequences, including the tenth target feature frame sequence...the ninety-third target feature frame sequence.

[0161] When the target risk level is the fourth risk level, the second target key frame set includes 20 target feature frame sequences, such as the tenth target feature frame sequence to the ninety-third target feature frame sequence. That is, all the target feature frames included in these 20 target feature frame sequences are second target key frames and form the second target key frame set.

[0162] When the target risk level is the third risk level, the second target key frame set includes 20 target feature frame sequences, such as the tenth target feature frame sequence...the ninety-third target feature frame sequence, and 20 target feature frame sequences, such as the sixth target feature frame sequence...the hundredth target feature frame sequence. That is, all the target feature frames included in these 40 target feature frame sequences are second target key frames and form the second target key frame set.

[0163] In the embodiments of the present invention, a second target key frame set is obtained based on the second determining module, which supplements the first target key frame set. When the second target key frame set is sent to the sensing device or called by other subsequent algorithms, the calculation accuracy of the sensing device is improved, thereby improving the accuracy and safety of autonomous driving.

[0164] like Figure 2As shown, in an optional embodiment of the present invention, a comparison module is further included; wherein the comparison module is connected to the acquisition module and the sensing device via a network.

[0165] The comparison module obtains actual trajectory data and predicted trajectory data based on the acquisition module. It obtains actual frame sequence based on actual trajectory data and predicted frame sequence based on predicted trajectory data.

[0166] The comparison module compares the actual frame sequence and the predicted frame sequence. Based on preset parameters, it filters out frames with large differences (the size of the difference can be identified or measured by preset parameters) to form a third target key frame set.

[0167] After obtaining the set of keyframes for the third target, the comparison module can annotate each keyframe for the third target using manual or automatic annotation methods, and then send it to the sensing device.

[0168] In the embodiments of the present invention, a third target keyframe set is obtained based on the comparison module, which supplements the first target keyframe set and the second target keyframe set, further improving the accuracy of subsequent algorithms called by perception devices, thereby greatly improving the accuracy and safety of autonomous driving.

[0169] like Figure 3 As shown, embodiments of the present invention also provide a method for determining keyframes, comprising:

[0170] Step S31: Obtain the original frame sequence and the historical frame sequence;

[0171] Step S32: Correct the original frame sequence to obtain the target frame sequence;

[0172] Specifically, the step of correcting the original frame sequence to obtain the target frame sequence includes:

[0173] Obtain the original frame sequence;

[0174] The original frame sequence is filtered to obtain the frame sequence to be corrected;

[0175] Determine the target label;

[0176] The frame sequence to be corrected is labeled based on the target label to obtain the corrected frame sequence;

[0177] The annotation unit obtains the target frame sequence based on the corrected frame sequence.

[0178] Step S33: Based on the comparison results of each information entropy with the threshold of the information entropy, the target frame sequence is filtered, and the first target key frame set is determined;

[0179] Specifically, the step of filtering the target frame sequence based on the comparison result of each information entropy with the threshold of the information entropy, and determining the first target keyframe set, includes:

[0180] Based on the target frame sequence, the original distribution result of the target frame sequence is obtained;

[0181] The original distribution results are extracted to obtain the target frame to be determined and the remaining distribution results;

[0182] Extract the target frame sequence to obtain the target frame to be determined and the remaining target frame sequence;

[0183] Traverse each of the remaining target frames in the target frame sequence to obtain the updated distribution result;

[0184] Based on the remaining distribution results and the updated distribution results, the target frame sequence is filtered, and the first target keyframe set is determined.

[0185] Optionally, the step of traversing each of the remaining target frame sequences to obtain the updated distribution result includes:

[0186] Obtain the original distribution result, extract the i-th target frame to be determined from the original distribution result, and obtain the i-th residual distribution result;

[0187] The i-th target frame to be determined is extracted from the target frame sequence, wherein the target frame sequence includes L target frames; where L≥i≥1, and L and i are both integers;

[0188] The remaining L-1 target frames are determined as the i-th remaining target frame sequence;

[0189] Traverse the i-th remaining target frame sequence to obtain the i-th update distribution result corresponding to the i-th target frame to be determined.

[0190] Further, the step of filtering the target frame sequence based on the remaining distribution results and the updated distribution results, and determining the first target keyframe set, includes:

[0191] Obtain the remaining distribution result and the updated distribution result for each of the target frames to be determined;

[0192] The remaining distribution results and the updated distribution results are calculated to obtain the information entropy of each target frame to be determined;

[0193] Each information entropy is compared with a threshold of the information entropy; if the information entropy is greater than the threshold of the information entropy, the target frame to be determined that matches the information entropy is determined as the first target key frame.

[0194] Further, the remaining distribution results and the updated distribution results are calculated to obtain the information entropy of each target frame to be determined, including:

[0195] Obtain the i-th updated distribution result and the i-th residual distribution result;

[0196] The i-th updated distribution result and the i-th remaining distribution result are calculated to obtain the i-th information entropy; if the i-th information entropy is greater than the threshold of the information entropy, then the i-th target frame to be determined corresponding to the i-th information entropy is determined as the first target key frame.

[0197] Step S34: Filter the historical frame sequence based on the target risk level to determine the second target key frame set.

[0198] Specifically, the step of filtering the historical frame sequence based on the target risk level to determine the second target key frame set includes:

[0199] Acquire multiple historical frame sequences;

[0200] Feature extraction is performed on multiple historical frame sequences to obtain feature data;

[0201] Cluster the feature data to obtain the target frame sequence;

[0202] Feature extraction is performed on the target frame sequence to obtain multiple target feature frame sequences;

[0203] Cluster the multiple target feature frame sequences to determine the risk level of each target feature frame sequence;

[0204] The target feature frame sequence is filtered based on the target risk level to determine the second target key frame set.

[0205] Embodiments of the present invention also provide an electronic device, including a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor executes the computer program to implement the system described above.

[0206] Embodiments of the present invention also provide a computer-readable storage medium comprising a stored computer program, wherein the computer program, when executed, controls the device containing the computer-readable storage medium to execute the system as described above.

[0207] Furthermore, other configurations and functions of the apparatus in the embodiments of the present invention are known to those skilled in the art, and will not be described in detail here to reduce redundancy.

[0208] It should be noted that the logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a sequenced list of executable instructions for implementing logical functions, and can be specifically implemented in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can fetch and execute instructions from, an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by, or in conjunction with, an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). Alternatively, the computer-readable medium may be paper or other suitable media on which the program can be printed, since the program can be obtained electronically, for example, by optically scanning the paper or other medium, followed by editing, interpreting, or otherwise processing as necessary, and then stored in a computer memory.

[0209] It should be understood that various parts of the present invention can be implemented in hardware, software, firmware, or a combination thereof. In the above embodiments, multiple steps or methods can be implemented in software or firmware stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.

[0210] In the description of this specification, references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of the invention. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples.

[0211] In the description of this invention, it should be understood that the terms "center," "longitudinal," "lateral," "length," "width," "thickness," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," "outer," "clockwise," "counterclockwise," "axial," "radial," and "circumferential" indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing this invention and simplifying the description, and are not intended to indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on this invention.

[0212] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this invention, "a plurality of" means at least two, such as two, three, etc., unless otherwise explicitly specified.

[0213] In this invention, unless otherwise explicitly specified and limited, the terms "installation," "connection," "linking," and "fixing," etc., should be interpreted broadly. For example, they can refer to a fixed connection, a detachable connection, or an integral part; they can refer to a mechanical connection or an electrical connection; they can refer to a direct connection or an indirect connection through an intermediate medium; they can refer to the internal communication of two components or the interaction between two components, unless otherwise explicitly limited. Those skilled in the art can understand the specific meaning of the above terms in this invention according to the specific circumstances.

[0214] In this invention, unless otherwise explicitly specified and limited, "above" or "below" the second feature can mean that the first feature is in direct contact with the second feature, or that the first feature is in indirect contact with the second feature through an intermediate medium. Furthermore, "above," "over," and "on top" of the second feature can mean that the first feature is directly above or diagonally above the second feature, or simply that the first feature is at a higher horizontal level than the second feature. "Below," "below," and "under" the second feature can mean that the first feature is directly below or diagonally below the second feature, or simply that the first feature is at a lower horizontal level than the second feature.

[0215] Although embodiments of the present invention have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting the present invention. Those skilled in the art can make changes, modifications, substitutions and variations to the above embodiments within the scope of the present invention.

Claims

1. A system for determining keyframes, characterized in that, include: The system comprises an acquisition module, a data correction module, a first determination module, and a second determination module; wherein the acquisition module, the data correction module, and the first determination module are sequentially and communicatively connected; and the acquisition module is communicatively connected to the second determination module. The acquisition module is used to acquire the original frame sequence and the historical frame sequence; The data correction module is used to correct the original frame sequence to obtain the target frame sequence; The first determining module is used to filter the target frame sequence based on the comparison result of each information entropy with the threshold of the information entropy, and determine the first target key frame set; The second determining module is used to filter the historical frame sequence based on the target risk level to determine a second target keyframe set; The first determining module includes an update distribution unit; The update distribution unit obtains the original distribution result of the target frame sequence based on the target frame sequence; The updated distribution unit extracts the original distribution results to obtain the target frame to be determined and the remaining distribution results; The update distribution unit extracts the target frame sequence to obtain the target frame to be determined and the remaining target frame sequence; The update distribution unit traverses each target frame of the remaining target frame sequence to obtain the update distribution result; The updated distribution unit obtains the original distribution result, extracts the i-th target frame to be determined from the original distribution result, and obtains the i-th remaining distribution result; The update distribution unit extracts the i-th target frame to be determined from the target frame sequence, wherein the target frame sequence includes L target frames; where L≥i≥1, and L and i are both integers; The update distribution unit determines the remaining L-1 target frames as the i-th remaining target frame sequence; The update distribution unit traverses the i-th remaining target frame sequence to obtain the i-th update distribution result corresponding to the i-th target frame to be determined; The first determining module further includes a calculation unit; The calculation unit obtains the remaining distribution result and the updated distribution result for each of the target frames to be determined; The calculation unit calculates the remaining distribution results and the updated distribution results to obtain the information entropy of each target frame to be determined; The calculation unit compares each information entropy with a threshold of the information entropy; if the information entropy is greater than the threshold of the information entropy, the target frame to be determined that matches the information entropy is determined as the first target key frame.

2. The system according to claim 1, characterized in that, The data correction module includes a labeling unit; The annotation unit acquires the original frame sequence; The annotation unit filters the original frame sequence to obtain the frame sequence to be corrected; The labeling unit determines the target label; The annotation unit annotates the frame sequence to be corrected based on the target label to obtain the corrected frame sequence; The annotation unit obtains the target frame sequence based on the corrected frame sequence.

3. The system according to claim 1, characterized in that, The computing unit obtains the i-th updated distribution result and the i-th residual distribution result; The computing unit calculates the i-th updated distribution result and the i-th remaining distribution result to obtain the i-th information entropy; If the i-th information entropy is greater than the information entropy threshold, then the i-th target frame to be determined corresponding to the i-th information entropy is determined as the first target key frame.

4. The system according to claim 1, characterized in that, The second determining module includes a classification unit; The classification unit acquires multiple historical frame sequences; The classification unit extracts features from multiple historical frame sequences to obtain feature data; The classification unit clusters the feature data to obtain the target frame sequence; The classification unit extracts features from the target frame sequence to obtain multiple target feature frame sequences; The classification unit clusters multiple target feature frame sequences to determine the risk level of each target feature frame sequence. The classification unit filters the target feature frame sequence based on the target risk level to determine the second target key frame set.

5. A method for determining keyframes, characterized in that, include: Obtain the original frame sequence and the historical frame sequence; The original frame sequence is corrected to obtain the target frame sequence; Based on the comparison results of each information entropy with the threshold of the information entropy, the target frame sequence is filtered, and the first target key frame set is determined; Based on the target risk level, the historical frame sequence is filtered to determine the second target key frame set; Based on the target frame sequence, the original distribution result of the target frame sequence is obtained; The original distribution results are extracted to obtain the target frame to be determined and the remaining distribution results; Extract the target frame sequence to obtain the target frame to be determined and the remaining target frame sequence; Iterate through each target frame in the remaining target frame sequence to obtain the updated distribution result; Obtain the original distribution result, extract the i-th target frame to be determined from the original distribution result, and obtain the i-th residual distribution result; The i-th target frame to be determined is extracted from the target frame sequence, wherein the target frame sequence includes L target frames; where L≥i≥1, and L and i are both integers; The remaining L-1 target frames are determined as the i-th remaining target frame sequence; Traverse the i-th remaining target frame sequence to obtain the i-th updated distribution result corresponding to the i-th target frame to be determined; Obtain the remaining distribution result and the updated distribution result for each of the target frames to be determined; The remaining distribution results and the updated distribution results are calculated to obtain the information entropy of each target frame to be determined; Each information entropy is compared with a threshold of the information entropy; if the information entropy is greater than the threshold of the information entropy, the target frame to be determined that matches the information entropy is determined as the first target key frame.

6. An electronic device, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the system as described in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the system as described in any one of claims 1 to 4.

Images