Method, device and equipment for determining key frame and storage medium

Abstract

The application discloses a kind of method, device and equipment for determining key frame and storage medium, the method includes: obtaining original sample set, and pixel point matching is carried out to each original sample, obtain target sample set;The target sample set is calculated, and first information entropy set is obtained;Characteristic vector set is obtained based on the target sample set, and the characteristic vector set is calculated, and second information entropy set is obtained;Each first information entropy and the second information entropy matched with the first information entropy are calculated, and target information entropy sequence is obtained;The target sample set is screened based on the target information entropy sequence, and target key frame set is obtained.The technical scheme of the present application can determine potential risks according to the target key frame set after obtaining the target key frame set, thereby improving the accuracy of data storage, and thereby improving the safety of autonomous driving and the experience of users.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic driving, and particularly relates to a method and device for determining key frames, equipment and a storage medium. BACKGROUND

[0002] An automatic driving system above L3 level needs to accumulate a large amount of data in the early development and late production processes, and it is very important to effectively use storage resources and store different data for different functions.

[0003] However, the current automatic driving system directly stores data obtained based on some data acquisition devices in order to facilitate the subsequent algorithm process calling, and does not correct or process the data, thereby causing poor precision of the stored data and reducing the safety of automatic driving and the experience of users. Meanwhile, each camera sensor of the current automatic driving system collects data of dozens of G (Gigabit) or more per hour (related to the collection frequency, sensor resolution, etc.), and there are a large amount of repetitions and noises, so that the subsequent function provides a small amount of picture data, thereby causing waste of storage resources and computing resources. SUMMARY

[0004] The present application aims to at least solve one of the technical problems in the related art. To this end, one object of the present application is to provide a method and device for determining key frames, equipment and a storage medium.

[0005] In order to solve the above technical problems, the embodiments of the present application provide the following technical solutions:

[0006] A method for determining key frames, comprising:

[0007] obtaining an original sample set and performing pixel point matching on each original sample to obtain a target sample set;

[0008] calculating the target sample set to obtain a first information entropy set;

[0009] obtaining a feature vector set based on the target sample set and calculating the feature vector set to obtain a second information entropy set;

[0010] calculating each first information entropy and the second information entropy matched with the first information entropy to obtain a target information entropy sequence;

[0011] filtering the target sample set based on the target information entropy sequence to obtain a target key frame set.

[0012] Optionally, the pixel point matching on each original sample to obtain a target sample set comprises:

[0013] obtaining the original sample set based on a perception device, wherein the original sample set comprises matched pairs of first-class samples and second-class samples;

[0014] performing pixel matching based on each pair of the first-class sample and the second-class sample to obtain a plurality of unmatched pixel point groups;

[0015] correcting each of the unmatched pixel point groups to obtain a plurality of corrected pixel point groups;

[0016] obtaining a pair of first-class corrected sample and second-class corrected sample based on the plurality of corrected pixel point groups;

[0017] obtaining the target sample set based on the plurality of pairs of first-class corrected sample and second-class corrected sample.

[0018] Optionally, the correcting each of the unmatched pixel point groups to obtain a plurality of corrected pixel point groups comprises:

[0019] calculating a first confidence of a first pixel point of the first-class sample of each of the unmatched pixel point groups;

[0020] calculating a second confidence of a second pixel point of the second-class sample of each of the unmatched pixel point groups;

[0021] comparing the first confidence and the second confidence to obtain a comparison result;

[0022] correcting each of the unmatched pixel point groups based on the comparison result to obtain a plurality of corrected pixel point groups.

[0023] Optionally, the calculating the target sample set to obtain a first information entropy set comprises:

[0024] the target sample set comprises L target samples; wherein L≥1, and L is an integer;

[0025] selecting an Mth target sample based on the target sample set, and generating an Mth reference sample set based on the remaining L-1 target samples; wherein L≥M≥1, and M is an integer;

[0026] calculating the Mth first information entropy based on the Mth target sample and the Mth reference sample set;

[0027] obtaining the first information entropy set based on a plurality of the first information entropy.

[0028] Optionally, the obtaining a feature vector set based on the target sample set, and calculating the feature vector set to obtain a second information entropy set comprises:

[0029] perform feature extraction on the Mth target sample to obtain an Mth feature vector;

[0030] perform calculation based on the Mth target sample and the Mth feature vector to obtain an Mth second information entropy;

[0031] obtain the second information entropy set based on the plurality of second information entropies.

[0032] Optionally, the performing calculation on each first information entropy and the second information entropy matched with the first information entropy to obtain a target information entropy sequence comprises:

[0033] performing calculation on each first information entropy and the second information entropy matched with the first information entropy to obtain an initial information entropy set;

[0034] obtain the initial information entropy sequence based on the initial information entropy set;

[0035] perform screening on the initial information entropy sequence to obtain a target information entropy sequence.

[0036] Optionally, the performing calculation on each first information entropy and the second information entropy matched with the first information entropy to obtain an initial information entropy set comprises:

[0037] obtain a first preset weight and a second preset weight;

[0038] determine an Mth first term based on the first preset weight and the Mth first information entropy;

[0039] determine an Mth second term based on the second preset weight and the Mth second information entropy;

[0040] calculate the Mth initial information entropy based on the Mth first term and the Mth second term;

[0041] obtain an initial information entropy set based on the plurality of initial information entropies.

[0042] Embodiments of the present application also provide a device for determining a key frame, comprising:

[0043] a matching module configured to obtain an original sample set and perform pixel point matching on each original sample to obtain a target sample set;

[0044] a first calculation module configured to perform calculation on the target sample set to obtain a first information entropy set;

[0045] an obtaining module configured to obtain a feature vector set based on the target sample set and perform calculation on the feature vector set to obtain a second information entropy set;

[0046] a second calculation module, configured to calculate each of the first information entropy and the second information entropy matched with the first information entropy to obtain a target information entropy sequence;

[0047] a screening module, configured to screen the target sample set based on the target information entropy sequence to obtain a target key frame set.

[0048] Embodiments of the present application also provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, and the processor implements the method as described above when executing the computer program.

[0049] Embodiments of the present application also provide a computer readable storage medium comprising a stored computer program, wherein the computer readable storage medium controls the device where the computer readable storage medium is located to execute the method as described above when the computer program runs.

[0050] Embodiments of the present application have the following technical effects:

[0051] The above technical solution of the present application has the following advantages: 1) The target sample set is globally processed to obtain a first information entropy set, and the target sample set is locally processed to obtain a second information entropy set. The first information entropy and the second information entropy corresponding to each target sample are calculated to obtain an initial information entropy set. The initial information entropy set is screened to obtain a target information entropy set. The target sample set is screened based on the target information entropy set to obtain a target key frame set. After obtaining the target key frame set, the potential risk can be determined according to the target key frame set, thereby improving the accuracy of data storage and the safety of autonomous driving and the user experience.

[0052] 2) Embodiments of the present application realize the emphasis and depiction of the edge of the obstacle in each frame of original sample based on the correction of each pair of first type sample and second type sample, that is, the obstacle region on each frame of picture sample is more accurate, which realizes the noise reduction of the data obtained by the perception device, thereby facilitating the accurate storage of data.

[0053] Additional aspects and advantages of the present application will be in part apparent and in part pointed out hereinafter. BRIEF DESCRIPTION OF DRAWINGS

[0054] Figure 1 is a structural schematic diagram of a system for determining a key frame provided by an embodiment of the present application;

[0055] Figure 2 is a flowchart of a method for determining a key frame provided by an embodiment of the present application;

[0056] Figure 3 is a principle diagram of mutual correction of the first type of samples and the second type of samples provided by an embodiment of the present application;

[0057] Figure 4 is a structural diagram of the device for determining key frames provided by an embodiment of the present application. DETAILED DESCRIPTION

[0058] Embodiments of the present application are described in detail below, examples of which are shown in the accompanying drawings, wherein the same or similar reference numerals represent the same or similar elements or elements having the same or similar functions throughout. The embodiments described below by referring to the accompanying drawings are exemplary and are intended to explain the present application, and cannot be understood as a limitation of the present application.

[0059] In order to facilitate those skilled in the art to understand the embodiments, part of the language is explained:

[0060] (1) Redis: a data structure server for memory storage.

[0061] (2) hive: a data warehouse tool.

[0062] (3) KL divergence: Kullback-Leibler Divergence, i.e. K-L divergence, is a way to quantify the difference between two probability distributions P and Q, also called relative entropy.

[0063] As shown in Figure 1 , an embodiment of the present application provides a system 100 for determining key frames, comprising:

[0064] a data acquisition device 101; wherein the data acquisition device comprises a plurality of vehicle-mounted sensors, the vehicle-mounted sensors monitor the running state of the target vehicle in real time to obtain monitoring data.

[0065] Further, the data acquisition device comprises a camera, a temperature sensor, a vehicle speed sensor and other sensors for monitoring the running state of the target vehicle.

[0066] a noise reduction device 102, which obtains the monitoring data sent by the data acquisition device and corrects the monitoring data to obtain corrected monitoring data.

[0067] a data processing device 103, which obtains the corrected monitoring data sent by the noise reduction device and performs calculation based on the corrected monitoring data to obtain a calculation result; based on the calculation result, a key frame with potential risk is obtained.

[0068] The data storage device 104 obtains data sent by the data processing device, labels key frames with potential risks, and then stores the key frames so as to facilitate subsequent algorithm calling.

[0069] Embodiments of the present application can manually or intelligently label part of data with potential risks, so as to improve accuracy and safety of subsequent automatic driving.

[0070] As shown in Figure 2 Embodiments of the present application provide a method for determining key frames, which comprises the following steps.

[0071] Step S21: Obtain an original sample set, and perform pixel point matching on each original sample to obtain a target sample set.

[0072] Embodiments of the present application, the data types contained in the original sample set include real-time data and offline imported data.

[0073] The sources of the original sample set can include vehicle-mounted sensors and shadow mode.

[0074] The scenes involved in the data of the original sample set can include natural scenes such as weather and terrain, and behavior scenes such as other vehicles cutting in, cutting out, overtaking or changing lanes.

[0075] The storage of the original sample set can include real-time writing to redis and offline processing and then falling to hive.

[0076] Specifically, the pixel point matching on each original sample to obtain a target sample set comprises the following steps.

[0077] The original sample set is obtained based on a perception device, wherein the original sample set includes a plurality of pairs of matched first type samples and second type samples.

[0078] A plurality of unmatched pixel point groups are obtained based on pixel matching of each pair of first type samples and second type samples.

[0079] A plurality of corrected pixel point groups are obtained by correcting each unmatched pixel point group.

[0080] A pair of first type corrected samples and second type corrected samples are obtained based on a plurality of corrected pixel point groups.

[0081] The target sample set is obtained based on a plurality of pairs of first type corrected samples and second type corrected samples.

[0082] Embodiments of the present application perform data noise reduction based on matching between a plurality of sensors of a perception device, so as to improve the accuracy of data.

[0083] For example, matching each frame of image samples acquired by a wide-angle camera (120°) and a narrow-angle camera (80°) respectively;

[0084] Among them, the image sample captured by the wide-angle camera at the same moment has a wide and close field of vision; the image sample captured by the narrow-angle camera has a narrow and far field of vision. Therefore, for the same obstacle at the same moment, the obstacle captured by the wide-angle camera may fill the entire frame of the image sample, while the obstacle captured by the narrow-angle camera may only be located in a small area of ​​the entire frame of the image sample.

[0085] Therefore, in the embodiments of the present invention, one frame of image samples captured by the narrow-angle camera and the wide-angle camera at the same time are obtained, and the obstacle areas of the two frame of image samples are matched pixel by pixel, and then the matching result is obtained. If there is a pixel mismatch in the matching result, mutual correction is performed based on the two frame of image samples.

[0086] Generally, the obstacle parts of the two image samples have been marked with bounding boxes. Therefore, during the matching process, the pixels within the obstacle bounding boxes can be directly obtained and matched, saving time and effort.

[0087] like Figure 3 As shown, specifically, in an embodiment of the present invention, the first type of sample corresponding to the obstacle in each frame of image samples captured by the narrow-angle camera is projected onto the second type of sample corresponding to the obstacle in each frame of image samples captured by the wide-angle camera based on a projection algorithm; wherein, the projection algorithm transforms all pixels of the first type of sample based on the camera's intrinsic and extrinsic parameter calibration data, and after performing coordinate transformation, it realizes that all pixels of the first type of sample are projected onto the second type of sample.

[0088] It should be noted that the specific implementation methods of the intrinsic and extrinsic parameter calibration data and coordinate transformation in the embodiments of the present invention are not within the protection scope of the present invention, and therefore will not be described in detail.

[0089] Further, the step of correcting each of the mismatched pixel groups to obtain multiple corrected pixel groups includes:

[0090] Calculate the first confidence level of the first pixel of the first class of samples in each group of mismatched pixels;

[0091] Calculate the second confidence level of the second pixel of the second type of sample in each group of mismatched pixels;

[0092] Compare the first confidence level and the second confidence level to obtain the comparison result;

[0093] Based on the comparison results, each mismatched pixel group is corrected to obtain multiple corrected pixel groups.

[0094] wherein each first pixel point of the first type of samples and a corresponding second pixel point of the second type of samples form a set of to-be-processed samples, and it is determined whether the two pixel points in the to-be-processed samples match;

[0095] If the first pixel point and the second pixel point in the to-be-processed samples match, that is, the first pixel point and the second pixel point are consistent, the to-be-processed sample is not processed.

[0096] If the first pixel point and the second pixel point in the to-be-processed samples do not match, that is, the first pixel point and the second pixel point do not match, the to-be-processed sample is determined as a set of to-be-corrected samples.

[0097] By analogy, based on a set of first type of samples and second type of samples at the same time, a plurality of sets of unmatched pixel points are determined.

[0098] For the set of unmatched pixel points, the first confidence X of the first pixel point and the second confidence Y of the second pixel point are respectively calculated and obtained.

[0099] The sizes of X and Y are compared, wherein:

[0100] If X>Y, the first pixel point of the first type of samples in the set of unmatched pixel points is corrected based on the second pixel point of the second type of samples in the set of unmatched pixel points, that is, the first pixel point of the first type of samples in the set of unmatched pixel points is replaced by the second pixel point of the second type of samples in the set of unmatched pixel points; the first pixel point of the first type of samples in the set of unmatched pixel points remains unchanged.

[0101] If XY, the first pixel point of the first type of samples in the set of unmatched pixel points is corrected based on the second pixel point of the second type of samples in the set of unmatched pixel points, that is, the first pixel point of the first type of samples in the set of unmatched pixel points is replaced by the second pixel point of the second type of samples in the set of unmatched pixel points; the second pixel point of the second type of samples in the set of unmatched pixel points remains unchanged.

[0102] By analogy, each set of unmatched pixel points of each pair of first type of samples and second type of samples is corrected to obtain a plurality of sets of corrected matching pixel points.

[0103] In an actual application scenario, after each pair of first type of samples and second type of samples of an original sample set in a time period (the specific length can be preset or manually adjusted, and the embodiments of the present application do not make specific limitations thereon) is corrected, each first type of corrected sample and second type of corrected sample is respectively restored to the corresponding original sample to obtain a corrected each frame of picture, that is, a target sample set is obtained.

[0104] The embodiment of the present application realizes the emphasis and depiction of the edge of the obstacle in each frame of original sample based on the correction of each pair of first type sample and second type sample, that is, makes the obstacle region on each frame of picture sample more accurate, realizes the noise reduction of the data obtained by the perception device, and further facilitates the accurate storage of the data.

[0105] Step S22: calculating the target sample set to obtain a first information entropy set;

[0106] Specifically, the target sample set is calculated to obtain a first information entropy set, including:

[0107] The target sample set includes L target samples; wherein L≥1, and L is an integer;

[0108] The Mth target sample is selected based on the target sample set, and an Mth reference sample set is generated based on the remaining L-1 target samples; wherein L≥M≥1, and M is an integer;

[0109] The Mth first information entropy is obtained based on the Mth target sample and the Mth reference sample set;

[0110] The first information entropy set is obtained based on a plurality of first information entropies.

[0111] The embodiment of the present application firstly performs global calculation on the target sample set, and obtains a first information entropy based on each target sample in the target sample set.

[0112] For example: the Mth target sample in the target sample set is obtained, and the remaining L-1 target samples in the target sample set after removing the Mth target sample are determined as the Mth reference sample set;

[0113] Then, the Mth first information entropy (Mth first KL divergence) corresponding to the Mth reference sample set and the Mth target sample is calculated; wherein the Mth first KL divergence can be calculated based on the following calculation formula:

[0114]

[0115] In the formula, K(Z) is the distribution of the target sample set, K(Z M ,Z) is the distribution of the target sample set after removing the Mth target sample, and the remaining L-1 target samples are traversed to form a new distribution;

[0116] In addition, the relationship between each physical quantity involved in the formula is explained as follows:

[0117] Wherein, Z represents the original sample set, which is equivalent to the universal set; Z MCorresponding to the subset of Z.

[0118] K(Z) is the distribution of the target sample set, corresponding to the universal set; K(Z M ,Z) corresponds to the subset of K(Z).

[0119] Based on the calculation of the above formula, the Mth first KL divergence I M .

[0120] By analogy, L first KL divergences I1, I2, I3, I4……I L .

[0121] Step S23: obtaining a feature vector set based on the target sample set, and calculating the feature vector set to obtain a second information entropy set;

[0122] Specifically, the feature vector set is obtained based on the target sample set, and the feature vector set is calculated to obtain the second information entropy set, including:

[0123] Feature extraction is performed on the Mth target sample to obtain an Mth feature vector;

[0124] Based on the Mth target sample and the Mth feature vector, the Mth second information entropy is obtained by calculation;

[0125] Based on a plurality of the second information entropy, the second information entropy set is obtained.

[0126] Embodiments of the present application perform local calculation on the target sample set, and obtain one second information entropy based on each target sample in the target sample set.

[0127] For example: obtaining the Mth target sample in the target sample set,

[0128] Then, the Mth second information entropy (Mth second ML divergence) corresponding to the Mth target sample and the Mth reference sample set is calculated; wherein the Mth second ML divergence can be obtained based on the following calculation formula:

[0129]

[0130] In the formula, P(X K , Y K ) is the distribution P(Y M ) of the feature vector X M of the Mth target sample Y M .

[0131] In addition, the relationship between each physical quantity involved in the formula is explained as follows:

[0132] Based on the Y of the Mth target sample M Convolution feature extraction is performed to obtain a feature vector X of the obstacle part M , wherein Y M is equivalent to the universal set; X M is a subset of Y M .

[0133] P(Y M ) is the distribution of Y M of the Mth target sample, which is equivalent to the universal set; P(X K , Y K ) is a subset of P(Y M ).

[0134] Based on the calculation of the above formula, the second KL divergence N K of the Mth target sample can be obtained.

[0135] By analogy, L second KL divergences N1, N2, N3, N4……N L can be obtained.

[0136] Step S24: calculating each of the first information entropy and the second information entropy matched with the first information entropy to obtain a target information entropy sequence;

[0137] Specifically, the calculation of each of the first information entropy and the second information entropy matched with the first information entropy to obtain a target information entropy sequence includes:

[0138] calculating each of the first information entropy and the second information entropy matched with the first information entropy to obtain an initial information entropy set;

[0139] Based on the initial information entropy set, the initial information entropy sequence is obtained;

[0140] The initial information entropy sequence is screened to obtain a target information entropy sequence.

[0141] Embodiments of the present application obtain a first information entropy set and a second information entropy set, wherein based on each target sample, the corresponding first information entropy and second information entropy thereof are obtained;

[0142] Then, the first information entropy and the second information entropy of the target sample are calculated to obtain an initial information entropy set;

[0143] Each initial information entropy in the initial information entropy set is sorted, and specifically, each initial information entropy in the initial information entropy set can be sorted in descending order to obtain an initial information entropy sequence;

[0144] A preset information entropy threshold is set.

[0145] The initial information entropy sequence is filtered based on a preset information entropy threshold, for example, if the preset information entropy threshold is 80%, the initial information entropy sequence is filtered from large to small, the first 80% is selected, and then a target information entropy sequence is obtained.

[0146] Further, the first information entropy and the second information entropy matched with the first information entropy are calculated to obtain an initial information entropy set, including:

[0147] A first preset weight and a second preset weight are obtained.

[0148] The Mth first item is determined based on the first preset weight and the Mth first information entropy.

[0149] The Mth second item is determined based on the second preset weight and the Mth second information entropy.

[0150] The Mth initial information entropy is calculated based on the Mth first item and the Mth second item.

[0151] The initial information entropy set is obtained based on a plurality of initial information entropies.

[0152] Embodiments of the present application correspond to one unique initial information entropy for each target sample, that is, based on the initial information entropy set, L arrays (target sample, initial information entropy) can be obtained.

[0153] Each initial information entropy can be calculated based on the following calculation method:

[0154] Taking the initial information entropy corresponding to the Mth target sample as an example:

[0155] Initial information entropy M =αI M +(1-α)N M .

[0156] Wherein, αI M is the first item; α is the first preset weight; (1-α)N M is the second item; 1-α is the second preset weight.

[0157] It should be noted that the value of α can be adjusted in real time according to actual needs, and embodiments of the present application do not make specific limitations.

[0158] By analogy, L initial information entropies, that is, an initial information entropy set, can be obtained:

[0159] Initial information entropy 1,

[0160] Initial information entropy 2.

[0161] …

[0162] initial information entropy L .

[0163] Correspondingly, L arrays can be obtained:

[0164] (target sample 1, initial information entropy 1),

[0165] (target sample 2, initial information entropy 2)

[0166] …

[0167] (target sample L , initial information entropy L ).

[0168] Step S25: screening the target sample set based on the target information entropy sequence to obtain a target key frame set.

[0169] Specifically, the L arrays are sorted from large to small based on the value of each initial information entropy, and then an initial information entropy sequence related to the initial information entropy is obtained;

[0170] Based on a preset information entropy threshold, the initial information entropy sequence is screened to obtain a target information entropy sequence;

[0171] Correspondingly, another array can be obtained:

[0172] (target sample 6, target information entropy 6),

[0173] (target sample 20 , target information entropy 20 )

[0174] …

[0175] (target sample 3, target information entropy 3).

[0176] Based on the array, the target key frame set can be determined:

[0177] target sample 6,

[0178] target sample 20

[0179] …

[0180] target sample 3.

[0181] Wherein, in order to improve the accuracy of data storage, the preset information entropy threshold can be increased, for example: increasing the preset information entropy threshold from 80% to 90%, more target key frames can be obtained, that is, a larger target key frame set is obtained;

[0182] Embodiments of the present application globally process a target sample set to obtain a first information entropy set; locally process the target sample set to obtain a second information entropy set; calculate the first information entropy and the second information entropy corresponding to each target sample to obtain an initial information entropy set; screen the initial information entropy set to obtain a target information entropy set; screen the target sample set based on the target information entropy set to obtain a target key frame set; wherein, after obtaining the target key frame set, potential risks can be determined according to the target key frame set, thereby improving the accuracy of data storage and the safety of autonomous driving and the experience of users.

[0183] As shown in Figure 4 Embodiments of the present application also provide a device 400 for determining a key frame, comprising:

[0184] A matching module 401 is configured to obtain an original sample set and perform pixel point matching on each original sample to obtain a target sample set;

[0185] A first calculation module 402 is configured to calculate the target sample set to obtain a first information entropy set;

[0186] An obtaining module 403 is configured to obtain a feature vector set based on the target sample set and calculate the feature vector set to obtain a second information entropy set;

[0187] A second calculation module 404 is configured to calculate each first information entropy and the second information entropy matched with the first information entropy to obtain a target information entropy sequence;

[0188] A screening module 405 is configured to screen the target sample set based on the target information entropy sequence to obtain a target key frame set.

[0189] Optionally, the pixel point matching on each original sample to obtain a target sample set comprises:

[0190] Obtaining the original sample set based on a perception device, wherein the original sample set comprises a plurality of matched pairs of first type samples and second type samples;

[0191] Performing pixel matching based on each pair of first type samples and second type samples to obtain a plurality of unmatched pixel point groups;

[0192] Correcting each unmatched pixel point group to obtain a plurality of corrected pixel point groups;

[0193] Obtaining a pair of first type corrected samples and second type corrected samples based on a plurality of corrected pixel point groups;

[0194] Obtaining the target sample set based on a plurality of pairs of first type corrected samples and second type corrected samples.

[0195] Optionally, the correction of each of the groups of unmatched pixel points obtains a plurality of groups of corrected pixel points, comprising:

[0196] The first confidence of the first pixel point of the first type of sample of each of the groups of unmatched pixel points is calculated.

[0197] The second confidence of the second pixel point of the second type of sample of each of the groups of unmatched pixel points is calculated.

[0198] The first confidence and the second confidence are compared to obtain a comparison result.

[0199] Based on the comparison result, each of the groups of unmatched pixel points is corrected to obtain a plurality of groups of corrected pixel points.

[0200] Optionally, the calculation of the target sample set obtains a first information entropy set, comprising:

[0201] The target sample set comprises L target samples; wherein L≥1, and L is an integer;

[0202] Based on the target sample set, the Mth target sample is selected, and based on the remaining L-1 target samples, the Mth reference sample set is generated; wherein L≥M≥1, and M is an integer;

[0203] Based on the Mth target sample and the Mth reference sample set, the Mth first information entropy is calculated.

[0204] Based on a plurality of the first information entropies, the first information entropy set is obtained.

[0205] Optionally, the calculation of the target sample set obtains a first information entropy set, comprising:

[0206] The feature extraction of the Mth target sample obtains the Mth feature vector;

[0207] Based on the Mth target sample and the Mth feature vector, the Mth second information entropy is calculated.

[0208] Based on a plurality of the second information entropies, the second information entropy set is obtained.

[0209] Optionally, the calculation of each of the first information entropies and the second information entropies matched with the first information entropies obtains a target information entropy sequence, comprising:

[0210] The calculation of each of the first information entropies and the second information entropies matched with the first information entropies obtains an initial information entropy set;

[0211] obtain the initial information entropy set based on the initial information entropy set;

[0212] screen the initial information entropy sequence to obtain a target information entropy sequence.

[0213] Optionally, the obtaining the initial information entropy set based on the first information entropy and the second information entropy matched with the first information entropy comprises:

[0214] obtaining a first preset weight and a second preset weight;

[0215] determining an Mth first term based on the first preset weight and the Mth first information entropy;

[0216] determining an Mth second term based on the second preset weight and the Mth second information entropy;

[0217] obtaining the Mth initial information entropy based on the Mth first term and the Mth second term;

[0218] obtaining the initial information entropy set based on a plurality of the initial information entropies.

[0219] Embodiments of the present application also provide an electronic device, comprising a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor implements the method as described above when executing the computer program.

[0220] Embodiments of the present application also provide a computer readable storage medium, comprising a stored computer program, wherein the computer readable storage medium controls a device where the computer readable storage medium is located to execute the method as described above when the computer program runs.

[0221] In addition, other configurations and functions of the device of the embodiments of the present application are known to those skilled in the art, and to reduce redundancy, they are not described here.

[0222] It is to be appreciated that the above description and the examples that follow are intended to be illustrative only and that changes can be made to the description, as represented by the above listed elements, by the steps recited in the flow charts, and by the examples that follow, without departing from the spirit of the application. Accordingly, the scope of the present application is intended to be defined only by the appended claims.

[0223] It should be understood that aspects of the application can be implemented in hardware, software, firmware or a combination thereof. In the above embodiments, various steps or methods can be implemented in software or firmware that is stored in memory and executed by a suitable instruction execution system. For example, if implemented in hardware, as in another embodiment, any of the following techniques, which are well known in the art of hardware implementation, can be used: a hybrid of the above techniques, a mixture of two or more of the above techniques, or a combination of the above techniques with other techniques not listed above.

[0224] In the description of the present specification, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" and the like means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the present application. In the present specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Also, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples.

[0225] In the description of the application, it should be understood that the orientation or positional relationship indicated by the terms "center", "longitudinal", "lateral", "length", "width", "thickness", "upper", "lower", "front", "back", "left", "right", "vertical", "horizontal", "top", "bottom", "inner", "outer", "clockwise", "counterclockwise", "axial", "radial", "circumferential" and the like is based on the orientation or positional relationship shown in the drawings, and is only for the convenience of describing the application and simplifying the description, and does not indicate or imply that the device or element referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as a limitation on the application.

[0226] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the technical features indicated. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "a plurality of" is at least two, such as two, three, etc., unless otherwise explicitly specified and limited.

[0227] In the present application, unless otherwise explicitly specified and limited, the terms "mounting", "connecting", "connecting", "fixing" and the like should be broadly understood, for example, it can be fixedly connected, or it can be detachably connected, or it can be integrated; it can be mechanically connected, or it can be electrically connected; it can be directly connected, or it can be indirectly connected through an intermediate medium; it can be the internal communication of two elements or the interaction relationship between two elements, unless otherwise explicitly limited. For those skilled in the art, the specific meaning of the above terms in the present application can be understood according to the specific circumstances.

[0228] In the present application, unless otherwise explicitly specified and limited, the first feature is "on" or "under" the second feature, which can be direct contact between the first and second features, or indirect contact between the first and second features through an intermediate medium. Moreover, the first feature "above", "above" and "above" the second feature can be directly above or obliquely above the first feature, or only indicate that the horizontal height of the first feature is higher than that of the second feature. The first feature "below", "below" and "below" the second feature can be directly below or obliquely below the first feature, or only indicate that the horizontal height of the first feature is less than that of the second feature.

[0229] Although the embodiments of the present application have been shown and described above, it can be understood that the above embodiments are exemplary and cannot be understood as a limitation on the present application, and those skilled in the art can make changes, modifications, replacements and variations to the above embodiments within the scope of the present application.

Claims

1. A method for determining keyframes, characterized in that, include: Obtain the original sample set and perform pixel matching on each original sample to obtain the target sample set; The target sample set is calculated to obtain the first information entropy set; The first information entropy set calculates the distribution difference between each target sample and the reference sample set based on KL divergence; A feature vector set is obtained based on the target sample set, and a second information entropy set is obtained by calculating the feature vector set; the second information entropy set calculates the distribution difference between the feature vector of each target sample and the target sample based on KL divergence. Calculate each of the first information entropy and the second information entropy that matches the first information entropy to obtain the target information entropy sequence; The target sample set is filtered based on the target information entropy sequence to obtain the target keyframe set; The step of performing pixel-by-pixel matching on each of the original samples to obtain the target sample set includes: The original sample set is obtained based on the sensing device, wherein the original sample set includes multiple pairs of matched first-class samples and second-class samples; Based on pixel matching of each pair of the first type of sample and the second type of sample, multiple groups of mismatched pixels are obtained. Each of the mismatched pixel groups is corrected to obtain multiple corrected pixel groups; Based on the multiple groups of corrected pixels, a pair of first-type correction samples and a pair of second-type correction samples are obtained; The target sample set is obtained based on multiple pairs of the first type of correction samples and the second type of correction samples.

2. The method according to claim 1, characterized in that, The step of correcting each of the mismatched pixel groups to obtain multiple corrected pixel groups includes: Calculate the first confidence level of the first pixel of the first class of samples in each group of mismatched pixels; Calculate the second confidence level of the second pixel of the second type of sample in each group of mismatched pixels; Compare the first confidence level and the second confidence level to obtain the comparison result; Based on the comparison results, each mismatched pixel group is corrected to obtain multiple corrected pixel groups.

3. The method according to claim 2, characterized in that, The target sample set is calculated to obtain a first information entropy set, including: The target sample set includes L target samples; where L≥1 and L is an integer; Based on the target sample set, select the Mth target sample, and based on the remaining L-1 target samples, generate the Mth reference sample set; where L≥M≥1, and M is an integer; The first information entropy is obtained by calculating based on the Mth target sample and the Mth reference sample set; The first information entropy set is obtained based on multiple first information entropies.

4. The method according to claim 1, characterized in that, The step of obtaining a feature vector set based on the target sample set and calculating a second information entropy set from the feature vector set includes: Feature extraction is performed on the Mth target sample to obtain the Mth feature vector; The Mth second information entropy is calculated based on the Mth target sample and the Mth feature vector. The second information entropy set is obtained based on multiple second information entropies.

5. The method according to claim 1, characterized in that, The step of calculating each of the first information entropies and the second information entropies matching the first information entropies to obtain the target information entropy sequence includes: Calculate each of the first information entropy and the second information entropy that matches the first information entropy to obtain an initial information entropy set; Based on the initial information entropy set, the initial information entropy sequence is obtained; The initial information entropy sequence is filtered to obtain the target information entropy sequence.

6. The method according to claim 5, characterized in that, The step of calculating each of the first information entropies and the second information entropies matching the first information entropies to obtain an initial information entropy set includes: Obtain the first preset weight and the second preset weight; Based on the first preset weight and the Mth first information entropy, the Mth first item is determined; Based on the second preset weight and the Mth second information entropy, the Mth second item is determined; Based on the Mth first item and the Mth second item, the Mth initial information entropy is calculated; Based on the multiple initial information entropies, an initial information entropy set is obtained.

7. An apparatus for determining keyframes, characterized in that, include: The matching module is used to obtain the original sample set and perform pixel matching on each original sample to obtain the target sample set. The first calculation module is used to calculate the target sample set to obtain a first information entropy set; the first information entropy set is calculated based on KL divergence to determine the distribution difference between each target sample and the reference sample set. The acquisition module is used to obtain a feature vector set based on the target sample set, and to calculate the feature vector set to obtain a second information entropy set; the second information entropy set calculates the distribution difference between the feature vector of each target sample and the target sample based on KL divergence. The second calculation module is used to calculate each of the first information entropy and the second information entropy that matches the first information entropy to obtain the target information entropy sequence. The filtering module is used to filter the target sample set based on the target information entropy sequence to obtain the target keyframe set; The matching module is specifically used to obtain the original sample set based on the sensing device, wherein the original sample set includes multiple pairs of matched first-class samples and second-class samples; perform pixel matching based on each pair of first-class samples and second-class samples to obtain multiple groups of mismatched pixels; correct each group of mismatched pixels to obtain multiple groups of corrected pixels; obtain a pair of first-class corrected samples and second-class corrected samples based on the multiple groups of corrected pixels; and obtain the target sample set based on multiple pairs of first-class corrected samples and second-class corrected samples.

8. An electronic device, characterized in that, The method includes a processor, a memory, and a computer program stored in the memory and configured to be executed by the processor, wherein the processor, when executing the computer program, implements the method as claimed in any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium includes a stored computer program, wherein, when the computer program is executed, it controls the device on which the computer-readable storage medium is located to perform the method as described in any one of claims 1 to 6.

Images