A frame extraction method and system for road test vehicle image information

Abstract

A kind of frame extraction method for road test vehicle image information, comprising: obtaining road test vehicle image information, and marking candidate key frame in road test vehicle image information;According to the marked candidate key frame, based on actual demand, relevant threshold setting and index calculation;According to the set threshold and calculated index, key frame is marked;According to the marked key frame and set threshold and calculated index, similar frame is marked;According to neural network model, no key target frame is marked;According to the marked key frame, similar frame and no key target frame, image information is frame extracted.The present application is highly adaptable to the business scene of automatic driving, and can more effectively solve the data redundancy problem of storage end.Free limited human resources, and then improve the execution efficiency of the entire automatic driving tool chain.And different content section image scene is processed by divide and conquer, and the frame extraction result is more reliable.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of video communication, in particular to a frame extraction method and system for road test vehicle image information. BACKGROUND

[0002] Under the background of continuous exploration of high-level automatic driving in the industry, the image data collected by the road test vehicle has a leap-like increase in size compared to the L1 / L2 automatic driving stage, and the image data storage management pressure is greater, which also brings greater overhead to the data storage server.

[0003] Since not all frame information in the image data contains valuable target object information, and since the image data collected by the vehicle has continuity in time, the target object information of adjacent frames may appear highly similar or even completely overlapping, which is particularly evident when the observed target in the vehicle's field of view is relatively stationary or moves slowly. These highly similar image data provide relatively little information gain, but occupy a considerable amount of storage resources, so a recognition means is needed to judge the above-mentioned scene and effectively filter similar frames and non-key target frames.

[0004] Prior to this, the industry has proposed several technologies for image duplicate frame recognition or key frame extraction, and the landing scenarios have basically focused on video monitoring or security fields. The judgment method of the value of the target object is obviously different from the data collection scenario of the automatic driving road test vehicle. In the traditional video monitoring or security field, the focus is generally on the presence of people, while for the image data collected by the road test vehicle, in addition to pedestrians, various motor vehicles, non-motor vehicles, other obstacles and traffic sign targets are also included in the focus. Whether each target object has a significant displacement, rotation or deformation is a potential basis for judgment. Therefore, there is an urgent need for a frame extraction method and system for road test vehicle image information. SUMMARY

[0005] In view of the above problems, the present application is proposed in order to provide a frame extraction method and system for road test vehicle image information which overcomes the above problems or at least partially solves the above problems.

[0006] In order to solve the above technical problems, the present application discloses the following technical solutions:

[0007] A frame extraction method for road test vehicle image information, comprising:

[0008] S100. Obtain road test vehicle image information and mark candidate key frames in the road test vehicle image information;

[0009] S200. According to the marked candidate key frame, based on the actual demand, the related threshold setting and index calculation;

[0010] S300. According to the set threshold and calculated index, the key frame is marked;

[0011] S400. According to the marked key frame and the set threshold and calculated index, the similar frame is marked;

[0012] S500. According to the neural network model, the non-key target frame is marked;

[0013] S600. According to the marked key frame, similar frame and non-key target frame, the image information is framed. Further, in S100, the method for marking the candidate key frame in the road test vehicle image information includes:

[0014] S101. The road test vehicle image is sampled at a rate of one frame per second, the first frame per second is selected as a representative frame, and all representative frames are combined to form a simplified data set, and in the simplified data set, the time second is used as the index of each representative frame;

[0015] S102. For the selected representative frame, the image data is divided into feature clusters according to the gray histogram using clustering method;

[0016] S103. The simplified data set is sorted according to the generation time, and for the clusters that are not continuous in time dimension, such clusters are further divided into multiple clusters according to time continuity, and each divided cluster maintains continuity in time dimension;

[0017] S104. Taking the sum of pixel point difference squares as distance measurement, the clustering center of each cluster is calculated, and the frame corresponding to the clustering center is marked as candidate key frame.

[0018] Further, in S200, based on the actual demand, the related threshold setting includes:

[0019] Pixel gray difference threshold T pd : used to judge whether there is significant difference between frames;

[0020] Frame significant difference absolute area threshold T ad : used to judge whether there is significant difference between frames according to absolute area;

[0021] Frame significant difference area ratio threshold T rd : used to judge whether there is significant difference between frames;

[0022] Image block gray maximum difference threshold T ms : used to judge whether there is local significant difference between frames;

[0023] average difference threshold T within a cluster ak : used to determine whether there is a high level of difference in the images within a cluster.

[0024] Further, in S200, based on actual needs, the relevant index calculation includes:

[0025] For each representative frame within a cluster, calculate the difference in gray value between the frame and the candidate key frame within the cluster to obtain a gray difference matrix;

[0026] Define the gray difference matrix of the representative frame with index i as M i , and based on the set threshold, calculate the following index for each matrix:

[0027] N i : represents the number of elements of the gray difference matrix M i , used to represent the size of the frame image;

[0028] S i : represents the sum of the elements of the gray difference matrix M i , used to represent the overall level of difference between frames;

[0029] AD i : represents the number of elements in the gray difference matrix M i whose value is greater than T pd , used to represent the absolute area of significant difference between frames;

[0030] RD i : represents the ratio AD i / N i of the number of elements of the gray difference matrix M i to the number of elements of the gray difference matrix M i whose value is greater than T pd , used to represent the relative area of significant difference between frames;

[0031] MaxSub i : represents the selection of a sub-matrix in the gray difference matrix Mi by traversing a window with size x by y, define the jth sub-matrix in the matrix Mi as SubM ij , compare the sum Sum ij of the elements of each sub-matrix SubM ij , and MaxSub i is the maximum value of the sum of the elements of each sub-matrix in the matrix M i , used to represent the maximum difference between the image blocks.

[0032] Further, in S200, based on actual needs, the relevant index calculation includes:

[0033] Define the cluster with index k as C. k For each cluster, calculate the following metrics:

[0034] A k : indicates C k Each gray-level difference matrix M i China RD i The mean of the values ​​is used to characterize the overall level of inter-frame differences within a cluster;

[0035] IQR k : indicates C k Each gray-level difference matrix M i China RD i The interquartile range is the difference between the upper quartile and the lower quartile.

[0036] SD k : indicates C k Each gray-level difference matrix M i China RD i Standard deviation;

[0037] V k : indicates C k Each gray-level difference matrix M i China RD i interquartile range and C k Each gray-level difference matrix M i China RD i The ratio of the standard deviations of the IQR k / SD k It is used to characterize the degree of dispersion of inter-frame differences within a cluster.

[0038] Furthermore, in S300, the methods for marking keyframes include:

[0039] S301. Determine the overall level A of intra-cluster inter-frame differences. k The degree of dispersion V of intra-cluster inter-frame differences k The magnitude of the preset threshold, when the dispersion V of the inter-frame difference within the cluster... k ≤1.5 or the overall level A of intra-cluster inter-frame difference k ≤Intra-cluster average difference threshold T ak When the conditions are met, candidate keyframes in the cluster are directly marked as keyframes;

[0040] S302. When the dispersion of inter-frame differences within a cluster is V k >1.5 and the overall level A of intra-cluster inter-frame difference k Intra-cluster average difference threshold T akWhen the time comes, the qualified representative frames in the cluster and the corresponding following frames of the representative frames are taken out as a subdivided data set, and the clustering and key frame selection actions are performed on each subdivided data set according to the operation process of step S100, to generate a subdivided new cluster to replace the original cluster, the selected frames are marked as key frames, and the gray difference matrix and related indicators of each frame image of all images in the generated new cluster are recalculated.

[0041] Further, in S400, according to the marked key frames, the set threshold and the calculated indicators, the similar frames are marked, specifically including: in each cluster, the gray difference matrix M i of each image frame and the key frame of the cluster is calculated one by one, and the following rules are calculated:

[0042] When the business side has defined the threshold T ad , the set R = (MaxSub i <T ms ) ∧ ((RD i <T rd ) ∨ (AD i <T ad ));

[0043] When the business side has not defined the threshold T ad , the set R = (MaxSub i <T ms ) ∧ (RD i <T rd );

[0044] When the set R is true, the image frame is marked as a similar frame.

[0045] Further, in S500, according to the neural network model, the non-key target frame is marked, and the specific method includes:

[0046] S501: In the way of deep learning model assisted manual marking of all key frame information, whether pedestrians, non-motor vehicles, motor vehicles, obstacles and traffic signs exist in the picture is identified, and whether the key frame contains potential high-value targets is identified on this basis;

[0047] S502: All key frames identified in step S501 and all frames corresponding in the cluster are marked as non-key target frames.

[0048] Further, in S600, according to the marked key frames, similar frames and non-key target frames, the image information is framed, and the specific method includes:

[0049] S601: All representative frames and the corresponding following frames of the representative frames marked as similar frames and non-key target frames are discarded;

[0050] S602: reserve the image frame marked as a key frame, discard the following frame corresponding to the key frame;

[0051] S603: all reserved frames are stored with their timestamp information, and the frame extraction is completed.

[0052] The application also discloses a frame extraction system for road test vehicle image information, comprising a candidate key frame marking unit, a correlation threshold setting and index calculation unit, a key frame marking unit, a similar frame marking unit, a non-key target frame marking unit and an image information frame extraction unit.

[0053] The candidate key frame marking unit is used to obtain road test vehicle image information and mark candidate key frames in the road test vehicle image information.

[0054] The correlation threshold setting and index calculation unit is used to set and calculate the correlation threshold and index based on actual requirements according to the marked candidate key frames.

[0055] The similar frame marking unit is used to mark the key frames according to the set threshold and calculated index.

[0056] The non-key target frame marking unit is used to mark the similar frames according to the marked key frames and set threshold and calculated index.

[0057] The image information frame extraction unit is used to extract the image information according to the marked key frames, similar frames and non-key target frames.

[0058] The above technical solutions provided by the embodiments of the application have at least the following advantages:

[0059] The application discloses a frame extraction method for road test vehicle image information, comprising:

[0060] Obtain road test vehicle image information and mark candidate key frames in the road test vehicle image information; set and calculate the correlation threshold and index based on actual requirements according to the marked candidate key frames; mark the key frames according to the set threshold and calculated index; mark the similar frames according to the marked key frames and set threshold and calculated index; mark the non-key target frames according to the neural network model; and extract the image information according to the marked key frames, similar frames and non-key target frames.

[0061] Compared with the conventional frame extraction technology applied in the video monitoring or security field, the application has the following advantages:

[0062] (1) highly adaptive to the business scenarios of automatic driving, fully considering the image preservation value in the field of automatic driving, and more effectively solving the data redundancy problem of the storage end.

[0063] (2) Fully embodies the design idea of automation, minimizes manual intervention, liberates limited human resources, and further improves the execution efficiency of the entire automatic driving tool chain.

[0064] (3) The image scenes of different content sections are processed by divide and conquer, and the frame extraction result is more reliable.

[0065] (4) Designed for algorithms, does not involve specific technical dependencies, and can be implemented using various development languages or tools, with a wide range of applications.

[0066] The technical solutions of the present application will be described in further detail below with reference to the accompanying drawings and examples. BRIEF DESCRIPTION OF DRAWINGS

[0067] The accompanying drawings are intended to provide a further understanding of the present application, and constitute a part of the specification, together with the embodiments of the present application, to explain the present application, and do not constitute a limitation on the present application. In the drawings:

[0068] Figure 1 For the flowchart of a frame extraction method for road test vehicle image information in Embodiment 1 of the present application. DETAILED DESCRIPTION

[0069] Exemplary embodiments of the present disclosure will be described in greater detail below with reference to the accompanying drawings. Although exemplary embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be limited by the embodiments set forth herein. On the contrary, these embodiments are provided so that the present disclosure can be more thoroughly understood and the scope of the present disclosure can be accurately conveyed to those skilled in the art.

[0070] To solve the problems in the prior art, the embodiments of the present application provide a frame extraction method and system for road test vehicle image information.

[0071] Embodiment 1

[0072] The present embodiment discloses a frame extraction method for road test vehicle image information, characterized by comprising:

[0073] S100. Obtain road test vehicle image information and mark candidate key frames in the road test vehicle image information;

[0074] Specifically, in S100 of the present embodiment, the method for marking candidate key frames in the road test vehicle image information comprises:

[0075] S101. Sample the road test vehicle image at a rate of one frame per second, select the first frame of each second as a representative frame, and select all representative frames to form a simplified dataset, and in the simplified dataset, use time in seconds as the index of each representative frame; Specifically, select the first frame of each second as a representative frame (for example, a video file with a frame rate of 30 fps can be regarded as 30 consecutive image data per second, then sample and select the 1st, 31st, 61st, 91st… frame image, and the rest is the same).

[0076] S102. For the selected representative frame, use clustering to divide the image data into feature clusters according to the gray level histogram; Specifically, calculate the gray level histogram of each frame image in the simplified dataset, use the gray level histogram as the feature, and use the hierarchical clustering method to cluster the simplified dataset, after clustering, the representative frame image is labeled with the cluster number it belongs to, and each cluster is sorted in chronological order, and the serial number is used as the index of each cluster.

[0077] S103. Sort the simplified dataset by generation time, and for clusters that are not continuous in time dimension (i.e. at least two frames in the cluster have a time index difference greater than 1 second), further divide such clusters into multiple clusters according to time continuity, and ensure that each cluster after division is continuous in time dimension.

[0078] S104. Calculate the cluster center for each representative frame in the cluster using the sum of squares of pixel point differences as the distance measure, and mark the frame corresponding to the cluster center as a candidate key frame.

[0079] S200. According to the marked candidate key frame, set the related threshold and calculate the index based on the actual demand; According to the business demand, a series of thresholds are set, which are respectively used to: ① judge whether there is a significant difference between frames; ② judge whether there is a significant difference between frames according to the absolute area (this threshold is set according to the actual demand of the business, for example, when the image resolution is low or the business needs to filter some fixed small size targets, this threshold can be considered to be set, or not to be set); ③ judge whether there is a significant difference between frames according to the area ratio; ④ judge whether there is a local significant difference between frames; ⑤ judge the overall difference level of the images in the cluster. The high and low of the five kinds of thresholds mainly depends on the strictness of the business demand for image frame extraction. Generally, the higher the absolute value of the threshold, the more relaxed the standard of judging the similarity between images, and more image data will be discarded because it is judged as a similar frame, which is suitable for scenes that are sensitive to data storage pressure; On the contrary, the more strict the standard of judging the similarity between images, the more image data will be retained, which is suitable for scenes that are sensitive to missing key information. The application party can select appropriate threshold level according to the actual business.

[0080] The gray scale difference is calculated between each representative frame in the cluster and the key frame, and a series of indicators of the representative frame are calculated, which are respectively used for: ① representing the overall level of inter-frame difference; ② representing the absolute area of significant difference between frames; ③ representing the area ratio of significant difference between frames; and ④ representing the maximum difference of image blocks between frames.

[0081] A series of indicators of the cluster are calculated, which are respectively used for: ① representing the overall level of inter-frame difference within the cluster (its business significance is that the lower the value, the smaller the inter-frame difference within the cluster, which corresponds to a scene where the picture in the field of view changes slowly or even does not change, represented by low-speed driving and parking; on the contrary, the higher the value, the larger the inter-frame difference within the cluster, which corresponds to a scene where the picture in the field of view changes significantly, represented by high-speed driving, turning or driving on a high-density traffic section); and ② representing the dispersion degree of inter-frame difference within the cluster (its business significance is that the lower the value, the higher the consistency of inter-frame difference within the cluster, which corresponds to a business scene where the picture in the field of view changes uniformly, represented by constant-speed cruising; on the contrary, the higher the value, the lower the consistency of inter-frame difference within the cluster, which corresponds to a scene where the picture in the field of view changes complexly, represented by vehicle turning, lane changing, driving at an intersection and driving on a high-density traffic section).

[0082] By comparing the level of the cluster indicators with the threshold value, clusters with large image change amplitude and frequency are selected, and the representative frames in the clusters and the following frames corresponding to the representative frames are again subjected to the clustering and key frame selection actions to mine new key frames. Since all the following frames in the clusters have been taken out for recalculation, each frame in the clusters is marked as a representative frame, and no following frame is set. The new clusters are generated, and all the images and clusters are again calculated for the aforementioned related indicators (its business significance is that in the foregoing process, the representative frames are used to replace the full image data for calculation, and the setting of the threshold value also cannot take into account the individualized features of each image, so there is inevitably a possibility of missing potential high-value targets. In this step, the key frames are again subdivided and marked from the images that change significantly in a short time, so as to retain as much image data containing potential high-value targets as possible without significantly affecting the execution performance).

[0083] Specifically, based on actual needs, related threshold settings and indicator calculations are specifically included.

[0084] S201: For each representative frame in each cluster, the gray scale difference matrix is obtained by calculating the gray scale difference between the representative frame and the candidate key frame in the cluster;

[0085] S202: The following threshold values are set based on actual business needs:

[0086] Pixel gray scale difference threshold T pd : used for judging whether there is a significant difference between pixels between frames;

[0087] Inter-frame significant difference absolute area threshold T adUsed to determine whether there are significant differences between images in different frames based on their absolute area;

[0088] Inter-frame significant difference area ratio threshold T rd Used to determine whether there are significant differences between images in different frames;

[0089] Image patch grayscale maximum difference threshold T ms Used to determine whether there are significant local differences between images in different frames;

[0090] Intra-cluster average difference threshold T ak Used to determine whether there is a generally high level of difference in images within a cluster.

[0091] S203: Define the gray-level difference matrix of the representative frame at index i as M. i And based on the threshold T d For each matrix, calculate the following index:

[0092] Define the grayscale difference matrix of the frame represented by index i as M. i And based on a set threshold, the following metrics are calculated for each matrix:

[0093] N i : Represents the gray-level difference matrix M i The number of elements used to characterize the size of the frame image;

[0094] S i : Represents the gray-level difference matrix M i The sum of the elements is used to characterize the overall level of inter-frame differences;

[0095] AD i : Represents the gray-level difference matrix M i Median greater than T pd The number of elements is used to characterize the absolute area where there are significant differences between frames;

[0096] RD i : Represents the gray-level difference matrix M i The number of elements and the gray-level difference matrix M i Median greater than T pd The ratio of the number of elements in AD i / N i , used to characterize the relative area where there are significant differences between frames;

[0097] MaxSub i This indicates that a submatrix is ​​selected by traversing the gray-level difference matrix Mi according to a window size of length x and width y. The j-th submatrix in matrix Mi is defined as SubM. ij Compare each submatrix SubM ij Sum of elements ij MaxSubi i.e. the maximum value of the sum of each element in the matrix M i i.e. the maximum value of the sum of each element in the matrix M

[0098] S204: define the cluster with index k as C k For each cluster, calculate the following indicators:

[0099] A k : represents the average value of RD k in each gray difference matrix M i in C i , used to represent the overall level of inter-frame difference within the cluster;

[0100] IQR k : represents the interquartile range of RD k in each gray difference matrix M i in C i , i.e. the difference between the upper quartile and the lower quartile;

[0101] SD k : represents the standard deviation of RD k in each gray difference matrix M i in C i ;

[0102] V k : represents the ratio of IQR k / SD i of RD i in each gray difference matrix M k in C i , used to represent the dispersion degree of inter-frame difference within the cluster.

[0103] S300. According to the set threshold value and the calculated indicators, the key frame is marked; specifically, in S300 of the embodiment, the method for marking the key frame includes:

[0104] S301. Determine whether the overall level A k of inter-frame difference within the cluster and the dispersion degree V k of inter-frame difference within the cluster are greater than the preset threshold value, when the dispersion degree V k of inter-frame difference within the cluster is ≤1.5 or the overall level A k of inter-frame difference within the cluster is ≤ the average difference threshold T ak of the cluster, the candidate key frame in the cluster meeting the condition is directly marked as a key frame;

[0105] S302. When the dispersion degree V k of inter-frame difference within the cluster is >1.5 or the overall level A k of inter-frame difference within the cluster is > the average difference threshold T ak of the cluster, the candidate key frame in the cluster meeting the condition is marked as a key frame according to the following rules:k >1.5 and the overall level of inter-frame variance within the cluster A k >the average variance threshold T within the cluster ak , the representative frame within the eligible cluster and the following frames corresponding to the representative frame are taken out as a subdivided dataset, and the clustering and key frame selection actions are performed on each subdivided dataset according to the operation process of step S100, to generate a subdivided new cluster to replace the original cluster, the selected frames are marked as key frames, and the generated new cluster is used to recalculate the gray difference matrix of each frame image and the related indicators.

[0106] S400. According to the marked key frames, the set threshold and the calculated indicators, similar frames are marked; in S400 of the embodiment, according to the marked key frames, the set threshold and the calculated indicators, similar frames are marked, which specifically includes: in each cluster, the gray difference matrix M i of each image frame and the key frame of the cluster is calculated one by one, and the following rules are used for calculation:

[0107] When the business side has defined the threshold T ad , the set R = (MaxSub i <T ms ) ∧ ((RD i <T rd ) ∨ (AD i <T ad ));

[0108] When the business side has not defined the threshold T ad , the set R = (MaxSub i <T ms ) ∧ (RD i <T rd );

[0109] When the set R is true, the image frame is marked as a similar frame.

[0110] S500. According to the neural network model, mark the non-key target frame; in S500 of the embodiment, according to the neural network model, mark the non-key target frame, and the specific method includes:

[0111] S501: In the manner of artificial marking assisted by a deep learning model, all key frame information is identified to identify whether there are pedestrians, non-motor vehicles, motor vehicles, obstacles and traffic signs in the picture, and on this basis, key frames without potential high-value targets are identified;

[0112] S502: All key frames without potential high-value targets identified in step S501 and all frames corresponding in the cluster are marked as non-key target frames.

[0113] S600. Frame the image information according to the marked key frame, similar frame and non-key target frame. Specifically, in S600, the image information is framed according to the marked key frame, similar frame and non-key target frame, and the specific method includes:

[0114] S601: all representative frames marked as similar frames and non-key target frames and the following frames corresponding to the representative frames are discarded;

[0115] S602: the image frame marked as a key frame is retained, and the following frame corresponding to the key frame is discarded;

[0116] S603: all retained frames are stored with their timestamp information, and the framing is completed.

[0117] The embodiment discloses a frame extraction method for image information of a road test vehicle, comprising:

[0118] Obtain the image information of the road test vehicle, and mark the candidate key frame in the image information of the road test vehicle; according to the marked candidate key frame, set the related threshold value and calculate the index based on the actual demand; according to the set threshold value and the calculated index, mark the key frame; according to the marked key frame and the set threshold value and the calculated index, mark the similar frame; according to the neural network model, mark the non-key target frame; and frame the image information according to the marked key frame, similar frame and non-key target frame.

[0119] Compared with the conventional frame extraction technology applied in the video monitoring or security field, the present application has the following advantages:

[0120] (1) Highly adaptive to the business scenario of automatic driving, fully considering the image retention value in the field of automatic driving, and more effectively solving the data redundancy problem of the storage end.

[0121] (2) Fully embodies the design idea of automation, minimizes human intervention, liberates limited human resources, and further improves the execution efficiency of the entire automatic driving tool chain.

[0122] (3) The image scenes of different content sections are processed by divide and conquer, and the frame extraction result is more reliable.

[0123] (4) Designed for algorithms, does not involve specific technical dependencies, and can be implemented using various development languages or tools, and has a wide application range.

[0124] Embodiment 2

[0125] The embodiment also discloses a frame extraction system for road test vehicle image information, comprising: a candidate key frame marking unit, a correlation threshold setting and index calculation unit, a key frame marking unit, a similar frame marking unit, a non-key target frame marking unit and an image information frame extraction unit; wherein:

[0126] The candidate key frame marking unit is used for acquiring the road test vehicle image information and marking candidate key frames in the road test vehicle image information.

[0127] The correlation threshold setting and index calculation unit is used for setting the correlation threshold and calculating the index based on actual requirements according to the marked candidate key frames.

[0128] The similar frame marking unit is used for marking the key frames according to the set threshold and the calculated index.

[0129] The non-key target frame marking unit is used for marking the similar frames according to the marked key frames and the set threshold and the calculated index.

[0130] The image information frame extraction unit is used for extracting the image information according to the marked key frames, similar frames and non-key target frames.

[0131] The working methods of the candidate key frame marking unit, the correlation threshold setting and index calculation unit, the key frame marking unit, the similar frame marking unit, the non-key target frame marking unit and the image information frame extraction unit have been described in detail in Embodiment 1, and will not be repeated here.

[0132] It should be understood that the specific order or hierarchy of steps in the processes disclosed should not be interpreted as reflecting an essential sequence for any claim. Based on design preference, it should be understood that the specific order or hierarchy of steps in the processes can be rearranged without departing from the scope of the present disclosure. The appended method claims list the elements of the various steps in the example order, and are not intended to be limited to the specific order or hierarchy presented.

[0133] In the above detailed description, various features are grouped together in single embodiments for the purpose of streamlining the disclosure. Such disclosed approaches should not be interpreted as reflecting an intention that the claimed embodiments require more features than are explicitly recited in each claim. On the contrary, as indicated above, the inventiveness lies in fewer than all features of the disclosed single embodiments. Accordingly, the claims are hereby expressly incorporated into this detailed description, with each claim standing on its own as a separate preferred embodiment of the present application.

[0134] Those of skill would further appreciate that the various illustrative logical blocks, modules, circuits, and algorithm steps described in connection with the embodiments disclosed herein can be implemented as electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various illustrative components, blocks, modules, circuits, and steps have been described above generally in terms of their functionality. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the overall system. Skilled artisans can implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present disclosure.

[0135] The steps of a method or algorithm described in connection with the embodiments disclosed herein can be embodied directly in hardware, in a software module executed by a processor, or in a combination of the two. A software module can reside in RAM memory, flash memory, ROM memory, EPROM memory, EEPROM memory, registers, hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor such that the processor can read information from, and write information to, the storage medium. In the alternative, the storage medium can be integral to the processor. The processor and the storage medium can reside in an ASIC. The ASIC can reside in a user terminal. In the alternative, the processor and the storage medium can reside as discrete components in a user terminal.

[0136] For a software implementation, the techniques described herein can be implemented with modules (e.g., procedures, functions, and so on) that perform the functions described herein. The software codes can be stored in memory units and executed by processors. The memory unit can be implemented within the processor or external to the processor, in which case it can be communicatively coupled to the processor via various means as is known in the art.

[0137] The above description includes one or more examples of the embodiments. Of course, not all possible combinations of components or method steps described above can be claimed as embodiments. One of ordinary skill in the art can recognize that modifications and variations of the embodiments described herein are possible and are within the scope of the present disclosure. It is therefore intended that the embodiments described herein be considered in all respects as only illustrative and not restrictive. Specifically, the description of the embodiments should be considered to be illustrative and not exhaustive, and should be considered in the light of the claims. Further, the use of the term "comprise" in the specification is to be construed in the same way as the term "comprise" as explained in the description of the claims. Also, the use of the term "or" in the claims is to be construed as "non-exclusive or" as explained in the description of the claims.

Claims

1. A frame extraction method for image information of a road test vehicle, characterized in that, Comprise: S100. Obtain road test vehicle image information, and mark candidate key frames in the road test vehicle image information; In S100, the method of marking candidate key frames in the road test vehicle image information comprises: S101. Sample the road test vehicle image at a rate of one frame per second, select the first frame per second as a representative frame, and form a simplified dataset by combining all representative frames, wherein the time number of seconds is used as the index of each representative frame in the simplified dataset; S102. For the selected representative frames, use clustering to divide the image data into feature clusters according to the gray histogram; S103. Sort the simplified dataset according to the generation time, and further divide the clusters that are not continuous in the time dimension into multiple clusters according to the time continuity, so that each divided cluster maintains continuity in the time dimension; S104. Calculate the cluster center of each cluster in the representative frame by taking the sum of the squares of the pixel point difference as the distance measure, and mark the frame corresponding to the cluster center as a candidate key frame; S200. According to the marked candidate key frames, set relevant thresholds and calculate indicators based on actual needs; S300. According to the set thresholds and calculated indicators, mark the key frames; S400. According to the marked key frames and set thresholds and calculated indicators, mark the similar frames; S500. According to the neural network model, mark the non-key target frames; S600. According to the marked key frames, similar frames and non-key target frames, frame the image information.

2. The frame extraction method for image information of a road test vehicle according to claim 1, wherein In S200, based on actual needs, relevant threshold settings include: Pixel gray scale difference threshold value T pd : for judging whether there is a significant difference between inter-frame pixels; inter-frame significant difference absolute area threshold T ad : for judging whether there is a significant difference between frames by absolute area; inter-frame significant difference area proportion threshold T rd : for judging whether there is a significant difference between frames of images; Image block gray level maximum difference threshold T ms : for judging whether the inter-frame image has local significant difference Intra-cluster average difference threshold T ak : used to determine whether images within a cluster generally exhibit a high level of difference.

3. The method for frame extraction of image information of a road test vehicle according to claim 2, wherein, In S200, based on actual needs, relevant index calculations include: For each representative frame in the cluster, calculate the gray value difference with the candidate key frame in the cluster to obtain a gray difference matrix; The gray level difference matrix of the representative frame with index i is defined as M i and based on a set threshold, the following index is calculated for each matrix: N i : represents the number of elements of the matrix M i of the difference in gray levels, used to characterize the size of the frame image; S i : represents the sum of the elements of the gray scale difference matrix M i , for representing the overall level of inter-frame difference; AD i : represents the gray scale difference matrix M i the number of elements whose median is greater than T pd for representing the absolute area of significant difference between frames; RD i : represents the element number of the gray difference matrix M i i : represents the element number of the gray difference matrix M i whose value is greater than T pd : represents the ratio AD i / N i of the element number of the gray difference matrix M i whose value is greater than T pd , used to represent the relative area of significant difference between frames; MaxSub i : represents the selection of a sub-matrix in the gray-scale difference matrix M by traversing the window size of x length and y width, and defines the jth sub-matrix in the matrix M as SubM ij , compares the sum of elements of each sub-matrix SubM ij , and MaxSub ij is the maximum value of the sum of elements of each sub-matrix in the matrix M i , which is used to represent the maximum difference of the inter-frame image block. i ​ 4. The frame extraction method for image information of a road test vehicle according to claim 3, wherein, In S200, based on actual needs, relevant index calculations further include: A cluster with index k is defined as C k For each cluster, the following index is calculated: Ak: represents the mean of RDi in each gray difference matrix Mi in Ck, and is used to represent the overall level of inter-frame difference in the cluster; IQRk: represents the interquartile range of RDi in each gray difference matrix Mi in Ck, i.e. the difference between the upper quartile and the lower quartile; SDk: represents the standard deviation of RDi in each gray difference matrix Mi in Ck; Vk: represents the ratio of the interquartile range of RDi in each of the gray difference matrixes Mi in Ck to the standard deviation of RDi in each of the gray difference matrixes Mi in Ck IQR k / SD k , for characterizing the degree of dispersion of the inter-frame differences within the cluster.

5. The method for frame extraction of image information of a road test vehicle according to claim 3, wherein, In S300, the method of marking key frames comprises: S301. judging a total level A of inter-frame difference within the cluster k and a dispersion degree V of inter-frame difference within the cluster k When the dispersion degree V of inter-frame difference within the cluster k ≤ 1.5 or the total level A of inter-frame difference within the cluster k ≤ a cluster average difference threshold T ak the candidate key frame in the qualified cluster is directly marked as a key frame. S302. When the discrete degree V k > 1.5 and the overall level A k > of the inter-frame difference within the cluster is greater than the average difference threshold T ak , the qualified representative frame within the cluster and the corresponding following frame are taken out as a subdivided dataset, and the clustering and key frame selection are performed on each subdivided dataset according to the operation process of step S100, to generate a subdivided new cluster to replace the original cluster, the selected frame is marked as a key frame, and the gray difference matrix and the related indexes of each frame image are recalculated for all the images in the generated new cluster.

6. The method for frame extraction of image information of a road test vehicle according to claim 3, wherein, In S400, the similar frames are marked according to the marked key frames, the set threshold and the calculated index, specifically including: in each cluster, the gray difference matrix M between the image frame and the key frame of the cluster is calculated one by one i , and the calculation is as follows: When the business side has defined a threshold T ad , the set R = (MaxSub i < T ms ) ∧ ((RD i < T rd ) ∨ (AD i <T ad )) When the business side does not define the threshold T ad , the set R = (MaxSub i < T ms ) ∧ (RD i < T rd ); When the set R is true, the image frame is marked as a similar frame.

7. The method for frame extraction of image information of a road test vehicle according to claim 1, wherein, In S500, according to the neural network model, the non-key target frames are marked, and the specific method comprises: S501: using a deep learning model assisted manual marking method on all key frame information, identifying whether there are pedestrians, non-motor vehicles, motor vehicles, obstacles and traffic signs in the picture, and identifying key frames without potential high-value targets based on this; S502: all key frames without potential high-value targets identified in step S501 and all frames corresponding in the cluster are marked as non-key target frames.

8. The method for frame extraction of image information of a road test vehicle according to claim 1, wherein, In S600, according to the marked key frames, similar frames and non-key target frames, the image information is framed, and the specific method comprises: S601: all representative frames marked as similar frames and non-key target frames and the following frames corresponding to the batch of representative frames are discarded; S602: the image frames marked as key frames are retained, and the following frames corresponding to the key frames are discarded; S603: all retained frames are stored with their timestamp information, and the frame extraction is completed.

9. A frame extraction system for road test vehicle image information, which adopts the frame extraction method for road test vehicle image information according to any one of claims 1-8, characterized in that, Comprise: a candidate key frame marking unit, a correlation threshold setting and index calculation unit, a key frame marking unit, a similar frame marking unit, a non-key target frame marking unit, and an image information frame extraction unit; wherein: the candidate key frame marking unit is configured to obtain road test vehicle image information and mark candidate key frames in the road test vehicle image information; the correlation threshold setting and index calculation unit is configured to set and calculate the correlation threshold and the index based on actual requirements according to the marked candidate key frames; the similar frame marking unit is configured to mark the key frames according to the set threshold and the calculated index; the non-key target frame marking unit is configured to mark the similar frames according to the marked key frames and the set threshold and the calculated index; the image information frame extraction unit is configured to extract the image information according to the marked key frames, similar frames, and non-key target frames.

Images