Frame type determination method, device, electronic device and storage medium

Abstract

The present disclosure relates to the field of video coding and decoding, and proposes a frame type determination method, device, electronic device and storage medium. The method extracts the features of each video frame in a video frame sequence, and obtains the matching degree between the video frame and the adjacent video frame according to the feature analysis of each video frame; the video frame sequence is divided into multiple groups of video frames, and for each group of video frames, a first threshold is determined according to the matching degree between each video frame in the group of video frames and the adjacent video frame; in the group of video frames, when the matching degree between any video frame and the adjacent video frame is less than the first threshold, the video frame is used as a candidate video frame corresponding to the group of video frames; and an intra-frame coded frame is determined in the candidate video frames. The process of the frame type determination method of the embodiment of the present disclosure is simpler, and the first threshold is determined in combination with the matching degree between the video frame in the group and the adjacent video frame, as the judgment threshold of the type of the video frame in the group, so that the accuracy of determining the intra-frame coded frame is higher.

Description

Technical Field

[0001] The present disclosure relates to the field of video coding and decoding, and in particular to a frame type determination method, device, electronic device and storage medium. Background Art

[0002] The amount of data in a video frame sequence is very large. In order to reduce transmission power consumption and storage cost, the video frame sequence is usually encoded before transmission or storage. In a video frame sequence, not all video frames must be encoded in the same way. Video frames can be divided into three types: intra coded picture, forward predictive coded frame, and bi-directional predictive interpolation coded frame.

[0003] Among them, intra-frame coded frames are also called I frames, forward predictive coded frames are also called P frames, and bidirectional predictive interpolation coded frames are also called B frames. I frames refer to video frames with scene switching (large changes in the content of the frame) compared with the previous frame. After being encoded, I frames still retain complete data and can be decoded independently without referring to other video frames. Therefore, the encoding method of I frames is intra-frame coding. P frames need to refer to the data of the previous video frame to complete encoding, and also need to refer to the encoding results of the previous video frame when decoding. Therefore, the encoding method of P frames is forward predictive coding. B frames need to refer to the data of the previous video frame and the next video frame to complete encoding. Also need to refer to the encoding results of the previous video frame and the next video frame when decoding. Therefore, the encoding method of B frames is bidirectional predictive interpolation coding. Obviously, I frames are the most important video frames. The higher the judgment accuracy of I frames, the better the video encoding effect.

[0004] The I frame determination methods proposed in the prior art are either relatively simple but have low determination accuracy, or have high accuracy but are too complicated. How to accurately determine the I frame in a video frame sequence in a simpler way has become a research hotspot in this field. Summary of the invention

[0005] In view of this, the present disclosure proposes a frame type determination method, device, electronic device and storage medium. The process of the frame type determination method of the embodiment of the present disclosure is simpler, and the first threshold is determined in combination with the matching degree between the video frame in the group and the adjacent video frame, which serves as the judgment threshold of the type of the video frame in the group, so that the accuracy of determining the intra-frame coding frame is higher.

[0006] According to one aspect of the present disclosure, a frame type determination method is provided, the method being used to determine an intra-frame coded frame in a video frame sequence, the method comprising: extracting features of each video frame in the video frame sequence, analyzing feature matching between each video frame and adjacent video frames, and obtaining, for each video frame, a degree of matching between the video frame and adjacent video frames; dividing the video frame sequence into a plurality of groups of video frames, wherein video frames belonging to the same group are continuous; for each group of video frames, determining a first threshold value based on a degree of matching between each video frame in the group of video frames and adjacent video frames; when a degree of matching between any one video frame in the group of video frames and an adjacent video frame is less than the first threshold value, taking the video frame as a candidate video frame corresponding to the group of video frames; and determining an intra-frame coded frame from the candidate video frames.

[0007] In a possible implementation, the video frame includes multiple pixel blocks, and extracting the features of each video frame in the video frame sequence includes: downsampling each video frame in the video frame sequence to obtain multiple sampling points, and each pixel block is downsampled into one sampling point; extracting at least one of a first feature, a second feature, and a third feature of each video frame as the feature of the video frame, wherein the first feature is the average absolute error of pixel points in the pixel block corresponding to each sampling point of the video frame; the second feature is the pixel value of each sampling point of the video frame; and the third feature is the average absolute error between each sampling point of the video frame and adjacent sampling points.

[0008] In a possible implementation, the analyzing feature matching between each video frame and adjacent video frames and obtaining the degree of matching between each video frame and adjacent video frames includes: analyzing feature matching between each video frame and adjacent video frames to obtain at least one type of matching point number between each video frame and adjacent video frames, wherein the matching point is a sampling point that matches between the video frame and the adjacent video frames; and taking the sum of at least one type of matching point number between each video frame and the adjacent video frames as the degree of matching between the video frame and the adjacent video frames.

[0009] In a possible implementation, determining the intra-frame coded frame in the candidate video frame includes: when the group of video frames corresponds to two or more candidate video frames, determining a second threshold corresponding to each type of matching point based on the number of matching points of each type between each video frame in the group of video frames and adjacent video frames; for each type of matching point, selecting, as the intra-frame coded frame, a candidate video frame with the smallest number of matching points of this type and less than the second threshold corresponding to this type of matching points.

[0010] In a possible implementation manner, determining the intra-coded frame in the candidate video frames includes: when the group of video frames corresponds to only one candidate video frame, determining the candidate video frame as the intra-coded frame.

[0011] In a possible implementation, analyzing the feature matching between each video frame and adjacent video frames to obtain the number of at least one type of matching points between each video frame and adjacent video frames includes: for any video frame, determining whether each feature of the sampling point of the video frame matches the sampling point of the adjacent video frame; when the first feature, the second feature, and the third feature of the first sampling point of the video frame match the first feature, the number of first-category matching points of the video frame increases by one; when the first feature and the third feature of the first sampling point of the video frame match the first feature and the third feature of the second sampling point of the adjacent video frame, but the second feature does not match, the number of second-category matching points of the video frame increases by one; when the first feature and the second feature of the first sampling point of the video frame match the first feature and the second feature of the second sampling point of the adjacent video frame, but the third feature does not match, the number of third-category matching points of the video frame increases by one; wherein the first sampling point is any sampling point of the video frame, and the second sampling point is any sampling point of the adjacent video frame.

[0012] In a possible implementation, determining, for any video frame, whether a sampling point of the video frame matches each feature of a sampling point of an adjacent video frame includes: for any video frame, generating a histogram corresponding to each feature of the video frame according to the features of the video frame; generating a histogram corresponding to each feature of the adjacent video frames according to the features of the adjacent video frames; wherein the interval intervals of the histograms corresponding to the same feature are the same, the abscissa of the histogram is the feature value, the ordinate of the histogram is the sampling point, and when the first sampling point matches any one feature of the second sampling point, the interval hit by the first sampling point in the histogram corresponding to the feature is the same as the interval hit by the second sampling point in the histogram corresponding to the same feature.

[0013] In a possible implementation, determining the first threshold value based on the degree of matching between each video frame in the group of video frames and adjacent video frames includes: calculating an average value of the degree of matching between each video frame in the group of video frames and adjacent video frames; and multiplying the calculated average value by a preset proportional coefficient as the first threshold value.

[0014] In a possible implementation, the method of determining the second threshold value corresponding to the number of each type of matching points between each video frame in the group of video frames and adjacent video frames includes: for each type of matching points, calculating the average number of the number of matching points of this type between each video frame in the group of video frames and adjacent video frames; and taking the product of the calculated average value and a preset proportional coefficient as the second threshold value corresponding to the number of matching points of this type.

[0015] According to another aspect of the present disclosure, a frame type determination device is provided, the device is used to determine intra-frame coded frames in a video frame sequence, the device includes: an analysis module, used to extract features of each video frame in the video frame sequence, analyze the feature matching between each video frame and adjacent video frames, and obtain the matching degree between each video frame and adjacent video frames for each video frame; a grouping module, used to divide the video frame sequence into multiple groups of video frames, and the video frames belonging to the same group are continuous; a first determination module, used to determine a first threshold for each group of video frames according to the matching degree between each video frame in the group of video frames and adjacent video frames; a second determination module, used to use any video frame in the group of video frames as a candidate video frame corresponding to the group of video frames when the matching degree between any video frame and the adjacent video frame is less than the first threshold; a third determination module, used to determine the intra-frame coded frame among the candidate video frames.

[0016] In a possible implementation, the video frame includes multiple pixel blocks, and the first analysis module is used to downsample each video frame in the video frame sequence to obtain multiple sampling points, and each pixel block is downsampled into one sampling point; extract at least one of a first feature, a second feature, and a third feature of each video frame as a feature of the video frame, wherein the first feature is the average absolute error of pixel points in the pixel block corresponding to each sampling point of the video frame; the second feature is the pixel value of each sampling point of the video frame; and the third feature is the average absolute error between each sampling point of the video frame and adjacent sampling points.

[0017] In a possible implementation, the first analysis module is further used to analyze the feature matching between each video frame and adjacent video frames, and obtain the number of at least one type of matching points between each video frame and adjacent video frames, wherein the matching points are sampling points that match between the video frame and the adjacent video frames; and the sum of the number of at least one type of matching points between each video frame and the adjacent video frames is used as the matching degree between the video frame and the adjacent video frames.

[0018] In a possible implementation, the third determination module is used to determine, when the group of video frames corresponds to two or more candidate video frames, a second threshold corresponding to the matching points of each type based on the number of matching points of each type between each video frame in the group of video frames and adjacent video frames; for each type of matching points, select the candidate video frame with the smallest number of matching points of this type and less than the second threshold corresponding to this type of matching points as the intra-frame coded frame.

[0019] In a possible implementation manner, the third determination module is configured to, when the group of video frames corresponds to only one candidate video frame, determine the candidate video frame as an intra-coded frame.

[0020] In a possible implementation, analyzing the feature matching between each video frame and adjacent video frames to obtain the number of at least one type of matching points between each video frame and adjacent video frames includes: for any video frame, determining whether each feature of the sampling point of the video frame matches the sampling point of the adjacent video frame; when the first feature, the second feature, and the third feature of the first sampling point of the video frame match the first feature, the number of first-category matching points of the video frame increases by one; when the first feature and the third feature of the first sampling point of the video frame match the first feature and the third feature of the second sampling point of the adjacent video frame, but the second feature does not match, the number of second-category matching points of the video frame increases by one; when the first feature and the second feature of the first sampling point of the video frame match the first feature and the second feature of the second sampling point of the adjacent video frame, but the third feature does not match, the number of third-category matching points of the video frame increases by one; wherein the first sampling point is any sampling point of the video frame, and the second sampling point is any sampling point of the adjacent video frame.

[0021] In a possible implementation, determining, for any video frame, whether a sampling point of the video frame matches each feature of a sampling point of an adjacent video frame includes: for any video frame, generating a histogram corresponding to each feature of the video frame according to the features of the video frame; generating a histogram corresponding to each feature of the adjacent video frames according to the features of the adjacent video frames; wherein the interval intervals of the histograms corresponding to the same feature are the same, the abscissa of the histogram is the feature value, the ordinate of the histogram is the sampling point, and when the first sampling point matches any one feature of the second sampling point, the interval hit by the first sampling point in the histogram corresponding to the feature is the same as the interval hit by the second sampling point in the histogram corresponding to the same feature.

[0022] In a possible implementation, the first determination module is used to calculate an average value of the degree of matching between each video frame in the group of video frames and adjacent video frames; and use the product of the calculated average value and a preset proportionality coefficient as the first threshold.

[0023] In a possible implementation, the method of determining the second threshold value corresponding to the number of each type of matching points between each video frame in the group of video frames and adjacent video frames includes: for each type of matching points, calculating the average number of the number of matching points of this type between each video frame in the group of video frames and adjacent video frames; and taking the product of the calculated average value and a preset proportional coefficient as the second threshold value corresponding to the number of matching points of this type.

[0024] According to another aspect of the present disclosure, an electronic device is provided, comprising: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to implement the above method when executing the instructions stored in the memory.

[0025] According to another aspect of the present disclosure, a non-volatile computer-readable storage medium is provided, on which computer program instructions are stored, wherein the computer program instructions implement the above method when executed by a processor.

[0026] According to another aspect of the present disclosure, a computer program product is provided, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0027] According to the frame type determination method of the embodiment of the present disclosure, by extracting the features of each video frame in the video frame sequence, analyzing the feature matching between each video frame and the adjacent video frames, and obtaining the matching degree between each video frame and the adjacent video frames for each video frame, the matching degree of the two video frames can reflect the possibility of scene switching between the two video frames; the video frame sequence is divided into multiple groups of video frames, and the video frames belonging to the same group are continuous. For each group of video frames, a first threshold is determined according to the matching degree between each video frame in the group of video frames and the adjacent video frames. When the matching degree between any video frame and the adjacent video frames in the group of video frames is less than the first threshold, the video frame is used as a candidate video frame corresponding to the group of video frames, so that when there is a large difference in the content of video frames collected in different time periods in the video frame sequence, the first threshold is more strongly correlated with the video frames in the group, and the possibility of scene switching between the candidate video frames determined based on the first threshold and the adjacent video frames is also greater; intra-frame coded frames are determined from the candidate video frames, so that the determination accuracy of intra-frame coded frames is higher. The frame type determination method of the embodiment of the present disclosure can also determine the intra-frame coded frame without pre-coding, so the frame type process is simpler; and the first threshold is determined in combination with the matching degree between the video frame in the group and the adjacent video frame as the judgment threshold of the type of the video frame in the group, and the correlation between frames is taken into account, so that the accuracy of determining the intra-frame coded frame is higher.

[0028] Further features and aspects of the present disclosure will become apparent from the following detailed description of exemplary embodiments with reference to the attached drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0029] The accompanying drawings, which are incorporated in and constitute a part of the specification, illustrate exemplary embodiments, features, and aspects of the disclosure and, together with the description, serve to explain the principles of the disclosure.

[0030] Figure 1 A schematic diagram showing the process of the third type of I frame detection method in the prior art.

[0031] Figure 2 An exemplary application scenario of the frame type determination method according to an embodiment of the present disclosure is shown.

[0032] Figure 3a A schematic diagram showing the process of a frame type determination method according to an embodiment of the present disclosure.

[0033] Figure 3b A schematic diagram showing the process of a frame type determination method according to an embodiment of the present disclosure.

[0034] Figure 4 A schematic diagram showing a method flow of extracting features of a video frame and calculating a matching degree between a video frame and adjacent video frames based on the features of the video frame according to an embodiment of the present disclosure.

[0035] Figure 5a An example of video frame sampling before and after according to an embodiment of the present disclosure is shown.

[0036] Figure 5b An example of a method of determining whether sampling points of video frame 2 and video frame 1 match according to an embodiment of the present disclosure is shown.

[0037] Figure 6 A schematic diagram showing the structure of a frame type determination device according to an embodiment of the present disclosure.

[0038] Figure 7 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. DETAILED DESCRIPTION

[0039] Various exemplary embodiments, features and aspects of the present disclosure will be described in detail below with reference to the accompanying drawings. The same reference numerals in the accompanying drawings represent elements with the same or similar functions. Although various aspects of the embodiments are shown in the accompanying drawings, the drawings are not necessarily drawn to scale unless otherwise specified.

[0040] The word “exemplary” is used exclusively herein to mean “serving as an example, example, or illustration.” Any embodiment described herein as “exemplary” is not necessarily to be construed as preferred or advantageous over other embodiments.

[0041] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the following specific embodiments. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the subject matter of the present disclosure.

[0042] The I frame determination methods proposed in the prior art mainly include the following three categories.

[0043] The first type is the I-frame determination method based on the content of adjacent frames. This method calculates the size of the pixel block that is different between the two frames based on the content of the previous and next frames. If it is larger than a certain preset value, then it can be considered that the scenes of the previous and next frames have switched. In this case, the next frame is the I-frame.

[0044] The disadvantage of this method is that it judges the I frame simply based on the size of the different pixel blocks in the two frames, ignoring the motion characteristics of the object or scene and the correlation between frames, which makes the evaluation of the similarities and differences of the frame contents prone to deviation, thereby reducing the accuracy of I frame judgment.

[0045] The second type is an I-frame determination method based on threshold comparison. This method sets a brightness threshold and compares the brightness of each pixel in the current video frame with the brightness threshold. If the brightness of the pixel is less than the brightness threshold, it is marked. When the proportion of marked pixels in all pixels included in the current video frame is greater than a preset value, it is considered that the scene has been switched, and the current video frame can be determined as an I frame. The brightness threshold can generally be taken as the average value of the three YUV channels or three RGB channels of each pixel in the video frame.

[0046] The disadvantage of this method is that it simply judges I frames by a fixed brightness threshold, which is only suitable for detecting transitional scene switches with obvious brightness changes such as fade-in and fade-out between scenes. The detection accuracy of other types of scene switches is not high, and the method has poor universality, which reduces the accuracy of I frame judgment.

[0047] The third category is the I-frame determination method based on intra-frame prediction cost and inter-frame prediction cost. Figure 1 A schematic diagram showing the process of the third type of I frame detection method in the prior art.

[0048] This method needs to complete two encodings for each frame to obtain the final encoding result. Among them, the second encoding is the formal encoding, and the first encoding is the pre-encoding. Before the formal encoding, pre-encoding can be performed to estimate the encoding effect. If the type of a video frame is an I frame, the video frame should use intra-frame encoding. If the type of a video frame is a P frame or a B frame, the video frame should use inter-frame encoding. The time domain correlation between the intra-frame coded frame and the adjacent frames is weak, so the cost of pre-encoding the intra-frame coded frame in the inter-frame coding method will be greater than the cost of pre-encoding in the intra-frame coding method.

[0049] like Figure 1As shown, for each video frame in a video frame sequence of unknown frame type, the method calculates the cost of pre-coding the video frame in an inter-frame coding mode and the cost of pre-coding the video frame in an intra-frame coding mode. If the cost of pre-coding the current video frame in an intra-frame coding mode is less than the cost of pre-coding the current video frame in an inter-frame coding mode, the current video frame is determined to be an I frame. If the cost of pre-coding the current video frame in an intra-frame coding mode is greater than or equal to the cost of pre-coding the current video frame in an inter-frame coding mode, it can be determined that the current video frame is not an I frame.

[0050] The disadvantage of this method is that the computational complexity of coding cost estimation is too high, which results in high power consumption and low coding efficiency.

[0051] In summary, how to accurately determine the I frame in a video frame sequence in a simpler way has become a research hotspot in this field.

[0052] In view of this, the present disclosure proposes a frame type determination method, device, electronic device and storage medium. The process of the frame type determination method of the embodiment of the present disclosure is simpler, and the first threshold is determined in combination with the matching degree between the video frame in the group and the adjacent video frame, which serves as the judgment threshold of the type of the video frame in the group, so that the accuracy of determining the intra-frame coding frame is higher.

[0053] Figure 2 An exemplary application scenario of the frame type determination method according to an embodiment of the present disclosure is shown.

[0054] like Figure 2 As shown, it is assumed that the video frame sequence stored on the first device needs to be transmitted to the second device. In this case, the first device can execute the frame type determination method of the embodiment of the present disclosure to determine the I frame, and then determine the P frame and the B frame according to the prior art. After the type of each video frame is determined, the first device can further determine the reference relationship of the video frame, that is, which video frames each video frame refers to and is referenced by which video frames. Based on the reference relationship, the encoding order of the video frames can be determined to ensure that when each video frame is encoded, the video frame referenced by the video frame has been encoded and the video frame referenced by the video frame has not yet started encoding.

[0055] The first device can encode each video frame according to the determined encoding order. When encoding each video frame, it is encoded according to the encoding method corresponding to the type of the video frame. For example, the encoding method of the I frame is intra-frame encoding, the encoding method of the P frame is forward prediction encoding, and the encoding method of the B frame is bidirectional prediction interpolation encoding. When the first device encodes each video frame, it can encode it only once to achieve higher encoding efficiency, or it can perform pre-coding and formal encoding, and adjust the formal encoding process according to the result of pre-coding to achieve better encoding effect. The embodiment of the present disclosure does not limit this. Intra-frame coding, forward prediction coding, and bidirectional prediction interpolation coding can all be implemented based on existing technologies, and adjusting the formal encoding process according to the result of pre-coding can also be implemented based on existing technologies. The specific encoding process will not be repeated here.

[0056] The first device can transmit the encoded code stream and the reference relationship, type and other information of each video frame to the second device. After receiving the code stream, the second device can decode it according to the reference relationship, type and other information of each video frame, thereby restoring each video frame, and then rearrange the video frames according to the acquisition order of the video frames themselves (such as recording by the video frame numbers) to restore the video frame sequence.

[0057] Figure 3a and Figure 3b A schematic diagram showing the process of a frame type determination method according to an embodiment of the present disclosure.

[0058] like Figure 3a As shown, in a possible implementation, the present disclosure proposes a frame type determination method, the method is used to determine an intra-coded frame in a video frame sequence, the method comprising steps S31-S35:

[0059] Step S31, extracting features of each video frame in the video frame sequence, analyzing feature matching between each video frame and adjacent video frames, and obtaining a matching degree between each video frame and adjacent video frames;

[0060] Step S32, dividing the video frame sequence into a plurality of groups of video frames, wherein the video frames belonging to the same group are continuous;

[0061] Step S33, for each group of video frames, determining a first threshold value according to a matching degree between each video frame in the group of video frames and an adjacent video frame;

[0062] Step S34, when the matching degree between any one video frame in the group of video frames and an adjacent video frame is less than a first threshold, the video frame is used as a candidate video frame corresponding to the group of video frames;

[0063] Step S35, determining an intra-coded frame among the candidate video frames.

[0064] For example, if Figure 3b As shown, it is assumed that the video frame sequence includes video frame 1 to video frame 12. Each video frame is numbered in the order of acquisition time, and the video frame with a smaller number is acquired earlier.

[0065] First, step S31 can be executed to extract the features of each video frame in the video frame sequence. The features of any extracted video frame are only associated with the video frame itself, and have nothing to do with other video frames. In this case, if the feature matching degree of two adjacent video frames is large, then the probability of scene switching between the latter video frame and the previous video frame will be smaller, and pixel blocks with higher similarity can be found by searching the previous video frame at a lower cost when encoding the video frame; if the feature matching degree of two adjacent video frames is small, then the probability of scene switching between the latter video frame and the previous video frame will be greater. Exemplary feature extraction methods can be found later and Figure 4 Related description.

[0066] On this basis, the feature matching between each video frame and the adjacent video frame can be analyzed, and the matching degree between the video frame and the adjacent video frame can be obtained for each video frame. The matching degree can reflect the feature matching degree of the two video frames (inter-frame correlation). The exemplary calculation method of the matching degree can be found in the following and Figure 4 Related description.

[0067] The adjacent video frame of a certain video frame may be the previous video frame of the video frame (that is, the first video frame whose acquisition time is before the current video frame), for example, the adjacent video frame of video frame 2 may be video frame 1. Those skilled in the art should understand that the adjacent video frame of a certain video frame may also be the next video frame of the video frame (that is, the first video frame whose acquisition time is after the current video frame), and the embodiments of the present disclosure do not limit whether the adjacent video frame is the previous video frame or the next video frame. For the sake of clarity, the adjacent video frame is described as an example in the following text.

[0068] Then, step S32 may be performed to divide the video frame sequence into multiple groups of video frames, and the video frames belonging to the same group are continuous. Figure 3b As shown, video frames 1 to 4 can be grouped into a group, corresponding to video frame group G1; video frames 5 to 8 can be grouped into a group, corresponding to video frame group G2; video frames 9 to 12 can be grouped into a group, corresponding to video frame group G3.

[0069] Those skilled in the art will appreciate that two different groups of video frames may also include repeated portions. For example, the grouping method may be to group video frames 1 to 6, video frames 4 to 9, and video frames 7 to 12. The number of video frames in each group may be preset, and the numbers of video frames in different groups may be the same or different. The specific grouping method of the video frame sequence in the disclosed embodiment is not limited, as long as the video frame sequence is divided into multiple groups of video frames, and the video frames belonging to the same group are continuous.

[0070] Step S31 and step S32 may be executed sequentially or simultaneously. The embodiment of the present disclosure does not limit the specific execution order of step S31 and step S32, as long as step S31 and step S32 can be completed before executing step S33.

[0071] Next, for each group of video frames, steps S33-S35 may be performed in sequence. For different groups of video frames, steps S33-S35 may be performed in parallel to improve the efficiency of determining intra-frame coded frames.

[0072] The following takes the video frame group G1 including video frames 1 to 4 as an example to introduce an exemplary implementation of steps S33 to S35.

[0073] Step S33 is performed for the video frame group G1, and the first threshold value may be determined according to the matching degree between each video frame in the video frame group G1 and the adjacent video frame. That is, the first threshold value is determined according to the matching degree P2-1 between video frame 2 and video frame 1, the matching degree P3-2 between video frame 3 and video frame 2, and the matching degree P4-3 between video frame 4 and video frame 3. An exemplary determination method of the first threshold value may be referred to in the further description of step S33 below.

[0074] The first threshold and the calculated matching degrees P2-1, P3-2, and P4-3 (matching degrees between each video frame in the video frame group G1 and the adjacent video frames) can be compared. If the matching degree between any video frame and the adjacent video frame is greater than or equal to the first threshold, it can be considered that the matching degree between the video frame and the adjacent video frame is large. If the matching degree between any video frame and the adjacent video frame is less than the first threshold, it can be considered that the matching degree between the video frame and the adjacent video frame is small. The probability of scene switching is high when the matching degree is small. Therefore, step S34 is performed for the video frame group G1. When the matching degree between any video frame and the adjacent video frame is less than the first threshold, the video frame is used as a candidate video frame corresponding to the video frame group G1. For example Figure 3b In the example of , when the matching degree P2-1 between the video frame 2 and the video frame 1 is less than the first threshold, the video frame 2 may be used as a candidate video frame corresponding to the video frame group G1.

[0075] Those skilled in the art should understand that for different groups of video frames, the first threshold determined by executing step S33 may be different, that is, the first threshold of the embodiment of the present disclosure is a threshold determined in an adaptive manner. When executing step S34 for any group of video frames, the first threshold used may be the first threshold determined by executing step S33 for the same group of video frames.

[0076] Step S35 is performed for the video frame group G1 to determine the intra-frame coded frame from the candidate video frames corresponding to the video frame group G1. Step S35 may be performed in different ways depending on whether the number of candidate video frames corresponding to the video frame group G1 is one, and the details are given later.

[0077] After a video frame is determined to be an intra-frame coded frame, a corresponding identifier of the intra-frame coded frame may be added to the video frame. In this case, when the video frame sequence is subsequently encoded, whether the video frame is an intra-frame coded frame may be identified based on the identifier of the video frame.

[0078] According to the frame type determination method of the embodiment of the present disclosure, by extracting the features of each video frame in the video frame sequence, analyzing the feature matching between each video frame and the adjacent video frames, and obtaining the matching degree between each video frame and the adjacent video frames for each video frame, the matching degree of the two video frames can reflect the possibility of scene switching between the two video frames; the video frame sequence is divided into multiple groups of video frames, and the video frames belonging to the same group are continuous. For each group of video frames, a first threshold is determined according to the matching degree between each video frame in the group of video frames and the adjacent video frames. When the matching degree between any video frame and the adjacent video frames in the group of video frames is less than the first threshold, the video frame is used as a candidate video frame corresponding to the group of video frames, so that when there is a large difference in the content of video frames collected in different time periods in the video frame sequence, the first threshold is more strongly correlated with the video frames in the group, and the possibility of scene switching between the candidate video frames determined based on the first threshold and the adjacent video frames is also greater; intra-frame coded frames are determined from the candidate video frames, so that the determination accuracy of intra-frame coded frames is higher. The frame type determination method of the embodiment of the present disclosure can also determine the intra-frame coded frame without pre-coding, so the frame type process is simpler; and the first threshold is determined in combination with the matching degree between the video frame in the group and the adjacent video frame as the judgment threshold of the type of the video frame in the group, and the correlation between frames is taken into account, so that the accuracy of determining the intra-frame coded frame is higher.

[0079] Figure 4 A schematic diagram showing a method flow of extracting features of a video frame and calculating a matching degree between a video frame and adjacent video frames based on the features of the video frame according to an embodiment of the present disclosure.

[0080] Combine the following Figure 4 First, an exemplary method for extracting features of a video frame according to an embodiment of the present disclosure is introduced.

[0081] like Figure 4 As shown, in a possible implementation, a video frame includes a plurality of pixel blocks, and in step S31, a feature of each video frame in a video frame sequence is extracted, including:

[0082] Step S311, down-sampling each video frame in the video frame sequence to obtain a plurality of sampling points, and each pixel block is down-sampled into one sampling point;

[0083] Step S312, extracting at least one of the first feature, the second feature, and the third feature of each video frame as the feature of the video frame, wherein the first feature is the average absolute error of the pixel points in the pixel block corresponding to each sampling point of the video frame; the second feature is the pixel value of each sampling point of the video frame; and the third feature is the average absolute error between each sampling point of the video frame and adjacent sampling points.

[0084] For example, each video frame in the video frame sequence may be downsampled to obtain a plurality of sampling points, and each pixel block may be downsampled to one sampling point. Figure 5a An example of video frame sampling before and after according to an embodiment of the present disclosure is shown.

[0085] like Figure 5a As shown, the video frame may include 4×4 pixel blocks, each pixel block includes 8×8 pixel points, and the video frame includes 32×32 pixel points in total. The video frame is downsampled, and each pixel block is downsampled to a sampling point; the sampled video frame may correspond to 4×4 sampling points c1-c16.

[0086] In an example, the sampling point is also a pixel point, and the pixel value of the sampling point may be equal to the average value of the absolute values ​​of the pixel values ​​of each pixel point in the pixel block corresponding to the sampling point (referred to as the pixel mean value of the pixel block corresponding to the sampling point), as shown in formula (1):

[0087]

[0088] In formula (1), pel k Indicates the pixel value of the kth pixel in the pixel block corresponding to the sampling point, 1≤k≤64 and is an integer. dspel represents the pixel value of the sampling point, and mean represents the pixel mean of the pixel block corresponding to the sampling point. The pixel value can be represented by brightness value or chromaticity value or other data that can reflect the gray value of the pixel point.

[0089] Those skilled in the art should understand that there are more ways to calculate the pixel value of the sampling point. For example, it can be equal to the pixel value of a random pixel in the pixel block corresponding to the sampling point, or equal to the weighted average of the pixel values ​​of each pixel in the pixel block corresponding to the sampling point, etc. The embodiment of the present disclosure does not limit the specific calculation method of the pixel value of each sampling point.

[0090] The extractable features include three types, among which the first feature can be the average absolute error of the pixels in the pixel block corresponding to each sampling point of the video frame. The average absolute error of the pixels in the pixel block corresponding to the sampling point can be equal to the average of the absolute values ​​of the differences between the pixel values ​​of each pixel in the pixel block corresponding to the sampling point and the pixel mean of the pixel block. Its exemplary calculation method is shown in formula (2):

[0091]

[0092] In formula (2), pel k represents the pixel value of the kth pixel in the pixel block corresponding to the sampling point, 1≤k≤64 and is an integer. mean represents the pixel mean of the pixel block corresponding to the sampling point, |pel k -mean| represents the absolute value of the difference between the pixel value of the kth pixel in the pixel block corresponding to the sampling point and the pixel mean of the pixel block. mad_pel represents the average of the absolute values ​​of the differences between the pixel values ​​of each pixel in the pixel block corresponding to the sampling point and the pixel mean of the pixel block, that is, the average absolute error of the pixels in the pixel block corresponding to the sampling point.

[0093] Formulas (1)-(2) are all based on the example that the pixel block includes 8×8=64 pixels. When the pixel block includes other numbers of pixels, 64 in formulas (1)-(2) can be replaced by the number of pixels included in the pixel block.

[0094] The second feature may be the pixel value of each sampling point of the video frame, and the calculation method thereof has been described above and will not be repeated here.

[0095] The third feature may be the mean absolute error between each sampling point and the adjacent sampling points of the video frame. The adjacent sampling points of a sampling point may refer to the sampling points around the sampling point. For example, see Figure 5a There are 8 adjacent sampling points of sampling point c6, namely sampling points c1, c2, c3, c5, c7, c9, c10, and c11.

[0096] The mean absolute error between a sampling point and its adjacent sampling points can be equal to the average of the absolute values ​​of the differences between the sampling point and each adjacent sampling point. The calculation method is shown in formula (3):

[0097]

[0098] In formula (3), pel curr Indicates the pixel value of the current sampling point, pel i Indicates the pixel value of the i-th adjacent sampling point of the sampling point, 1≤i≤8 and is an integer. |pel curr -pel i | represents the absolute value of the difference between the sampling point and the i-th adjacent sampling point, and mad_arround represents the average absolute value of the difference between the sampling point and each adjacent sampling point, that is, the average absolute error between the sampling point and the adjacent sampling points.

[0099] Formula (3) takes the current sampling point corresponding to 8 adjacent sampling points as an example. When the current sampling point corresponds to other numbers of adjacent sampling points, 8 in formula (3) can be replaced by the number of adjacent sampling points corresponding to the current sampling point.

[0100] In this case, the first feature can reflect the texture complexity of the pixel block corresponding to the sampling point, the second feature can reflect the image content brightness or chromaticity of the pixel block corresponding to the sampling point, and the third feature can reflect the content correlation between the pixel block corresponding to the sampling point and the adjacent pixel blocks in the same video frame.

[0101] Since each first feature, each second feature, and each third feature corresponds to a sampling point, when extracting the features of the video frame, the corresponding number of first features, second features, and third features can be extracted according to the number of sampling points. For example, when there are 4×4 sampling points in the video frame, there are also 4×4 first features, second features, and third features of the video frame.

[0102] Those skilled in the art should understand that when the first feature is the average absolute error of pixel points in a pixel block corresponding to each sampling point of a video frame and the third feature is the average absolute error between each sampling point of a video frame and adjacent sampling points, the first feature and the third feature may also be calculated using other formulas of the prior art, and the embodiments of the present disclosure are not limited to this.

[0103] Those skilled in the art should understand that the first feature, the second feature, and the third feature may also be calculated in other ways, as long as the first feature can reflect the texture complexity of the pixel block corresponding to the sampling point, the second feature can reflect the image content brightness or chromaticity of the pixel block corresponding to the sampling point, and the third feature can reflect the degree of content correlation between the pixel block corresponding to the sampling point and the adjacent pixel blocks in the same video frame. The embodiments of the present disclosure do not limit the specific calculation methods of the first feature, the second feature, and the third feature.

[0104] In this way, it is possible to extract features of a variety of video frames, so that the accuracy of further determining the matching degree of the video frames based on the extracted features is higher, thereby making the accuracy of determining the intra-coded frames higher.

[0105] Combine the following Figure 4 An exemplary method for analyzing feature matching of a video frame and obtaining a matching degree between a video frame and adjacent video frames according to an embodiment of the present disclosure is introduced.

[0106] like Figure 4 As shown, in a possible implementation, in step S31, the feature matching between each video frame and the adjacent video frame is analyzed, and the matching degree between the video frame and the adjacent video frame is obtained for each video frame, including:

[0107] Step S313, analyzing the feature matching between each video frame and adjacent video frames, and obtaining the number of at least one type of matching points between each video frame and adjacent video frames, wherein the matching points are the matching sampling points between the video frame and the adjacent video frames;

[0108] Step S314: The sum of the number of at least one type of matching points between each video frame and the adjacent video frames is used as the matching degree between the video frame and the adjacent video frames.

[0109] For example, since each video frame is downsampled during feature extraction, analyzing the feature matching between each video frame and adjacent video frames can be analyzing whether the features of the sampling points of each video frame and adjacent video frames match. When the features of the sampling points of two video frames match, the sampling points can be considered to be matching sampling points, i.e., matching points.

[0110] In the disclosed embodiment, the sampling point may have at least one feature matching mode, and different feature matching modes indicate different content change modes of the pixel block. Accordingly, according to different feature matching modes, the matching point may have at least one type. Therefore, after analyzing whether the features of the sampling points of each video frame and the adjacent video frame match, the number of at least one type of matching points of each video frame and the adjacent video frame can be obtained.

[0111] The disclosed embodiment provides an example of a method for calculating the number of three types of matching points, and the specific calculation method is given below. Those skilled in the art should understand that in actual applications, there should be more than three types of matching points, as long as the matching points are sampling points for feature matching.

[0112] The sum of the number of at least one type of matching points between each video frame and the adjacent video frames may be used as the matching degree between the video frame and the adjacent video frames.

[0113] In this way, the matching degree is obtained by integrating multiple feature matching methods, so the matching degree between video frames is more accurate when used to reflect the feature matching situation between video frames.

[0114] The following introduces examples of feature matching methods corresponding to three types of matching points, as well as an exemplary method for determining the number of three types of matching points.

[0115] In a possible implementation, analyzing the feature matching between each video frame and adjacent video frames to obtain the number of at least one type of matching points between each video frame and adjacent video frames includes:

[0116] For any video frame, determine whether each feature of the sampling point of the video frame matches each feature of the sampling point of the adjacent video frame;

[0117] When the first feature, the second feature, and the third feature of the first sampling point of the video frame match the first sampling point of the adjacent video frame, the number of first-category matching points of the video frame is increased by one;

[0118] When the first sampling point of the video frame matches the first feature and the third feature of the second sampling point of the adjacent video frame, but the second feature does not match, the number of the second type of matching points of the video frame is increased by one;

[0119] When the first feature and the second feature of the first sampling point of the video frame match the second sampling point of the adjacent video frame, but the third feature does not match, the number of the third type of matching points of the video frame is increased by one;

[0120] The first sampling point is any sampling point of the video frame, and the second sampling point is any sampling point of an adjacent video frame.

[0121] For example, taking the first feature, the second feature, and the third feature extracted from each video frame as an example, for each video frame, it can be determined whether each feature of the sampling point of the video frame matches that of the sampling point of the adjacent video frame. For example, for video frame 1, when its adjacent video frame is video frame 2, it can be determined whether each feature of each sampling point of video frame 1 matches that of each sampling point of video frame 2.

[0122] In the embodiments of the present disclosure, the feature matching of the sampling points is determined by using the histogram method as an example. Those skilled in the art should understand that the feature matching of the sampling points can also be determined by key point matching, hash algorithm, structural similarity (SSIM) algorithm, cosine similarity algorithm, etc. in the prior art. The embodiments of the present disclosure do not limit the specific algorithm used to determine the feature matching of the sampling points. The details of the histogram method for determining the feature matching of the sampling points are given later.

[0123] Assume that any sampling point of the video frame is the first sampling point, and any sampling point of the adjacent video frame is the second sampling point. There are three feature matching methods as follows:

[0124] First feature matching method: the first sampling point of the video frame matches the first feature, the second feature, and the third feature of the second sampling point of the adjacent video frame. As described above, the first feature can reflect the texture complexity of the pixel block corresponding to the sampling point, the second feature can reflect the image content brightness or chromaticity of the pixel block corresponding to the sampling point, and the third feature can reflect the content correlation degree between the pixel block corresponding to the sampling point and the adjacent pixel blocks in the same video frame. Therefore, at this time, the texture complexity, image content brightness or chromaticity of the pixel block corresponding to the first sampling point and the pixel block corresponding to the second sampling point are the same, and the content correlation degree with the adjacent pixel blocks in the same video frame is also the same. The first sampling point can be considered as a moving background point. When the image includes a stationary object and a moving object, the pixel block corresponding to the first sampling point may include the image content corresponding to the stationary object. The sampling points that meet the first feature matching method are regarded as the first type of matching points, and the number of the first type of matching points of the video frame is increased by one.

[0125] The second feature matching method: The first sampling point of the video frame matches the first feature and the third feature of the second sampling point of the adjacent video frame, but the second feature does not match. At this time, the texture complexity of the pixel block corresponding to the first sampling point and the pixel block corresponding to the second sampling point are the same, and the content correlation degree with the adjacent pixel blocks in the same video frame is also the same, but the image content brightness or chromaticity of the pixel blocks is different. The first sampling point can be considered as a brightness transition point. When the image includes a stationary object and a moving object, the pixel block corresponding to the first sampling point may include image content whose brightness changes due to the light changes caused by the movement of the moving object. The sampling points that meet the second feature matching method are regarded as the second type of matching points. At this time, the number of the second type of matching points of the video frame is increased by one.

[0126] The third feature matching method: the first sampling point of the video frame matches the first feature and the second feature of the second sampling point of the adjacent video frame, but the third feature does not match. At this time, the texture complexity, image content brightness or chromaticity of the pixel block corresponding to the first sampling point and the pixel block corresponding to the second sampling point are the same, but the degree of correlation with the content of the adjacent pixel blocks in the same video frame is different. The first sampling point can be considered as a motion foreground point. When the image includes a stationary object and a moving object, the pixel block corresponding to the first sampling point may include the image content of the moving object. The sampling points that meet the third feature matching method are regarded as the third type of matching points. At this time, the number of the third type of matching points of the video frame is increased by one.

[0127] It should be noted that the number of three types of matching points determined based on the feature matching of video frame 1 and video frame 2 (as adjacent video frames) is the number of three types of matching points of video frame 1. The number of three types of matching points of video frame 2 can be determined by the feature matching of video frame 2 and video frame 3 (as adjacent video frames). That is, the number of three types of matching points of each video frame is determined by the feature matching of the video frame and the adjacent video frames.

[0128] Those skilled in the art should understand that there may be more types of matching points, such as new types of matching points when the first feature matches, the second feature, and the third feature do not match, and so on. The embodiments of the present disclosure do not limit the types of matching points, as long as the matching points are sampling points that match at least one feature.

[0129] In this way, the feature matching methods of the sampling points are more diverse and more consistent with the changes in the actual image content.

[0130] The following is an example of using the histogram method to determine the feature matching of sampling points.

[0131] In a possible implementation, for any video frame, determining whether each feature of a sampling point of the video frame matches each feature of a sampling point of an adjacent video frame includes:

[0132] For any video frame, a histogram corresponding to each feature of the video frame is generated according to the features of the video frame;

[0133] Generate a histogram corresponding to each feature of adjacent video frames according to the features of adjacent video frames;

[0134] Among them, the interval intervals of the histograms corresponding to the same feature are the same, the abscissa of the histogram is the feature value, the ordinate of the histogram is the sampling point, and when the first sampling point matches any one feature of the second sampling point, the interval hit by the first sampling point in the histogram corresponding to the feature is the same as the interval hit by the second sampling point in the histogram corresponding to the same feature.

[0135] For example, for any video frame, a histogram corresponding to each feature of the video frame may be generated according to the features of the video frame, and a histogram corresponding to each feature of adjacent video frames may be generated according to the features of adjacent video frames.

[0136] Figure 5b An example of a method of determining whether sampling points of video frame 2 and video frame 1 match according to an embodiment of the present disclosure is shown.

[0137] like Figure 5bAs shown, assuming that the first feature, the second feature, and the third feature are extracted for each video frame, a histogram 11 can be generated according to the first feature of video frame 1, a histogram 12 can be generated according to the second feature of video frame 1, and a histogram 13 can be generated according to the third feature of video frame 1. A histogram 21 can be generated according to the first feature of video frame 2, a histogram 22 can be generated according to the second feature of video frame 2, and a histogram 23 can be generated according to the third feature of video frame 2.

[0138] The horizontal axis of each histogram may be a feature value, and the vertical axis may be a sampling point. The horizontal axis of the histogram may be divided into multiple intervals, and the intervals between adjacent intervals may be preset, such as 1 or 2. The embodiment of the present disclosure does not limit the intervals of the histogram.

[0139] The interval intervals of the histograms corresponding to the same feature are the same. In one example, for the second feature, its feature value is usually between 0 and 255, and the feature value of 0 to 255 can be divided into 64 intervals at intervals of 4. Then the histogram 12 of video frame 1 and the histogram 22 of video frame 2 are both divided into 64 intervals at intervals of 4.

[0140] In this case, in each histogram, each sampling point hits a unique interval. The intervals hit by each sampling point of the current video frame in the histogram corresponding to a certain feature can be compared with the intervals hit by each sampling point of the adjacent video frame in the histogram corresponding to the same feature. If the interval hit by the first sampling point of the current video frame in the histogram corresponding to a certain feature is the same as the interval hit by the second sampling point of the adjacent video frame in the histogram corresponding to the same feature, then the first sampling point can be considered to match the second sampling point. That is, when the first sampling point matches any one of the features of the second sampling point, the interval hit by the first sampling point in the histogram corresponding to the feature is the same as the interval hit by the second sampling point in the histogram corresponding to the same feature.

[0141] like Figure 5b As shown, assuming that it is determined that the interval hit by sampling point 1 of video frame 1 in histogram 11 is the same as the interval hit by sampling point 2 of video frame 2 in histogram 21, then the first feature of sampling point 1 matches with that of sampling point 2. Similarly, assuming that it is determined that the interval hit by sampling point 1 in histogram 12 is the same as the interval hit by sampling point 2 in histogram 22, then the second feature of sampling point 1 matches with that of sampling point 2. Similarly, assuming that it is determined that the interval hit by sampling point 1 in histogram 13 is different from the interval hit by sampling point 2 in histogram 23, then the third feature of sampling point 1 does not match with that of sampling point 2.

[0142] In this way, the accuracy and reference of the histogram method in determining the matching of sampling point features can be increased. In addition, the histogram method has a smaller amount of calculation, which can reduce the data processing cost.

[0143] An exemplary method for determining the first threshold is described below.

[0144] In a possible implementation, determining the first threshold according to a matching degree between each video frame in the group of video frames and an adjacent video frame includes:

[0145] Calculate the average value of the matching degree between each video frame and adjacent video frames in the group of video frames;

[0146] The product of the calculated average value and the preset proportionality coefficient is used as the first threshold.

[0147] For example, step S31 obtains the matching degree between each video frame and the adjacent video frame for each video frame, so for any group of video frames, the average value of the matching degree between each video frame in the group of video frames and the adjacent video frames can be calculated. The calculated average value is multiplied by the preset proportionality coefficient 1 / x, and the product is used as the first threshold corresponding to the group of video frames. In this case, the values ​​of the first threshold corresponding to different video frame groups may be different, and will be determined by the matching degree between each video frame in the group of video frames and the adjacent video frames. The value of x can be set according to the application scenario requirements, and the present disclosure does not limit this.

[0148] In this way, the value of the first threshold corresponding to each group of video frames can be adaptively adjusted according to the matching degree between each video frame in the group of video frames and adjacent video frames.

[0149] A deeper level of analysis may be performed on the candidate video frames to determine intra-coded frames from the candidate video frames. An exemplary method of determining intra-coded frames from the candidate video frames is described below.

[0150] In a possible implementation, determining an intra-coded frame in a candidate video frame includes:

[0151] When the group of video frames corresponds to only one candidate video frame, the candidate video frame is determined as an intra-coded frame.

[0152] For example, if a group of video frames corresponds to only one candidate video frame, the candidate video frame can be directly determined as an intra-coded frame. In this case, no additional analysis and processing for the group of video frames for the purpose of determining the intra-coded frame is required, thus saving data processing costs.

[0153] In a possible implementation, determining an intra-coded frame in a candidate video frame includes:

[0154] When the group of video frames corresponds to two or more candidate video frames, determining a second threshold corresponding to the matching points of each type according to the number of matching points of each type between each video frame in the group of video frames and an adjacent video frame;

[0155] For each type of matching points, a candidate video frame having the smallest number of matching points in this type and a number smaller than a second threshold corresponding to the matching points in this type is selected as an intra-frame coding frame.

[0156] For example, if a group of video frames corresponds to two or more candidate video frames, it can be considered that which candidate video frames have better effects as intra-frame coding frames can be further determined. The second threshold corresponding to each type of matching point can be determined based on the number of matching points of each type between each video frame in the group of video frames and the adjacent video frames. For example, when there are three types of matching points, three second thresholds corresponding to the three types of matching points are determined.

[0157] Correspondingly, the determined three second thresholds also correspond to the group of video frames. That is, each group of video frames corresponds to three second thresholds, and the three second thresholds corresponding to each group of video frames also correspond to three types of matching points respectively.

[0158] For each type of matching point, a candidate video frame with the smallest number of matching points in this type and less than the second threshold value corresponding to the matching points in this type can be selected as an intra-frame coded frame. The candidate video frames selected for different types of matching points may be the same. If the candidate video frames selected for each type of matching point are the same, it can be considered that the most suitable intra-frame coded frame is only the selected candidate video frame. It is also possible that multiple candidate video frames are selected for a certain type of matching point, and the selected multiple candidate video frames can all be used as intra-frame coded frames.

[0159] When the frames with the lowest three matching points are the same frame, it means that there is only one most suitable scene switching frame.

[0160] Taking the video frame group G1 as an example, it is assumed that the candidate video frames corresponding to the video frame group G1 include video frame 1, video frame 2, and video frame 3, where:

[0161] The number of first-category matching points in video frame 1 is 5, the number of second-category matching points is 7, and the number of third-category matching points is 3;

[0162] The number of first-category matching points in video frame 2 is 6, the number of second-category matching points is 5, and the number of third-category matching points is 9;

[0163] The number of first-category matching points of video frame 3 is 2, the number of second-category matching points is 4, and the number of third-category matching points is 3;

[0164] It can be seen that the video frame 3 has the smallest number of matching points in the first category, the video frame 3 has the smallest number of matching points in the second category, and the video frames 1 and 3 have the smallest number of matching points in the third category.

[0165] Assume that among the three second thresholds corresponding to the video frame group G1, the second threshold corresponding to the first type of matching points is equal to 6, the second threshold corresponding to the second type of matching points is equal to 3, and the second threshold corresponding to the third type of matching points is equal to 5.

[0166] For the first type of matching points, since the number of the first type of matching points of video frame 3 is the smallest and is smaller than the second threshold corresponding to the first type of matching points, video frame 3 may be selected as the intra-frame coding frame.

[0167] For the second type of matching points, since the number of the second type of matching points of video frame 3 is the smallest but greater than the second threshold corresponding to the second type of matching points, the intra-coded frame is not selected.

[0168] For the third type of matching points, since the number of the third type of matching points of video frame 1 and video frame 3 is the smallest and is smaller than the second threshold corresponding to the third type of matching points, video frame 1 and video frame 3 may be selected as intra-frame coding frames.

[0169] Therefore, in the video frame group G1, the finally determined intra-frame coded frames may be video frame 1 and video frame 3.

[0170] In this way, the situation in which the frequent occurrence of intra-coded frames causes high bit rate consumption in subsequent encoding can be avoided, the redundancy of intra-coded frames can be reduced, the encoding cost can be saved, and the application scenarios of the frame type determination method disclosed in the present invention can be expanded.

[0171] An exemplary method for determining the second threshold corresponding to each type of matching point is described below.

[0172] In a possible implementation, determining a second threshold corresponding to each type of matching point according to the number of each type of matching points between each video frame in the group of video frames and an adjacent video frame includes:

[0173] For each type of matching point, calculate the average number of matching points of this type between each video frame in the group of video frames and the adjacent video frames;

[0174] The product of the calculated average value and the preset proportionality coefficient is used as the second threshold corresponding to the number of matching points of this type.

[0175] The calculation method of the second threshold is similar to the calculation method of the first threshold. When calculating the second threshold corresponding to a certain type of matching point, the average number of matching points of this type between each video frame in the group of video frames and the adjacent video frames can be calculated first, and then the calculated average value is multiplied by a preset proportionality coefficient, and the product is used as the second threshold corresponding to the number of matching points of this type.

[0176] The proportionality coefficient used when calculating the second threshold may be the same as or different from the proportionality coefficient used when calculating the first threshold, and the present disclosure does not impose any limitation on this.

[0177] In this way, the value of the second threshold is also adaptively adjusted, so the accuracy of determining the intra-coded frame from the candidate video frames based on the second threshold is higher.

[0178] The present disclosure also provides a frame type determination device, Figure 6 A schematic diagram showing the structure of a frame type determination device according to an embodiment of the present disclosure.

[0179] like Figure 6 As shown, in a possible implementation manner, the apparatus is used to determine an intra-coded frame in a video frame sequence, and the apparatus includes:

[0180] An analysis module 61 is used to extract features of each video frame in the video frame sequence, analyze the feature matching between each video frame and adjacent video frames, and obtain the matching degree between each video frame and adjacent video frames;

[0181] A grouping module 62, configured to divide the video frame sequence into a plurality of groups of video frames, wherein the video frames belonging to the same group are continuous;

[0182] A first determination module 63 is used to determine a first threshold for each group of video frames according to a matching degree between each video frame in the group of video frames and an adjacent video frame;

[0183] A second determination module 64 is configured to, when a matching degree between any one video frame and an adjacent video frame in the group of video frames is less than a first threshold, use the video frame as a candidate video frame corresponding to the group of video frames;

[0184] The third determination module 65 is configured to determine an intra-coded frame in the candidate video frames.

[0185] In a possible implementation, the video frame includes multiple pixel blocks, and the first analysis module is used to downsample each video frame in the video frame sequence to obtain multiple sampling points, and each pixel block is downsampled into one sampling point; extract at least one of a first feature, a second feature, and a third feature of each video frame as a feature of the video frame, wherein the first feature is the average absolute error of pixel points in the pixel block corresponding to each sampling point of the video frame; the second feature is the pixel value of each sampling point of the video frame; and the third feature is the average absolute error between each sampling point of the video frame and adjacent sampling points.

[0186] In a possible implementation, the first analysis module is further used to analyze the feature matching between each video frame and adjacent video frames, and obtain the number of at least one type of matching points between each video frame and adjacent video frames, wherein the matching points are sampling points that match between the video frame and the adjacent video frames; and the sum of the number of at least one type of matching points between each video frame and the adjacent video frames is used as the matching degree between the video frame and the adjacent video frames.

[0187] In a possible implementation, the third determination module is used to determine, when the group of video frames corresponds to two or more candidate video frames, a second threshold corresponding to the matching points of each type based on the number of matching points of each type between each video frame in the group of video frames and adjacent video frames; for each type of matching points, select the candidate video frame with the smallest number of matching points of this type and less than the second threshold corresponding to this type of matching points as the intra-frame coded frame.

[0188] In a possible implementation manner, the third determination module is configured to, when the group of video frames corresponds to only one candidate video frame, determine the candidate video frame as an intra-coded frame.

[0189] In a possible implementation, analyzing the feature matching between each video frame and adjacent video frames to obtain the number of at least one type of matching points between each video frame and adjacent video frames includes: for any video frame, determining whether each feature of the sampling point of the video frame matches the sampling point of the adjacent video frame; when the first feature, the second feature, and the third feature of the first sampling point of the video frame match the first feature, the number of first-category matching points of the video frame increases by one; when the first feature and the third feature of the first sampling point of the video frame match the first feature and the third feature of the second sampling point of the adjacent video frame, but the second feature does not match, the number of second-category matching points of the video frame increases by one; when the first feature and the second feature of the first sampling point of the video frame match the first feature and the second feature of the second sampling point of the adjacent video frame, but the third feature does not match, the number of third-category matching points of the video frame increases by one; wherein the first sampling point is any sampling point of the video frame, and the second sampling point is any sampling point of the adjacent video frame.

[0190] In a possible implementation, determining, for any video frame, whether a sampling point of the video frame matches each feature of a sampling point of an adjacent video frame includes: for any video frame, generating a histogram corresponding to each feature of the video frame according to the features of the video frame; generating a histogram corresponding to each feature of the adjacent video frames according to the features of the adjacent video frames; wherein the interval intervals of the histograms corresponding to the same feature are the same, the abscissa of the histogram is the feature value, the ordinate of the histogram is the sampling point, and when the first sampling point matches any one feature of the second sampling point, the interval hit by the first sampling point in the histogram corresponding to the feature is the same as the interval hit by the second sampling point in the histogram corresponding to the same feature.

[0191] In a possible implementation, the first determination module is used to calculate an average value of the degree of matching between each video frame in the group of video frames and adjacent video frames; and use the product of the calculated average value and a preset proportionality coefficient as the first threshold.

[0192] In a possible implementation, the method of determining the second threshold value corresponding to the number of each type of matching points between each video frame in the group of video frames and adjacent video frames includes: for each type of matching points, calculating the average number of the number of matching points of this type between each video frame in the group of video frames and adjacent video frames; and taking the product of the calculated average value and a preset proportional coefficient as the second threshold value corresponding to the number of matching points of this type.

[0193] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0194] The embodiment of the present disclosure also provides a computer-readable storage medium on which computer program instructions are stored, and the computer program instructions implement the above method when executed by a processor. The computer-readable storage medium can be a volatile or non-volatile computer-readable storage medium.

[0195] The present disclosure also provides an electronic device, comprising: a processor; a memory for storing instructions executable by the processor; wherein the processor is configured to implement the above method when executing the instructions stored in the memory. Figure 2 And the first device in the related description.

[0196] The embodiments of the present disclosure also provide a computer program product, including a computer-readable code, or a non-volatile computer-readable storage medium carrying the computer-readable code. When the computer-readable code runs in a processor of an electronic device, the processor in the electronic device executes the above method.

[0197] Figure 7 1 is a block diagram of an electronic device 1900 according to an embodiment of the present disclosure. For example, the electronic device 1900 may be provided as a server or a terminal device. Figure 7 , the electronic device 1900 includes a processing component 1922, which further includes one or more processors, and a memory resource represented by a memory 1932 for storing instructions executable by the processing component 1922, such as an application. The application stored in the memory 1932 may include one or more modules, each corresponding to a set of instructions. In addition, the processing component 1922 is configured to execute instructions to perform the above method.

[0198] The electronic device 1900 may also include a power supply component 1926 configured to perform power management of the electronic device 1900, a wired or wireless network interface 1950 configured to connect the electronic device 1900 to a network, and an input / output interface 1958 (I / O interface). The electronic device 1900 may operate based on an operating system stored in the memory 1932, such as Windows Server 2003. TM , Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM or similar.

[0199] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions, which can be executed by the processing component 1922 of the electronic device 1900 to perform the above method.

[0200] The present disclosure may be a system, a method and / or a computer program product. The computer program product may include a computer-readable storage medium carrying computer-readable program instructions for causing a processor to implement various aspects of the present disclosure.

[0201] A computer-readable storage medium may be a tangible device that can hold and store instructions used by an instruction execution device. A computer-readable storage medium may be, for example, but not limited to, an electrical storage device, a magnetic storage device, an optical storage device, an electromagnetic storage device, a semiconductor storage device, or any suitable combination of the foregoing. More specific examples of computer-readable storage media (a non-exhaustive list) include: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a static random access memory (SRAM), a portable compact disk read-only memory (CD-ROM), a digital versatile disk (DVD), a memory stick, a floppy disk, a mechanical encoding device, such as a punch card or a raised structure in a groove on which instructions are stored, and any suitable combination of the foregoing. As used herein, a computer-readable storage medium is not to be interpreted as a transient signal per se, such as a radio wave or other freely propagating electromagnetic wave, an electromagnetic wave propagating through a waveguide or other transmission medium (e.g., a light pulse through a fiber optic cable), or an electrical signal transmitted through a wire.

[0202] The computer-readable program instructions described herein can be downloaded from a computer-readable storage medium to each computing / processing device, or downloaded to an external computer or external storage device via a network, such as the Internet, a local area network, a wide area network, and / or a wireless network. The network can include copper transmission cables, optical fiber transmissions, wireless transmissions, routers, firewalls, switches, gateway computers, and / or edge servers. The network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards the computer-readable program instructions for storage in the computer-readable storage medium in each computing / processing device.

[0203] The computer program instructions for performing the operation of the present disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-related instructions, microcode, firmware instructions, state setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages, such as Smalltalk, C++, etc., and conventional procedural programming languages, such as "C" language or similar programming languages. Computer-readable program instructions may be executed completely on a user's computer, partially on a user's computer, as an independent software package, partially on a user's computer, partially on a remote computer, or completely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., using an Internet service provider to connect via the Internet). In some embodiments, an electronic circuit, such as a programmable logic circuit, a field programmable gate array (FPGA), or a programmable logic array (PLA), may be customized by utilizing the state information of the computer-readable program instructions, and the electronic circuit may execute the computer-readable program instructions, thereby realizing various aspects of the present disclosure.

[0204] Various aspects of the present disclosure are described herein with reference to the flowcharts and / or block diagrams of the methods, devices (systems) and computer program products according to the embodiments of the present disclosure. It should be understood that each box in the flowchart and / or block diagram and the combination of each box in the flowchart and / or block diagram can be implemented by computer-readable program instructions.

[0205] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, thereby producing a machine, so that when these instructions are executed by the processor of the computer or other programmable data processing device, a device that implements the functions / actions specified in one or more boxes in the flowchart and / or block diagram is generated. These computer-readable program instructions can also be stored in a computer-readable storage medium, and these instructions cause the computer, programmable data processing device, and / or other equipment to work in a specific manner, so that the computer-readable medium storing the instructions includes a manufactured product, which includes instructions for implementing various aspects of the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0206] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device so that a series of operating steps are performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to implement the functions / actions specified in one or more boxes in the flowchart and / or block diagram.

[0207] The flow chart and block diagram in the accompanying drawings show the possible architecture, function and operation of the system, method and computer program product according to multiple embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a part of a module, program segment or instruction, and a part of the module, program segment or instruction includes one or more executable instructions for realizing the specified logical function. In some alternative implementations, the function marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two continuous square boxes can actually be executed substantially in parallel, and they can sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs the specified function or action, or can be implemented with a combination of special hardware and computer instructions.

[0208] The embodiments of the present disclosure have been described above, and the above description is exemplary, not exhaustive, and is not limited to the disclosed embodiments. Many modifications and changes will be apparent to those of ordinary skill in the art without departing from the scope and spirit of the described embodiments. The selection of terms used herein is intended to best explain the principles of the embodiments, practical applications, or technical improvements in the market, or to enable other persons of ordinary skill in the art to understand the embodiments disclosed herein.

Claims

1. A method for determining a frame type, characterized in that: The method is used to determine an intra-coded frame in a sequence of video frames, the method comprising: Extract the features of each video frame in the video frame sequence, analyze the feature matching between each video frame and adjacent video frames, and obtain the matching degree between the video frame and adjacent video frames for each video frame; Dividing the video frame sequence into a plurality of groups of video frames, wherein the video frames belonging to the same group are continuous; For each group of video frames, determining a first threshold according to a matching degree between each video frame in the group of video frames and an adjacent video frame; When the matching degree between any one video frame and an adjacent video frame in the group of video frames is less than a first threshold, the video frame is used as a candidate video frame corresponding to the group of video frames; Intra-coded frames are determined among the candidate video frames.

2. The method according to claim 1, characterized in that The video frame includes a plurality of pixel blocks, and the step of extracting a feature of each video frame in the video frame sequence includes: Downsampling each video frame in the video frame sequence to obtain a plurality of sampling points, wherein each pixel block is downsampled into one sampling point; At least one of a first feature, a second feature, and a third feature of each video frame is extracted as a feature of the video frame, wherein the first feature is an average absolute error of pixel points in a pixel block corresponding to each sampling point of the video frame; the second feature is a pixel value of each sampling point of the video frame; and the third feature is an average absolute error between each sampling point of the video frame and adjacent sampling points.

3. The method according to claim 2, characterized in that The analyzing the feature matching between each video frame and adjacent video frames, and obtaining the matching degree between each video frame and adjacent video frames, includes: Analyze the feature matching between each video frame and adjacent video frames to obtain the number of at least one type of matching points between each video frame and adjacent video frames, wherein the matching points are sampling points that match between the video frame and the adjacent video frames; The sum of the number of at least one type of matching points between each video frame and the adjacent video frames is taken as the matching degree between the video frame and the adjacent video frames.

4. The method according to claim 3, characterized in that The determining of an intra-frame coding frame in the candidate video frame comprises: When the group of video frames corresponds to two or more candidate video frames, determining a second threshold corresponding to the matching points of each type according to the number of matching points of each type between each video frame in the group of video frames and an adjacent video frame; For each type of matching points, a candidate video frame having the smallest number of matching points in this type and a number smaller than a second threshold corresponding to the matching points in this type is selected as an intra-frame coding frame.

5. The method according to claim 1, characterized in that The determining of an intra-frame coding frame in the candidate video frame comprises: When the group of video frames corresponds to only one candidate video frame, the candidate video frame is determined as an intra-coded frame.

6. The method according to claim 3, characterized in that The analyzing the feature matching between each video frame and adjacent video frames to obtain the number of at least one type of matching points between each video frame and adjacent video frames includes: For any video frame, determine whether each feature of the sampling point of the video frame matches each feature of the sampling point of the adjacent video frame; When the first feature, the second feature, and the third feature of the first sampling point of the video frame match the first sampling point of the adjacent video frame, the number of first-category matching points of the video frame is increased by one; When the first sampling point of the video frame matches the first feature and the third feature of the second sampling point of the adjacent video frame, but the second feature does not match, the number of the second type of matching points of the video frame is increased by one; When the first feature and the second feature of the first sampling point of the video frame match the second sampling point of the adjacent video frame, but the third feature does not match, the number of the third type of matching points of the video frame is increased by one; The first sampling point is any sampling point of the video frame, and the second sampling point is any sampling point of an adjacent video frame.

7. The method according to claim 6, characterized in that The step of determining, for any video frame, whether each feature of a sampling point of the video frame matches each feature of a sampling point of an adjacent video frame includes: For any video frame, a histogram corresponding to each feature of the video frame is generated according to the features of the video frame; Generating a histogram corresponding to each feature of the adjacent video frames according to the features of the adjacent video frames; Among them, the interval intervals of the histograms corresponding to the same feature are the same, the abscissa of the histogram is the feature value, the ordinate of the histogram is the sampling point, and when the first sampling point matches any one feature of the second sampling point, the interval hit by the first sampling point in the histogram corresponding to the feature is the same as the interval hit by the second sampling point in the histogram corresponding to the same feature.

8. The method according to claim 1, characterized in that The determining of the first threshold according to the matching degree between each video frame in the group of video frames and the adjacent video frame comprises: Calculate the average value of the matching degree between each video frame and adjacent video frames in the group of video frames; The product of the calculated average value and the preset proportionality coefficient is used as the first threshold.

9. The method according to claim 4, characterized in that The step of determining the second threshold value corresponding to each type of matching point according to the number of each type of matching points between each video frame in the group of video frames and the adjacent video frames includes: For each type of matching point, calculate the average number of matching points of this type between each video frame in the group of video frames and the adjacent video frames; The product of the calculated average value and the preset proportionality coefficient is used as the second threshold corresponding to the number of matching points of this type.

10. A frame type determination device, characterized in that: The apparatus is used to determine an intra-coded frame in a video frame sequence, the apparatus comprising: An analysis module is used to extract features of each video frame in the video frame sequence, analyze the feature matching between each video frame and adjacent video frames, and obtain the matching degree between each video frame and adjacent video frames; A grouping module, used for dividing the video frame sequence into a plurality of groups of video frames, wherein the video frames belonging to the same group are continuous; A first determination module is used to determine a first threshold for each group of video frames according to a matching degree between each video frame in the group of video frames and an adjacent video frame; A second determination module is used to, when a matching degree between any video frame and an adjacent video frame in the group of video frames is less than a first threshold, use the video frame as a candidate video frame corresponding to the group of video frames; The third determination module is used to determine an intra-frame coded frame in the candidate video frames.

11. An electronic device, characterized in that: include: processor; a memory for storing processor-executable instructions; Wherein, the processor is configured to implement the method described in any one of claims 1 to 9 when executing the instructions stored in the memory.

12. A non-volatile computer-readable storage medium having computer program instructions stored thereon, characterized in that: When the computer program instructions are executed by a processor, the method according to any one of claims 1 to 9 is implemented.

Images