Frame skip detection device, learning device, and frame skip detection program

Abstract

To provide a device capable of automatically detecting a frame skip.SOLUTION: A frame skip detection device 1 includes: an even number and odd number conversion unit 11 for dividing frames constituting an input video into even frames and odd frames; and a determination unit 14 for inputting pixel values extracted from the even frames and the odd frames to a learning model that has learned a label of whether or not a frame skip has occurred as an input of pixel values of two consecutive frames and outputs determination results.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an apparatus, a method, and a program for detecting a frame skip. [Background technology]

[0002] In recent years, frame rates have been increasing in line with the increasing resolution of video. The frame frequency of current high definition broadcasting is 29.97 Hz (hereinafter referred to as 30 Hz), but the frame frequency of 8K Super Hi-Vision, which began broadcasting in December 2018, is 59.94 Hz (hereinafter referred to as 60 Hz). Furthermore, ARIB STD-B32 version 3.9 (Non-Patent Document 1), the standard that specifies the encoding parameters of 8K broadcasting, standardizes the frame frequency up to 119.88 Hz (hereinafter referred to as 120 Hz). By increasing the frame frequency, it becomes possible to provide a higher quality viewing experience, for example by making it possible to portray fast-moving subjects in sports footage more clearly and smoothly. [Prior art documents] [Non-patent literature]

[0003] [Non-Patent Document 1] ARIB STD-B32, "Video Coding, Audio Coding and Multiplexing Methods for Digital Broadcasting", Version 3.9 Summary of the Invention [Problem to be solved by the invention]

[0004] Video is a collection of images (frames), and when a video is played, these frames are displayed consecutively on a display device such as a television or monitor, but frame skipping can occur, where some frames are dropped (skipped) due to slow processing, etc. Frame skipping can be caused by display devices such as televisions and monitors, but it is not rare for frames to be skipped at the video output stage by a recorder, decoder, etc., due to a bug or slow processing in the device.

[0005] Frame skipping can have a large impact on subjective image quality, so it is desirable to avoid it, but frame skip detection is often done manually. This makes it difficult for a human eye to determine frame skipping over an extended period of time, and it is also difficult for one person to check on a large monitor. Furthermore, depending on the image, it can be difficult for even the human eye to determine frame skipping.

[0006] An object of the present invention is to provide an apparatus capable of automatically detecting a frame skip. [Means for solving the problem]

[0007] The frame skip detection device of the present invention comprises an even-odd conversion unit that divides frames constituting an input video into even frames and odd frames, and a judgment unit that inputs pixel values ​​extracted from each of the even frames and the odd frames to a learning model that uses pixel values ​​of two consecutive frames as input and learns a label indicating whether or not a frame skip has occurred, and outputs a judgment result.

[0008] The learning model may further receive a frame rate of the video as an input, and the determination unit may obtain the frame rate of the input video and input it to the learning model together with the pixel value.

[0009] The learning model may further include a conversion unit that receives data converted from the pixel values ​​as input and converts the pixel values ​​extracted from each of the even frames and the odd frames, and the determination unit may input the data obtained by the conversion unit to the learning model together with the pixel values.

[0010] The data may include at least one of a pixel difference value, a motion vector, a frequency decomposition value, and an image feature amount.

[0011] The determination unit may output a determination result based on an output ratio when the pixel values ​​are input to the learning model in units of blocks obtained by dividing the frame into a predetermined size.

[0012] The learning device of the present invention is a learning device that learns the learning model used in the frame skip detection device, and includes a teacher data acquisition unit that acquires teacher data representing a correct answer corresponding to learning image data for the learning model, and a learning unit that calculates an error between the teacher data and the output of the learning model, and updates the parameters of the learning model based on the error.

[0013] A frame skip detection program according to the present invention causes a computer to function as the frame skip detection device. Effect of the Invention

[0014] According to the present invention, frame skipping can be detected automatically. [Brief description of the drawings]

[0015] [Figure 1] 1 is a diagram illustrating a configuration of a frame skip detection device according to an embodiment. [Diagram 2] FIG. 1 is a diagram illustrating a configuration of a convolutional neural network, which is an example of a learning model in an embodiment. [Diagram 3] FIG. 2 is a diagram illustrating a configuration of a learning device according to an embodiment. [Figure 4] FIG. 2 is a diagram illustrating an example of a 1 of K vector that is training data in the embodiment. [Diagram 5] 1 is a diagram illustrating an example of a relationship between time-series image frames as learning data and teacher data in an embodiment. FIG. [Figure 6] FIG. 4 is a diagram showing a flow of a learning process performed by the learning device in the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0016] An example of an embodiment of the present invention will now be described. The frame skip detection device 1 in this embodiment automatically determines whether or not a frame skip has occurred by using a learning model that has learned various types of information related to frame skipping through machine learning. In this embodiment, an example will be described in which information based on pixel values ​​of frames constituting a video and a frame rate are used as inputs to the learning model.

[0017] FIG. 1 is a diagram showing the configuration of a frame skip detection device 1 according to this embodiment. The frame skip detection device 1 includes an even / odd conversion unit 11, an even frame memory 12, an odd frame memory 13, and a determination unit .

[0018] The even-odd conversion unit 11 divides an input video, that is, consecutive frames, into even-numbered frames and odd-numbered frames.

[0019] An even frame memory 12 stores the even frames separated by the even / odd conversion unit 11, and an odd frame memory 13 stores the odd frames, and adjusts the timing of inputting the frames to a determination unit .

[0020] The determination unit 14 receives as input the even and odd frames obtained from the even and odd frame memories 12 and 13, and information on the frame rate. Based on the input data, the determination unit 14 determines whether or not a frame skip has occurred, or the probability that a frame skip has occurred, using a learning model configured by a neural network or the like, and outputs the determination result.

[0021] FIG. 2 is a diagram illustrating a configuration of a convolutional neural network, which is an example of a learning model in this embodiment. The learning model can have a general structure including one or more convolutional layers, a pooling layer, an activation layer, a softmax function, etc. In addition, the video data to be input to the convolutional neural network may be a variety of data converted from pixel values, such as frequency decomposition values, image features, and motion vectors, in addition to pixel values ​​of the frames.

[0022] Here, the learning model is one in which parameters are learned by a learning device 2, which will be described next, using learning image data and teacher data. The learning device 2 may be configured as an information processing device separate from the frame skip detection device 1, and the learned parameters may be provided to the learning model of the frame skip detection device 1, or it may be integrated with the frame skip detection device 1, and the parameters of the learning model may be directly updated.

[0023] In this embodiment, in addition to pixel values, pixel difference values ​​between frames, motion vectors, and frame rates are adopted as inputs to the learning model, and the determination unit 14 is also provided with a pixel value conversion function for each part of the learning device 2.

[0024] FIG. 3 is a diagram showing the configuration of the learning device 2 in this embodiment. The learning device 2 includes an even frame acquisition unit 21 , an odd frame acquisition unit 22 , an inter-frame pixel difference calculation unit 23 , a motion vector calculation unit 24 , a frame rate acquisition unit 25 , a teacher data acquisition unit 26 , and a learning unit 27 .

[0025] The even frame acquisition unit 21 acquires even-numbered learning image frames, and the odd frame acquisition unit 22 acquires odd-numbered learning image frames, and each inputs pixel values ​​in the frames to the learning unit 27. The even frame acquisition unit 21 and the odd frame acquisition unit 22 also provide pixel values ​​to the inter-frame pixel difference calculation unit 23 and the motion vector calculation unit 24 in a similar manner.

[0026] At this time, the even frame acquisition unit 21 and the odd frame acquisition unit 22 alternately provide pixel values ​​of the frames in a state in which the temporal relationship of the frames constituting the video can be identified, i.e., accompanied by information on the order of even and odd, or odd and even.

[0027] The inter-frame pixel difference calculation unit 23 calculates pixel difference values ​​between consecutive frames for even-numbered and odd-numbered learning image frames, and inputs the calculated values ​​to the learning unit 27 .

[0028] The motion vector calculation unit 24 calculates motion vectors between consecutive frames for even-numbered and odd-numbered learning image frames, and inputs the calculated motion vectors to the learning unit 27 .

[0029] The frame rate acquisition unit 25 acquires information on the frame rate of the learning images and inputs it to the learning unit 27 .

[0030] The teacher data acquisition unit 26 acquires teacher data (e.g., a 1 of K vector) indicating the correct answer as to whether or not a frame skip has occurred corresponding to the learning image frame, and provides it to the learning unit 27 for learning of the learning model.

[0031] The learning unit 27 calculates the error between the output (judgment result) obtained when the pixel value, pixel difference value, motion vector, and frame rate are input to the learning model and the teacher data representing the correct answer, and updates the parameters of the learning model based on this error.

[0032] Then, the learning unit 27 repeats the learning until a predetermined end condition is satisfied, and outputs the optimized parameters. Here, the termination condition may be when the number of learning times reaches a predetermined value (for example, 1 million times), or when the error value falls below a predetermined value. Alternatively, the termination condition may be when the number of learning times reaches a predetermined value (for example, 1 million times), or when the error value falls below a predetermined value. Alternatively, the termination condition may be when the number of learning times reaches a predetermined value (for example, 100 learning data items are used for 100 learning data items), or when the number of learning data items is 10000 times.

[0033] The training data used is a combination of pre-created video in which frame skipping has occurred and teacher data. FIG. 4 is a diagram illustrating an example of 1 of K vectors that are training data in this embodiment. Here, either a vector [1] indicating that a frame skip has occurred, or a vector [0] indicating that a frame skip has not occurred, is assigned to each combination of two frames, one before and one after the other.

[0034] FIG. 5 is a diagram illustrating an example of the relationship between time-series image frames as learning data and teacher data in this embodiment. For video that is composed of a group of frames with no frame skipping between them, all of the training data is set to [0]. On the other hand, in a video where a frame skip has occurred, if the frame indicated by the dashed line is missing, [1] is assigned as training data to the frame immediately following this frame (and the combination of the frame immediately preceding it).

[0035] FIG. 6 is a diagram showing the flow of the learning process by the learning device 2 in this embodiment. In step S1 , the learning unit 27 acquires pixel values ​​of learning image frames from the even-numbered frame acquisition unit 21 and the odd-numbered frame acquisition unit 22 .

[0036] In step S2, the learning unit 27 acquires from the inter-frame pixel difference calculation unit 23 the pixel difference values ​​between the consecutive frames acquired in step S1.

[0037] In step S3, the learning unit 27 acquires, from the motion vector calculation unit 24, the motion vector between the consecutive frames acquired in step S1.

[0038] In step S4, the learning unit 27 acquires from the frame rate acquisition unit 25 the frame rate of the video composed of the learning image frames.

[0039] In step S5, the learning unit 27 acquires, from the teacher data acquisition unit 26, teacher data for the combination of consecutive frames acquired in step S1.

[0040] In step S6, the learning unit 27 inputs the learning data acquired in steps S1 to S4 into a learning model, and obtains an output of a determination result.

[0041] In step S7, the learning unit 27 calculates the error between the output obtained in step S6 and the teacher data acquired in step S5.

[0042] In step S8, the learning unit 27 updates the parameters of the learning model based on the error calculated in step S7.

[0043] In step S9, the learning unit 27 determines whether or not all of the prepared learning data has been used for learning. If the determination is YES, the process ends, and if the determination is NO, the process proceeds to step S10.

[0044] In step S10, the learning unit 27 determines whether or not learning has been performed a predetermined number of times. If the determination is YES, the process ends, and if the determination is NO, the process returns to step S1.

[0045] According to this embodiment, the frame skip detection device 1 divides the frames constituting the input video into even frames and odd frames, inputs pixel values ​​extracted from each of the even frames and odd frames into a learning model that has learned a label indicating whether or not a frame skip has occurred by inputting the pixel values ​​of two consecutive frames, and outputs a determination result. Therefore, the frame skip detection device 1 can automatically detect frame skips using a learning model based on the pixel values ​​of each consecutive frame. As a result, the amount of work that was previously done manually is reduced. Furthermore, since it can detect frame skips that would be overlooked by the human eye, it becomes possible to debug devices more efficiently.

[0046] The learning model can further input data converted from pixel values ​​such as the video frame rate, pixel difference values, motion vectors, frequency decomposition values, and image features, thereby increasing the amount of information required to determine frame skipping and improving the accuracy of the determination.

[0047] Although the embodiments of the present invention have been described above, the present invention is not limited to the above-described embodiments. Furthermore, the effects described in the above-described embodiments are merely a list of the most preferable effects resulting from the present invention, and the effects of the present invention are not limited to those described in the embodiments.

[0048] In the above embodiment, the presence or absence of a frame skip is determined by a learning model that receives as input pixel values ​​of all frames constituting a video, but the unit of input data is not limited to this. For example, the frame skip detection device 1 may output a determination result based on the output ratio when data such as pixel values ​​is input to a learning model in units of blocks obtained by dividing a frame into a predetermined size.

[0049] Specifically, the judgment unit 14 may detect frame skipping in block units such as 8x8 or 16x16, and depending on the number of blocks in which it is determined that frame skipping has occurred, may determine that frame skipping has occurred for the entire frame when, for example, the learning model estimates that frame skipping has occurred in the majority of the blocks. Furthermore, the motion vector may be calculated in units of blocks such as 8×8 or 16×16. This allows the frame skip detection device 1 to integrate the determination results based on local information and detect frame skips with high accuracy.

[0050] Furthermore, the frames input to the learning model may not only be one even-numbered frame and one odd-numbered frame, but also multiple frames before and after them. This makes it possible to more accurately detect a frame skip based on transition information between multiple frames.

[0051] The information to be input into the learning model is not limited to the data described above. For example, various information related to video, such as resolution, can be used as a feature. Furthermore, the above-mentioned learning algorithm is just an example, and various existing machine learning algorithms can be applied.

[0052] In this embodiment, the configuration and operation of the frame skip detection device 1 and the learning device 2 have been mainly described, but the present invention is not limited to this and may be configured as a method or program for detecting frame skips, including each component.

[0053] Furthermore, the functions of the frame skip detection device 1 and the learning device 2 may be realized by recording a program on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system.

[0054] The term "computer system" here includes hardware such as the OS and peripheral devices. Additionally, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage devices such as hard disks built into computer systems.

[0055] Furthermore, the term "computer-readable recording medium" may include a recording medium that dynamically stores a program for a short period of time, such as a communication line when transmitting a program via a network such as the Internet or a communication line such as a telephone line, and a recording medium that stores a program for a certain period of time, such as a volatile memory in a computer system that serves as a server or client in such a case. The above program may be a program for realizing part of the above-mentioned functions, or may be a program that can realize the above-mentioned functions in combination with a program already recorded in the computer system. [Explanation of symbols]

[0056] 1. Frame skip detection device 2 Learning device 11 Even-odd conversion section 12 Even Frame Memory 13 Odd Frame Memory 14 Judgment section 21 Even frame acquisition section 22 Odd frame acquisition section 23 Interframe pixel difference calculation unit (conversion unit) 24 Motion vector calculation unit (conversion unit) 25 Frame rate acquisition section 26 Teacher Data Acquisition Department 27 Learning Department

Claims

1. an even-odd conversion unit that divides frames constituting an input video into even frames and odd frames; a determination unit that inputs the frame rate of an input video along with pixel values ​​extracted from each of the even frames and the odd frames to a learning model that has learned a label indicating whether or not a frame skip has occurred using the pixel values ​​of two consecutive frames and the frame rate of the video as input, and outputs a determination result.

2. The learning model further receives data converted from the pixel values ​​as input, a conversion unit that converts pixel values ​​extracted from each of the even-numbered frames and the odd-numbered frames, The frame skip detection device according to claim 1 , wherein the determination unit inputs the data obtained by the conversion unit into the learning model together with the pixel values.

3. The frame skip detection device according to claim 2 , wherein the data includes at least one of a pixel difference value, a motion vector, a frequency decomposition value, and an image feature amount.

4. A frame skip detection device as described in any one of claims 1 to 3, wherein the judgment unit outputs a judgment result based on the output ratio when the pixel values ​​are input to the learning model in block units obtained by dividing the frame into blocks of a predetermined size.

5. A learning device that learns the learning model used in the frame skip detection device according to any one of claims 1 to 4, comprising: A teacher data acquisition unit that acquires teacher data representing a correct answer corresponding to the learning image data for the learning model; A learning device comprising: a learning unit that calculates an error between the output of the learning model and the teacher data, and updates parameters of the learning model based on the error.

6. A frame skip detection program for causing a computer to function as the frame skip detection device according to any one of claims 1 to 4.

Images