Bit rate prediction model generating device, bit rate prediction device, image data compression system, bit rate prediction model generating method, and bit rate prediction model generating program

Abstract

To provide a bit rate prediction model generating device capable of setting a suitable bit rate for each portion of image data without regard to a situation where photographic data is photographed.SOLUTION: A bit rate prediction model generating device 5 comprises: an input unit for inputting image data; a bit rate extraction unit 512 which when the image data is encoded, extracts a bit rate capable of keeping predetermined image quality; a feature amount extraction unit 513 which extracts a feature amount from the image data; and a mechanical learning unit 51 which sets the extracted bit rate as a target variable and learns a mechanical learning model with the feature amount as an explanation variable.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a bitrate prediction model generation device, a bitrate prediction device, a video data compression system, a bitrate prediction model generation method, and a bitrate prediction model generation program, and in particular to a bitrate prediction model generation device, a bitrate prediction device, a video data compression system, a bitrate prediction model generation method, and a bitrate prediction model generation program that predict the optimal bitrate for each portion of video data. [Background technology]

[0002] Currently, video data is compressed and recorded in real time using a specified standard (for example, MPEG-4 AVC (Advanced Video Coding) (also known as H.264)). In this case, the compression rate tends to be set at a relatively high bit rate to accommodate scenes that are difficult to compress. As a result, the size of the compressed video data is not sufficiently small, and there is room for improvement in terms of increasing the storage capacity for storing the compressed video data.

[0003] Patent Document 1, for example, is a known invention for improving the above points. The on-board device described in Patent Document 1 is mounted on a vehicle, estimates the state of the vehicle from the vehicle speed and surrounding obstacles, compresses sensor data at a compression rate according to the vehicle state, and transmits the data to the outside. In Patent Document 1, an obstacle detection unit receives sensor data transmitted from the on-board device and generates a signal indicating the presence or absence of obstacles around the vehicle and the vehicle speed according to the compression rate. Patent Document 1 also describes reducing the compression rate of the sensor data when the vehicle speed is relatively high, thereby reducing degradation of the sensor data.

[0004] The vehicle monitoring device described in Patent Document 2 photographs passing vehicles with a camera and detects traffic conditions based on the photographed images. If the vehicle monitoring device described in Patent Document 2 determines that the traffic conditions are "abnormal," it sets a lower compression rate for the images than if it determines that the traffic conditions are "normal." Patent Document 2 discloses that, for example, if a vehicle is detected that has been stopped continuously for a certain period of time or more even though there are no vehicles stopped or moving slowly downstream, this condition is determined to be abnormal. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Publication No. 2023-71336 [Patent Document 2] Japanese Patent Application Publication No. 9-305891 Summary of the Invention [Problem to be solved by the invention]

[0006] However, even if a vehicle is traveling at high speeds, for example, on a highway, the speed remains constant, and data degradation may not be a problem even if the compression rate of sensor data is increased. Furthermore, even when a vehicle is traveling at low speeds or is stopped, the surrounding environment may change significantly. According to the invention disclosed in Patent Document 1, in such cases, it is possible that the compression rate is lowered more than necessary, or the quality of the sensor data detecting the changing environment may deteriorate, making it impossible to adequately identify the changes in the environment.

[0007] Furthermore, the invention described in Patent Document 2 captures images of traffic conditions and determines whether the traffic conditions are normal or abnormal from the captured images, so it waits for the time required for the determination to elapse before switching the compression rate. As a result, the invention disclosed in Patent Document 2 causes a delay in determining whether to switch the compression rate, making it difficult to set the optimal compression rate for each scene, especially when compressing video data in real time.

[0008] The present invention has been made in consideration of the above points, and relates to a bit rate prediction model generation device, a bit rate prediction device, a video data compression system, a bit rate prediction model generation method, and a bit rate prediction model generation program that are capable of setting an appropriate bit rate for each portion of video data regardless of the circumstances under which the video data was captured. [Means for solving the problem]

[0009] A bitrate prediction model generation device according to an aspect of the present invention that solves the above problems includes an input unit that inputs video data, a bitrate extraction unit that extracts a bitrate that can maintain a predetermined image quality when the video data is encoded, a feature extraction unit that extracts features from the video data, and a machine learning unit that sets the bitrate as a target variable and trains a machine learning model using the features as explanatory variables.

[0010] Furthermore, a bitrate prediction device according to an aspect of the present invention is a bitrate prediction device that compresses video data using a bitrate prediction model generated using the above-described bitrate prediction model generation device, and includes a video data input unit that inputs video data, a feature extraction unit that extracts features from the video data, and the features are input to the bitrate prediction model, which predicts the bitrate based on the features.

[0011] A video data compression system according to one embodiment of the present invention includes a bitrate prediction model generated by a bitrate prediction model generation device, the bitrate prediction model generation device including a learning input unit that inputs learning video data, a bitrate extraction unit that extracts a bitrate that can maintain a predetermined image quality when the learning video data is encoded, a learning feature extraction unit that extracts learning features from the learning video data, and a machine learning unit that sets the bitrate as a target variable and trains a machine learning model using the learning features as explanatory variables; a processing video data input unit that inputs processing video data, and a processing feature extraction unit that extracts processing features from the processing video data; the processing features are input to the bitrate prediction model, and the bitrate prediction model predicts the processing bitrate.

[0012] In addition, the bit rate prediction model generation method of the present invention includes the steps of inputting video data, extracting a bit rate that can maintain a predetermined image quality when the video data is encoded, extracting features from the video data, and setting the bit rate as a target variable and training a machine learning model using the features as explanatory variables.

[0013] The bit rate prediction model generation program of the present invention causes a computer to implement the above bit rate prediction model generation method. [Effects of the Invention]

[0014] According to the above aspects, it is possible to provide a bit rate prediction model generation device, a bit rate prediction device, a video data compression system, a bit rate prediction model generation method, and a bit rate prediction model generation program that are capable of setting an appropriate high-speed bit rate for each portion of video data regardless of the circumstances under which the video data was captured. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a functional block diagram for explaining a data compression system according to an embodiment of the present invention; [Figure 2]1 is a functional block diagram illustrating a bit rate prediction model generation device according to an embodiment of the present invention. [Figure 3] FIG. 10 is a diagram showing a list of data extracted from training data according to an embodiment of the present invention. [Figure 4] 4 is a header for explaining other data corresponding to the list of data shown in FIG. 3. [Figure 5] 3 is a flowchart for explaining a method for forming the bit rate prediction model generation device shown in FIG. 2. [Figure 6] 2 is a flowchart illustrating a bit rate prediction method executed by the bit rate prediction device of FIG. 1. [Figure 7] FIG. 1 is a diagram illustrating a computer that executes a bitrate prediction model generation method according to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0016] A bitrate prediction model generation device, a bitrate prediction device, a video data compression system, a bitrate prediction model generation method, and a bitrate prediction model generation program according to one embodiment of the present invention will be described below with reference to the drawings. Note that this embodiment will be described by giving examples of the bitrate prediction model generation device, the bitrate prediction device, the video data compression system, the bitrate prediction model generation method, and the bitrate prediction model generation program, and does not limit the specific configuration, processing procedure, etc. of this embodiment.

[0017] (Video Data Compression System) FIG. 1 is a functional block diagram illustrating a data compression system 1 including a bit rate prediction device 10 according to one embodiment of the present invention. The video data compression system shown in FIG. 1 includes a decoder 21 functioning as the bit rate prediction device 10, a bit rate prediction model 27, and an encoder 25. The bit rate described in this specification refers to the amount of data that can be transmitted and received per second, and is calculated by multiplying the number of pixels of the image sensor that captured the video data by the number of frames by the compression rate. In this embodiment, the bit rate prediction may be performed by the bit rate prediction model 27 predicting the bit rate, or by returning the compression rate to the encoder 25, which then extracts the bit rate based on the compression rate.

[0018] The bitrate prediction model 27 is trained using previously prepared training data 103 (used for training the bitrate prediction model, and therefore corresponds to "video data for training data" in this embodiment). The bitrate prediction model 27 is trained to extract a bitrate that can maintain a predetermined video quality according to features (training features) included in the training video data 103. Here, the predetermined video quality may be determined based on, for example, the score of a VMAF image quality index. In such a case, the score is represented by a predetermined acceptable score. A generating device and a generating method for the bitrate prediction model 27 will be described in detail later.

[0019] The data compression system 1 also includes a decoder 21 and an encoder 25. The decoder 21 functions as the bit rate prediction device of this embodiment, extracting feature quantities 101 (processing feature quantities) from video data 107 and inputting the feature quantities to a bit rate prediction model 27. The bit rate prediction model 27 predicts an appropriate bit rate for video data 23 (processing video data) based on the feature quantities 101 and notifies the encoder 25 of the predicted bit rate 105. At this time, the decoded processing video data 23 is input to the encoder 25, and the encoder 25 encodes the processing video data 23 according to the notified bit rate 105, compresses it using, for example, H.264, and stores it in a database (DB) 13. The decoder 21 can extract feature quantities from the video data 107 when decoding the video data 107. Such feature extraction can be performed in parallel with the conversion of the video data 107, and can be executed in a short time. However, this embodiment is not limited to extracting features when decoding the video data 107, and another program may be executed in parallel with the decoding.

[0020] In this embodiment, video data to be processed is compressed using, for example, H.264 and stored in a database (DB) 11. The video data stored in DB 11 is divided at regular intervals and input to a video data compression system 1. Then, video data encoded by an encoder 25 is divided at regular intervals and stored in DB 13. Each unit (frame group) of divided video data is also called a GOP (Group of Picture). In FIG. 1, the GOP input to the video data compression system 1 is GOP 17, and the GOP output from the video data compression system 1 is GOP 19. GOP 19 may be recombined as necessary. GOPs 17 and 19 are set to approximately 10 to 15 frames. This processing allows the video data to be divided into characteristic scenes.

[0021] Input of GOP 17 to video data compression system 1 may be performed, for example, through a series of operations from acquiring video data to compressing it, temporarily saving it in DB 11, and dividing it, or may be performed by compressing and dividing the video data after all of it has been acquired. GOPs 17 and 19 in video data compression system 1 are not limited to those generated outside decoder 21 and encoder 25 as shown in FIG. 1; decoder 21 and encoder 25 may have a function for generating GOPs internally. The input unit for GOP 17 (video data) to decoder 21 may be an input port that inputs GOP 107 to decoder 21, a communication device that receives video data via a network or the like, or an external memory connected to video data compression system 1 (none of which are shown).

[0022] (Bitrate prediction model generator) Fig. 2 is a functional block diagram for explaining a bitrate prediction model generation device 5 that generates the bitrate prediction model 27 shown in Fig. 1. The bitrate prediction model generation device 5 includes a machine learning unit 51 and a machine learning model 52 that is trained by the machine learning unit 51. As the machine learning model 52, for example, a linear regression model of Scikit-learn may be used.

[0023] The machine learning unit 51 includes an input unit (not shown) that inputs video data DP (learning video data), a bit rate extraction unit 512 that extracts a bit rate that can maintain a predetermined image quality when the learning video data DP is encoded, and a feature extraction unit (learning feature extraction unit) 513 that extracts features (learning features) from the learning video data DP. The machine learning unit 51 also includes a learning data extraction unit 511 that divides the learning video data DP into groups of frame images with the same number as GOPs 17 and 19 shown in FIG. 1 to extract learning data (learning video data). The learning data is, for example, compressed according to the H.264 standard, containerized in MP4, and created by dividing it into GOPs (groups of 10 to 15 frames).

[0024] The bitrate extraction unit 512 extracts the bitrate to be set in the encoder 25 shown in FIG. 1 using, for example, VMAF, a video evaluation library. Specifically, the bitrate set in the encoder 25 is gradually adjusted for compression, and the minimum bitrate at which a VMAF image quality index score (hereinafter referred to as the "reference score") predetermined by the designer is maintained upon decoding is determined. The bitrate adjustment is performed, for example, by increasing or decreasing the bitrate in increments of 100 kbps. The bitrate extraction unit 512 uses the extracted bitrate as a target variable when training the machine learning model 52. Note that this process of adjusting the bitrate takes a relatively long time. However, in this embodiment, the machine learning model 52 is trained by adjusting the bitrate. After training of the machine learning model 52 is completed, the feature extracted during decoding by the decoder 21 can be set as an explanatory variable of the machine learning model 52, thereby speeding up processing.

[0025] The feature extraction unit 513 also extracts, as a feature, at least one of the proportion of intra macroblocks included in the video data and the proportion of skipped macroblocks included in the video data. Furthermore, this embodiment also extracts the magnitude of the quantization scale of the training data as a feature. Here, an intra macroblock is a block that is compressed using information within one frame, regardless of other frames. A skipped macroblock is a macroblock whose video data is identical to that of a macroblock at the same position in another frame (mainly the immediately preceding frame). For such macroblocks, extremely high compression efficiency can be achieved. The quantization scale of the training data can be set in units of macroblocks (or, more precisely, in units of even smaller blocks). A larger quantization scale is a characteristic parameter that increases the compression rate but reduces data accuracy during decoding (i.e., the image becomes rougher). However, this embodiment is not limited to using the proportion of intra macroblocks and the proportion of skipped macroblocks included in the video data as feature quantities. The feature quantity may be any factor as long as it is an index that estimates the relationship between the degree of compression of video data and the image quality after encoding.

[0026] As described above, the bitrate prediction model generation device 5 of this embodiment can train the machine learning model 52 so that feature values ​​correspond to bitrates that can maintain a predetermined video quality. Therefore, the video data compression system 1 can obtain a bitrate that can maintain a predetermined video quality by setting feature values ​​in the machine learning model. The bitrate prediction of this embodiment can be completed using only the video data without using external data, thereby reducing the processing load and contributing to a smaller configuration (device) that executes the processing and increasing processing speed. Furthermore, feature values ​​can be extracted in this embodiment by, for example, setting a log in an H.264 decoder or tracer program and inputting training data into this program. Therefore, the feature extraction process is completed in a relatively short time, making this embodiment suitable for processing video data for processing in real time. Furthermore, this embodiment can be implemented faster and with fewer resources than estimating bitrates using a machine learning framework, and therefore has an advantage over known techniques in terms of processing cost performance.

[0027] (Data List) Next, a list of data extracted from the training data will be described. FIG. 3 is a diagram showing a list of data extracted from the training data. FIG. 3 includes data 31, 32, and 33. Data 31 includes a training data name (file name). The training data name may be a name indicating a driving route, such as High way_1, High way_2, High way_3, Drive_1, Drive_2, Drive_3, etc. Data 32 includes a bit rate corresponding to the file name and a video quality that is equal to or exceeds the VMAF standard score achieved by the bit rate. The bit rate included in data 32 is the minimum bit rate that achieves the standard score mentioned above in data 32.

[0028] Data 33 includes a pixel size and frame rate corresponding to the file name. The pixel size and frame rate are used to normalize the target bit rate when processing data with different pixel sizes and frame rates. Of data 31, 32, and 33, data Va is learning data extracted from video captured while a vehicle was traveling on a highway. Video captured while traveling on a highway has little change and is generally data that is easy to compress (has a low bit rate), with a bit rate of approximately 500 to 600 kbps.

[0029] Data Vb is learning data extracted from video data taken while the vehicle is traveling on an ordinary road (urban area). When traveling in an urban area, the vehicle repeatedly starts and stops in a relatively short period of time. Because the video footage changes in a complex manner while traveling, the bit rate for playback and compression of the video is approximately 2000 kbps. Furthermore, when the vehicle is stopped at a traffic light, the bit rate for playback and compression drops to approximately 700 to 900 kbps.

[0030] Data Vc is learning data extracted from video data taken while driving on an ordinary road (suburban). When driving in the suburbs, the vehicle repeatedly starts and stops at longer intervals than when driving in urban areas. The video changes less while driving than when driving in urban areas, so the bit rate for playback and compression of the video is approximately 600 to 700 kbps. In addition, the bit rate for playback and compression when stopped at traffic lights is approximately 300 kbps, because there is less change in the video around the vehicle than when driving in urban areas.

[0031] FIG. 4 shows data 34, 35, and 36, each containing a plurality of data items corresponding to each training data item with a file name shown in FIG. 3. Data 34 indicates the proportion of intra macroblocks in the training data for the entire GOP. Data 35 indicates the proportion of skip macroblocks in the training data for the entire GOP. Data 36 indicates the average quantization scale of the training data for the entire GOP. Of data 34, 35, and 36, data Vaa corresponds to data Va in data 31, data Vbb corresponds to data Vb in data 31, and data Vcc corresponds to data Vc in data 31.

[0032] For video with a quality that exceeds the VMAF standard score, the bitrate tends to be positively proportional to the percentage of intra macroblocks. Also, for video with a quality that exceeds the VMAF standard score, the bitrate tends to be negatively proportional to the percentage of skipped macroblocks. Furthermore, for video with a quality that exceeds the VMAF standard score, the bitrate tends to be positively proportional to the average value of the quantization scale.

[0033] The extracted features are merged with existing data (such as the minimum bit rate to maintain image quality). The inference model used for machine learning is generated using, for example, the data list and linear regression model shown in Figures 3 and 4.

[0034] The intra-macroblock rate (intramb_rate), skip-macroblock rate (skipmb_rate), and quantization scale mean (qscale_mean) in the data list in Figure 3 have an approximately linear relationship with the bit rate. Therefore, we assume that the bit rate can be expressed by the following linear formula: Bitrate = A1*intramb_rate + A2*skipmb_rate + A3*qscale_mean + A0 (An is a coefficient) When a bitrate prediction model is trained using a large amount of training data, the coefficients (An) are optimized, and the prediction accuracy for unknown data increases (if the prediction accuracy does not increase, the data = features need to be reviewed). A linear equation using coefficients with sufficiently high prediction accuracy is used as the inference model.

[0035] (Method for forming a bit rate prediction device) FIG. 5 is a flowchart for explaining a method for forming the bitrate prediction model generation device 5 shown in FIG. 2. The bitrate prediction model generation device 5 reads training video data DP (step S401). The training data extraction unit 511 divides the training video data DP into GOPs to create training data (step S402). The bitrate extraction unit 512 then encodes and decodes the training data to extract a bitrate that can maintain video quality at a predetermined VMAF standard score (step S403). The feature extraction unit 513 extracts average values ​​of intra macroblocks, skip macroblocks, and quantization scales as features from the training data (step S404). Furthermore, the machine learning unit 51, which includes the training data extraction unit 511, bitrate extraction unit 512, and feature extraction unit 513, trains the machine learning model 52 using the bitrate as a target variable and the features as explanatory variables (step S405).

[0036] 6 is a flowchart illustrating a bit rate prediction method executed by the bit rate prediction device 10 included in the video data compression system 1 of FIG. 1. The video data compression system 1 reads video data from the DB 11 (step S601). The read video data is divided into GOPs 17 in the video data compression system 1 (step S602). Each GOP 17 becomes video data to be processed. The decoder 21 decodes the video data to be processed. As a result of the decoding, features of the video data to be processed are extracted (step S603). The decoded video data to be processed 23 is input to the encoder 25.

[0037] Meanwhile, the feature values ​​extracted in step S603 are input to the bit rate prediction model 27. The bit rate prediction model 27 predicts the bit rate based on the input feature values ​​using an inference program that has been improved in accuracy through learning (step S604). The video data compression system 1 encodes the processing video data 23 in accordance with the predicted bit rate. The encoded video data is divided into GOPs 19 and stored in the DB 13.

[0038] (Bitrate prediction model generation program) The bitrate prediction model generation method of this embodiment shown in FIG. 5 is executed by a computer using a program. FIG. 7 is a diagram showing a computer 110 that executes the program. The computer 110 includes known hardware such as a CPU (Central Processor Unit) 111, a memory device 113, and a user interface 115. The program may be stored in the memory device 113, recorded on a computer-readable medium, or downloaded via a network. When the program loaded into the working memory (memory device) of the computer is executed by the CPU 111, the CPU 111 executes the functions of a training data extraction unit 511, a bitrate extraction unit 512, and a feature extraction unit 513, thereby realizing the bitrate prediction model generation method described above.

[0039] As described above, the bit rate prediction model generation device, bit rate prediction device, video data compression system, bit rate prediction model generation method, and bit rate prediction model generation program of the present embodiment predict the bit rate when compressing video according to feature quantities such as intra macroblocks and skip macroblocks contained in frame images that make up the video. Therefore, the present embodiment can predict an appropriate bit rate for each GOP regardless of the situation (vehicle speed, etc.) of the vehicle in which the video was captured. Furthermore, the present embodiment trains a machine learning model with training data in advance to create a bit rate prediction model, and then inputs the video data to be processed into the bit rate prediction model, thereby enabling high-speed prediction and setting of the bit rate of the video data to be processed. [Explanation of symbols]

[0040] 1 Data Compression System 5. Bitrate prediction model generator 10 Bitrate Predictor 11,12 Database 21 Decoder 23 Video data for processing 25 Encoder 27 Bitrate Prediction Model 31, 32, 33, 34, 35, 36 Data 51 Machine Learning Department 52 Machine Learning Models 110 Computer DP learning video data

Claims

1. an input unit for inputting video data; a bit rate extraction unit that extracts a bit rate that can maintain a predetermined image quality when the video data is encoded; a feature extraction unit that extracts features from the video data; a machine learning unit that sets the bit rate as a target variable and trains a machine learning model using the feature amount as an explanatory variable, Bitrate prediction model generator.

2. The bitrate prediction model generating device according to claim 1 , further comprising a training data extracting unit that divides the video data at regular intervals to extract training data.

3. 2. The bitrate prediction model generating device according to claim 1, wherein the feature extracting unit extracts at least one of a ratio of intra macroblocks included in the video data and a ratio of skip macroblocks included in the video data as the feature.

4. 2. A bit rate prediction device that compresses video data using a bit rate prediction model generated by the bit rate prediction model generation device according to claim 1, a video data input unit for inputting the video data; a feature extraction unit that extracts features from the video data; the feature amount is input to the bitrate prediction model, and the bitrate prediction model predicts the bitrate based on the feature amount. Bitrate predictor.

5. The bit rate prediction device according to claim 4 , further comprising a frame group generation unit that divides the video data at regular intervals to generate frame groups.

6. a learning input unit for inputting learning video data; a bit rate extraction unit that extracts a bit rate that can maintain a predetermined image quality when the learning video data is encoded; a learning feature extraction unit that extracts learning features from the learning video data; a machine learning unit that sets the bit rate as a target variable and trains a machine learning model using the learning features as explanatory variables; and a processing image data input unit for inputting processing image data; a processing feature extraction unit that extracts processing features from the processing video data; the processing feature is input to the bitrate prediction model, and the bitrate prediction model predicts a processing bitrate. Video data compression system.

7. inputting video data; extracting a bit rate that can maintain a predetermined image quality when the video data is encoded; extracting features from video data; and a step of training a machine learning model by setting the bit rate as a target variable and using the feature as an explanatory variable. A method for generating a bitrate prediction model.

8. A bitrate prediction model generating program that causes a computer to implement the bitrate prediction model generating method according to claim 7.

Images