Program, information processing device, and information processing method

Abstract

Execute more stable rate control for video encoding. 【Solution means】A program for causing a computer to execute an expansion step and a learning step. The expansion step is the encoding state of the (t-1)-th image included in the video, which is the buffer utilization rate b t-1 and the state s t-1 including the reward r representing the evaluation value of the encoding of the (t-1)-th image t and the state s t-1 and the action a obtained by a model that inputs the state s t-1 and outputs an action used to determine the encoding parameters, and the encoding state of the t-th image included in the video, which is the buffer utilization rate b t and the state s t and based on the experience data including, at least the buffer utilization rate b t-1 included in the experience data, generate one or more extended experience data with different values of the buffer utilization rate b. The learning step learns the model using the experience data and the extended experience data.

Description

[Technical Field]

[0001] Embodiments of the present invention relate to a program, an information processing device, and an information processing method. [Background technology]

[0002] Because video data is large, it is compressed using video encoding when transmitted over a network and when stored in a storage device. While it is desirable to minimize image quality degradation due to encoding, the amount of data after encoding must be kept below a certain level to meet the limitations of expected network bandwidth and storage capacity.

[0003] To minimize image quality degradation under data volume constraints, encoding parameters (rate control) are implemented. Video encoding is a complex process utilizing inter-frame and intra-frame correlations and frequency component biases, making it difficult to accurately predict image quality and data volume before encoding. Therefore, manually optimizing rate control is challenging.

[0004] Therefore, a technique has been proposed to optimize rate control using reinforcement learning. This technique optimizes rate control from data obtained through trial and error in video encoding (empirical data), resulting in better rate control than when designed manually. [Prior art documents] [Patent Documents]

[0005] [Patent Document 1] Special Publication No. 2023-535290 [Non-patent literature]

[0006] [Non-Patent Document 1] Mingliang Zhou, Xuekai Wei, Sam Kwong, Weijia Jia, Bin Fang, “Rate Control Method Based on Deep Reinforcement Learning for Dynamic Video Sequences in HEVC”, IEEE Transactions on Multimedia, volume: 23, pp.1106-1121, 2021. [Non-Patent Document 2] Junqi Liao, Li Li, Dong Liu, Houqiang Li, “Reinforcement Learning-based Frame-level Bit Allocation for VVC”, IEEE International Conference on Visual Communications and Image Processing (VCIP), December, 2023. [Overview of the project] [Problems that the invention aims to solve]

[0007] The present invention aims to provide a program, an information processing device, and an information processing method that can perform more stable rate control for the encoding of moving images. [Means for solving the problem]

[0008] The program of the embodiment is a program that causes a computer to perform an extension step and a learning step. The extension step is the encoding state of the (t-1)th image contained in the video, with buffer usage b t-1 State s including t-1 And the reward r represents the evaluation value of the encoding of the (t-1)th image. t And, state s t-1 The action a is determined by a model that takes input as input and outputs the action used to determine the encoding parameters. t-1 The encoding state of the t-th image included in the video, and the buffer usage bt State s including t Based on empirical data including and, at least the buffer usage b included in the empirical data t-1 Generate one or more augmented experience data sets by changing the value of . The learning step trains the model using the experience data and augmented experience data. [Brief explanation of the drawing]

[0009] [Figure 1] Block diagram of an information processing device according to an embodiment. [Figure 2] Block diagram of the rate control unit. [Figure 3] A diagram showing examples of empirical data and augmented empirical data. [Figure 4] Flowchart of the extension process in the embodiment. [Figure 5] A diagram illustrating the effects of data augmentation. [Figure 6] A diagram illustrating the effects of data augmentation. [Figure 7] A diagram showing an example of the specified screen. [Figure 8] Hardware configuration diagram of the information processing device according to the embodiment. [Modes for carrying out the invention]

[0010] A preferred embodiment of the information processing device according to this invention will be described in detail below with reference to the attached drawings.

[0011] In reinforcement learning for rate control, rate control is optimized based on the empirical data obtained through trial and error during training. Therefore, rate control is not learned for image changes that did not occur during training. For example, consider training rate control on video with little change, such as surveillance images of an empty room or surveillance images of stationary equipment. When training on such video, once reinforcement learning has progressed to a certain extent and rate control is possible, the amount of encoded data for each frame position within the GOP (Group of Pictures) will be concentrated within a narrow range.

[0012] When transmitting video over a network, data before transmission is temporarily stored in a transmit buffer. If the amount of data is concentrated within a specific range, the amount of data in the transmit buffer will also be concentrated within a narrow range for each frame position in the GOP (Group of Processors). The amount of data in the transmit buffer can be expressed as the transmit buffer utilization rate (hereinafter referred to as buffer utilization rate). Buffer utilization rate is one of the pieces of information used to determine the encoding parameters in rate control. As described above, when dealing with video with little change, the buffer utilization rate only takes values ​​within a specific range during training, making it impossible to properly learn control outside of that range.

[0013] As a result, when rate control is implemented using a rate control model obtained through reinforcement learning, for example, image changes that did not occur during training may occur, causing buffer usage to deviate from the learned range and potentially making proper rate control impossible. A rate control model is a model that is learned to take the state of encoding as input and output the action used to determine the encoding parameters. Training the rate control model to optimize the action for each state is equivalent to optimizing (learning) rate control.

[0014] The information processing device of this embodiment generates data on various buffer usage rates (extended empirical data) from a single set of empirical data by augmenting data on buffer usage rates, and uses this data for learning. As a result, it becomes possible to learn rate control at buffer usage rates that did not occur during the initial learning process without reducing the efficiency of the learning process, enabling more stable rate control.

[0015] Figure 1 is a block diagram showing an example of the configuration of an information processing device 100 according to an embodiment. As shown in Figure 1, the information processing device 100 includes a transmission buffer 121, an acquisition unit 101, an encoding unit 102, an evaluation unit 103, an output control unit 104, and a rate control unit 110.

[0016] The transmission buffer 121 temporarily stores the data (compressed data) encoded by the encoding unit 102 before it is output (transmitted) as output data by the output control unit 104.

[0017] The acquisition unit 101 acquires various types of information used in the information processing apparatus 100. For example, the acquisition unit 101 acquires a moving image to be encoded and outputs the images included in the moving image to the encoding unit 102 in units of one frame. The method for acquiring information by the acquisition unit 101 may be any method, and for example, a method of receiving from an external device via a network and a method of reading information from a storage medium can be applied.

[0018] The encoding unit 102 encodes the image input from the acquisition unit 101 to reduce the data amount. The encoding method may be any method, and for example, MPEG-2, MPEG-4, H.264, and H.265 etc. can be used. The encoding unit 102 stores the encoded data (compressed data) in the transmission buffer 121.

[0019] When encoding the image I input as the t-th from the acquisition unit 101 t the encoding state s t is sent to the rate control unit 110. The state s t is data including, for example, the following elements, and is used for the rate control unit 110 to determine the action a t . · Image I t · Buffer utilization rate b t · Image I t Position p in the GOP of t · Image I t The M (M is an integer greater than or equal to 1) images I t-1 , ···, I t-M corresponding to the data amounts v t-1 , ···, v t-M · Image I t-1 , ···, I t-MThe encoding parameter q used for encoding t-1 ,···,q t-M

[0020] Encoding parameters include, for example, quantization parameters (QP) and lambda. Buffer usage b t This is image I t This is the ratio of the amount of data stored in the transmit buffer 121 to a fixed value at the moment immediately before encoding. The fixed value is, for example, the upper limit of the amount of data that can be stored in the transmit buffer 121. The fixed value may be set to a value that does not cause delay in the processing (display, etc.) of the compressed data on the receiving side.

[0021] The encoding unit 102 processes state s t Action a decided in response t The rate control unit 110 receives this. Action a t This is, for example, a continuous value within a certain range, such as -1 to 1, or an integer value within a certain range, such as 0 to 51. The coding unit 102 performs a predetermined transformation process (linear transformation, integer approximation, etc.) to perform action a t Convert the data into encoding parameters, and use the encoding parameters obtained from the conversion to create image I. t Encoding is performed. The conversion process may be performed within the rate control unit 110.

[0022] Furthermore, the encoding unit 102 outputs evaluation data to the evaluation unit 103. The evaluation data is data for evaluating image quality and includes, for example, the image before encoding, the image after decoding the compressed data, and the buffer usage rate.

[0023] The evaluation unit 103 evaluates the image quality of the compressed data using evaluation data. For example, the evaluation unit 103 evaluates the reward r used for training the rate control model. t The reward r is calculated and output to the rate control unit 110. t This represents the evaluation value of the encoding (result of rate control) for images prior to the t-th image. Reward r tThe value is calculated such that the greater the distortion caused by image encoding, the smaller the value, and the smaller the distortion, the larger the value. Distortion is calculated, for example, by the mean squared error between the image before encoding and the image after decoding the compressed data, PSNR (Peak Signal to Noise Ratio), SSIM (Stractural SIMilarity), etc. Also, the reward r t This value is calculated such that when the buffer usage exceeds a predetermined threshold, the value decreases as the amount exceeding the threshold increases, and increases as the amount exceeding the threshold decreases. The threshold is determined, for example, from the buffer usage value at which a delay is expected to occur in image display on the receiving end when compressed data is transmitted over a network.

[0024] The output control unit 104 controls the output of various types of information used by the information processing device 100. For example, the output control unit 104 retrieves some or all of the compressed data stored in the transmission buffer 121 in the order in which it was stored and transmits it to the network.

[0025] When the rate control unit 110 performs inference using the learned rate control model, it outputs a state s from the encoding unit 102. t Action a t The rate control unit 110 determines the reward r output by the evaluation unit 103 during learning. t We will use this to train a rate control model.

[0026] At least a portion of each of the above components (acquisition unit 101, encoding unit 102, evaluation unit 103, output control unit 104, and rate control unit 110) may be implemented by one or more processing units. Each of the above components may be implemented by, for example, one or more processors. For example, each of the above components may be implemented by having a processor such as a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) execute a program, i.e., by software. Each of the above components may be implemented by a processor such as a dedicated IC (Integrated Circuit), i.e., by hardware. Each of the above components may be implemented by using both software and hardware. When multiple processors are used, each processor may implement one of the above components, or two or more of the above components.

[0027] Next, the details of the configuration of the rate control unit 110 will be described. Figure 2 is a block diagram showing an example of the configuration of the rate control unit 110. As shown in Figure 2, the rate control unit 110 includes a state acquisition unit 111, an inference unit 112, a reward acquisition unit 113, an extension unit 114, a playback buffer 115, and a learning unit 116.

[0028] For example, during the training of the rate control model, the components shown in Figure 2 are used. During inference using the trained rate control model, from the components shown in Figure 2, for example, the state acquisition unit 111 and the inference unit 112 are used.

[0029] The state acquisition unit 111 receives the state s from the encoding unit 102. t The data is obtained and sent to the inference unit 112 and the extension unit 114.

[0030] During inference, the inference unit 112 receives a state s in the rate control model. t The input is action a, which is the output of the rate control model. t The encoding unit 102 outputs the result. During learning, the inference unit 112 processes action a t The data is sent to the encoding unit 102 and the extension unit 114.

[0031] The rate control model can be any form of model, but examples include models using neural networks, linear functions, decision trees, and Gaussian processes.

[0032] The reward acquisition unit 113 receives the reward r output by the evaluation unit 103. t The data is obtained and sent to the extension unit 114.

[0033] The expansion unit 114 generates empirical data that serves as the basis for data expansion, as well as expanded empirical data, and stores each of the generated data in the playback buffer 115. The empirical data and expanded empirical data are data (training data) used to train the rate control model.

[0034] For example, the extension unit 114 determines the encoding state s of the (t-1)th image included in the video. t-1 And the reward r for encoding the (t-1)th image t And, state s t-1 The action a is determined by the rate control model based on the input. t-1 And the encoding state of the t-th image contained in the video, s t and, including empirical data e t It generates and stores it in the playback buffer 115.

[0035] Furthermore, the extension unit 114 uses the experience data e t Based on empirical data e t The states included t-1 Of which, buffer usage b t-1 One or more extended experience data e' with different values t It generates and stores it in the playback buffer 115.

[0036] The details of how the extended experience data is generated will be explained. The extension unit 114 first calculates the buffer usage rate b t-1 Extended buffer usage b' changed to a different value t-1 The extension unit 114 calculates the buffer usage rate b. t-1 Based on the value obtained by adding a random number (noise) to this, the extended buffer usage b't-1 Calculate.

[0037] Random numbers can be generated in any way, but for example, a method of sampling from a normal distribution with mean 0 and variance σ can be applied. Other probability distributions, such as a uniform distribution within a certain range, may also be used.

[0038] The extension unit 114 has an extended buffer usage rate b' t-1 Using reward r' t Calculate the reward r'. t The calculation method is the same as the reward calculation method by the evaluation unit 103. In the extension unit 114, the buffer usage rate b t-1 Instead, extended buffer usage b' t-1 Using the reward r' t The following is calculated: That is, the extension unit 114 calculates the extension buffer usage rate b' t-1 When the amount exceeds a predetermined threshold, the reward r' is set such that the larger the amount exceeding the threshold, the smaller the value, and the smaller the amount exceeding the threshold, the larger the value. t Calculate.

[0039] The extension unit 114 has an extended buffer usage rate b' t-1 Using the extended buffer usage b' t Calculate the extended buffer usage b'. t b' is the extended buffer usage from the previous frame. t-1 This corresponds to the buffer usage obtained by simulating the encoding of video in that case.

[0040] For example, the extension unit 114 has an extension buffer usage rate b' t-1 The network bandwidth and the amount of compressed data for one or more images prior to the (t-1)th image are input to the extended buffer usage b'. t Using an update model that outputs the extended buffer usage b' t The update model is, for example, the extended buffer usage b'. t-1 Compressed data volume v t-1, using the assumed network bandwidth B, is represented by the following equation (1). max(x,y) is a function that outputs the larger value between x and y. b’ t = max(b’ t-1 - B, 0) + v t-1 ···(1)

[0041] The extension part 114 replaces the buffer utilization rate b t included in the experience data e t-1 , the reward r t and the buffer utilization rate b t with the extended buffer utilization rate b’ t-1 , the reward r’ t and the extended buffer utilization rate b’ t respectively to generate the extended experience data e’ t .

[0042] Figure 3 is a diagram showing examples of the experience data e t and the extended experience data e’ t . Data 301 and data 302 are examples of the experience data e t and the extended experience data e’ t respectively.

[0043] In the example of Figure 3, the experience data e t includes the state s t-1 , the state s t , the action a t-1 , and the reward r t . The extended experience data e’ t corresponds to data that simulates a buffer utilization rate different from that of the experience data e t . The extended experience data e’ t is data for enabling the rate control model to learn control at various buffer utilization rates.

[0044] Data 302 is an example of extended experience data generated by extending data 301 (experience data e t ). The data included in the state s’ t-1 of data 302 is the state s t-1 of data 301.The data included is almost the same, but buffer usage b t-1 The extended buffer usage b' t-1 The difference lies in the changes made to data 302's state s'. t The data included is the state of data 301 s t The data included is almost the same, but buffer usage b t The extended buffer usage b' t The difference lies in the fact that it is structured differently.

[0045] Returning to the explanation of Figure 2, the playback buffer 115 is a buffer for storing data input from the extension unit 114. The extension unit 114 inputs both experience data and extended experience data, but the playback buffer 115 does not distinguish between the two and stores both as experience data. When the number of experience data to be stored reaches the upper limit, the playback buffer 115 deletes the oldest experience data and stores new experience data. When the learning unit 116 performs learning of the rate control model, the playback buffer 115 samples a certain number of experience data and inputs them to the learning unit 116.

[0046] The learning unit 116 uses the experience data (experience data, extended experience data) output from the playback buffer 115 to train a rate control model. The rate control model to be trained is the same as the rate control model used by the inference unit 112 to calculate actions. As a training algorithm, reinforcement learning algorithms using experience data, such as DQN (Deep Q-Network), SAC (Soft Actor Critic), DDPG (Deep Deterministic Policy Gradient), and TD3 (Twin Delayed DDPG), can be used.

[0047] The transmit buffer 121 and the playback buffer 115 can be composed of any commonly used storage medium, such as flash memory, memory card, RAM (Random Access Memory), HDD (Hard Disk Drive), and optical disc.

[0048] The transmit buffer 121 and the replay buffer 115 may be implemented as physically different storage media, or as different storage areas on the same physically storage media. Furthermore, the transmit buffer 121 and the replay buffer 115 may each be implemented using multiple physically different storage media.

[0049] Furthermore, the information processing device 100 may be composed of one physical device or multiple physical devices. For example, the information processing device 100 may be built on a cloud environment. Also, each part of the information processing device 100 may be distributed and provided on multiple devices.

[0050] Next, the extended processing by the information processing device 100 of the embodiment will be described. Figure 4 is a flowchart showing an example of the extended processing in the embodiment.

[0051] Augmentation is the process of extending one set of experience data to generate one or more augmented experience data. Augmentation is performed on each of the one or more experience data before performing reinforcement learning on a rate-controlled model.

[0052] The expansion unit 114 is the experience data e that is subject to data expansion. t This is stored in the playback buffer 115 (step S101). Steps S102 to S105 correspond to the process of generating extended experience data.

[0053] The extension unit 114 is used for experiential data e t Buffer usage b included t-1 Add a random number to this to get the extended buffer usage b' t-1 Calculate (Step S102).

[0054] The extension unit 114 has an extended buffer usage rate b' t-1 Therefore, using an update model such as equation (1), the extended buffer usage b' t Calculate (Step S103).

[0055] The extension unit 114 has an extended buffer usage rate b' t Using this, the extended buffer usage is b' t The reward r' in this case t Calculate (Step S104).

[0056] The extension unit 114 is used for experiential data e t Buffer usage b included t-1 , buffer usage b t , and reward r t These are the extended buffer usage b', respectively. t-1 , extended buffer usage is b' t , and reward r' t Extended empirical data e' replaced with t It is generated and stored in the playback buffer (step S105).

[0057] The expansion unit 114 determines whether or not to terminate the expansion (step S106). For example, the expansion unit 114 determines whether or not to terminate the expansion based on a predetermined number of expansion experience data e' t experience data e t If generated from this source, it is determined that the extension has ended.

[0058] If the extension is not terminated (step S106: No), the extension unit 114 returns to step S102 and repeats the process. If the extension is terminated (step S106: Yes), the extension unit 114 terminates the extension process.

[0059] Figures 5 and 6 illustrate the effects of data augmentation. Figure 5 plots the buffer usage for each frame position in the GOP when learning rate control without using augmented experience data for video footage with little screen change. As learning progresses and a certain degree of rate control becomes possible, the buffer usage becomes concentrated within a narrow range for each frame position in the GOP, and almost no data is generated outside of that range.

[0060] Figure 6 plots the buffer usage for each frame position in the GOP when learning rate control by adding augmented experience data generated by the extension unit 114. The squares represent the buffer usage corresponding to the augmented experience data. Because augmented experience data containing various buffer usages is added to the training data, it becomes possible to learn control for screen changes that did not occur during training.

[0061] Next, we will describe an example of a user interface (specification screen) for specifying the settings used to generate augmented experience data. Figure 7 shows an example of the specification screen.

[0062] For example, the output control unit 104 displays a designated screen 700, as shown in Figure 7, on a display device provided by the information processing device 100, or on an external display device. The extension unit 114 generates extended experience data using the setting values ​​specified on the designated screen. The setting values ​​specified on the designated screen include, for example, a method for changing the number of extended experience data to be generated and the buffer usage rate to different values.

[0063] In the example in Figure 7, the data augmentation corresponds to the number of augmented experience data points generated, and the augmentation method corresponds to changing the buffer usage rate to a different value. Figure 7 shows an example where, as an augmentation method, it is possible to specify whether to use a normal distribution or a uniform distribution when sampling the random numbers assigned to the buffer usage rate.

[0064] If a normal distribution is selected, the standard deviation can be specified. If a uniform distribution is selected, the upper and lower limits can be specified. Input devices such as keyboards, mice, and touch panels can be used to select the probability distribution and input numerical values.

[0065] Thus, the information processing device of this embodiment generates augmented experience data of various buffer usage rates and uses it to train the rate control model. This makes it possible to learn rate control at buffer usage rates that did not occur during training, and enables more stable rate control for video encoding.

[0066] Next, the hardware configuration of the information processing device of the embodiment will be described using Figure 8. Figure 8 is an explanatory diagram showing an example of the hardware configuration of the information processing device of the embodiment.

[0067] The information processing device of this embodiment includes a control device such as a CPU (Central Processing Unit) 51, a storage device such as a ROM (Read Only Memory) 52 and a RAM (Random Access Memory) 53, a communication interface 54 that connects to a network for communication, and a bus 61 that connects each part.

[0068] The program to be executed in the information processing device of this embodiment is provided pre-installed in a ROM 52 or the like.

[0069] The program executed by the information processing device of this embodiment may be configured to be provided as a computer program product by recording it in an installable or executable file format onto a computer-readable recording medium such as a CD-ROM (Compact Disk Read Only Memory), a flexible disk (FD), a CD-R (Compact Disk Recordable), or a DVD (Digital Versatile Disk).

[0070] Furthermore, the program executed by the information processing device of the embodiment may be stored on a computer connected to a network such as the Internet and provided by downloading it via the network. Alternatively, the program executed by the information processing device of the embodiment may be provided or distributed via a network such as the Internet.

[0071] The program executed in the information processing device of this embodiment can cause the computer to function as a component of the information processing device described above. This computer can read the program from a computer-readable storage medium onto the main memory and execute it using the CPU 51.

[0072] While several embodiments of the present invention have been described, these embodiments are presented as examples only and are not intended to limit the scope of the invention. These novel embodiments can be carried out in a variety of other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their variations are included in the scope and spirit of the invention, as well as in the claims of the invention and its equivalents. [Explanation of Symbols]

[0073] 100 Information Processing Devices 101 Acquisition Department 102 Encoding section 103 Evaluation Department 104 Output Control Unit 110 Rate Control Unit 111 Status acquisition unit 112 Reasoning Department 113 Compensation Acquisition Department 114 Expansion section 115 Playback buffer 116 Learning Department 121 Transmit Buffer

Claims

1. On the computer, The encoding state of the (t-1)th image included in the video, with buffer usage b. ｔ－１ state s ｔ－１ The reward r represents the evaluation value of the encoding of the (t-1)-th image. ｔ And, the aforementioned state s ｔ－１ Action a is determined by a model that takes input as input and outputs the action used to determine the encoding parameters. ｔ－１ The encoding state of the t-th image included in the video, and the buffer usage b ｔ state s ｔ Based on empirical data including the above, the buffer usage rate b included in the empirical data is determined. ｔ－１ An extension step that generates one or more extended experience data sets in which the value is changed to a different value, A learning step in which the model is trained using the aforementioned empirical data and the aforementioned augmented empirical data, A program to execute.

2. The aforementioned extension step is, The buffer usage rate b ｔ－１ is changed to a different value to obtain an extended buffer usage rate b' ｔ－１ and The aforementioned extended buffer usage rate b' ｔ－１ Using reward r' ｔ Calculate, The aforementioned extended buffer usage rate b' ｔ－１ Using the extended buffer usage b' ｔ Calculate, The buffer usage rate b included in the aforementioned empirical data ｔ－１ , the aforementioned reward r ｔ and the buffer usage rate b ｔ The extended buffer usage rate b' ｔ－１ , the aforementioned reward r' ｔ and the extended buffer usage rate b' ｔ To generate the extended experience data that has been replaced with the above, The program according to claim 1.

3. The aforementioned extension step is, The aforementioned buffer usage rate b ｔ－１ Based on the value obtained by adding a random number to the above, the extended buffer usage rate b' ｔ－１ Calculate The program according to claim 2.

4. The aforementioned extension step is, The aforementioned extended buffer usage rate b' ｔ－１ The network bandwidth and the amount of compressed data for one or more images prior to the (t-1)th image are input to the extended buffer usage b'. ｔ Using a model that outputs the extended buffer usage b' ｔ Calculate The program according to claim 2.

5. The aforementioned reward r ｔ This value is calculated such that it decreases as the distortion caused by encoding increases, and as the amount by which the buffer usage exceeds the threshold increases. The program according to claim 1.

6. The computer is further instructed to execute an output control step that outputs a specification screen for specifying the setting values ​​used to generate the aforementioned extended experience data, The extension step generates the extended experience data using the setting values ​​specified using the designated screen. The program according to claim 2.

7. The aforementioned setting values ​​are the number of extended experience data to be generated and the buffer usage rate b. ｔ－１ Including how to change to a different value, The program according to claim 6.

8. The encoding state of the (t-1)th image included in the video, with buffer usage b. ｔ－１ state s ｔ－１ The reward r represents the evaluation value of the encoding of the (t-1)-th image. ｔ And, the aforementioned state s ｔ－１ Action a is determined by a model that takes input as input and outputs the action used to determine the encoding parameters. ｔ－１ The encoding state of the t-th image included in the video, and the buffer usage b ｔ state s ｔ Based on empirical data including the above, the buffer usage rate b included in the empirical data is determined. ｔ－１ An extension unit that generates one or more extended experience data by changing to different values, A learning unit that learns the model using the aforementioned empirical data and the aforementioned extended empirical data, An information processing device equipped with the following features.

9. An information processing method performed by an information processing device, The encoding state of the (t-1)th image included in the video, with buffer usage b. ｔ－１ state s ｔ－１ The reward r represents the evaluation value of the encoding of the (t-1)-th image. ｔ And, the aforementioned state s ｔ－１ Action a is determined by a model that takes input as input and outputs the action used to determine the encoding parameters. ｔ－１ The encoding state of the t-th image included in the video, and the buffer usage b ｔ state s ｔ Based on empirical data including the above, the buffer usage rate b included in the empirical data is determined. ｔ－１ An extension step that generates one or more extended experience data sets in which the value is changed to a different value, A learning step in which the model is trained using the aforementioned empirical data and the aforementioned augmented empirical data, Information processing methods including

Images