Image processing system, information processing device, image processing method, and program

Abstract

In order to suitably estimate bit rate or image recognition accuracy, this video processing system (1) comprises: an acquisition unit (11) that acquires at least one among past video frames, video compression parameters in an encoding process of the video frames, and a video recognition result of decoded video frames obtained by decoding the encoded video frames; and a determination unit (12) that calculates at least one among an estimated value of the bit rate of video frames in the future, and an estimated value of video recognition accuracy.

Description

[Technical field]

[0001] The present invention relates to a video processing system, an information processing device, a video processing method, and program Regarding. [Background technology]

[0002] There are known techniques for improving the quality of experience (QoE), which is the communication quality perceived by a user. Patent Document 1 describes a technique for allocating resources of a communication device according to the user QoE and service priority. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2011-172150 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology of Patent Document 1, the relationship between the bit rate and the QoE is determined in advance. However, in the case of a service in which a video is received and a machine such as an object detector recognizes the video, the QoE index, which indicates the accuracy of recognizing the video, may be high or low depending on the information contained in the video, even if the bit rate is the same. Therefore, a technology that can appropriately estimate the bit rate or image recognition accuracy is desired, but the technology of Patent Document 1 cannot perform such an estimation.

[0005] One aspect of the present invention has been made in consideration of the above problems, and one object of the present invention is to provide a technique that can suitably estimate a bit rate or image recognition accuracy. [Means for solving the problem]

[0006] A video processing system according to one aspect of the present invention includes an acquisition means for acquiring at least one of one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and an estimation means for calculating at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy by referring to the data acquired by the acquisition means.

[0007] An information processing device according to one aspect of the present invention includes an acquisition means for acquiring at least one of one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and a feature calculation means for calculating features from the data acquired by the acquisition means, the feature calculation means calculating features referenced to calculate at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy.

[0008] An information processing device according to one aspect of the present invention includes a feature acquisition means for acquiring features calculated with reference to at least one or more video frames in the past, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and an estimate calculation means for calculating at least one of an estimate of a bit rate of one or more video frames in the future and an estimate of video recognition accuracy from the features acquired by the feature acquisition means.

[0009] An information processing method according to one aspect of the present invention includes acquiring at least one of one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and calculating at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy by referring to the data acquired in the acquiring process.

[0010] According to one aspect of the present invention program causes a computer to function as an acquisition means for acquiring at least one of one or more past video frames, video compression parameters in encoding the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames, and a feature calculation means for calculating a feature from the data acquired by the acquisition means, the feature calculation means calculating a feature to be referenced for calculating at least one of an estimated value of a bit rate of one or more future video frames and an estimated value of video recognition accuracy. do.

[0011] According to one aspect of the present invention program causes a computer to function as: a feature acquisition means for acquiring a feature calculated by referring to at least one or more past video frames, a video compression parameter in encoding the video frames, and a video recognition result of the decoded video frames obtained by decoding the encoded video frames; and an estimate calculation means for calculating at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy from the feature acquired by the feature acquisition means. do. Effect of the Invention

[0012] According to one aspect of the present invention, it is possible to suitably estimate a bit rate or image recognition accuracy. [Brief description of the drawings]

[0013] [Figure 1] 1 is a block diagram showing a configuration of a video processing system according to a first exemplary embodiment of the present invention. [Diagram 2] 1 is a flow chart showing the flow of a video processing method according to a first exemplary embodiment of the present invention. [Diagram 3] FIG. 11 is a block diagram showing a configuration of a video processing system according to a second exemplary embodiment of the present invention. [Figure 4] FIG. 11 is a block diagram showing a configuration of a video processing system according to an exemplary embodiment 3 of the present invention. [Diagram 5] FIG. 11 is a diagram illustrating an example of processing executed by a feature amount calculation unit and an estimated value calculation unit according to the third exemplary embodiment of the present invention. [Figure 6] 11 is a graph showing a method for calculating video compression parameters by an optimization unit according to an exemplary embodiment 3 of the present invention; [Figure 7] FIG. 11 is a flow chart showing the flow of a video processing method according to an exemplary embodiment 3 of the present invention. [Figure 8] FIG. 11 is a block diagram showing a configuration of a video processing system according to an exemplary embodiment 4 of the present invention. [Figure 9] FIG. 11 is a diagram illustrating an example of a process in which a learning unit according to the fourth exemplary embodiment of the present invention causes an estimated value calculation unit to learn. [Figure 10] FIG. 11 is a block diagram showing a configuration of a video processing system according to an exemplary embodiment 5 of the present invention. [Figure 11] 13 is a graph showing a method for calculating video compression parameters by an optimization unit according to an exemplary embodiment 5 of the present invention; [Figure 12] FIG. 13 is a block diagram showing a configuration of a video processing system according to a sixth exemplary embodiment of the present invention. [Figure 13] FIG. 13 is a block diagram showing a configuration of a video processing system according to an exemplary embodiment 7 of the present invention. [Figure 14] FIG. 13 is a block diagram showing a configuration of a video processing system according to an eighth exemplary embodiment of the present invention. [Figure 15]FIG. 13 is a block diagram showing a configuration of a video processing system according to an exemplary embodiment 9 of the present invention. [Figure 16] FIG. 2 is a block diagram showing an example of a hardware configuration of an information processing device in each exemplary embodiment of the present invention. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] Example 1 A first exemplary embodiment of the present invention will be described in detail with reference to the drawings. This exemplary embodiment is a basic form of the exemplary embodiments described later.

[0015] (Configuration of video processing system 1) The configuration of the image processing system 1 according to this exemplary embodiment will be described with reference to FIG. 1. FIG. 1 is a block diagram showing the configuration of the image processing system 1 according to this exemplary embodiment. The image processing system 1 is a system that estimates and calculates at least one of a bit rate in image compression and image recognition accuracy in image recognition. Specifically, the image processing system 1 calculates at least one of an estimated value of a bit rate of one or more frames of image frames in the future and an estimated value of an image recognition accuracy of one or more frames of image frames in the future. As an example, the image processing system 1 calculates an estimated value of a bit rate of one or more frames of image frames output from a camera in the future and an estimated value of an accuracy of an image recognition result output from a recognition engine that uses the image frames as input and recognizes a subject included in the image frames.

[0016] 1, the video processing system 1 includes an acquisition unit 11 and a determination unit 12. The acquisition unit 11 and the determination unit 12 are components that respectively realize an acquisition means and an estimation means in this exemplary embodiment.

[0017] The acquisition unit 11 acquires at least one of the following data: One or more frames of video from the past -Video compression parameters in the encoding process of video frames -Video recognition results for decoded video frames obtained by decoding encoded video frames Examples of video compression parameters include, but are not limited to, a Constant Rate Factor (CRF) and a Quantization parameter (QP).

[0018] The video recognition result of the decoded video frame is a video recognition result output from a recognition engine that uses the decoded video frame as an input and recognizes a subject included in the decoded video frame. The video recognition result of the decoded video frame may also be information obtained as a result of human judgment.

[0019] Furthermore, the acquiring unit 11 may be configured to acquire at least any of the following data in addition to the above-mentioned data. - Code length of one or more past video frames - Decoded video frame obtained by decoding the encoded video frame - Intermediate feature values ​​in video recognition processing of decoded video frames Here, the intermediate feature in the video recognition process of the decoded video frame is a feature obtained in an intermediate layer of a recognition engine that executes the video recognition process. The feature is a numerical representation of a feature extracted from the video frame. An example of the feature is a feature vector in which the extracted features are arranged in a vector, but is not limited to this.

[0020] Moreover, the recognition engine is, for example, configured with a neural network consisting of an input layer, one or more intermediate layers, and an output layer. In the recognition engine, when a video frame is input to the input layer, features in the video frame are extracted in one or more intermediate layers, and a video recognition result is output from the output layer. Here, the features extracted in at least one of the one or more intermediate layers are intermediate features in the video recognition process of the video frame.

[0021] The determination unit 12 calculates at least one of an estimated value of a bit rate of one or more future video frames and an estimated value of video recognition accuracy by referring to the data acquired by the acquisition unit 11. More specifically, the determination unit 12 calculates at least one of an estimated value of a transmission bit rate of one or more future encoded video frames as the estimated bit rate and an estimated value of video recognition accuracy of one or more future decoded video frames as the estimated video recognition accuracy.

[0022] A CNN (Convolution Neural Network), an RNN (Recurrent Neural Network), or a combination thereof can be used as an example of a specific configuration of the determination unit 12. A non-neural network type model such as a random forest or a support vector machine may also be used.

[0023] As described above, in the video processing system 1 according to this exemplary embodiment, the acquisition unit 11 acquires at least one of one or more past video frames, video compression parameters in encoding the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames. The determination unit 12 is configured to refer to the data acquired by the acquisition unit 11 and calculate at least one of an estimated value of the bit rate of one or more future video frames and an estimated value of the video recognition accuracy.

[0024] Therefore, according to the video processing system 1 of this exemplary embodiment, at least one of an estimated value of the bit rate of the video frame and the video recognition accuracy is calculated by referring to at least one or more video frames from the past, video compression parameters in the encoding process of the video frames, and the video recognition result of the decoded video frame obtained by decoding the encoded video frame, so that the bit rate or image recognition accuracy can be suitably estimated.

[0025] (Flow of video processing method S1) The flow of the image processing method S1 according to this exemplary embodiment will be described with reference to Fig. 2. Fig. 2 is a flow diagram showing the flow of the image processing method S1.

[0026] (Step S11) In step S11, the acquisition unit 11 acquires at least one of one or more past video frames, video compression parameters in the encoding process of the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames.

[0027] (Step S12) In step S12, the determination unit 12 refers to the data acquired in step S11 and calculates at least one of an estimate of the bit rate of one or more future video frames and an estimate of the video recognition accuracy.

[0028] As described above, in the video processing method S1 according to this exemplary embodiment, in step S11, the acquisition unit 11 acquires at least one or more past video frames, video compression parameters in the encoding process of the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames. Then, in step S12, the determination unit 12 calculates at least one of an estimated value of a bit rate of one or more future video frames and an estimated value of video recognition accuracy by referring to the data acquired in step S11. Therefore, the video processing method S1 according to this exemplary embodiment has the same effect as the video processing system 1.

[0029] Exemplary embodiment 2 A second exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first exemplary embodiment are given the same reference numerals, and the description thereof will be omitted as appropriate.

[0030] (Configuration of video processing system 1A) 3 is a block diagram showing a configuration of a video processing system 1A according to a second exemplary embodiment of the present invention. The video processing system 1A includes a first video processing device 10A and a second video processing device 20A.

[0031] The first video processing device 10A and the second video processing device 20A are connected to each other so as to be able to communicate with each other.

[0032] In the video processing system 1A, the first video processing device 10A acquires data related to video frames and calculates feature amounts from the data. Also, in the video processing system 1A, the second video processing device 20A acquires the feature amounts calculated by the first video processing device 10A and calculates at least one of an estimated value of a bit rate of one or more future frames of video frames and an estimated value of video recognition accuracy from the feature amounts.

[0033] (Configuration of the first image processing device 10A) 3, the first video processing device 10A includes an acquisition unit 11 and a feature amount calculation unit 121. The feature amount calculation unit 121 is configured to realize the feature amount calculation means in this exemplary embodiment. The acquisition unit 11 is as described above.

[0034] The feature amount calculation unit 121 calculates feature amounts from the data acquired by the acquisition unit 11. The feature amounts calculated by the feature amount calculation unit 121 are referenced to calculate at least one of an estimated value of a bit rate of one or more future video frames and an estimated value of video recognition accuracy. The feature amount calculation unit 121 outputs the calculated feature amount to the second video processing device 20A.

[0035] (Configuration of the second image processing device 20A) 3, the second video processing device 20A includes a feature acquisition unit 21 and an estimated value calculation unit 122. The feature acquisition unit 21 and the estimated value calculation unit 122 are configured to realize a feature acquisition means and an estimated value calculation means, respectively, in this exemplary embodiment.

[0036] The feature amount acquiring unit 21 acquires the feature amount output from the first video processing device 10A.

[0037] The estimate calculation unit 122 calculates, from the features acquired by the feature acquisition unit 21, at least one of an estimate of the bit rate of one or more future video frames and an estimate of the video recognition accuracy.

[0038] Here, it can be said that the feature amount calculation unit 121 and the estimate value calculation unit 122 configure the above-mentioned determination unit 12. In other words, the determination unit 12 includes the feature amount calculation unit 121 and the estimate value calculation unit 122.

[0039] As described above, in the video processing system 1A according to the present exemplary embodiment, the first video processing device 10A acquires at least one or more past video frames, video compression parameters in the encoding process of the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames, and calculates feature quantities from the acquired data. The second video processing device 20A acquires the feature quantities calculated by the first video processing device 10A, and calculates at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy from the feature quantities. Therefore, the video processing system 1A according to the present exemplary embodiment has the same effects as the video processing system 1 according to the exemplary embodiment 1.

[0040] Exemplary embodiment 3 A third exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first and second exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

[0041] (Configuration of video processing system 1B) 4 is a block diagram showing a configuration of a video processing system 1B according to a third exemplary embodiment of the present invention. The video processing system 1B includes a first video processing device 10B and a second video processing device 20B.

[0042] The first video processing device 10B and the second video processing device 20B are connected via a network N so as to be able to communicate with each other.

[0043] The specific configuration of network N is not limited to this embodiment, but as an example, a wireless LAN (Local Area Network), a wired LAN, a WAN (Wide Area Network), a public line network, a mobile data communication network, or a combination of these networks can be used.

[0044] In the video processing system 1B, a first video processing device 10B acquires video data, and outputs video frames obtained by encoding the video data and feature amounts of the video frames.

[0045] Moreover, in the video processing system 1B, the second video processing device 20B executes the following process. Acquires the feature amount output from the first video processing device 10B, and calculates at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy from the feature amount. Acquires a video frame output from the first video processing device 10B, decodes the video frame, and outputs the decoded video frame. Obtain network statistics and calculate video compression parameters for one or more future video frames Here, the network statistical data is data indicating the usage environment of the network N, and an example thereof includes, but is not limited to, the available bandwidth in the network N.

[0046] (Configuration of the first image processing device 10B) As shown in FIG. 4, the first video processing device 10B includes an acquisition unit 11, a feature amount calculation unit 121, an encoding unit 13, and a communication unit .

[0047] The acquisition unit 11 acquires video data and supplies the acquired video data to the feature amount calculation unit 121 and the encoding unit 13.

[0048] The feature amount calculation unit 121 calculates a feature amount by referring to at least one of the video data supplied from the acquisition unit 11, video compression parameters in the encoding process of the video frame supplied from the encoding unit 13 described later, and a video recognition result of the decoded video frame output from the second video processing device 20B described later. The feature amount calculation unit 121 supplies the calculated feature amount to the communication unit 14. The process of calculating the feature amount by the feature amount calculation unit 121 will be described in detail later.

[0049] The encoding unit 13 encodes the video data supplied from the acquisition unit 11. More specifically, the encoding unit 13 encodes video frames constituting the video data using video compression parameters. The encoding unit 13 supplies the encoded video frames to the communication unit 14. The encoding unit 13 also supplies the video compression parameters used in the encoding process to the feature calculation unit 121.

[0050] The communication unit 14 is a communication module that communicates with the second video processing device 20B via the network N. As an example, the communication unit 14 outputs the feature amount supplied from the feature amount calculation unit 121 and the encoded video frame supplied from the encoding unit 13 to the second video processing device 20B via the network N, and obtains the video recognition result of the decoded video frame from the second video processing device 20B.

[0051] (Configuration of the second image processing device 20B) 4, the second video processing device 20B includes a feature acquisition unit 21, a communication unit 23, an optimization unit 24, a decoding unit 25, a recognition unit 26, an output unit 27, an available bandwidth prediction unit 28, and an estimate calculation unit 122. The optimization unit 24 is configured to realize a parameter calculation means in this exemplary embodiment.

[0052] The communication unit 23 is a communication module that communicates with the first video processing device 10B via the network N. As an example, the communication unit 23 acquires features and encoded video frames output from the first video processing device 10B, and outputs a video recognition result of a decoded video frame supplied from a recognition unit 26 (to be described later) to the first video processing device 10B.

[0053] The feature amount acquiring unit 21 acquires the feature amount supplied from the communication unit 23. The feature amount acquiring unit 21 supplies the acquired feature amount to the estimated value calculation unit 122.

[0054] The estimate calculation unit 122 calculates at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy from the features acquired by the feature acquisition unit 21. Details of the process in which the estimate calculation unit 122 calculates at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy will be described later.

[0055] The optimization unit 24 calculates video compression parameters in encoding processing of one or more future video frames by referring to the bit rate estimate and the video recognition accuracy estimate calculated by the estimate calculation unit 122. Details of the process in which the optimization unit calculates video compression parameters in encoding processing of one or more future video frames will be described later.

[0056] In addition to the above data, the optimization unit 24 may further refer to the available bandwidth that can be used for communication between the first video processing device 10B and the second video processing device 20B in the network N and the compression parameters of future segments supplied by the available bandwidth prediction unit 28 described later, to calculate the video compression parameters.

[0057] Furthermore, the optimization unit 24 derives compression parameter candidate values ​​to be referenced by a later-described predicted value calculation unit 1221. The optimization unit 24 supplies the derived compression parameter candidate values ​​to the predicted value calculation unit 1221.

[0058] The optimization unit 24 may use at least any one of a plurality of preset values ​​as the compression parameter candidate values, may use compression parameter candidate values ​​derived to have random values, or may derive compression parameter candidate values ​​by referring to past compression parameters.

[0059] The available bandwidth prediction unit 28 obtains network statistical data and predicts an available bandwidth available for communication in the network N between the first video processing device 10B and the second video processing device 20B.

[0060] The decoding unit 25 obtains the encoded video frame supplied from the communication unit 23, and decodes the video frame. The decoding unit 25 supplies the decoded video frame to the recognition unit 26 and the output unit 27.

[0061] The recognition unit 26 receives the decoded video frame supplied from the decoding unit 25 as an input and acquires a video recognition result output from a recognition engine that recognizes a subject included in the video frame. The recognition unit 26 supplies the acquired video recognition result to the communication unit 23. The recognition unit 26 may also supply the decoded video frame to the output unit 27.

[0062] Moreover, the recognition engine is, for example, configured with a neural network consisting of an input layer, one or more intermediate layers, and an output layer. In the recognition engine, when a video frame is input to the input layer, features in the video frame are extracted in one or more intermediate layers, and a video recognition result is output from the output layer. Here, the features extracted in at least one of the one or more intermediate layers are intermediate features in the video recognition process of the video frame.

[0063] The recognition unit 26 may be configured to obtain the result of human video recognition of the decoded video frames supplied from the decoding unit 25.

[0064] The output unit 27 outputs the decoded video frame supplied from the decoding unit 25 or the recognition unit 26. An example of the output unit 27 outputting the decoded video frame includes, but is not limited to, a configuration in which the output unit 27 outputs the decoded video frame to a display panel to display an image.

[0065] (Example of Processing Executed by the Feature Calculation Unit 121 and the Estimated Value Calculation Unit 122) An example of processing executed by the feature amount calculation unit 121 and the estimated value calculation unit 122 will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of processing executed by the feature amount calculation unit 121 and the estimated value calculation unit 122 according to the third exemplary embodiment of the present invention.

[0066] In the diagram shown in Fig. 5, a schematic time axis is set vertically, and each pre-compression video is shown on the time axis. Also, as an example, the current time is indicated by time t1, and video frames PF1, PF2, ..., PFn of the pre-compression video are past segments (one or more video frames in the past) that are a collection of past video frames, and video frames FF1, ..., FFn are future segments (one or more video frames in the future) that are a collection of future video frames.

[0067] In the diagram shown in FIG. 5, in order to explain the processing executed by the feature calculation unit 121 and the estimate value calculation unit 122 in FIG. 4, blocks such as the communication unit 14 and the communication unit 23 that transmit and receive data between the feature calculation unit 121 and the estimate value calculation unit 122 are omitted.

[0068] As shown in FIG. 5, the feature calculation unit 121 includes a feature extraction unit 1211, a first aggregation unit 1212, and a second aggregation unit 1213.

[0069] The feature amount extraction unit 1211 obtains the video frames PF1 to PFn that constitute the past segment, and calculates the feature amounts in the video frames PF1 to PFn. The feature amount extraction unit 1211 supplies the calculated feature amounts to the first aggregation unit 1212.

[0070] The first aggregation unit 1212 acquires the feature amount and the compression parameter CP of the past segment supplied from the feature amount calculation unit 121, and aggregates the acquired feature amount and compression parameter CP for each frame. As an example, the first aggregation unit 1212 repeatedly processes the feature amount supplied from the feature amount calculation unit 121 and the compression parameter CP of the past segment for each frame in a convolution layer and a ReLU (Rectified Linear Unit) layer to calculate aggregated data. The first aggregation unit 1212 supplies the aggregated data to the second aggregation unit 1213.

[0071] The second aggregation unit 1213 acquires the data supplied from the first aggregation unit 1212 and aggregates the data on the time axis. As an example, the second aggregation unit 1213 repeatedly processes the aggregated data of each of the video frames PF1 to PFn in the convolution layer and the ReLU layer on the time axis, and calculates a feature amount SV aggregated on the time axis. The second aggregation unit 1213 outputs the calculated feature amount SV.

[0072] A CNN (Convolution Neural Network), an RNN (Recurrent Neural Network), or a combination thereof may be used as a specific example of the configuration of the first aggregation unit 1212 and the second aggregation unit 1213. A non-neural network type model such as a random forest or a support vector machine may also be used.

[0073] 5, the estimated value calculation unit 122 includes a predicted value calculation unit 1221. The predicted value calculation unit 1221 is configured to realize a candidate value acquisition means in this exemplary embodiment.

[0074] The predicted value calculation unit 1221 acquires the feature value SV output from the second aggregation unit 1213 and the compression parameter candidate values ​​FCPC, which are candidate values ​​of the video compression parameters for one or more frames in the future, and calculates at least one of a predicted value (estimated value) of the bit rate of the future segment and a predicted value (estimated value) of the video recognition accuracy of the future segment. As an example, the predicted value calculation unit 1221 performs a process of repeatedly calculating predicted values ​​in the FC (Fully Connected) layer or the ReLU layer, and calculates at least one of a predicted value of the bit rate of the future segment and a predicted value of the video recognition accuracy of the future segment for each of the compression parameter candidate values ​​FCPC of the future segment supplied from the optimization unit 24.

[0075] (Example of processing executed by the optimization unit 24) An example of the process performed by the optimization unit 24 will be described with reference to Fig. 6. Fig. 6 is a graph illustrating a method for calculating video compression parameters by the optimization unit 24 according to a third exemplary embodiment of the present invention.

[0076] First, the optimization unit 24 stores in advance at least one of the estimated value of the bit rate of the future segment and the estimated value of the video recognition accuracy of the future segment, which are calculated for the compression parameter candidate value FCPC of the future segment by the estimate calculation unit 122. The upper diagram of Fig. 6 is a graph showing the relationship between the estimated value of the bit rate of the future segment and the estimated value of the video recognition accuracy of the future segment stored by the optimization unit 24.

[0077] Next, the optimization unit 24 refers to the estimated bit rate of the future segment and the estimated video recognition accuracy of the future segment, and calculates video compression parameters for encoding one or more future video frames. As an example, the optimization unit 24 uses a gradient method or the like to calculate video compression parameters for encoding one or more future video frames. For example, the optimization unit 24 uses the following formula (1) to calculate video compression parameters for encoding one or more future video frames. Formula (1) is a predetermined loss function that takes the estimated bit rate and the estimated video recognition accuracy as arguments.

[0078]

number

[0079] Next, the optimization unit 24 optimizes the plurality of video compression parameters λ acquired by the prediction value calculation unit 1221. C For each of the candidate values ​​of C and the estimated image recognition error D C (in other words, the inverse of the estimated value of the video recognition accuracy) so that the video compression parameter λ becomes smaller. * Calculate λ calculated by formula (1). * corresponds to OB in the lower graph of FIG.

[0080] More specifically, in a certain video segment, the predicted value calculation unit 1221 calculates the compression parameter candidate value λ C1 ,λ C2 ,λ C3 , ··· Next, the predicted value calculation unit 1221 obtains the obtained compression parameter candidate value λ C1 ,λ C2 ,λ C3 , ···For each of the estimates of the bit rate R C , and the estimated image recognition error D C Then, the optimization unit 24 calculates each estimated value R of the bit rate calculated by the prediction value calculation unit 1221. C , and the estimated value of each image recognition error D C Using the gradient method, the video compression parameter λ * Calculate.

[0081] In this way, the prediction value calculation unit 1221 and the optimization unit 24 calculate the video compression parameter λ for a plurality of candidate values ​​of the video compression parameters. * Calculate the smallest video compression parameter λ * Calculate.

[0082] (Flow of image processing method S1B) The flow of the image processing method S1B according to this exemplary embodiment will be described with reference to Fig. 7. Fig. 7 is a flow diagram showing the flow of the image processing method S1B.

[0083] (Step S101) In step S101, the acquisition unit 11 of the first video processing device 10B acquires video data of a target segment.

[0084] (Step S102) In step S102, the encoding unit 13 encodes the video data acquired by the acquisition unit 11 using the video compression parameters to generate encoded data.

[0085] (Step S103) In step S103, the feature amount calculation unit 121 refers to at least one of the video data acquired by the acquisition unit 11, the video compression parameters used in the encoding process by the encoding unit 13, and the video recognition result of the decoded video frame output from the second video processing device 20B, and calculates the feature amount of the video data. The specific process by the feature amount calculation unit 121 has been described above, so a description thereof will be omitted here.

[0086] (Step S104) In step S104, the communication unit 14 transmits the features calculated by the feature calculation unit 121 and the encoded data used when the encoding unit 13 performed the encoding process to the second video processing device 20B via the network N.

[0087] (Step S201) In step S201, the communication unit 23 of the second video processing device 20B receives the feature amount and the encoded data transmitted from the first video processing device 10B.

[0088] (Step S202) In step S202, the decoding unit 25 decodes the encoded data received by the communication unit 23 to restore the video data.

[0089] (Step S203) In step S203, the recognition unit 26 executes a video recognition process on the video data decoded by the decoding unit 25.

[0090] (Step S204) In step S204, the output unit 27 outputs the video data decoded by the decoding unit 25. Furthermore, the recognition unit 26 supplies the video recognition result indicating the result of the video recognition process executed in step S203 to the communication unit 23. The communication unit 23 transmits the supplied video recognition result to the first video processing device 10B via the network N. Upon receiving the video recognition result, the communication unit 14 of the first video processing device 10B supplies the video recognition result to the feature amount calculation unit 121. The feature amount calculation unit 121 refers to the supplied video recognition result and calculates the feature amount.

[0091] (Step S205) In step S205, feature acquisition unit 21 acquires the feature received by communication unit 23. Next, estimate calculation unit 122 calculates an estimate of the bit rate of one or more future frames of video and an estimate of the video recognition accuracy from the feature acquired by feature acquisition unit 21. Also, in step S205, estimate calculation unit 122 may acquire video compression parameter candidate values ​​FCPC for one or more future frames, and further refer to the video compression parameter candidate values ​​FCPC to calculate at least one of the estimate of the bit rate of one or more future frames of video and an estimate of the video recognition accuracy. The specific processing by estimate calculation unit 122 has been described above, and therefore will not be described here.

[0092] (Step S206) In step S206, the optimization unit 24 calculates video compression parameters for the encoding process of the future segment by referring to the bit rate estimate and the video recognition accuracy estimate calculated by the estimate calculation unit 122. The specific process by the optimization unit 24 has been described above, and therefore will not be described here.

[0093] (Step S207) In step S207, the communication unit 23 transmits the video compression parameters calculated by the optimization unit 24 to the first video processing device 10B via the network N. Upon receiving the video compression parameters, the communication unit 14 of the first video processing device 10B supplies the video compression parameters to the encoding unit 13. The encoding unit 13 uses the supplied video compression parameters in the encoding process of the future segment.

[0094] As described above, in the video processing system 1B according to the present exemplary embodiment, the optimization unit 24 in the second video processing device 20B calculates video compression parameters in the encoding process of one or more future video frames by referring to the estimated bit rate and the estimated video recognition accuracy in step S206. Therefore, according to the video processing system 1B according to the present exemplary embodiment, it is possible to suitably calculate suitable video compression parameters in the encoding process of one or more future video frames.

[0095] Furthermore, in the video processing system 1B according to this exemplary embodiment, the estimate calculation unit 122 in the second video processing device 20B acquires the video compression parameter candidate values ​​FCPC for one or more future frames in step S205, and further refers to the video compression parameter candidate values ​​FCPC to calculate the bit rate estimate value and the video recognition accuracy estimate value. Therefore, according to the video processing system 1B according to this exemplary embodiment, it is possible to preferably calculate the suitable bit rate estimate value and the suitable video recognition accuracy estimate value.

[0096] Exemplary embodiment 4 A fourth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to third exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

[0097] (Configuration of video processing system 1C) 8 is a block diagram showing a configuration of a video processing system 1C according to a fourth exemplary embodiment of the present invention. The video processing system 1C includes a first video processing device 10C and a second video processing device 20C instead of the first video processing device 10B and the second video processing device 20B in the above-mentioned video processing system 1B.

[0098] In the video processing system 1C, the first video processing device 10B and the second video processing device 20B each acquire teacher data, and cause the feature amount calculation unit 121 and the estimated value calculation unit 122 to learn.

[0099] (Configuration of the first video processing device 10C) 8, the first video processing device 10C includes a learning unit 19 in addition to the components included in the first video processing device 10B described above. The acquisition unit 11 and the learning unit 19 are components that respectively realize an acquisition means and a learning means in this exemplary embodiment.

[0100] The acquisition unit 11 acquires teacher data including a set of at least one of one or more video frames from the past, video compression parameters in the encoding process of the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames, and at least one of an actual bit rate of one or more video frames at a point in time later than the past, and an actual video recognition accuracy.

[0101] The learning unit 19 uses the teacher data to train the feature amount calculation unit 121. Details of the processing by the learning unit 19 will be described later.

[0102] (Configuration of the second image processing device 20C) 8, the second image processing device 20C includes a learning unit 29 in addition to the components included in the second image processing device 20B described above. The learning unit 29 is a component that realizes an acquisition unit and a learning unit in this exemplary embodiment.

[0103] The learning unit 29 acquires the teacher data from the acquisition unit 11 of the first video processing device 10C. More specifically, the learning unit 29 acquires teacher data including a set of at least one of one or more past video frames, video compression parameters in the encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and at least one of an actual bit rate of one or more past video frames and an actual video recognition accuracy at a time later than the past.

[0104] Then, the learning unit 29 uses the acquired teacher data to train the estimated value calculation unit 122. Details of the processing by the learning unit 29 will be described later.

[0105] (Example of learning process by learning unit 19 and learning unit 29) A processing example in which the learning unit 29 trains the estimated value calculation unit 122 and a processing example in which the learning unit 19 trains the feature calculation unit 121 will be described with reference to Fig. 9. Fig. 9 is a diagram illustrating a schematic diagram of a processing example in which the learning unit 29 trains the estimated value calculation unit 122 and a processing example in which the learning unit 19 trains the feature calculation unit 121 according to a fourth exemplary embodiment of the present invention.

[0106] 9, a schematic time axis is set vertically, and each pre-compressed video is shown on the time axis. As an example, a time point after video frames PF1 to PFn and before video frames FF1 to FFn is shown as time t1, and a time point after video frames PF1 to PFn is shown as time t2.

[0107] First, similarly to the process described with reference to FIG. 5, the estimate calculation unit 122 refers to at least one of past video frames PF1-PFn, video compression parameters in the encoding process of the video frames PF1-PFn, and video recognition results of decoded video frames obtained by decoding the encoded video frames at time t1 to calculate an estimate of video recognition accuracy and an estimate of bit rate for video frames FF1-FFn, which are future segments. Then, in the exemplary embodiment, the estimate calculation unit 122 stores the calculated estimate of video recognition accuracy and the estimated bit rate in a storage unit (not shown). Here, the calculation process of each of the above values ​​by the estimate calculation unit 122 may be based on an instruction from the learning unit 29, for example. Moreover, each of the above data referred to by the estimate calculation unit 122 constitutes a part of the above-mentioned teacher data.

[0108] Next, at time t2, the encoding unit 13 encodes the video frames FF1-FFn. The video frames FF1-FFn were future video frames at time t1, but are past video frames at time t2. The encoding unit 13 outputs the actual bit rate of the encoded video frames FF1-FFn and the encoded data obtained by encoding the video frames FF1-FFn to the second video processing device 20C.

[0109] The decoding unit 25 of the second video processing device 20C acquires the coded data output from the coding unit 13, and decodes the coded data to restore the video data. The decoding unit 25 also stores the actual bit rate acquired from the coding unit 13 in a storage unit (not shown). The recognition unit 26 then performs video recognition processing on the video data decoded by the decoding unit 25, and acquires the actual video recognition accuracy in the video recognition. The recognition unit 26 stores the acquired actual video recognition accuracy in a storage unit (not shown). The above-mentioned actual bit rate and actual video recognition accuracy constitute part of the above-mentioned teacher data.

[0110] The learning unit 29 is The estimated values ​​of the video recognition accuracy and the estimated values ​​of the bit rate for the above-mentioned video frames FF1 to FFn, The actual image recognition accuracy and the actual bit rate for the above-mentioned image frames FF1 to FFn are The parameters in the estimate calculation unit 122 are updated so that the difference between the estimated values ​​and the bit rate is reduced, thereby causing the estimate calculation unit 122 to learn. Here, the estimated values ​​of the video recognition accuracy and the estimated values ​​of the bit rate for the video frames FF1 to FFn are calculated by the estimate calculation unit 122 through the above-mentioned process.

[0111] Similarly, the learning unit 19 The estimated values ​​of the video recognition accuracy and the estimated values ​​of the bit rate for the above-mentioned video frames FF1 to FFn, The actual image recognition accuracy and the actual bit rate for the above-mentioned image frames FF1 to FFn are The parameters in the feature calculation unit 121 are updated so that the difference between the features is reduced, thereby causing the feature calculation unit 121 to learn. Here, the estimated values ​​of the video recognition accuracy and the estimated values ​​of the bit rate for the video frames FF1 to FFn are calculated by the estimate calculation unit 122 through the above-mentioned process.

[0112] In this way, the learning unit 19 and the learning unit 29 At least one of past video frames PF1 to PFn, video compression parameters in the encoding process of the video frames PF1 to PFn, and a video recognition result of a decoded video frame obtained by decoding the encoded video frames; The actual video recognition accuracy and the actual bit rate of the video frames FF1 to FFn at the later time point in the past, Using training data including The estimated values ​​of the video recognition accuracy and the estimated values ​​of the bit rate for the above-mentioned video frames FF1 to FFn, The actual image recognition accuracy and the actual bit rate for the above-mentioned image frames FF1 to FFn are The feature amount calculation unit 121 and the estimated value calculation unit 122 are trained so that the difference between the 。

[0113] Therefore, according to the video processing system 1C of this exemplary embodiment, the feature calculation unit 121 and the estimate calculation unit 122 are trained, so that an estimate of the bit rate of one or more future video frames and an estimate of the video recognition accuracy are preferably calculated.

[0114] Exemplary embodiment 5 The fifth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to fourth exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

[0115] (Configuration of video processing system 1D) 10 is a block diagram showing a configuration of an image processing system 1D according to a fifth exemplary embodiment of the present invention. The image processing system 1D includes a plurality of first image processing devices 10B in the image processing system 1B, and further includes a second image processing device 20D instead of the second image processing device 20B.

[0116] In the video processing system 1D, for each of a plurality of videos to be transmitted, the first video processing device 10B-1 acquires video data 1, the first video processing device 10B-2 acquires video data 2, and the first video processing device 10B-3 acquires video data 3. Also, in the video processing system 1D, the second video processing device 20D calculates video compression parameters for encoding one or more future video frames for each of a plurality of videos to be transmitted.

[0117] In the example shown in Fig. 10, three first image processing devices 10B-1, 10B-2, and 10B-3 are illustrated, but this does not limit the present exemplary embodiment. The image processing system 1D can be configured to include any number of first image processing devices. In the present exemplary embodiment, the multiple first image processing devices have similar main components.

[0118] (Configuration of the second image processing device 20D) As shown in FIG. 10, the second video processing device 20D includes an estimated value calculation section 122a and an optimization section 24a instead of the estimated value calculation section 122 and the optimization section 24 in the second video processing device 20B.

[0119] The estimated value calculation unit 122a refers to each feature amount transmitted from the first video processing devices 10B-1 to 10B-3 for each of the multiple transmission target videos, and calculates each estimated value of the bit rate of one or more frames of video frames in the future and each estimated value of the video recognition accuracy. Here, each feature amount transmitted from the first video processing devices 10B-1 to 10B-3 is data calculated by the feature amount calculation unit 121 with reference to at least one of one or more frames of video frames in the past acquired by each acquisition unit 11, video compression parameters in the encoding process of the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames, as described above.

[0120] The optimization unit 24a calculates video compression parameters for the encoding process of one or more future video frames for each of the multiple videos to be transmitted, by referring to the estimated bit rate and the estimated video recognition accuracy for each of the multiple videos to be transmitted.

[0121] (Example of processing executed by the optimization unit 24a) An example of the process executed by the optimization unit 24a will be described with reference to Fig. 11. Fig. 11 is a graph showing a method for calculating video compression parameters by the optimization unit 24a according to the fifth exemplary embodiment of the present invention.

[0122] The optimization unit 24a accumulates in advance at least one of the estimated value of the bit rate of the future segment and the estimated value of the video recognition accuracy of the future segment, which are calculated by the estimate calculation unit 122 for each of the compression parameter candidate values ​​FCPC of the future segment, for each of the multiple videos to be transmitted. The upper diagram in Fig. 11 is a graph showing the relationship between the estimated value of the bit rate of the future segment and the estimated value of the video recognition accuracy of the future segment accumulated by the optimization unit 24a for each of the video data 1, video data 2, and video data 3.

[0123] Next, the optimization unit 24a calculates video compression parameters for encoding one or more future video frames for each of the multiple transmission target videos, with reference to the estimated value of the bit rate of the future segment and the estimated value of the video recognition accuracy of the future segment, etc. As an example, the optimization unit 24a calculates video compression parameters for encoding one or more future video frames for each of the multiple transmission target videos, using the following formula (2).

[0124]

number

[0125] That is, first, the prediction value calculation unit 1221 calculates a compression parameter candidate value λ , which is a candidate value of a video compression parameter for one or more frames in the future. C Then, the prediction value calculation unit 1221 obtains a plurality of the obtained video compression parameters λ C For each of the candidate values ​​of X, for each of the multiple transmission target videos, at least one of one or more past video frames, video compression parameters in the encoding process of the video frames, and a video recognition result of the decoded video frame obtained by decoding the encoded video frame is selected. C Refer to the bit rate estimate R C , and the estimated value D of the video recognition error (in other words, the inverse of the estimated value of the video recognition accuracy) C In this way, the prediction value calculation unit 1221 calculates the bit rate estimate R C The distribution of the image recognition error estimate D C The distribution of can be obtained.

[0126] Next, the optimization unit 24 optimizes the plurality of video compression parameters λ obtained by the prediction value calculation unit 1221 for each of the plurality of videos to be transmitted. C For each of the candidate values ​​of C and the estimated image recognition error D C(in other words, the inverse of the estimated value of the video recognition accuracy) so that the video compression parameter λ becomes smaller. * Calculate the λ of each image data calculated by Equation (2). * In the graph at the bottom of FIG. 11, video data 1 corresponds to OB1, video data 2 corresponds to OB2, and video data 3 corresponds to OB3.

[0127] More specifically, in a certain video segment in each of the plurality of videos to be transmitted, the predicted value calculation unit 1221 calculates the compression parameter candidate value λ C1 ,λ C2 ,λ C3 , ··· Next, the predicted value calculation unit 1221 obtains the obtained compression parameter candidate value λ C1 ,λ C2 ,λ C3 , ··· For each of the estimates of the bit rate R C , and the estimated image recognition error D C Then, the optimization unit 24 calculates each estimated value R of the bit rate calculated by the prediction value calculation unit 1221. C , and the estimated value of each image recognition error D C The video compression parameter λ is calculated by using the gradient method or the like. * Calculate.

[0128] As described above, in the video processing system 1D according to the present exemplary embodiment, the acquisition unit 11 of the first video processing device 10B acquires at least one of one or more past video frames, video compression parameters in the encoding process of the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames, for each of the multiple transmission target videos. Next, in the video processing system 1D according to the present exemplary embodiment, the feature calculation unit 121 of the first video processing device 10B and the estimate calculation unit 122a of the second video processing device 20B calculate an estimate of the bit rate of one or more future video frames and an estimate of the video recognition accuracy, for each of the multiple transmission target videos, by referring to the data acquired by the acquisition unit 11. In the video processing system 1D according to the present exemplary embodiment, the optimization unit 24a of the second video processing device 20B refers to the estimated bit rate and the estimated video recognition accuracy for each of the plurality of videos to be transmitted, and calculates video compression parameters for encoding one or more frames of video frames in the future for each of the plurality of videos to be transmitted. Therefore, according to the video processing system 1D according to the present exemplary embodiment, it is possible to calculate suitable video compression parameters for encoding one or more frames of video frames in the future for each of the plurality of videos to be transmitted.

[0129] Exemplary Embodiment 6 A sixth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to fifth exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

[0130] (Configuration of video processing system 1E) 12 is a block diagram showing a configuration of a video processing system 1E according to a sixth exemplary embodiment of the present invention. The video processing system 1E includes a second video processing device 20E instead of the second video processing device 20B in the video processing system 1B.

[0131] In the image processing system 1E, the second image processing device 20E includes: - Code length of one or more past video frames - Decoded video frame obtained by decoding the encoded video frame - The result of video recognition processing of the decoded video frame, or intermediate features in the video recognition processing and calculates at least one of an estimate of the bit rate of one or more future video frames and an estimate of the video recognition accuracy.

[0132] (Configuration of the second video processing device 20E) 12, in the second video processing device 20E, the estimate calculation unit 122 is configured to obtain data from the decoding unit 25 and the recognition unit 26. The estimate calculation unit 122 is configured to realize the obtaining means in this exemplary embodiment.

[0133] As an example, the estimate calculation unit 122 acquires at least one of the code length of one or more past video frames and the decoded video frames from the decoding unit 25, and acquires the video recognition processing result of the decoded video frames or intermediate features in the video recognition processing from the recognition unit 26. Then, in addition to the features acquired by the feature acquisition unit 21, the estimate calculation unit 122 further refers to the decoded video frames and at least one of the video recognition processing result of the decoded video frames or intermediate features in the video recognition processing, and calculates at least one of an estimate of the bit rate of one or more future video frames and an estimate of the video recognition accuracy.

[0134] Here, the intermediate feature in the video recognition process is a feature obtained in an intermediate layer of a recognition engine that executes the video recognition process. The feature is a numerical representation of a feature extracted from a video frame. An example of the feature is a feature vector in which the extracted features are arranged in a vector, but is not limited to this.

[0135] Moreover, the recognition engine is, for example, configured with a neural network consisting of an input layer, one or more intermediate layers, and an output layer. In the recognition engine, when a video frame is input to the input layer, features in the video frame are extracted in one or more intermediate layers, and a video recognition result is output from the output layer. Here, the features extracted in at least one of the one or more intermediate layers are intermediate features in the video recognition process of the video frame.

[0136] As described above, in the video processing system 1E according to this exemplary embodiment, the estimate calculation unit 122 of the second video processing device 20E further refers to at least one of the code length of one or more past video frames, the decoded video frame obtained by decoding the encoded video frame, and the video recognition processing result of the decoded video frame or the intermediate feature value in the video recognition processing, and calculates at least one of the bit rate estimate and the video recognition accuracy estimate of one or more future video frames. Therefore, the video processing system 1E according to this exemplary embodiment can suitably calculate a more suitable bit rate estimate and a more suitable video recognition accuracy estimate.

[0137] Exemplary Embodiment 7 The seventh exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to sixth exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

[0138] (Configuration of the video processing system 1F) 13 is a block diagram showing a configuration of a video processing system 1F according to a seventh exemplary embodiment of the present invention. The video processing system 1F includes a first video processing device 10F and a second video processing device 20F instead of the first video processing device 10B and the second video processing device 20B in the video processing system 1B.

[0139] In the video processing system 1F, the feature amount of the video frame is calculated in the second video processing device 20F.

[0140] (First image processing device 10F) 13, the first video processing device 10F does not include the feature amount calculation unit 121 in the first video processing device 10B. That is, the first video processing device 10F transmits, via the communication unit 14, to the second video processing device 20F, the encoded data encoded by the encoding unit 13, the video compression parameters used in the encoding process by the encoding unit 13, and the data acquired by the acquisition unit 11.

[0141] (Second image processing device 20F) 13, the second video processing device 20F includes a feature amount calculation unit 121 instead of the feature amount acquisition unit 21 in the second video processing device 20B. In the second video processing device 20F, the feature amount calculation unit 121 is configured to acquire data from a decoding unit 25 and a recognition unit 26.

[0142] The feature calculation unit 121 acquires the decoded video frames supplied from the decoding unit 25, the video recognition results of the decoded video frames supplied from the recognition unit 26, the video compression parameters transmitted from the first video processing device 10F, and the data acquired by the acquisition unit 11. The feature calculation unit 121 then calculates feature amounts by referring to the acquired decoded video frames, the video recognition results, the video compression parameters, and the data acquired by the acquisition unit 11. The feature calculation unit 121 supplies the calculated feature amounts to the estimate calculation unit 122. The estimate calculation unit 122 calculates at least one of an estimate of the bit rate of one or more future video frames and an estimate of the video recognition accuracy from the feature amounts acquired by the feature acquisition unit 21.

[0143] As described above, in the video processing system 1F according to this exemplary embodiment, the second video processing device 20F includes the feature amount calculation unit 121. Therefore, according to the video processing system 1F according to this exemplary embodiment, the calculation processes such as the calculation of the feature amount, the calculation of at least one of the estimated value of the bit rate of one or more future frames of video frames and the estimated value of the video recognition accuracy, and the calculation of the video compression parameters in the encoding process of one or more future frames of video frames can be performed only by the second video processing device 20F.

[0144] Exemplary embodiment 8 An eighth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to seventh exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

[0145] (Configuration of video processing system 1G) 14 is a block diagram showing a configuration of a video processing system 1G according to an eighth exemplary embodiment of the present invention. The video processing system 1G includes a second video processing device 20G instead of the second video processing device 20B in the video processing system 1B, and further includes a third video processing device 30G.

[0146] In the video processing system 1G, a second video processing device 20G calculates at least one of an estimated value of a bit rate of one or more future frames of video frames and an estimated value of a video recognition accuracy, and calculates a video compression parameter in encoding processing of one or more future frames of video frames. A third video processing device 30G decodes the encoded data, performs video recognition processing of the decoded video frames, and outputs the decoded video frames.

[0147] (Configuration of the second image processing device 20G) As shown in Fig. 14, the second video processing device 20G does not include the decoding unit 25, the recognition unit 26, and the output unit 27 in the second video processing device 20B. That is, in the second video processing device 20G, the estimate value calculation unit 122 calculates at least one of an estimate of a bit rate of one or more future frames of video frames and an estimate of video recognition accuracy, and the optimization unit 24 calculates video compression parameters for encoding processing of one or more future frames of video frames. Then, the second video processing device 20G transmits the calculated video compression parameters to the first video processing device 10B.

[0148] (Configuration of the third image processing device 30G) 14, the third video processing device 30G includes a decoding unit 25, a recognition unit 26, and an output unit 27. That is, in the third video processing device 30G, the decoding unit 25 decodes the encoded data, the recognition unit 26 performs video recognition processing on the decoded video frames, and the output unit 27 outputs the video frames.

[0149] As described above, in the image processing system 1G according to the present exemplary embodiment, data is transmitted and received between the first image processing device 10B and the second image processing device 20G, but the third image processing device 30G only receives data. Therefore, by shortening the physical distance between the first image processing device 10B and the second image processing device 20G or by reducing the number of relay devices between the first image processing device 10B and the second image processing device 20G, the RTT (Round Trip Time) in the image processing system 1F can be shortened.

[0150] Exemplary Embodiment 9 A ninth exemplary embodiment of the present invention will be described in detail with reference to the drawings. Note that components having the same functions as those described in the first to eighth exemplary embodiments are denoted by the same reference numerals, and the description thereof will not be repeated.

[0151] (Configuration of video processing system 1H) 15 is a block diagram showing a configuration of a video processing system 1H according to a ninth exemplary embodiment of the present invention. The video processing system 1H includes a second video processing device 20H instead of the second video processing device 20B in the video processing system 1B.

[0152] In the video processing system 1H, the second video processing device 20H predicts the occurrence rate of packet loss, and calculates video compression parameters for encoding processing of one or more future video frames by referring to the predicted occurrence rate of packet loss.

[0153] (Configuration of the second image processing device 20H) As shown in FIG. 15, the second video processing device 20H has a configuration in which a packet loss prediction unit 31 and a packet loss detection unit 32 are added to the second video processing device 20B.

[0154] The packet loss prediction unit 31 predicts a packet loss rate, which is the frequency at which packet loss occurs during data transmission and reception. The packet loss prediction unit 31 supplies the predicted packet loss rate to the estimate value calculation unit 122.

[0155] The packet loss detection unit 32 detects packet loss that occurred in past data transmission and reception. The packet loss detection unit 32 supplies the packet loss detection result to the estimate value calculation unit 122.

[0156] The estimate calculation unit 122 further refers to the packet loss rate supplied by the packet loss prediction unit 31 and the detection result supplied by the packet loss detection unit 32 in addition to the features acquired by the feature acquisition unit 21 described above, and calculates at least one of an estimate of the bit rate of one or more future video frames and an estimate of the video recognition accuracy.

[0157] As an example, when the detection result supplied by the packet loss detection unit 32 indicates that a packet loss has occurred, the estimate calculation unit 122 refers to a parameter indicating how to set a GOP (Group Of Picture) in the received encoded data, and sets the estimate of the video recognition accuracy of the GOP including the frame in which the packet loss has occurred to be lower than the estimate of the video recognition accuracy of other GOPs.

[0158] As another example, the estimate calculation unit 122 sets the estimate value of the video recognition accuracy of a GOP including a frame whose packet loss rate supplied by the packet loss prediction unit 31 is higher than a predetermined value to be lower than the estimate values ​​of the video recognition accuracy of other GOPs.

[0159] As described above, in the video processing system 1H according to this exemplary embodiment, the second video processing device 20H includes a packet loss prediction unit 31 that predicts a packet loss rate, which is the frequency at which packet loss occurs in data transmission and reception, and a packet loss detection unit 32 that detects packet loss that occurred in past data transmission and reception. Therefore, according to the video processing system 1H according to this exemplary embodiment, an estimate of video recognition accuracy is calculated by referring to whether or not a packet loss has occurred or whether or not the predicted packet loss rate is lower than a predetermined value, so that a suitable estimate of video recognition accuracy can be suitably calculated.

[0160] (Additional Notes) The technology in each embodiment is widely applicable to one or more AGVs (Automatic Guided Vehicles), construction machines, autonomous vehicles, monitoring systems, and the like.

[0161] For example, the technology in each embodiment can be used in a system that receives video from multiple cameras installed at multiple work sites, recognizes the video captured by the multiple cameras, and detects and notifies danger, to calculate an appropriate bit rate to be assigned to each of the multiple cameras.

[0162] [Software implementation example] Some or all of the functions of the image processing system 1, the first image processing devices 10A, 10B, 10C, 10F, the second image processing devices 20A, 20B, 20C, 20D, 20E, 20F, 20G, 20H, and the third image processing device 30G may be realized by hardware such as an integrated circuit (IC chip), or by software.

[0163] In the latter case, the image processing system 1, the first image processing device 10A, 10B, 10C, 10F, the second image processing device 20A, 20B, 20C, 20D, 20E, 20F, 20G, 20H, and the third image processing device 30G are realized by, for example, a computer that executes instructions of a program, which is software that realizes each function. An example of such a computer (hereinafter, referred to as computer C) is shown in FIG. 16. The computer C includes at least one processor C1 and at least one memory C2. The memory C2 stores a program P for operating the computer C as the image processing system 1, the first image processing device 10A, 10B, 10C, 10F, the second image processing device 20A, 20B, 20C, 20D, 20E, 20F, 20G, 20H, and the third image processing device 30G. In computer C, processor C1 reads and executes program P from memory C2 to realize the functions of video processing system 1, first video processing devices 10A, 10B, 10C, 10F, second video processing devices 20A, 20B, 20C, 20D, 20E, 20F, 20G, 20H, and third video processing device 30G.

[0164] The processor C1 may be, for example, a central processing unit (CPU), a graphic processing unit (GPU), a digital signal processor (DSP), a micro processing unit (MPU), a floating point number processing unit (FPU), a physics processing unit (PPU), a microcontroller, or a combination of these. The memory C2 may be, for example, a flash memory, a hard disk drive (HDD), a solid state drive (SSD), or a combination of these.

[0165] The computer C may further include a RAM (Random Access Memory) for expanding the program P during execution and for temporarily storing various data. The computer C may further include a communication interface for transmitting and receiving data to and from other devices. The computer C may further include an input / output interface for connecting input / output devices such as a keyboard, a mouse, a display, and a printer.

[0166] Furthermore, the program P can be recorded on a non-transitory tangible recording medium M that can be read by the computer C. Such a recording medium M can be, for example, a tape, a disk, a card, a semiconductor memory, or a programmable logic circuit. The computer C can obtain the program P via such a recording medium M. Furthermore, the program P can be transmitted via a transmission medium. Such a transmission medium can be, for example, a communication network or broadcast waves. The computer C can also obtain the program P via such a transmission medium.

[0167] [Additional Note 1] The present invention is not limited to the above-described embodiment, and various modifications are possible within the scope of the claims. For example, embodiments obtained by appropriately combining the technical means disclosed in the above-described embodiment are also included in the technical scope of the present invention.

[0168] [Additional Note 2] Some or all of the above-described embodiments can be described as follows. However, the present invention is not limited to the following described aspects.

[0169] (Appendix 1) A video processing system comprising: an acquisition means for acquiring at least one of one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames; and an estimation means for calculating at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy by referring to the data acquired by the acquisition means.

[0170] (Appendix 2) The video processing system of claim 1, wherein the estimation means calculates at least one of an estimate of a transmission bit rate of encoded video frames for one or more frames in the future as the estimated bit rate, and an estimate of video recognition accuracy of decoded video frames for one or more frames in the future as the estimated video recognition accuracy.

[0171] (Appendix 3) The acquisition means further acquires at least any of a code length of one or more past video frames, a decoded video frame obtained by decoding the encoded video frame, and an intermediate feature in video recognition processing of the decoded video frame, and the estimation means calculates at least any of an estimate of the bit rate and an estimate of the video recognition accuracy by referring to the data acquired by the acquisition means.

[0172] (Appendix 4) The video processing system according to any one of appendixes 1 to 3, wherein the estimation means includes a feature calculation means for calculating a feature from the data acquired by the acquisition means, and an estimate calculation means for calculating at least one of an estimate of the bit rate and an estimate of the video recognition accuracy from the feature calculated by the feature calculation means.

[0173] (Appendix 5) The video processing system according to any one of appendixes 1 to 4, further comprising a parameter calculation means for calculating video compression parameters for encoding processing of one or more future video frames by referring to the bit rate estimate calculated by the estimation means and the video recognition accuracy estimate.

[0174] (Appendix 6) The video processing system of claim 5, further comprising a candidate value acquisition means for acquiring candidate values ​​of video compression parameters for one or more frames in the future, wherein the estimation means further refers to the candidate values ​​of video compression parameters for the one or more frames in the future acquired by the acquisition means to calculate an estimate of the bit rate and an estimate of the video recognition accuracy.

[0175] (Appendix 7) The video processing system of claim 6, wherein the candidate value acquisition means acquires a plurality of candidate values ​​for the video compression parameter, the estimation means calculates an estimate of the bit rate and an estimate of the video recognition accuracy for each of the plurality of candidate values ​​of the video compression parameter acquired by the acquisition means, and the parameter calculation means calculates the video compression parameter so as to reduce a predetermined loss function having the estimate of the bit rate and the estimate of the video recognition accuracy as arguments for each of the plurality of candidate values ​​of the video compression parameter acquired by the acquisition means.

[0176] (Appendix 8) The video processing system of any one of appendixes 5 to 7, wherein the acquisition means acquires, for each of a plurality of videos to be transmitted, at least one of one or more video frames in the past, video compression parameters in the encoding process of the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames; the estimation means calculates, for each of the plurality of videos to be transmitted, an estimate of a bit rate of one or more video frames in the future and an estimate of video recognition accuracy, by referring to the data acquired by the acquisition means; and the parameter calculation means calculates, for each of the plurality of videos to be transmitted, video compression parameters in the encoding process of one or more video frames in the future, by referring to the estimate of the bit rate and the estimate of the video recognition accuracy,

[0177] (Appendix 9) The video processing system of any one of Appendices 1 to 8, wherein the acquisition means acquires teacher data including a set of at least one of one or more video frames in the past, video compression parameters in the encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and at least one of an actual bit rate of one or more video frames at a point in time later than the past and an actual video recognition accuracy, and the video processing system further comprises a learning means for training the estimation means using the teacher data.

[0178] (Appendix 10) A video processing system including a first video processing device and a second video processing device, wherein the first video processing device comprises an acquisition means for acquiring at least one of one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and a feature calculation means for calculating a feature from the data acquired by the acquisition means, and the second video processing device comprises a feature acquisition means for acquiring the feature, and an estimate calculation means for calculating at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy from the feature acquired by the feature acquisition means.

[0179] (Appendix 11) An information processing device comprising: an acquisition means for acquiring at least one of one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of the decoded video frames obtained by decoding the encoded video frames; and a feature calculation means for calculating features from the data acquired by the acquisition means, the feature calculation means calculating features referenced to calculate at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy.

[0180] (Appendix 12) An information processing device comprising: a feature acquisition means for acquiring features calculated with reference to at least one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames; and an estimate calculation means for calculating at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy from the features acquired by the feature acquisition means.

[0181] (Appendix 13) A video processing method including: acquiring at least one of one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames; and calculating at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy by referring to the data acquired in the acquiring step.

[0182] (Appendix 14) A computer-readable recording medium having recorded thereon a program causing a computer to function as an acquisition means for acquiring at least one of one or more past video frames, video compression parameters in the encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and a feature calculation means for calculating features from the data acquired by the acquisition means, the feature calculation means calculating features referenced to calculate at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy.

[0183] (Appendix 15) A computer-readable recording medium having recorded thereon a program causing a computer to function as: a feature acquisition means for acquiring features calculated by referring to at least one or more past video frames, video compression parameters in the encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames; and an estimate calculation means for calculating at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy from the features acquired by the feature acquisition means.

[0184] [Additional Note 3] A part or all of the above-described embodiments can be further expressed as follows.

[0185] A video processing system comprising at least one processor, the processor executing an acquisition process to acquire at least one of one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and an estimation process to calculate at least one of an estimated bit rate of one or more future video frames and an estimated video recognition accuracy by referring to the data acquired in the acquisition process.

[0186] The image processing system may further include a memory that stores a program for causing the processor to execute the acquisition process and the estimation process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium.

[0187] An information processing device comprising at least one processor, the processor executing a feature acquisition process that acquires features calculated with reference to at least one or more past video frames, video compression parameters in an encoding process of the video frames, and video recognition results of decoded video frames obtained by decoding the encoded video frames, and an estimate calculation process that calculates at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy from the features acquired in the feature acquisition process.

[0188] The information processing device may further include a memory that stores a program for causing the processor to execute the feature acquisition process and the estimated value calculation process. The program may be recorded in a computer-readable, non-transitory, tangible recording medium. [Explanation of symbols]

[0189] 1, 1A, 1B, 1C, 1D, 1E, 1F, 1G, 1H Video processing system 10, 10A, 10B, 10C, 10F First image processing device 11 Acquisition Department 12 Judgment section 13 Encoding section 14, 23 Communications Department 19, 29 Learning Department 20A, 20B, 20C, 20D, 20E, 20F, 20G, 20H Second image processing device 21 Feature acquisition unit 24, 24a Optimization section 25 Decoding section 26 Recognition part 27 Output section 28 Available Bandwidth Predictor 30G 3rd Generation Video Processing Device 31 Packet loss prediction unit 32 Packet loss detection unit 121 Feature Calculation Unit 122, 122a Estimated value calculation unit 1211 Feature Extraction Unit 1212 First Consolidation Section 1213 Second Consolidation Section 1221 Prediction value calculation unit

Claims

1. One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; An acquisition means for acquiring at least one of the above. an estimation means for calculating at least one of an estimate of a bit rate of one or a plurality of future video frames and an estimate of video recognition accuracy by referring to the data acquired by the acquisition means; a parameter calculation means for calculating a video compression parameter in a coding process of one or a plurality of video frames in the future by referring to the estimated value of the bit rate and the estimated value of the video recognition accuracy calculated by the estimation means; a candidate value acquisition means for acquiring candidate values ​​of video compression parameters for one or more future frames; Equipped with The estimation means further refers to the candidate values ​​of the video compression parameters for one or more future frames acquired by the acquisition means to calculate the estimated value of the bit rate and the estimated value of the video recognition accuracy. Video processing system.

2. The estimation means includes: as the bit rate estimate, an estimate of a transmission bit rate of one or more encoded video frames in the future; and As the estimated value of the video recognition accuracy, an estimated value of the video recognition accuracy of one or more decoded video frames in the future. Calculate at least one of The video processing system according to claim 1 .

3. A video processing system, comprising: One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; An acquisition means for acquiring at least one of the above. an estimation means for calculating at least one of an estimated value of a bit rate of one or a plurality of future video frames and an estimated value of video recognition accuracy by referring to the data acquired by the acquisition means; Equipped with The acquisition means includes: One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; and at least one of an actual bit rate of one or more frames of video frames at a time point later than the past, and an actual video recognition accuracy. The video processing system includes: The method further includes a learning means for learning the estimation means using the teacher data. Video processing system.

4. An information processing device, One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; An acquisition means for acquiring at least one of the above. a feature amount calculation means for calculating a feature amount from the data acquired by the acquisition means, the feature amount calculation means calculating a feature amount to be referenced for calculating at least one of an estimated value of a bit rate of one or a plurality of future video frames and an estimated value of video recognition accuracy; Equipped with The acquisition means includes: One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; and at least one of an actual bit rate of one or more frames of video frames at a time point later than the past, and an actual video recognition accuracy. The information processing device includes: The method further includes a learning means for learning the feature amount calculation means using the teacher data. Information processing device.

5. One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; A feature acquisition means for acquiring a feature calculated by referring to at least one of the above. an estimate calculation means for calculating at least one of an estimate of a bit rate of one or a plurality of future video frames and an estimate of video recognition accuracy from the feature acquired by the feature acquisition means; a parameter calculation means for calculating a video compression parameter in a coding process of one or a plurality of video frames in the future by referring to the bit rate estimate and the video recognition accuracy estimate calculated by the estimate calculation means; a candidate value acquisition means for acquiring candidate values ​​of video compression parameters for one or more future frames; Equipped with The estimated value calculation means further refers to the candidate values ​​of the video compression parameters for one or more frames in the future acquired by the candidate value acquisition means, and calculates the estimated value of the bit rate and the estimated value of the video recognition accuracy. Information processing device.

6. An information processing device, One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; A feature acquisition means for acquiring a feature calculated by referring to at least one of the above. an estimate calculation means for calculating at least one of an estimate of a bit rate of one or a plurality of future video frames and an estimate of video recognition accuracy from the feature acquired by the feature acquisition means; Equipped with The feature acquisition means includes: One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; and at least one of an actual bit rate of one or more frames of video frames at a time point later than the past, and an actual video recognition accuracy. The information processing device includes: The estimation means further includes a learning means for learning the estimation value calculation means using the teacher data. Information processing device.

7. One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; and Calculating at least one of an estimate of a bit rate of one or more future video frames and an estimate of video recognition accuracy by referring to the data acquired in the acquiring step; calculating a video compression parameter for encoding one or more future video frames by referring to the estimated bit rate and the estimated video recognition accuracy calculated in calculating at least one of the estimated bit rate and the estimated video recognition accuracy; Obtaining candidate video compression parameter values ​​for one or more future frames; Including, In calculating at least one of the estimated value of the bit rate and the estimated value of the video recognition accuracy, the estimated value of the bit rate and the estimated value of the video recognition accuracy are calculated by further referring to the candidate values ​​of the video compression parameters for one or more frames in the future acquired in acquiring the candidate values ​​of the video compression parameters. Image processing method.

8. A video processing method, comprising: One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; and calculating, by using an estimation means, at least one of an estimated value of a bit rate of one or more future video frames and an estimated value of a video recognition accuracy by referring to the data acquired in the acquiring; Including, In the obtaining, One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; and at least one of an actual bit rate of one or more frames of video frames at a time point later than the past, and an actual video recognition accuracy. The video processing method includes: The method further includes training the estimation means using the training data. Image processing method.

9. Computer, One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; An acquisition means for acquiring at least one of the above. a feature calculation means for calculating a feature from the data acquired by the acquisition means, the feature calculation means calculating a feature to be referenced for calculating at least one of an estimated value of a bit rate of one or a plurality of future video frames and an estimated value of video recognition accuracy; Functioning as a The acquisition means includes: One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; and at least one of an actual bit rate of one or more frames of video frames at a time point later than the past, and an actual video recognition accuracy. The computer, The method further functions as a learning means for learning the feature amount calculation means using the teacher data. program.

10. Computer, One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; A feature acquisition means for acquiring a feature calculated by referring to at least one of the above. an estimate calculation means for calculating at least one of an estimate of a bit rate of one or a plurality of future video frames and an estimate of video recognition accuracy from the feature acquired by the feature acquisition means; a parameter calculation means for calculating a video compression parameter in a coding process of one or a plurality of video frames in the future by referring to the bit rate estimate and the video recognition accuracy estimate calculated by the estimate calculation means; a candidate value acquisition means for acquiring candidate values ​​of video compression parameters for one or more future frames; Functioning as a The estimated value calculation means further refers to the candidate values ​​of the video compression parameters for one or more frames in the future acquired by the candidate value acquisition means, and calculates the estimated value of the bit rate and the estimated value of the video recognition accuracy. program.

11. A computer comprising: One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; A feature acquisition means for acquiring a feature calculated by referring to at least one of the above. an estimate calculation means for calculating at least one of an estimate of a bit rate of one or a plurality of future video frames and an estimate of video recognition accuracy from the feature acquired by the feature acquisition means; Functioning as a The feature acquisition means includes: One or more video frames from the past; video compression parameters in the encoding process of the video frame; and a video recognition result of the decoded video frame obtained by decoding the encoded video frame; and at least one of an actual bit rate of one or more frames of video frames at a time point later than the past, and an actual video recognition accuracy. The computer, The estimated value calculation means further functions as a learning means for learning the estimated value calculation means using the teacher data. program.

Images