Decoding device and program

Abstract

To provide a decoding device capable of flexibly coping with various delay requirements of a video signal and being realized at a low cost.SOLUTION: A decoding device includes a reception processing unit, a decoding unit, a transmission processing unit, and a resource management unit. The reception processing unit receives a video stream encoded by a predetermined encoding method. The decoding unit decodes the video stream received by the reception processing unit and outputs the decoded video stream. The transmission processing unit transmits the decoded video stream output by the decoding unit. The resource management unit determines resources for processing by the decoding unit according to a delay request, which is a request for a delay due to decoding processing of the video stream by the decoding unit, and assigns the determined resources for the decoding process of the video stream.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a decoding device and a program. [Background technology]

[0002] Standardization for packetizing video signals into IP packets and transmitting them over general-purpose Ethernet networks is currently under discussion at SMPTE (Society of Television and Motion Picture Engineers). By transmitting video signals in IP packets in this way, it is expected that a program production system that can efficiently share video signals using existing infrastructure, an IP production system, will be realized.

[0003] Program production using this IP production system uses a compression method called light compression, which allows for low latency compression with little degradation in image quality. Unlike codecs with high compression rates such as HEVC (High Efficiency Video Coding), this compression method achieves low line-level latency by keeping the compression rate to around 1 / 4 to 1 / 10. Specifically, methods such as LLVC (Low Latency Video Codec), VC2, TICO (Tiny Codec), and JPEG XS (XS stands for eXtra Speed ​​eXtra Small) exist and are used.

[0004] Of these methods, only JPEG XS has been internationally standardized. Packetization of compressed signals in IP production systems using JPEG XS is specified in ISO / IEC 21122.

[0005] The light compression techniques described above can be implemented in hardware on an FPGA or ASIC, or in software running on a CPU. When implemented in hardware, the processing delay during compression can be reduced to line-level low delay, which is the theoretical value for light compression techniques. When implemented in software, the amount of processing delay depends heavily on the program implementation and operating environment.

[0006] In the prior art, equipment has been developed that compresses and decompresses (decompresses) video using JPEG XS on dedicated hardware, primarily for the purpose of compressing and transmitting with as little delay as possible.

[0007] Non-Patent Documents 1 and 2 describe the implementation of JPEG XS. [Prior art documents] [Non-patent literature]

[0008] [Non-Patent Document 1] "JPEG XS - A new low-complexity codec standard for professional film production", Fraunhofer IIS, [online], [downloaded March 4, 2021], Internet<URL:https: / / www.iis.fraunhofer.de / ja / ff / amm / content-production / jpegxs.html> [Non-Patent Document 2] "Open Format JPEG XS, the First Paradigm Shift in Image Coding History", PRONEWS, April 23, 2018, [online], [downloaded March 4, 2021], Internet<URL:https: / / www.pronews.jp / news / 20180423181595052.html> Summary of the Invention [Problem to be solved by the invention]

[0009] As IP production systems become more widespread, various television programs are beginning to be produced using IP production systems. This has led to an increasing demand for compressed video transmission. However, it is difficult to equip all broadcasting equipment, such as CCUs (camera control units), video control consoles, and switchers, with encoding and decoding functions. In other words, it is necessary to realize the encoding and decoding functions in hardware or software separate from the broadcasting equipment and connect it to the broadcasting equipment.

[0010] In order to reduce the amount of delay in the encoding and decoding processes in JPEG XS, conventional techniques have implemented light compression as a function of dedicated hardware.

[0011] However, as the variety of programs produced by IP production systems increases, there are also more and more cases where the degree of delay requirement differs depending on the nature of the programs being produced. When delay requirements are relaxed, equipment that can accommodate a slightly larger amount of delay and is less expensive than conventional hardware implementations is required. There is also a demand for equipment that can flexibly accommodate various delay requirements depending on the nature of the programs, etc.

[0012] In addition, if encoding and decoding functions are realized using dedicated hardware, a dedicated machine is required for each video format. Furthermore, dedicated hardware has low flexibility in handling programs with video formats with various pixel counts, such as 2K / 4K / 8K. Preparing sufficient equipment for the production of all these video formats would be extremely costly.

[0013] The present invention has been made based on the recognition of the above-mentioned problems, and aims to provide a decoding device and a program therefor that can flexibly respond to various delay requirements for video signals and can be implemented at low cost. [Means for solving the problem]

[0014] [1] In order to solve the above problems, a decoding device according to one aspect of the present invention comprises a receiving processing unit that receives a video stream encoded using a predetermined encoding method, a decoding unit that performs a decoding process on the video stream received by the receiving processing unit and outputs the decoded video stream, a transmitting processing unit that transmits the decoded video stream output by the decoding unit, and a resource management unit that determines resources for processing by the decoding unit in response to a delay request, which is a request for a delay caused by the decoding process of the video stream in the decoding unit, and allocates the determined resources for processing the video stream.

[0015] [2] Furthermore, one aspect of the present invention is that the above-mentioned decoding device further includes a request processing unit that notifies the resource management unit of the delay request based on request data from a video processing device that receives the decoded video stream transmitted from the transmission processing unit.

[0016] [3] Also, in one aspect of the present invention, in the above-mentioned decoding device, the resource management unit determines resources for processing by the decoding unit also in accordance with the type of video format of the video stream, and allocates the determined resources for decoding processing of the video stream.

[0017] [4] Moreover, according to one aspect of the present invention, in the decoding device described above, the resource management unit receives a notification of a type of video format of the video stream from the reception processing unit.

[0018] [5] Also, one aspect of the present invention is that in the above-mentioned decoding device, the resource management unit determines the number of threads in a processor that executes a program in which the processing of the decoding unit is described as resources for the decoding processing of the video stream, and allocates the determined resources for the decoding processing of the video stream.

[0019] [6] Also, one aspect of the present invention is a program for causing a computer to function as a decoding device including: a receiving processing unit that receives a video stream encoded using a predetermined encoding method; a decoding unit that performs a decoding process on the video stream received by the receiving processing unit and outputs the decoded video stream; a transmitting processing unit that transmits the decoded video stream output by the decoding unit; and a resource management unit that determines resources for processing by the decoding unit in response to a delay request, which is a request for a delay caused by the decoding process of the video stream in the decoding unit, and allocates the determined resources for processing the decoding of the video stream. Effect of the Invention

[0020] According to the present invention, the decoding device is able to determine and allocate computational resources for the decoding process of an individual video stream depending on the delay requirement for that video stream. [Brief description of the drawings]

[0021] [Figure 1] 1 is a block diagram showing a configuration of a video transmission system according to an embodiment of the present invention and a schematic functional configuration of each device in the system. [Diagram 2] 11 is a sequence chart showing an example of an operation procedure of the video transmission system according to the embodiment. [Diagram 3] 11 is a sequence chart showing another example of the operation procedure of the video transmission system according to the embodiment. [Figure 4] 11 is a schematic diagram showing an example of the format of requested function data included in an SDP file used in the embodiment. FIG. [Diagram 5] This is a schematic diagram showing the measurement results of the delay (time) due to the decoding process for combinations of the video stream format and the number of threads allocated to the decoding process of the video stream when a specific operating environment (computer) functions as a decoding device in the same embodiment, and when a specific operating environment (computer) functions as decoding device 3. [Figure 6]2 is a block diagram showing an example of the internal configuration of each device constituting the video transmission system according to the embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0022] Next, an embodiment of the present invention will be described with reference to the drawings. The video transmission system according to this embodiment includes a function for performing encoding processing of a video to be transmitted, and a function for performing decoding processing (also called a decoding process).

[0023] First, the prerequisites for the embodiment will be described.

[0024] When implementing encoding and decoding functions in software, various delay requirements can be met by controlling the resources allocated through multi-threaded processing. Also, when implementing in software, it is possible to process multiple video formats such as 2K / 4K / 8K with almost the same program. In addition, resources can be shared with software functions other than the encoding and decoding functions, which allows for efficient resource use and can lead to cost reduction.

[0025] However, the amount of computation required for encoding processing in software is large compared to the amount of computation required for decoding processing. Due to the large amount of computation required for encoding processing, it is difficult to reduce the processing delay of the encoding processing. Even if maximum resources (CPU resources, etc.) are allocated to software-based encoding processing, there are many restrictions on the achievable processing delay.

[0026] When using dedicated hardware, the FPGA IP core has a smaller amount of processing power for decoding than for encoding, and the circuit scale for decoding is smaller. However, the reduction in circuit scale for decoding is only 20-30% compared to the circuit scale for encoding.

[0027] Even when encoding and decoding are performed by software, the amount of processing required for decoding is smaller than the amount of processing required for encoding. When performed by software, the amount of processing required for decoding can be reduced by 40 to 50 percent compared to the amount of processing required for encoding.

[0028] Furthermore, when encoding is performed by software, a delay of several frames may occur, but when decoding is performed by software, the amount of delay can be less than one frame.

[0029] As described above, when the decoding function is realized by software, the effects of being able to relatively reduce the amount of processing, to suppress costs, and to appropriately control the computational resources for each thread are great. On the other hand, even if an encoding function is realized by software, it is difficult to obtain the same effect as that obtained by the decoding function, and it is also impossible to reduce the amount of delay. In other words, when the encoding process is implemented by dedicated hardware and the decoding process is implemented by software, it is possible to perform processing according to a wide range of delay requirements while suppressing costs.

[0030] When the decoding function is realized by software, it is assumed that resources are allocated with a margin in equipment operation in order to be able to respond to various delay requirements. For example, even in the transmission of video that does not mind a relatively large amount of delay, low-delay resources may be allocated. Also, the operator of the decoding process is required to judge the appropriate resource allocation for each request. This embodiment automates and optimizes the allocation of resources to such decoding processes. As a result, the video transmission system of this embodiment makes it possible to operate the equipment while efficiently allocating resources.

[0031] The specific configuration will be described below.

[0032] 1 is a block diagram showing the configuration of a video transmission system according to this embodiment and the schematic functional configuration of each device. In the process of transmitting a video signal, the video transmission system 100 performs signal encoding and decoding using a light compression method. As shown in the figure, the video transmission system 100 includes an encoding device 1, a network switch 2, a decoding device 3, and video processing devices 4A and 4B.

[0033] The encoding device 1 encodes and outputs an input video signal. The encoding device 1 transmits the encoded data as IP packets. That is, the encoding device 1 generates a compressed video stream and transmits it to the network switch 2 side. This compressed video stream is IP packetized based on the SMPTE ST2110 standard, for example. The encoding device 1 can transmit the IP packets by, for example, a multicast method. However, the encoding device 1 may also transmit the IP packets by a unicast method.

[0034] The network switch 2 is a device that constitutes a network, and transfers IP packets. Specifically, the network switch 2 receives IP packets sent from the encoding device 1, and transfers them to the decoding device 3. The network switch 2 has multiple input ports and multiple output ports. The network switch 2 references the header of the IP packet, and outputs the received packet to an output port that corresponds to the destination.

[0035] The decoding device 3 receives IP packets from the network switch 2 and decodes data contained in the IP packets. For example, based on a request from the video processing devices 4A and 4B, the decoding device 3 transmits the decoded data to the video processing devices 4A and 4B.

[0036] The video processing devices 4A and 4B are devices that process video for purposes such as producing a broadcast program. The video processing devices 4A and 4B may be collectively referred to as the video processing device 4. When the video processing device 4 requests decoded video of video that has been encoded, it transmits requested function data to the decoding device 3. The requested function data includes delay request data. The delay request data is expressed, for example, by an integer value between 0 and 10. A delay request value of 0 represents the minimum delay. A delay request value of 10 represents the maximum delay. The delay request value represents the magnitude of the delay request within the range between 0 and 10.

[0037] In this embodiment, the encoding function of the encoding device 1 is realized by using dedicated hardware. This allows the encoding device 1 to perform encoding processing with a small amount of delay. The encoding device 1 performs encoding using, for example, JPEG XS as an encoding method. By realizing the encoding function by dedicated hardware, it is possible to reduce the amount of delay in the encoding processing compared to when the encoding function is realized by a general-purpose processor (CPU, etc.).

[0038] On the other hand, the decoding function of the decoding device 3 of this embodiment is realized by a general-purpose processor (CPU, etc.) and a program (software). As a result, the decoding device 3 can individually allocate resources for the decoding process to each of the multiple video streams that are decoded in parallel. The resources allocated here are mainly CPU cores, threads, and CPU time. In other words, the decoding device 3 can allocate resources to the decoding process of each video stream in units of CPU cores, threads, CPU time, etc. However, the allocated resources may include other resources (for example, a main memory area realized by a semiconductor memory). The amount of delay allowed in the decoding process differs depending on the type of video stream (for example, the contents of the content). By realizing the decoding device 3 by software, it becomes easy to flexibly allocate resources to each video stream and change the allocation.

[0039] As an example, the decoding device 3 can allocate a different number of threads as a resource for calculation to each video stream to be decoded.

[0040] The internal functional configuration of the encoding device 1 is as follows. The encoding device 1 includes an encoding unit 11 and a transmission processing unit 12. The encoding device 1 is realized using electronic circuits. A part of the encoding device 1 may be realized by a computer and a program. However, the encoding device 1 in this embodiment is realized by dedicated hardware. The functions of each unit of the encoding device 1 are as follows.

[0041] The encoding unit 11 encodes the input video stream and outputs IP packets.

[0042] The transmission processing unit 12 transmits the IP packet generated by the encoding unit 11 to the outside (to the network switch 2 in the configuration of FIG. 1).

[0043] The internal functional configuration of the decoding device 3 is as follows. As shown in the figure, the decoding device 3 is configured to include a reception processing unit 31, a decoding unit 32, a transmission processing unit 33, a resource management unit 34, a request processing unit 35, and a reception request unit 36. Each of these functional units can be realized, for example, by a computer and a program. Each functional unit also has a storage means as necessary. The storage means is, for example, a variable in the program or a memory allocated by the execution of the program. Also, non-volatile storage means such as a magnetic hard disk device or a solid state drive (SSD) may be used as necessary. Also, at least a part of the functions of each functional unit may be realized as a dedicated electronic circuit rather than a program. The functions of each unit are as follows.

[0044] The reception processing unit 31 receives IP packets from the outside (from the network switch 2 in the configuration of FIG. 1). The IP packets received by the reception processing unit 31 may transmit any type of data. However, the IP packets received by the reception processing unit 31 include packets for transmitting video stream data encoded by the encoding device 1. In other words, the reception processing unit receives a video stream encoded by a predetermined encoding method.

[0045] The decoding unit 32 decodes the video stream received by the reception processing unit 31 and outputs it. That is, the decoding unit 32 decodes the video stream received by the reception processing unit 31 and outputs the decoded video stream. The decoding unit 32 in this embodiment is realized by a computer and a program (software). The decoding unit 32 can simultaneously perform the decoding process of a plurality of video streams in parallel. The calculation resources allocated to the decoding process of each video stream in the decoding unit 32 are variable, and the allocation is performed by the resource management unit 34 described later. That is, the resources for decoding a specific video stream in the decoding unit are determined and allocated by the resource management unit 34. When the processing of the decoding unit 32 is realized by a computer and a program, the allocation of resources to the processing (process (also called a task, etc.) managed by the operating system) can be performed relatively flexibly and is variable. The decoding unit 32 passes the decoded video data to the transmission processing unit 33 as an IP packet.

[0046] The transmission processing unit 33 transmits the IP packets to the outside. The transmission processing unit 33 transmits at least the IP packets carrying the video data passed from the decoding unit 32 to the video processing devices 4A and 4B. In other words, the transmission processing unit 33 transmits the decoded video stream output by the decoding unit 32 to the outside.

[0047] The resource management unit 34 manages and allocates resources for processing in the decoding unit 32. The resource management unit 34 determines the amount of resources to be allocated for each video stream to be decoded by the decoding unit 32. The resources here are, for example, CPU resources. More specifically, the resource management unit 34 determines the number of threads to be allocated for each video stream and allocates them. When the decoding unit 32 simultaneously performs decoding processing for a plurality of video streams, the resource management unit 34 can determine and allocate resources for the decoding processing for each video stream. Note that the resource management unit 34 may further allocate resources other than the CPU to each video stream to be processed.

[0048] Specifically, the resource management unit 34 determines resources for the processing of the decoding unit 32 at least in response to a delay request, which is a request for a delay caused by the decoding process in the decoding unit of the video stream, and allocates the determined resources for the decoding process of the video stream. The resource management unit 34 may further determine resources for the processing of the decoding unit 32 in response to the type of video format of the video stream, and allocate the determined resources for the decoding process of the video stream. The video format specifies, for example, the number of pixels (the number of pixels in each of the vertical and horizontal directions) of a frame included in a video. If the video format differs, the amount of calculation required for the process of decoding the encoded video naturally changes. The resource management unit 34 may receive the delay request included in the request data from the video processing device 4 (the video processing device that receives the decoded video stream transmitted from the transmission processing unit 33). The resource management unit 34 also receives a notification of the type of video format of the video stream from the reception processing unit 31 (for example, via the request processing unit 35).

[0049] Specifically, the resource management unit 34 determines the number of threads in a processor (called a CPU, GPU, core, etc.) that executes a program in which the processing of the decoding unit 32 is described, as resources for the decoding processing of the video stream, and allocates the determined resources for the decoding processing of the video stream.

[0050] As described above, the resource management unit 34 determines the amount of resources to be allocated to each video stream based on at least a delay-related request from the destination video processing device 4. When the video streams to be processed by the decoding unit 32 are all in the same format, the resource management unit 34 may determine the amount of resources to be allocated to each video stream based only on the delay request from the video processing device 4.

[0051] The resource management unit 34 may further determine the amount of resources to be allocated to each video stream according to the type of video format. This is because different video formats result in different processing loads for decoding. In other words, when the decoding unit 32 simultaneously decodes a plurality of types of video formats, the resource management unit 34 determines the amount of resources to be allocated to each video stream based not only on the delay request from the video processing device 4 but also on the type of video format.

[0052] The resource management unit 34 acquires delay request information from the video processing device 4 to determine resources to be allocated. Also, the resource management unit 34 acquires information indicating the type of video format from the network switch 2 (encoding device 1) to determine resources to be allocated. The procedure for acquiring such information will be described later with reference to Figs. 2 and 3.

[0053] The request processing unit 35 receives a decoding request from the video processing device 4. At this time, the request from the video processing device 4 includes information on the request regarding delay. That is, the request processing unit 35 can notify the resource management unit 34 of the delay request based on the request data from the video processing device 4.

[0054] If a video requested by the video processing device 4 has not been received, the reception request unit 36 ​​requests the network switch 2 to receive the video stream.

[0055] Next, the operation procedure of the video transmission system 100 will be described with reference to FIGS.

[0056] 2 is a sequence chart showing an example of an operation procedure of the video transmission system 100. In this operation procedure, the video processing device 4A requests the video of program A from the decoding device 3. The video processing device 4A uses a multicast address to receive the video. At the time when the video processing device 4A requests the video, the reception processing unit 31 of the decoding device 3 has not yet received the video. The following will be described with reference to this flowchart.

[0057] In step S11, the video processing device 4A transmits requested function data to the request processing unit 35 of the decoding device 3. The requested function data includes information on an address for receiving the requested video. The request processing unit 35 receives this requested function data.

[0058] In step S12, the request processing unit 35 queries the receiving processing unit 31 about the address in the requested function data. In response to this query, the receiving processing unit 31 checks whether the requested video has already been received. In this example, the decoding device 3 has not yet received the video, so the receiving processing unit 31 responds to the request processing unit 35 that the video has not yet been received. The request processing unit 35 receives the response.

[0059] In step S13, the request processing unit 35 transmits the requested function data from the video processing device 4A to the reception request unit 36.

[0060] In step S14, the reception request unit 36 ​​requests the network switch 2 to multicast the video of program A. At this time, the reception request unit 36 ​​requests the multicast transmission using IGMP (Internet Group Management Protocol). IGMP is a protocol used by the decoding device 3 on the receiving side to notify the network switch 2 of participation in, maintenance of, and withdrawal from a multicast group.

[0061] In step S15, the network switch 2 starts transmitting the requested video. The video data transmitted from the network switch 2 to the decoding device 3 is data encoded by the encoding unit 11 of the encoding device 1. The receiving processing unit 31 of the decoding device 3 receives the video data transmitted by the network switch 2 via multicast.

[0062] In step S16, the reception processor 31 passes a confirmation that the video data has been received and parameters of the video data to the request processor 35. The request processor 35 receives the confirmation and parameters. The parameters here include information indicating the type of video format.

[0063] In step S17, based on the parameter information received in step S16, the request processing unit 35 requests the resource management unit 34 to allocate resources for executing the processing of the decoding unit 32. Specifically, the resources allocated by the resource management unit 34 to the processing of the decoding unit 32 are, for example, the processing time (CPU time) of the processor. Alternatively, the resource management unit 34 may allocate real memory or other computational resources.

[0064] In step S18, after the resource allocation in step S17 is completed, the resource management unit 34 causes the decoding unit 32 to start the decoding process. Based on this, the decoding unit 32 starts processing using the resources allocated by the resource management unit 34. Specifically, the decoding unit 32 starts decoding the video data received by the reception processing unit 31. The decoding unit 32 sequentially passes the decoded video data to the transmission processing unit 33. The transmission processing unit 33 transmits the decoded video data passed from the decoding unit 32 to the video processing device 4A that made the request.

[0065] As described above, according to the procedure shown in FIG. 2, the decoding device 3 starts receiving video data based on a request from the video processing device 4A, decodes the received video data, and transmits the decoded video data to the video processing device 4A.

[0066] 3 is a sequence chart showing another example of the operation procedure of the video transmission system 100. The following description will be given with reference to this flowchart. In this operation procedure, the video processing device 4B requests the video of program B from the decoding device 3. The video processing device 4B uses a multicast address to receive the video. At the time when the video processing device 4B requests the video, the receiving processing unit 31 of the decoding device 3 has already received the video data. The following description will be given with reference to this flowchart.

[0067] In step S21, the video processing device 4B transmits requested function data to the request processing unit 35 of the decoding device 3. The requested function data includes information on an address for receiving the requested video. The request processing unit 35 receives this requested function data.

[0068] In step S22, the request processing unit 35 queries the receiving processing unit 31 about the address in the requested function data. In response to this query, the receiving processing unit 31 checks whether the requested video has already been received. In this example, the decoding device 3 has already received that video. The receiving processing unit 31 responds to the request processing unit 35 that it has already received the video. The receiving processing unit 31 also passes the parameters of that video to the request processing unit 35. The request processing unit 35 receives these responses (including parameter information).

[0069] In step S23, the request processing unit 35 requests the resource management unit 34 to allocate resources for executing the processing of the decoding unit 32, based on the parameter information received in step S22. Here, the resources allocated by the resource management unit 34 to the processing of the decoding unit 32 are, for example, the processing time (CPU time) of the processor, as in the case described with reference to Fig. 2. Alternatively, the resource management unit 34 may allocate real memory or other computational resources.

[0070] In step S24, after the resource allocation in step S23 is completed, the resource management unit 34 causes the decoding unit 32 to start the decoding process. Based on this, the decoding unit 32 starts processing using the resources allocated by the resource management unit 34. Specifically, the decoding unit 32 starts decoding the video data received by the reception processing unit 31. The decoding unit 32 sequentially passes the decoded video data to the transmission processing unit 33. The transmission processing unit 33 transmits the decoded video data passed from the decoding unit 32 to the video processing device 4B that made the request.

[0071] In the above process, an example has been described in which the network switch 2 distributes video data by multicast, but the video data may be distributed by unicast.

[0072] In the video transmission system 100, the encoding device 1 and the decoding device 3 transmit a compressed video stream conforming to the SMPTE ST 2110 standard. For this purpose, an SDP file describing transmission-related parameters is transmitted from the encoding device 1 to the decoding device 3. Note that SDP is an abbreviation for Session Description Protocol, and is a format for describing streaming parameters. The amount (size) of resources required for processing in the encoding device 1 and the decoding device 3 depends on parameters such as the size and format of the video included in the compressed video stream, as well as on the delay requirement for the transmission of the stream.

[0073] In this embodiment, the resource management unit 34 of the decoding device 3 allocates resources based on the parameters of the video data in the SDP file passed from the encoding device 1 and the required function data passed from the video processing device 4 side.

[0074] In this embodiment, an SDP file is also used to describe requested function data passed from the video processing device 4 to the decoding device 3. The SDP file passed from the video processing device 4 to the decoding device 3 is an extension of the RFC4566 standard, and includes the address of the compressed video stream and delay request data.

[0075] FIG. 4 is a schematic diagram showing an example of the format of requested function data included in an SDP file. The requested function data is described in the media description section. Each line of data shown in the figure is session information described with "a=". The data on the first line of the figure indicates the address of a compressed video stream requested by the video processing device 4.<payload type> The payload value is written in the first line.<IP address> The IP address indicating the location of the video stream is described in the portion of the figure. The data in the second line of the figure indicates the level of delay required by the video processing device 4.<payload type> The payload value is written in the <0~10> section of the second line. An integer value between 0 and 10 is written in the <0~10> section of the second line. This value is a number that represents the delay request. 0 represents the minimum delay, and 10 represents the maximum delay. In other words, the smaller this value, the smaller the delay requested. Also, the larger this value, the larger the delay that is tolerated as a delay request.

[0076] The descriptions in the SDP file other than the descriptions of the video stream address and delay request shown in FIG. 4 are based on RFC4566.

[0077] The resource management unit 34 of the decoding device 3 refers to the delay request value (any value from 0 to 10) in the requested function data described in the SDP file received from the video processing device 4, and the parameters of the video data in the SDP file related to the compressed video stream passed from the encoding device 1. Based on the above-mentioned parameters and other referenced values, the resource management unit 34 allocates resources to the decoding unit 32 that decodes the video stream so as to satisfy the delay request.

[0078] Specifically, the resource management unit 34 allocates, for each video stream, the number of threads for the decoding unit 32 to perform the decoding process. Note that the resource management unit 34 stores in advance, for example, an appropriate number of threads corresponding to a combination of video data parameters and a delay request in the decoding device 3. Alternatively, the resource management unit 34 may be capable of executing a procedure for calculating an appropriate number of threads corresponding to a combination of video data parameters and a delay request.

[0079] As a premise, the decoding device 3 is realized by, for example, one computer. One computer has one or more CPUs. One CPU may have multiple cores. Specifically, one CPU may have, for example, one core, four cores, eight cores, ten cores, twelve cores, etc. Furthermore, one core may be configured to execute a single thread or multiple threads. Depending on the hardware resources of the computer, for example, a thread configuration such as 4 cores / 4 threads, 4 cores / 8 threads, 8 cores / 64 threads, 10 cores / 80 threads, 12 cores / 96 threads, etc. may be possible. Furthermore, other thread configurations different from those exemplified here may be used.

[0080] Furthermore, the relationship between threads and processes (tasks) is as follows. For example, one thread may be occupied by one process (task). Alternatively, multiple threads may be occupied by one process (task). Alternatively, one thread may be shared by multiple processes (tasks). Here, a process (task) is a logical unit of processing managed by an operating system (also called a monitor, etc.) that manages resources on a computer.

[0081] Each core operates at a given clock frequency and executes instructions contained in an executable program.

[0082] The decoding device 3 holds in advance information on the number of threads to be allocated according to the video format and delay request. The number of threads to be allocated is determined in advance according to the processing amount (calculation amount) of the decoding process and the capacity of the hardware resources, and is written in a predetermined storage means in the decoding device 3. The number of threads to be allocated is verified in advance theoretically or by demonstration according to the actual environment. The resource management unit 34 of the decoding device 3 can determine the number of threads to be allocated by referring to the storage means after grasping the video format and the delay request.

[0083] 5 is a schematic diagram showing measurement results of delay (time) due to decoding processing for combinations of video stream format (2K or 4K in this case) and the number of threads allocated to the decoding processing of the stream (processing of the decoding unit 32) when a certain operating environment (computer) is made to function as the decoding device 3. Here, 2K is a video format configured with a pixel array of 1920 pixels horizontal by 1080 pixels vertical. Also, 4K is a video format configured with a pixel array of 3840 pixels horizontal by 2160 pixels vertical. Also, the unit of delay shown in the figure is milliseconds (msec).

[0084] As shown in the figure, FIG. 5 shows the delay amount (time) of the decoding process when the number of threads is 1, 2, 4, 8, and 12 for each of the 2K and 4K video formats. When the video format is 2K, the delay amount is 19.3 milliseconds, 11.6 milliseconds, 6.0 milliseconds, 4.9 milliseconds, and 3.5 milliseconds for each of the above numbers of threads. When the video format is 4K, the delay amount is 73.7 milliseconds, 40.8 milliseconds, 22.5 milliseconds, 18.6 milliseconds, and 12.6 milliseconds for each of the above numbers of threads. Note that the data shown in FIG. 5 is merely an example. The decoding device 3 determines the number of threads to be assigned to the decoding process according to the actual processing capacity of its own device.

[0085] Assuming the delay in Figure 5, for video with a 2K video format and a progressive frame rate of 60 fps (frames per second), in order to keep the delay (time length) due to the decoding process to one frame (1 / 60 seconds, or approximately 16.667 milliseconds) or less, it is necessary to allocate two or more threads. Also, in order to keep the delay due to the decoding process for the same video to two frames (2 / 60 seconds, or approximately 33.333 milliseconds) or less, it is necessary to allocate one or more threads.

[0086] Also, assuming the delay in Figure 5, for a 4K video format with a frame rate of 60 fps (frames per second) progressive video, in order to keep the delay (time length) due to the decoding process to one frame (1 / 60 seconds, or about 16.667 milliseconds) or less, it is necessary to allocate 12 or more threads. Also, in order to keep the delay due to the decoding process to two frames (2 / 60 seconds, or about 33.333 milliseconds) or less for the same video, it is necessary to allocate 4 or more threads.

[0087] In this way, the resource management unit 34 of the decoding device 3 can determine the number of threads to be allocated for the decoding process based on the video format and the delay request. Furthermore, the resource management unit 34 controls the decoding unit 32 so as to perform the decoding process of the video with the determined number of threads. This allows the decoding unit 32 to execute the decoding process of the video data passed from the reception processing unit 31 with an amount of delay commensurate with the request from the video processing device 4 side, and pass the decoded video data to the transmission processing unit 33. Furthermore, the transmission processing unit 33 can transmit the decoded video data to the video processing device 4 that originated the request.

[0088] As described above, the resource management unit 34 of the decoding device 3 can allocate resources (such as a CPU) suitable for the decoding process to the decoding unit 32 according to the contents of the data (SDP file) passed from the encoding device 1 or the video processing device 4. In other words, the resource management unit 34 can automatically allocate resources for the decoding process that satisfy the delay requirement based on the data received from the outside. Also, the decoding unit 32 can perform the decoding process of the video stream while satisfying the delay requirement.

[0089] According to the processing method of the present embodiment, the decoding device 3 can automatically allocate resources required for the decoding process based on a request from the side requesting the video (the video processing device 4 side). That is, the decoding device 3 can perform processing that efficiently utilizes the processing capacity of a computer. That is, the decoding device 3 can flexibly allocate computational resources to the decoding process compared to a case where the decoding device is configured to perform processing only by hardware. Also, compared to a case where the decoding device is configured only by hardware, the decoding device 3 of the present embodiment does not require a dedicated device for each video format. The decoding device 3 of the present embodiment can be realized using general-purpose equipment (computer), so it is highly flexible and can reduce costs. Also, the decoding device 3 of the present embodiment can flexibly respond to various delay requirements. That is, it is possible to realize a transmission system (including an encoding device and a decoding device) that can be used for various programs with various delay requirements at the site where broadcast programs are produced, etc.

[0090] FIG. 6 is a block diagram showing an example of the internal configuration of each device constituting the video transmission system of this embodiment. At least a part of each device (encoding device 1, network switch 2, decoding device 3, video processing device 4A, 4B) can be realized using a computer. As shown in the figure, the computer is configured to include a central processing unit 901, a RAM 902, an input / output port 903, input / output devices 904 and 905, etc., and a bus 906. The computer itself can be realized using existing technology. The central processing unit 901 executes instructions included in a program read from the RAM 902, etc. The central processing unit 901 writes data to the RAM 902, reads data from the RAM 902, and performs arithmetic and logical operations according to each instruction. The RAM 902 stores data and programs. Each element included in the RAM 902 has an address and can be accessed using the address. Note that RAM is an abbreviation for "random access memory." The input / output port 903 is a port through which the central processing unit 901 exchanges data with an external input / output device, etc. The input / output devices 904 and 905 are input / output devices. The input / output devices 904 and 905 exchange data with the central processing unit 901 via the input / output port 903. The bus 906 is a common communication path used inside the computer. For example, the central processing unit 901 reads and writes data from and to the RAM 902 via the bus 906. Also, for example, the central processing unit 901 accesses the input / output port via the bus 906.

[0091] At least one of the functions of the encoding device 1, the network switch 2, the decoding device 3, and the video processing devices 4A and 4B in the above-mentioned embodiment can be realized by a computer and a program. In this case, the program for realizing the function may be recorded in a computer-readable recording medium, and the program recorded in the recording medium may be read into a computer system and executed to realize the function. Note that the term "computer system" here includes hardware such as an OS and peripheral devices. Furthermore, the term "computer-readable recording medium" refers to portable media such as flexible disks, optical magnetic disks, ROMs, CD-ROMs, DVD-ROMs, and USB memories, and storage devices such as hard disks built into computer systems. In other words, the term "computer-readable recording medium" may be a non-transitory computer-readable recording medium. Furthermore, the term "computer-readable recording medium" may include a medium that temporarily and dynamically holds a program, such as a communication line when a program is transmitted via a network such as the Internet or a communication line such as a telephone line, and a medium that holds a program for a certain period of time, such as a volatile memory inside a computer system that is a server or client in such a case. Furthermore, the above program may be for realizing part of the functions described above, and may further be capable of realizing the functions described above in combination with a program already recorded in the computer system.

[0092] As described above, the video transmission system 100 of this embodiment can automatically allocate appropriate resources based on a delay request from the video processing device 4 side.

[0093] In this embodiment, the encoding device 1 is realized by dedicated hardware, and the decoding device 3 is realized by running software (programs) on a general-purpose computer. This makes it possible to meet a wide range of delay requirements while suppressing the total cost.

[0094] Furthermore, in this embodiment, requested function data including a delay request (data expressing an allowable amount of delay, for example, as a numerical value (an integer from 0 to 10)) is sent from the video processing device 4 (broadcasting equipment, etc.) to the resource management unit 34 of the decoding device 3. As a result, the resource management unit 34 automatically allocates resources to satisfy the request for the decoding process by the decoding unit 32. Therefore, the video transmission system 100 of this embodiment enables efficient resource allocation.

[0095] According to the present embodiment, the function of the encoding process in the encoding device 1 is realized by dedicated hardware, and the function of the decoding process in the decoding device 3 is realized by a general-purpose computer and software. In addition, in the production of a broadcast program using video transmission by IP, a broadcasting device (video processing device 4) that requests video decoding requests the decoding device 3 to perform video decoding. In the decoding device 3, the resource management unit 34 can automatically allocate resources (CPU cores, threads, CPU time) required for the decoding process according to a delay request received from the broadcasting device (video processing device 4). That is, the resources of a general-purpose computer for implementing the decoding device 3 can be automatically allocated to the decoding process. This enables efficient operation of the encoding device 1 and the decoding device 3. In addition, the decoding device can respond to various delay requests from the broadcasting device (video processing device 4). In addition, the decoding device can flexibly respond to decoding processes of various video formats. This enables reduction in equipment costs in the entire video transmission system 100.

[0096] Although the embodiment has been described above, the present invention can also be embodied in the following modified examples.

[0097] In the video transmission system 100, IP is used for communication between the devices. However, a protocol other than IP may be used for communication in a similar facility.

[0098] In the above embodiment, the resource management unit 34 determines and allocates resources for encoding processing based on delay request information included in data transmitted from the video processing device 4. However, the resource management unit 34 may determine the resources based on delay request information issued from a device other than the video processing device 4.

[0099] In the above embodiment, the information on the type of video format is notified to the resource management unit 34 from the reception processing unit 31 via the request processing unit 35. However, the resource management unit 34 may obtain the information on the type of video format from another location (information source).

[0100] In addition, multiple modified examples may be combined to the extent that they are possible to combine.

[0101] Although an embodiment of the present invention has been described in detail above with reference to the drawings, the specific configuration is not limited to this embodiment, and designs that do not deviate from the gist of the present invention are also included. [Industrial Applicability]

[0102] The present invention can be used in video decoding processing. For example, the present invention can be used in the production of broadcast programs using IP transmission. However, the scope of use of the present invention is not limited to the examples given here. [Explanation of symbols]

[0103] 1 Encoding device 2. Network Switch 3. Decoding device 4A, 4B Image processing device 11 Encoding section 12 Transmission processing unit 31 Receiving processing unit 32 Decoding section 33 Transmission processing unit 34 Resource Management Department 35 Request Processing Unit 36 Reception request section 100 Video Transmission System 901 Central Processing Unit 902 RAM 903 Input / Output Ports 904,905 Input / Output Devices 906 Bus

Claims

1. a reception processing unit that receives a video stream encoded in a predetermined encoding format; a decoding unit that performs a process of decoding the video stream received by the reception processing unit and outputs the decoded video stream; a transmission processing unit that transmits the decoded video stream output by the decoding unit; a resource management unit that determines resources for processing by the decoding unit in response to a delay request that is a request for a delay caused by a decoding process of the video stream in the decoding unit, and allocates the determined resources for the decoding process of the video stream; a request processing unit that notifies the resource management unit of the delay request based on request data from a video processing device that receives the decoded video stream transmitted from the transmission processing unit; A decoding device comprising:

2. the resource management unit determines resources for processing by the decoding unit according to a type of a video format of the video stream, and allocates the determined resources for decoding the video stream.

2. The decoding device according to claim 1.

3. a reception processing unit that receives a video stream encoded in a predetermined encoding format; a decoding unit that performs a process of decoding the video stream received by the reception processing unit and outputs the decoded video stream; a transmission processing unit that transmits the decoded video stream output by the decoding unit; a resource management unit that determines resources for processing by the decoding unit in response to a delay request that is a request for a delay caused by a decoding process of the video stream in the decoding unit, and allocates the determined resources for the decoding process of the video stream; Equipped with the resource management unit determines resources for processing by the decoding unit according to a type of a video format of the video stream, and allocates the determined resources for processing the decoding of the video stream; the resource management unit receives a notification of a type of a video format of the video stream from the reception processing unit, The type of the video format indicates that the video format is one of 2K, 4K, or 8K. Decoding device.

4. the resource management unit determines the number of threads in a processor that executes a program in which the processing of the decoding unit is described as resources for the processing of decoding the video stream, and allocates the determined resources for the processing of decoding the video stream. Decoding device according to any one of claims 1 to 3.

5. A decoding device according to any one of claims 1 to 4. A program that makes a computer function as a

Images