Control device, video transmission system, control method, and program

Abstract

Provided is a control device for executing control on a network, which includes a video processing unit that processes video data transmitted from a video transmission device and a network device that executes processing of transferring the video data, the control device comprising a processing determination unit that uses a predetermined plurality of sets of layout information and layout information selected by each client terminal from among the plurality of sets of layout information to determine video processing to be executed by the video processing unit and transfer processing to be executed by the network device using multicast.

Description

[Technical Field]

[0001] The present invention relates to a technique for distributing video to a client terminal. [Background technology]

[0002] Live video streaming is becoming increasingly common, and services requiring ultra-low latency are emerging. However, common video compression technologies such as H.265 and H.264 introduce a certain amount of latency during encoding and decoding, and additional latency can occur when switching or compositing video.

[0003] In recent years, there have been protocols for transmitting uncompressed video or lightly compressed video that is close to uncompressed, such as ST2110. These protocols are highly compatible with video production, and can transmit high-quality video with low latency compared to transmission using general compressed video.

[0004] In the future, advances in network technology will likely make it possible to deliver video at the multi-Gbps level to users. In that case, protocols such as ST2110, which have been used mainly in the video production field until now, can be applied to video distribution, enabling end-to-end ultra-low latency distribution.

[0005] Furthermore, in recent years, technologies have emerged that process ST2110 on a packet basis to perform tasks such as video replacement (e.g., Non-Patent Document 1, Non-Patent Document 2). Such technologies identify the frame number and position information of pixels contained in the payload described in the packet header, and perform processes such as discarding packets and rewriting headers in network equipment, thereby achieving video switching and composition processing through packet-based processing. This makes it possible to achieve low-latency video distribution and video processing using only network equipment, without using dedicated video editing facilities.

[0006] Furthermore, as the needs of video viewers become more diverse, there is a demand for more personalized video services. Until now, networks have only transmitted pre-packaged video data, but by utilizing the above-mentioned technology, personalized video distribution for each user can be realized even over the network. [Prior art documents] [Non-patent literature]

[0007] [Non-Patent Document 1] Paul Briscoe, "Efficient Carriage of Sub-Rasters With ST 2110-20", IP SHOWCASE THEATER AT IBC2019 : 13-17 SEPT 2019. [Non-patent document 2] K. Kitada, M. Fukudome, M. Okuyama, M. Cho, M. Kurozumi, M. Nishide, M. Nishimura, and M. Nishimoto, "A Study on In-Network Video Processing Methods for Lightly Compressed Video," IEICE General Conference 2022, B-6-26, March 2022. Summary of the Invention [Problem to be solved by the invention]

[0008] When packet-based video processing is performed using technologies such as ST2110, video data is typically transmitted to each user in the following order: encoder, video processor, and decoder. To deliver personalized content to each user, the video processing content must be adjusted to accommodate various video processing requirements. This requires preparing video processing functions for each client and securing bandwidth for each client, which requires a significant amount of resources.

[0009] The present invention has been made in consideration of the above points, and aims to provide a technology for reducing the resources required for video processing and video transmission when delivering personalized video to individual users. [Means for solving the problem]

[0010] According to the disclosed technology, there is provided a control device that executes control in a network having a video processing unit that processes video data transmitted from a video transmission device and a network device that performs transfer processing of the video data, the control device comprising: a processing determination unit that determines video processing in the video processing unit and transfer processing using multicast in the network device based on a predetermined plurality of pieces of layout information and layout information selected by each client terminal from the plurality of pieces of layout information; The image processing in the image processing unit is grouped, and image processing is performed for each group. A control device is provided. [Effects of the Invention]

[0011] The disclosed technology provides a technology for reducing resources required for video processing and video transmission when delivering personalized video to individual users. [Brief explanation of the drawings]

[0012] [Figure 1] FIG. 1 is a diagram illustrating an example of the overall configuration of a system. [Figure 2] FIG. 2 illustrates an example of the configuration of a control unit 120. [Figure 3] FIG. 10 is a diagram illustrating an example of layout information. [Figure 4] FIG. 10 is a diagram illustrating an example of client terminal information. [Figure 5] FIG. 10 is a diagram showing an example of video data information. [Figure 6] FIG. 2 illustrates an example of the configuration of a network device 110. [Figure 7]FIG. 2 is a diagram illustrating an example of the configuration of a video processing unit 130. [Figure 8] 1 is a flowchart of the first embodiment. [Figure 9] 10 is a flowchart of a second embodiment. [Figure 10] FIG. 10 is a diagram illustrating an example of a multicast tree. [Figure 11] FIG. 10 is a diagram illustrating an example of a multicast tree. [Figure 12] FIG. 10 is a diagram showing an example of setting parameters of the video processing unit 130. [Figure 13] FIG. 2 is a diagram illustrating an example of setting parameters of the network device 110. [Figure 14] FIG. 2 illustrates an example of a hardware configuration of the apparatus. DETAILED DESCRIPTION OF THE INVENTION

[0013] Hereinafter, an embodiment of the present invention (the present embodiment) will be described with reference to the drawings. The embodiment described below is merely an example, and the embodiment to which the present invention is applied is not limited to the following embodiment.

[0014] (System Configuration) An example of the overall configuration of a system according to this embodiment is shown in Fig. 1. As shown in Fig. 1, in this system, a video transmission system 100 is provided on a distribution platform 200.

[0015] The video transmission system 100 includes network devices 110-1 to 110-3, a control unit 120, and a video processing unit 130. Note that the video transmission system 100 does not necessarily have to include the network devices 110-1 to 110-3. The control unit 120 and the video processing unit 130 may be referred to as the control device 120 and the video processing device 130, respectively.

[0016] 1, an encoder 210 or a decoder 220 is connected to each network device 110. Also, there is a video sending device 300 and client terminals 400-1 to 400-2. Each device can communicate with other devices via a network (e.g., an IP network).

[0017] The number of devices shown in Fig. 1 is an example. For example, there are generally multiple video sending devices 300. In this embodiment, it is assumed that the video sending devices 300 are devices of a content provider. The video sending devices 300 may also be called video sources.

[0018] In this system, video transmission is performed using a protocol, such as ST2110, for transmitting uncompressed video or lightly compressed video that is nearly uncompressed. As an example, ST2110 converts multiple pixels into IP packets. The video processing unit 130 uses a programmable switch or other function to determine pixels from packet information in the packet stream transmitted from the video sending device 300, and rewrites headers, thereby enabling video composition within the network. The video processing unit 130 may be the network device 110. That is, the network device 110 may include the functions of the video processing unit 130.

[0019] The network device 110 is a device capable of forwarding packets, such as a router or a switch.

[0020] The encoder 210 is a device that encodes video. The decoder 220 is a device that decodes the encoded video data to obtain video.

[0021] The control unit 120 executes a control algorithm (control program) to determine the video processing content and the transfer processing content. The control unit 120 may also be called a control device.

[0022] The client terminal 400 is a device of a video viewing user, such as a PC, a tablet, a smartphone, etc. The client terminal 400 may be a CPE to which a plurality of terminals are connected under its control.

[0023] In this embodiment, it is assumed that a viewer user selects a desired pattern from a range of video processing patterns previously permitted by a content provider. Under this assumption, a system having the above configuration groups the video processing for each viewer user and processes it for each group, and performs routing control by multicasting transmission and video processing for each video data, thereby realizing a control method that achieves both video processing and video transmission.

[0024] In this embodiment, the method of video processing (resizing and compositing) performed by the video processor 130 is not limited to a specific method, but may be, for example, the method described in Non-Patent Document 2. As an example, by replacing part of the header information of the video packets in video source b with the header information of the video packets of video source a, it is possible to composite the video of video source a onto video source b.

[0025] Furthermore, each part or device in the video transmission system may be a physical device or a virtual machine on a virtual platform.

[0026] The configuration of each device (each unit) will be explained below, and detailed operations will be explained in Examples 1 and 2 thereafter.

[0027] (control unit 120) 2 shows the configuration of the control unit 120. As shown in FIG. 2, the control unit 120 includes a network control unit 121, a control request receiving unit 122, a video processing control unit 123, a processing content determination unit 124, and a DB 125.

[0028] The network control unit 121 sets transfer processing details for each network device 110. The video processing control unit 123 sets video processing details for the video processing unit 130. The network control unit 121 and the video processing control unit 123 may be collectively referred to as the "control unit." Furthermore, the network control unit 121 and the video processing control unit 123 may be located outside the control unit 120.

[0029] The control request receiving unit 122 communicates with the client terminal 400 and the video sending device 130. The processing content determining unit 124 determines the video processing content and resources in the video processing unit 130, and the processing content in the network device 110.

[0030] Various types of data are stored in the database (DB) 125. Examples of data (parameters) stored in the database (DB) 125 are shown in Figures 3 to 5. Examples of how each piece of data is used will be explained in the examples described later.

[0031] An example of layout information is shown in Fig. 3. As shown in Fig. 3, the layout information has a layout number, a size, and division information. The division information includes the following information, with the top left corner of the video as the origin of the coordinates (0,0):

[0032] "(Video transmission device identifier).(Display coordinate start point).(Display coordinate end point),..." When multiple videos are superimposed and composited, they are described in order from back to front, separated by commas. The example shown in the bottom right of Figure 3 shows the division information for layout number 1.

[0033] An example of client terminal information is shown in Fig. 4. As shown in Fig. 4, the client terminal information includes a client number, an identifier of the client terminal, and layout information (number) selected in the client terminal. Note that in Fig. 4, for the sake of simplicity, the layout information differs for each client terminal, but in reality, it is assumed that one layout information is selected by multiple client terminals.

[0034] Fig. 5 shows an example of video data information. This video data information is information that the control unit 120 acquires in advance from each video sending device 300. As shown in Fig. 5, the video data information includes a video sending device identifier, the original size of the video data, and a video data identifier.

[0035] (Network device 110) Fig. 6 shows the configuration of the network device 110. As shown in Fig. 6, the network device 110 has a transfer unit 111. The transfer unit 111 transfers video data (packets) received from one device to another device in accordance with the transfer processing content set by the control unit 120.

[0036] (Video processing unit 130) Fig. 7 shows the configuration of the video processing unit 130. As shown in Fig. 7, the video processing unit 130 has a video processing function unit 131. The video processing function unit 131 processes packets received from a certain network device in accordance with the video processing content set by the control unit 120, and transfers the processed packets to another network device.

[0037] The operation of the system having the above configuration will be described below in Examples 1 and 2. Example 2 is a detailed example of the processing in the processing determination unit 124. The control described in Examples 1 and 2 is dynamically performed in response to a request (selection) from the client terminal 400.

[0038] Example 1 The first embodiment will be described with reference to the flowchart of Fig. 8. In S101, the control request receiving unit 122 in the control unit 120 receives layout information from the video sending device 300 (content provider) and registers the received layout information in the DB 125. An example of the registered layout information is as shown in Fig. 3.

[0039] In S102 of FIG. 8, the control request receiving unit 122 of the control unit 120 reads out the registered layout information from the DB 125 and presents the registered layout information to the client terminal 400.

[0040] In the client terminal 400, certain layout information is selected and notified to the control unit 120. The control request receiving unit 122 receives the selected layout information. The received layout information is registered in the DB 125 as client terminal information. An example of the client terminal information is as shown in FIG. 4.

[0041] In S103, the processing determination unit 124 of the control unit 120 determines, based on the client terminal information and the layout information, the video processing content in the video processing unit 130 and the transfer processing content in the network devices 110 to 130, using a control algorithm. The determined processing content is passed to the network control unit 121 and the video processing control unit 123.

[0042] In S104, the network control unit 121 sets the transfer processing content determined in S103 for each network device 110 to be set, and the video processing control unit 123 sets the video processing content determined in S103 for each video processing unit 130 to be set.

[0043] In S105, when the transfer unit 111 of the network device 110 receives video data from the encoder 110 (or another network device), it identifies the video data based on the processing content set by the control unit 120 and transmits the video data to the video processing unit 130 (or another network device or decoder) set for the identified video data.

[0044] In S106, when the video processing unit 130 receives the video data from the network device 110, it identifies the video data based on the processing content set by the control unit 120, processes the video data, and transmits the video data to the next network device.

[0045] Example 2 Next, as a second embodiment, a process of determining video processing content and transfer processing content in the control unit 120 will be described with reference to the flowchart in Fig. 9. It is assumed that layout information (e.g., Fig. 3), client terminal information (e.g., Fig. 4), and video data information (e.g., Fig. 5) are stored in the DB 125. It is also assumed that network configuration information required for determining the transfer processing content is stored in the DB 125. The process determination unit 124 can use the network configuration information to perform tasks such as building a multicast tree, as will be described later.

[0046] 9, steps S202 to S205 are executed for each registered layout information (layout number). In step S201, the processing determining unit 124 initializes i, which indicates the layout number, to 1.

[0047] In S202, the processing determination unit 124 acquires the i-th layout information from the DB 125.

[0048] In S203, the processing determination unit 124 compares the layout information acquired in S202 with the client terminal information in the DB 125, and determines whether one or more client terminals have selected the layout information acquired in S202. For example, if the layout number in Fig. 3 is 1, then the client number in Fig. 4 is a.

[0049] In S204, the processing determination unit 124 determines the video processing content. First, the processing determination unit 124 determines the processing for the video data included in the division information based on the division information of the i-th layout information and the video data information. A specific example is as follows.

[0050] If the division information is made up of only one piece of video data and the size of the video data in the layout information indicated by the division information is no different from the original size, the processing determination unit 124 determines that no video processing is to be performed.

[0051] The processing determination unit 124 determines to perform resizing processing on video data in which the division information is composed of only one piece of video data and the size of the video data in the layout information indicated by the division information is different from the original size.

[0052] If the division information is made up of a plurality of video data, the processing determination unit 144 determines to perform synthesis processing.

[0053] The above is just an example, and processing other than resizing and combining may also be performed.

[0054] If there is unprocessed layout information, i is incremented and the above process is performed on the next layout information (Yes in S205), and if there is no unprocessed layout information (No in S205), the process proceeds to S206.

[0055] In S206, the process determining unit 125 determines the transfer process content, specifically as follows.

[0056] First, the processing determination unit 124 constructs a multicast tree between each video source (video sending device) requested by the client terminal and the requesting client terminal based on the layout information, client terminal information, and the video processing determination result at S206.

[0057] As an example, Fig. 10 shows an example of a multicast tree when the client terminal information shown in Fig. 4 is obtained under the layout information shown in Fig. 3. In Fig. 10, capital letters such as A indicate video sources. Lowercase letters such as a indicate client terminals. (A, B) below a client terminal indicates, for example, that client terminal a is requesting video from A and video from B. The same applies to other client terminals.

[0058] In addition, the black dots (branching points of the tree) in the figure represent copy points (network devices that function as copy points) of the multicast tree. In other words, the video data sent from the video sending device 300 is copied at the copy points and transferred to each branched destination.

[0059] In FIG. 10, for the sake of simplicity, an example is shown in which one client terminal is connected for each pattern such as (A, B), but in general, multiple client terminals are connected for each pattern such as (A, B), and each client terminal receives the corresponding video via a decoder.

[0060] Next, the processing determination unit 124 selects, for each video source, a video processing unit 130 that is closest to the multicast copy point. Note that one video processing unit 130 may be selected for one video source, or one video processing unit 130 may be selected for multiple video sources.

[0061] Furthermore, for example, when the video processing unit 130 is realized by a virtual machine, the resources allocated to the video processing unit 130 (virtual machine) may be increased or decreased depending on the number of video sources processed by the video processing unit 130. In other words, the greater the number of video sources, the greater the amount of resources allocated. Fig. 11 shows an image in which three video processing units 130-1 to 130-3 are selected in the state of Fig. 10.

[0062] "Selecting a video processing unit that is close to the multicast copy point" means, for example, focusing on copy point 1, if there are three video processing units, that of the three video processing units is the video processing unit that is closest to copy point 1. Also, "close to the copy point" may mean close physical distance, or close network distance (e.g., the smallest number of network devices to pass through).

[0063] Next, based on the result of the video processing determination in S204, the video processing control unit 123 sets the processing content, the video sending device identifier of the video data that needs to be processed, and destination information in the video processing unit 130.

[0064] Furthermore, in a video transmission system, when video data is sent to the video processing unit 130, a route is set so that the data travels from a network device immediately before the copy point (on the video source side) to the network device at the copy point via the video processing unit 130. However, this route setting is just one example. When setting the route, the network control unit 121 sets setting parameters in each network device 110 so that a desired route is constructed.

[0065] Fig. 12 shows an example of setting parameters for the video processing unit 130. This example shows a case where a certain video processing unit 130 processes video data from four video transmission devices. Fig. 12 shows that, for example, video data transmitted from the video transmission device at 160.10.10.10 is resized and transferred to the network device at the copy point.

[0066] 13 shows an example of setting parameters for a certain network device 110. In this example, it is indicated that video data transmitted from a video transmission device at 160.10.10.10 is to be transferred to the video processing unit 130, and video data transmitted from a video transmission device at 160.10.11.11 is to be transferred to another network device.

[0067] In the example of Figure 11, for example, with regard to video processing unit 130-1, for video data (packets) from video source A, a packet that has been subjected to a synthesis process of (A, B) and a packet that has been subjected to a synthesis process of (A, C) are generated, and these are sent to the network device at the copy point.

[0068] For client terminal a, the video that has been composited (A, B) by a decoder located before it is delivered to client terminal a. For client terminal b, the video that has been composited (A, C) by a decoder located before it is delivered to client terminal b.

[0069] (Example of hardware configuration) The network device 110, control unit 120, and video processing unit 130 in this embodiment can all be realized, for example, by causing a computer to execute a program. This computer may be a physical computer or a virtual machine on the cloud. When a virtual machine on the cloud is used, the configuration described below is a virtual configuration. Hereinafter, the network device 110, control unit 120, and video processing unit 130 will be collectively referred to as "devices."

[0070] That is, the device can be realized by executing a program corresponding to the processing performed by the device using hardware resources such as a CPU and memory built into a computer. The program can be recorded on a computer-readable recording medium (such as a portable memory) and stored or distributed. The program can also be provided via a network such as the Internet or email.

[0071] Fig. 14 is a diagram showing an example of the hardware configuration of the computer. The computer in Fig. 14 includes a drive device 1000, an auxiliary storage device 1002, a memory device 1003, a CPU 1004, an interface device 1005, a display device 1006, an input device 1007, an output device 1008, and the like, all of which are interconnected by a bus BS.

[0072] A program for realizing processing on the computer is provided by a recording medium 1001 such as a CD-ROM or a memory card. When the recording medium 1001 storing the program is set in the drive device 1000, the program is installed from the recording medium 1001 to the auxiliary storage device 1002 via the drive device 1000. However, the program does not necessarily have to be installed from the recording medium 1001, but may be downloaded from another computer via a network. The auxiliary storage device 1002 stores the installed program as well as necessary files, data, etc.

[0073] The memory device 1003 reads and stores a program from the auxiliary storage device 1002 when an instruction to start the program is received. The CPU 1004 realizes the functions related to the device in accordance with the program stored in the memory device 1003. The interface device 1005 is used as an interface for connecting to a network, etc. The display device 1006 displays a GUI (Graphical User Interface) or the like according to the program. The input device 1007 is composed of a keyboard, mouse, buttons, a touch panel, etc., and is used to input various operation instructions. The output device 1008 outputs the results of calculations.

[0074] (Effects of the embodiment) The technology of this embodiment makes it possible to reduce resources for video processing performance by grouping and executing video processing, and also makes it possible to carry out video distribution using minimal network bandwidth resources by dynamically controlling video processing resources and transmission paths.

[0075] (Addendum) The following additional clauses are disclosed in relation to the above-described embodiment. (Additional note 1) A control device that executes control in a network having a video processing unit that processes video data transmitted from a video transmission device and a network device that performs transfer processing of the video data, a processing determination unit that determines video processing in the video processing unit and transfer processing using multicast in the network device based on a predetermined plurality of pieces of layout information and layout information selected by each client terminal from the plurality of pieces of layout information; A control device comprising: (Additional note 2) a control unit that executes setting of the video processing in the video processing unit and setting of the transfer processing in the network device; The control device according to claim 1, further comprising: (Additional note 3) The processing determination unit determines the video processing for each piece of layout information selected by at least one client terminal. 3. The control device according to claim 1 or 2. (Additional note 4) The processing determination unit constructs a multicast tree between each video source of video data constituting layout information selected by at least one client terminal and the plurality of client terminals that are the selection sources, and determines the transfer processing so that the video data reaches a network device that is a copy point in the multicast tree via the video processing unit. 4. The control device according to any one of claims 1 to 3. (Additional note 5) A video transmission system including the control device according to any one of appended items 1 to 4 and the video processing unit. (Additional note 6) A control method in a control device that executes control in a network having a video processing unit that processes video data transmitted from a video transmission device and a network device that performs transfer processing of the video data, comprising: Based on a predetermined plurality of pieces of layout information and layout information selected by each client terminal from the plurality of pieces of layout information, video processing in the video processing unit and transfer processing using multicast in the network device are determined. Control method. (Additional note 7) A non-transitory storage medium storing a program for causing a computer to function as each unit in the control device described in any one of appendixes 1 to 4.

[0076] Although the present embodiment has been described above, the present invention is not limited to such a specific embodiment, and various modifications and changes are possible within the scope of the gist of the present invention described in the claims. [Explanation of symbols]

[0077] 100 Video Transmission System 200 distribution platforms 110 Network equipment 111 Transfer Department 120 control section 121 Network Control Unit 122 Control request reception unit 123 Video processing control unit 124 Processing content determination unit 125 DB 130 Video Processing Unit 131 Image processing function unit 210 Encoder 220 decoder 300 Video transmission device 400 client terminal 1000 Drive Device 1001 Recording media 1002 Auxiliary storage device 1003 Memory device 1004 CPU 1005 Interface device 1006 Display device 1007 Input Device 1008 Output Device

Claims

1. A control device that executes control in a network having a video processing unit that processes video data transmitted from a video transmission device and a network device that performs transfer processing of the video data, a processing determination unit that determines video processing in the video processing unit and transfer processing using multicast in the network device based on a predetermined plurality of pieces of layout information and layout information selected by each client terminal from the plurality of pieces of layout information; The image processing in the image processing unit is grouped, and image processing is performed for each group. Control device.

2. a control unit that executes setting of the video processing in the video processing unit and setting of the transfer processing in the network device; The control device of claim 1 further comprising:

3. The processing determination unit determines the video processing for each piece of layout information selected by at least one client terminal. The control device according to claim 1 .

4. The processing determination unit constructs a multicast tree between each video source of video data constituting layout information selected by at least one client terminal and the plurality of client terminals that are the selection sources, and determines the transfer processing so that the video data reaches a network device that is a copy point in the multicast tree via the video processing unit. The control device according to claim 1 .

5. A video transmission system comprising the control device according to claim 1 and the video processing unit.

6. A control method in a control device that executes control in a network having a video processing unit that processes video data transmitted from a video transmission device and a network device that performs transfer processing of the video data, comprising: a control method for determining video processing in the video processing unit and transfer processing using multicast in the network device based on a predetermined plurality of pieces of layout information and layout information selected by each client terminal from the plurality of pieces of layout information, The image processing in the image processing unit is grouped, and image processing is performed for each group. Control method.

7. A program for causing a computer to function as each unit in the control device according to any one of claims 1 to 4.

Images