Transmitting device, receiving device, transmitting program, and receiving program

Abstract

To reduce image quality deterioration due to light compression processing caused by video division while suppressing processing delay due to video division processing.SOLUTION: A transmission device includes a video acquisition unit that acquires a video stream captured by an imaging device, a division unit that divides a video frame of the acquired video stream into a divided video frame having a group of pixels that includes a plurality of pixels that is continuous in the vertical direction and the horizontal direction of the video frame, and that exists at at least two discrete positions, a compression unit that compresses the divided video frames, and a transmission unit that transmits the compressed divided video frames.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to a transmitting device, a receiving device, a transmitting program, and a receiving program. [Background technology]

[0002] The Society of Television and Motion Picture Engineers (SMPTE) is currently discussing standardization for packetizing video signals over IP and transmitting them over general-purpose Ethernet networks. Packaging video signals over IP and transmitting them in this way is expected to lead to the realization of a program production system, an IP production system, that can efficiently share video signals using existing infrastructure.

[0003] This method of live program production using these standards and technologies is called IP remote production. Conventionally, broadcast programs were produced by transporting program production equipment to the broadcast site, setting up the equipment, and then producing the program at the broadcast site. On the other hand, with IP remote production, the program materials necessary for program production, such as video and audio, are transmitted from the broadcast site to the broadcast station, where the program is produced. With IP remote production, the program production equipment remains installed at the broadcast station, and program production can be carried out from the station.

[0004] To maintain the quality of traditional live broadcast programs even with IP remote production, it is necessary to transmit high-quality video from the live broadcast site to the broadcast station in real time. To maintain the quality of traditional live broadcast programs, it is specifically required to transmit video with a latency of one video frame (16.6 ms) or less. Because the video data to be transmitted is large, it is necessary to compress the video before transmission in order to maintain high image quality and use the line efficiently.

[0005] Video transmitted from live broadcast locations is in either 2K, 4K, or 8K video format, and some programs may contain a mixture of video formats. Conventional video compression transmission equipment can only handle one video format, so it was necessary to use dedicated equipment for each format. In addition, the need to prepare dedicated equipment for each program posed challenges in equipment operation.

[0006] One solution is to split 4K or 8K video into 2K video streams and treat them as 2K x 4ch and 2K x 16ch video streams, respectively. By treating 4K and 8K video streams as multiple 2K compressed video streams in this way, video compression resources can be unified into a 2K video processing format. Furthermore, even if the video formats are different, it becomes possible to compress and transmit them using a single common transmission device, greatly improving operability.

[0007] In the case of 4K and 8K video, even after compression, the data volume is large, requiring a broadband network with a bandwidth of 10Gbps (Giga bits per second) or 100Gbps. Because narrowband networks such as 10Gbps or 1Gbps are cheaper than 100Gbps networks, handling 4K and 8K video as multiple 2K compressed video streams allows for a more cost-effective system configuration. Narrowband networks are also easier to use, as they are generally available at any broadcast location.

[0008] Here, 2SI (2-Sample Interleave Division) and SQD (Square Division) are known as conventional division methods (see, for example, Non-Patent Document 1). [Prior art documents] [Non-patent literature]

[0009] [Non-Patent Document 1] Takaaki Ishikawa, Journal of the Institute of Image Information and Television Engineers, Vol. 74, No. 1, pp. 87-92 (2020), JPEG Lightweight Image Compression Technology Summary of the Invention [Problem to be solved by the invention]

[0010] When 2SI is used, the delay in the video division process is small, but the image quality degradation is greater than in the non-division case. On the other hand, when SQD is used, the image quality degradation is smaller than in the non-division case, but the delay in the video division process is greater. Therefore, there is a need for a video division method that can suppress both processing delay and image quality degradation.

[0011] Therefore, the present invention aims to provide a transmitting device, a receiving device, a transmitting program, and a receiving program that can reduce image quality degradation due to light compression processing caused by video division while suppressing processing delays caused by video division processing. [Means for solving the problem]

[0012] [1] In order to solve the above problem, a transmission device according to one aspect of the present invention includes a video acquisition unit that acquires a video stream captured by an imaging device, and a pixel group that includes a plurality of pixels that are consecutive in the vertical and horizontal directions of the video frame. The division divides the video frame at least vertically and horizontally, and among the divided pixel groups, pixels that are not consecutive in the vertical and horizontal directions are A divided video frame having the pixel groups located at at least two or more discrete positions. Generate a compression unit that compresses the divided video frames; and a transmission unit that transmits the compressed divided video frames.

[0013] [2] Also, one aspect of the present invention is that, in the above-mentioned transmitting device, the dividing unit divides the video frame into four divided video frames, and the transmitting unit transmits the divided video frames as a video stream of 2K video.

[0014] [3] Also, one aspect of the present invention is that in the above-mentioned transmitting device, the video stream is either 8K video, 4K video, or 2K video, and the dividing unit divides the video frames into the divided video frames of 4K video if the video frames are 8K video, and then divides them into the divided video frames of 2K video if the video frames are 4K video, and does not divide them if the video frames are 2K video.

[0015] [4] In addition, according to one aspect of the present invention, in the above-mentioned transmission device, the dividing unit divides the video frame vertically. By dividing at least Line up horizontally Divide the pixel group into two groups horizontally. By dividing at least Vertically aligned The image is divided into four pixel groups, and a plurality of the pixel groups are combined to form the divided video frame.

[0016] [5] Furthermore, the transmitting device according to one aspect of the present invention further includes a division line number acquisition unit that acquires a division line number, which is the number of horizontal lines that the pixel group divided by the division unit has, and the division unit divides the video frame horizontally for each of the acquired division line number.

[0017] [6] According to another aspect of the present invention, a receiving device includes a receiving unit that receives a video stream of 2K video transmitted by a transmitting device; an expanding unit that expands the acquired video stream to restore divided video frames; an integrating unit that integrates a plurality of pixel groups included in each of the restored divided video frames into an integrated video frame by integrating the pixel groups into at least two or more discrete positions; and an output unit that outputs the integrated integrated video frame. The divided video frame is a video frame included in the video stream that includes a plurality of pixel groups divided at least vertically and horizontally, and the divided pixel groups include pixel groups that exist in at least two or more discrete positions that are not consecutive in the vertical and horizontal directions. It is something.

[0018] [7] Another aspect of the present invention is a method for controlling a computer to perform a video acquisition step of acquiring a video stream captured by an imaging device, and dividing a video frame of the acquired video stream into pixel groups each including a plurality of pixels that are consecutive in the vertical and horizontal directions of the video frame. The division divides the video frame at least vertically and horizontally, and among the divided pixel groups, pixels that are not consecutive in the vertical and horizontal directions are A divided video frame having the pixel groups located at at least two or more discrete positions. Generate The transmission program executes a division step, a compression step of compressing the divided video frames, and a transmission step of transmitting the compressed divided video frames.

[0019] [8] Also, one aspect of the present invention is a method for causing a computer to execute a receiving step of receiving a video stream of 2K video transmitted by a transmitting device, an expanding step of restoring divided video frames by performing an expanding process on the acquired video stream, an integrating step of integrating a plurality of pixel groups included in each of the restored divided video frames into an integrated video frame by integrating them into at least two or more discrete positions, and an output step of outputting the integrated integrated video frame, The divided video frame is a video frame included in the video stream that includes a plurality of pixel groups divided at least vertically and horizontally, and the divided pixel groups include pixel groups that exist in at least two or more discrete positions that are not consecutive in the vertical and horizontal directions. This is a receiving program. [Effects of the Invention]

[0020] According to the present invention, it is possible to reduce the degradation of image quality caused by light compression processing resulting from video division while suppressing the processing delay due to video division processing. [Brief explanation of the drawings]

[0021] [Figure 1] 1 is a block diagram showing a schematic functional configuration of a program production system according to a first embodiment of the present invention. [Figure 2] FIG. 2 is a block diagram showing an example of a functional configuration of a transmission device according to the embodiment. [Figure 3] FIG. 2 is a block diagram showing an example of a functional configuration of a receiving device according to the embodiment. [Figure 4] 10A and 10B are diagrams illustrating an example in which a dividing unit according to the embodiment divides a video frame of 4K video into video frames of 2K video. [Figure 5] 10A and 10B are diagrams illustrating an example in which the dividing unit according to the embodiment divides a video frame of an 8K video into a video frame of a 4K video. [Figure 6] 10 is a diagram illustrating a buffer required to operate compression processing resources in the embodiment. FIG. [Figure 7] 10 is a flowchart showing a series of operations of the transmitting device according to the embodiment. [Figure 8] 10 is a flowchart showing a series of operations of the receiving device according to the embodiment. [Figure 9] FIG. 2 is a diagram showing the relationship between compression ratio and PSNR according to the embodiment. [Figure 10] FIG. 10 is a diagram showing the amount of delay according to the embodiment. [Figure 11] FIG. 10 is a diagram illustrating an example in which a dividing unit according to the second embodiment of the present invention divides a video frame of a 4K video into video frames of a 2K video. [Figure 12] FIG. 2 is a block diagram showing an example of a functional configuration of a transmission device according to the embodiment. [Figure 13] FIG. 10 is a diagram showing the delay amount for each divided line according to the embodiment. [Figure 14] FIG. 1 is a diagram for explaining image quality degradation caused by conventional technology. [Figure 15] FIG. 10 is a diagram for explaining delays in video division processing according to the prior art. [Figure 16] FIG. 10 is a diagram showing the relationship between the compression ratio and PSNR when using 2SI and SQD according to the conventional technology. [Figure 17] FIG. 10 is a diagram showing the amount of delay when 2SI and SQD are used according to the conventional technology. DETAILED DESCRIPTION OF THE INVENTION

[0022] [Prior art] First, problems that arise when video is divided using the conventional techniques of 2SI (2-Sample Interleave Division) or SQD (Square Division) will be described with reference to FIGS.

[0023] 14A and 14B are diagrams for explaining image quality degradation caused by conventional techniques. An example of dividing an 8K video frame into four 4K video frames using 2SI will be described with reference to Fig. 14A, and an example of dividing an 8K video frame into four 4K video frames using SQD will be described with reference to Fig. 14B.

[0024] 14(A) and 14(B) both show an 8K video frame consisting of 4320 pixels (vertical) x 7680 pixels (horizontal). Each pixel is assigned a number from 1 to 4. The number assigned to each pixel indicates which of the four 4K video frames the pixel is divided into.

[0025] First, an example of 2SI will be described with reference to Fig. 14(A). 2SI maps pixels in an 8K video frame before division in the horizontal direction, two pixels at a time, to generate a 4K video frame after division. In this process, pixels in one column are alternately mapped in two-pixel increments onto two video frames. Note that even-numbered columns and odd-numbered columns are mapped onto different video frames.

[0026] With 2SI, because every two pixels are divided into different video frames, the proportion of high-frequency components in the spatial frequency components of the divided 4K video frames increases compared to the proportion of high-frequency components in the spatial frequency characteristics of the 8K video frames before division. Therefore, when the divided 4K video frames are compressed and expanded using a specified method, there is a problem that the image quality deteriorates.

[0027] In the above example, an example of dividing an 8K video frame into four 4K video frames was described, but the same problem occurs when dividing a 4K video frame into four 2K video frames.

[0028] Next, an example of SQD will be described with reference to Fig. 14(B). SQD divides a video frame before division into a square-shaped pattern to generate divided video frames. When dividing an 8K video frame into 16 2K video frames, the 8K video frame is first divided into four 4K video frames in a square-shaped pattern, and each of the divided 4K video frames is then further divided into four 2K video frames in a square-shaped pattern.

[0029] With SQD, continuous pixel information is maintained even after division, so it is possible to compress and expand while preserving spatial frequency components. In other words, SQD can suppress image quality degradation compared to 2SI.

[0030] Fig. 15 is a diagram for explaining delays in video division processing according to conventional technology. With reference to Fig. 15(A), an example will be described in which an 8K video frame is divided into four 4K video frames using 2SI, and each of the divided 4K video frames is further divided into four (i.e., 16) 2K video frames, and with reference to Fig. 15(B), an example will be described in which an 8K video frame is divided into four 4K video frames using SQD, and each of the divided 4K video frames is further divided into four (i.e., 16) 2K video frames.

[0031] 15(A) and 15(B) both show an 8K video frame consisting of 4320 [px] vertically and 7680 [px] horizontally. Each pixel is assigned a number nm (n and m are natural numbers from 1 to 4). Of the numbers assigned to each pixel, n indicates which of the four 4K video frames into which the 8K video frame is divided. Furthermore, m indicates which of the four 2K video frames into which the 4K video frame specified by n is divided. In other words, if a pixel is assigned "2-3," it indicates that it will be divided into a second 4K video frame, which will then be divided into a third 2K video frame.

[0032] Here, the necessary and appropriate functional requirement for compression processing resources to compress 2K frame rate 60p video is the ability to process 1920 x 1080 [px] video frames in 1 / 60 [seconds]. If there are insufficient compression processing resources, the 60p frame rate cannot be maintained. Conversely, if there are too many compression processing resources, the 60p frame rate can be maintained, but high-speed operation and standby will be repeated, resulting in wait time. It is difficult to allocate this wait time to other resources, and as a result, excessive resources will be allocated. In other words, the performance of compression resources means that, for example, a 2K60p compression processing resource will allocate appropriate resources while maintaining the 60p frame rate for 2K video.

[0033] Specifically, the processing performance of a compression processing resource for processing 2K60p, 4:2:2, 10-bit video is expressed as a data processing capacity of 1920 x 1080 x 20 x 60, which is approximately 2.5 Gbps. In the case of 8K, the compression processing resource required to process 8K is 40 Gbps. If 8K is divided into 16 2K images and processed, the overall compression processing resource will still be 40 Gbps, so 16 2K compression processing resources with a performance of 2.5 Gbps each must be operated in parallel to achieve a total pseudo performance of 40 Gbps. Similarly, in the case of 4K, four 2K compression processing resources with a performance of 2.5 Gbps each must be operated in parallel to achieve a total performance of 10 Gbps.

[0034] Therefore, buffering is required to divide the video and perform video compression processing, and the time required for buffering is the delay time in the video division processing. In the following explanation, the time required for buffering (i.e., the time until all compression processing resources start operating) may be simply referred to as delay time.

[0035] First, an example of 2SI will be described with reference to Figure 15(A). According to 2SI, video signals are output line by line on one line of a video frame from left to right and from top to bottom, so processing of the 16th 2K video frame begins only after processing has progressed 4 [px] in the vertical direction and 7 [px] in the horizontal direction. In other words, all resources are idle while processing the pixels indicated by the thick lines in the figure.

[0036] Next, an example of SQD will be described with reference to Figure 15(B). With SQD, video signals are output line by line on one line of a video frame from left to right and from top to bottom, so processing of the 16th 2K video frame only begins when processing has progressed 3241 [px] vertically and 5761 [px] horizontally. In other words, all resources are not operating while processing the pixels indicated by the thick lines in the figure. Therefore, SQD has the problem of larger processing delays compared to 2SI.

[0037] Figure 16 shows the relationship between compression ratio and PSNR when using 2SI and SQD according to conventional technologies. The figure shows the results of a simulation evaluation of image quality degradation when using 2SI or SQD, using PSNR (Peak Signal-to-Noise Ratio). Specifically, an 8K video frame is divided into 16 2K video frames using 2SI or SQD, then compressed and decompressed using JPEG XS, a light compression technology, and the decompressed divided video frames are combined together to form a video frame. The figure shows the results of an evaluation using PSNR of the original 8K video frame. Figures 16(B) and 16(D) show tables showing the measurement results, and Figures 16(A) and 16(C) show tables plotting the measurement results. Each figure also shows the results when the video was compressed and decompressed using JPEG XS without dividing it, in order to evaluate the degree of image quality degradation.

[0038] The examples in Figures 16(A) and 16(B) are examples of when one video frame from "No. 1 Water polo (goal)" from the Ultra High Definition Wide Color Gamut Standard Video B Series provided by the Institute of Image Information and Television Engineers (ITE) was used as the evaluation video (hereinafter referred to as evaluation video 1). The examples in Figures 16(C) and 16(D) are examples of when one video frame from "No. 3 Horse race (dirt)" from the same series was used as the evaluation video (hereinafter referred to as evaluation video 2). Comparing the spatial frequency components of evaluation video 1 and evaluation video 2, evaluation video 2 has more high spatial frequency components.

[0039] In the case of 2SI, the image quality of evaluation video 1 deteriorates by approximately 1.5 to 1.7 dB (decibels) compared to when the video was not divided. Also, the image quality of evaluation video 2 deteriorates by approximately 4.0 to 4.2 dB compared to when the video was not divided. In other words, with 2SI, the image quality degradation of the video compression / expansion process due to video division is significant. It should be noted that evaluation video 2 contains more high spatial frequency components than evaluation video 1, and is therefore considered to have suffered from greater image quality degradation.

[0040] On the other hand, SQD suppresses the degradation of image quality to a maximum of approximately 0.2 dB compared to when no division is performed, for both evaluation video 1 and evaluation video 2. In other words, with SQD, the degradation of image quality due to the video compression / expansion process caused by video division is small.

[0041] Figure 17 shows the amount of delay when using 2SI and SQD according to conventional technology. The figure shows the processing delay caused by the division process of 2SI and SQD. For both 2SI and SQD, the figure shows the processing delay when dividing 4K video into 2K video and the processing delay when dividing 8K video into 2K video.

[0042] According to 2SI, the processing delay is 15.4 μs (microseconds) for 4K video and 15.4 μs for 8K video, which are sufficiently small. On the other hand, according to SQD, the processing delay is 8.3 ms (milliseconds) for 4K video and 12.5 ms for 8K video, which are larger than 2SI. The delay required for IP remote production is 16.6 ms, so taking into account other transmission delays such as the fiber distance required for transmission, the processing delay caused by SQD is too large, which is an issue.

[0043] Therefore, an object of the present invention is to reduce the degradation in image quality that occurs when 2SI is used, while suppressing the processing delay that occurs when SQD is used.

[0044] [Embodiment] First, the prerequisites for the embodiment will be described.

[0045] A program production system 100 according to this embodiment transmits program materials such as video and audio necessary for program production, which have been filmed and recorded at a filming location 2, to a program production location 3. The program production system 100 includes a video transmission system 1 and an audio transmission system 4. It is assumed that the video materials necessary for program production are a mixture of 8K, 4K, and 2K video materials.

[0046] The filming location 2 is equipped with dedicated cameras for shooting 8K, 4K, and 2K video materials. The video transmission system 1 divides the video materials shot in 8K into 16 2K video streams and the video materials shot in 4K into four 2K video streams. The video transmission system 1 does not divide the video materials shot in 2K.

[0047] The video transmission system 1 compresses the video stream unified into a 2K video stream and transmits it to the program production site 3. Here, since the video stream compressed by the video transmission system 1 is unified into 2K, it can be compressed using compression resources for 2K. In other words, the video transmission system 1 does not need to be provided with compression resources for 8K and compression resources for 4K.

[0048] The compression technology used here is a so-called light compression method, which allows for low latency and minimal degradation of image quality. Unlike codecs with high compression rates such as HEVC (High Efficiency Video Coding), this compression method achieves line-level low latency by keeping the compression rate to around 1 / 4 to 1 / 10. Specific methods that are used include LLVC (Low Latency Video Codec), VC2, TICO (Tiny Codec), and JPEG XS (XS stands for eXtra Speed ​​eXtra Small).

[0049] Of these methods, only JPEG XS has been internationally standardized. Packetization of compressed signals in IP production systems using JPEG XS is clarified in ISO / IEC 21122. In this embodiment, an example of compression using JPEG XS will be described.

[0050] The video transmission system 1 lightly compresses the 2K video stream using JPEG XS, then converts it into IP packets and Ethernet frames, and transmits it to the program production site 3. The program production site 3 performs IP depacketization and Ethernet deframing, decodes the video, and then combines the divided images to output the video. At the program production site 3, programs are produced using the integrated 8K, 4K video material, or 2K video material.

[0051] [First embodiment] A first embodiment of the present invention will be described below with reference to the drawings.

[0052] [Video transmission system functional configuration] 1 is a block diagram showing the schematic functional configuration of a program production system according to a first embodiment of the present invention. The schematic functional configuration of the program production system 100 will be described with reference to the figure. The program production system 100 includes a plurality of imaging devices (cameras) 10, a plurality of sound collection devices (microphones) 11, a video transmission system 1, an audio transmission system 4, and a program editing system 38. The video transmission system 1 includes a transmitting device 20 and a network switch 27 at a filming location 2. The video transmission system 1 also includes a network switch 37 and a receiving device 30 at a program production location 3. The audio transmission system 4 includes an IP gateway 26 and a network switch 27 at the filming location 2. The audio transmission system 4 also includes a network switch 37 and an IP gateway 36 at the program production location 3.

[0053] Imaging device 10-1, imaging device 10-2, and imaging device 10-3 are connected to video transmission system 1 as multiple imaging devices 10. Imaging device 10-1 captures a 2K video stream, imaging device 10-2 captures a 4K video stream, and imaging device 10-3 captures an 8K video stream. Furthermore, sound collection device 11-1, sound collection device 11-2, and sound collection device 11-3 are connected to the audio transmission system 4 as a plurality of sound collection devices 11. Sound collection device 11-1, sound collection device 11-2, and sound collection device 11-3 collect audio information corresponding to imaging device 10-1, imaging device 10-2, and imaging device 10-3, respectively, and record the collected audio information by a predetermined method.

[0054] The transmitting device 20 divides the 8K and 4K video streams captured by the imaging device 10 into 2K video streams and compresses the 2K video streams directly using a light compression method. The transmitting device 20 also converts the compressed video streams into IP packets and Ethernet frames. The transmitting device 20 transmits the IP packetized and Ethernet framed video streams to the program production site 3 via the network switch 27. The detailed functional configuration of the transmitting device 20 will be described later.

[0055] The IP gateway 26 converts the audio data recorded by the sound collection device 11 into IP packets and Ethernet frames. The IP gateway 26 transmits the IP packetized and Ethernet framed audio data to the program production site 3 via the network switch 27.

[0056] The network switch 27 and the network switch 37 are devices that constitute the video transmission system 1 and the audio transmission system 4, and transfer IP packets. The network switch 27 and the network switch 37 each have a plurality of input ports and a plurality of output ports. Specifically, network switch 27 receives IP packets sent from transmitting device 20 and IP packets sent from IP gateway 26, and transfers them to network switch 37 installed at program production site 3. Network switch 27 references the header of the IP packet and outputs the received packet to an output port corresponding to the destination.

[0057] The network switch 37 receives the IP packets transmitted by the network switch 27. The network switch 37 outputs the received IP packets to the receiving device 30 and the IP gateway .

[0058] The receiving device 30 performs Ethernet deframing and IP depacketizing on the acquired information. The receiving device 30 temporarily decompresses the video stream obtained by IP depacketization, which is the video stream compressed by the transmitting device 20, into a 2K video stream, and integrates the decompressed video stream corresponding to the 2K video stream generated by dividing the 8K or 4K video stream by the transmitting device 20 into the 8K or 4K video stream. The receiving device 30 also outputs the integrated video stream to the program editing system 38. A detailed functional configuration of the receiving device 30 will be described later.

[0059] The IP gateway 36 performs Ethernet deframing and IP depacketization on the acquired information, and outputs the audio data obtained by IP depacketization to the program editing system 38.

[0060] The program editing system 38 produces a program based on the video stream and audio data transmitted from the shooting location 2. Specifically, the program editing system 38 includes a display device 40, a video switching device 381, a video editing device 382, ​​and an audio synthesizing device 383.

[0061] The display device 40 displays the 8K, 4K, or 2K video stream received from the receiving device 30. The video switching device 381 switches which video of the video materials displayed on the display device 40 is to be used for the program. The video editing device 382 edits the video materials received from the receiving device 30. The audio synthesizer 383 synthesizes the audio data received from the IP gateway 36 with the video materials.

[0062] [Functional configuration of the transmitter] 2 is a block diagram showing an example of the functional configuration of a transmission device according to the embodiment. An example of the functional configuration of the transmission device 20 will be described with reference to the figure. The transmission device 20 includes a video acquisition unit 210, a division unit 220, a compression unit 230, and a transmission buffer (transmission unit) 240.

[0063] The video acquisition unit 210 acquires a video stream VS captured by the imaging device 10. The video acquisition unit 210 acquires an 8K, 4K, or 2K video stream VS.

[0064] The dividing unit 220 divides the video frames VF of the video stream VS acquired by the video acquisition unit 210 into divided video frames DVF. Specifically, if the video frames VF of the acquired video stream VS are 8K, the dividing unit 220 divides them into 4K divided video frames DVF and then into 2K divided video frames DVF. Also, if the video frames of the acquired video stream are 4K, the dividing unit 220 divides them into 2K divided video frames DVF. In other words, the dividing unit 220 divides the video frame VF into four divided video frames DVF. It should be noted that the dividing unit 220 does not divide the acquired video stream VS if the video frames VF are 2K.

[0065] The compression unit 230 compresses the divided video frames DVF divided into 2K video frames VF by the division unit 220. The compression unit 230 compresses the divided video frames DVF by light compression processing.

[0066] The transmission buffer 240 transmits the compressed video stream CVS, which is the divided video frames DVF compressed by the compression unit 230. Since the compressed video stream CVS has been divided into 2K video streams by the division unit 220, the transmission buffer 240 transmits the divided video frames DVF as a video stream of 2K video.

[0067] [Functional configuration of receiving device] 3 is a block diagram showing an example of the functional configuration of a receiving device according to the embodiment. An example of the functional configuration of the receiving device 30 will be described with reference to the figure. The receiving device 30 includes a receiving buffer (receiving unit) 310, a decompression unit 320, a integrating unit 330, and an output unit 340.

[0068] The receiving buffer 310 receives the compressed video stream CVS of 2K video transmitted by the transmitting device 20.

[0069] The decompression unit 320 performs decompression processing on the compressed video stream CVS acquired by the receiving buffer 310 to restore the divided video frames DVF.

[0070] The integrating unit 330 integrates the restored divided video frames DVF into an integrated video frame IVF. Specifically, the integrating unit 330 integrates four divided video frames DVF into a 4K video frame, and integrates 16 divided video frames DVF into an 8K video frame.

[0071] The output unit 340 outputs the integrated video frame IVF to the display device 40.

[0072] [Video division by MCD] 4 is a diagram illustrating an example in which a dividing unit according to the embodiment divides a 4K video frame into 2K video frames. The video dividing method in this embodiment is referred to as MCD (Multi-Cross Division). The video dividing method using the MCD method will be described with reference to the diagram. An example of dividing a 4K video frame into 2K video frames using MCD will be described with reference to the same figure. Figure 4(A) shows the 4K video frame before division, and Figure 4(B) shows the 2K video frame after division.

[0073] The MCD method generates divided video frames by dividing the left and right halves of a video frame into specific dividing lines starting from the top. The figure shows an example of dividing every 540 lines. In the following explanation, multiple pixels that are consecutive in the horizontal and vertical directions will be referred to as a pixel group. Pixel group 1 includes multiple pixels that are consecutive in both the vertical and horizontal directions of the video frame. In the figure, each pixel group is numbered 1 to 8. Each pixel group may be referred to as pixel group 1 to pixel group 8.

[0074] Looking at the left half of the video frame, lines 1 to 540 (pixel group 1) are assigned to the first divided video frame, lines 541 to 1080 (pixel group 3) are assigned to the third divided video frame, lines 1081 to 1620 (pixel group 5) are assigned to the first divided video frame, and lines 1621 to 2160 (pixel group 7) are assigned to the third divided video frame.

[0075] Similarly, if we look at the right half of the video frame, lines 1 to 540 (pixel group 2) are assigned to the second divided video frame, lines 541 to 1080 (pixel group 4) are assigned to the fourth divided video frame, lines 1081 to 1620 (pixel group 6) are assigned to the second divided video frame, and lines 1621 to 2160 (pixel group 8) are assigned to the fourth divided video frame.

[0076] Here, the number of horizontal pixels in each pixel group is half the number of horizontal pixels in the video frame, and the number of vertical pixels in each pixel group can be any number, but is preferably 135 lines or more (for example, 135 lines, 270 lines, or 540 lines).

[0077] A divided video frame DVF has pixel groups located at at least two or more discrete positions. For example, as shown in Figure 4(B), a case will be described in which a 4K video frame VF is divided into four 2K divided video frames DVF, divided video frames DVF1 to DVF4. In this case, divided video frame DVF1 has pixel groups located at two discrete positions, such as pixel group 1 and pixel group 5, divided video frame DVF3 has pixel group 3 and pixel group 7, divided video frame DVF2 has pixel group 2 and pixel group 6, and divided video frame DVF4 has pixel group 4 and pixel group 8.

[0078] That is, the division unit 220 divides the video frame VF into divided video frames DVF, each having a pixel group including a plurality of pixels that are consecutive in both the vertical and horizontal directions of the video frame VF, and that are present at at least two or more discrete positions. Furthermore, the integrating unit 330 integrates a plurality of pixel groups included in each of the plurality of divided video frames DVF into at least two or more discrete positions, thereby integrating them into an integrated video frame IVF.

[0079] More specifically, the dividing unit 220 divides the video frame VF vertically into at least two pixel groups. The dividing unit 220 also divides the video frame VF horizontally into at least four pixel groups. The dividing unit 220 combines the divided pixel groups to form a divided video frame DVF.

[0080] 5 is a diagram illustrating an example of a case where a dividing unit according to the embodiment divides an 8K video frame into 4K video frames. With reference to the diagram, an example of a case where an 8K video frame is divided into 4K video frames by MCD will be described. Fig. 5(A) shows the 8K video frame before division, and Fig. 5(B) shows the 4K video frame after division.

[0081] When dividing an 8K video frame into 2K video frames, the dividing unit 220 first divides it into four using the same process as for 4K. Next, the dividing unit 220 divides the divided 4K video frames again into 2K video frames using the same process as for 4K. In the description given with reference to FIG. 5, an example will be described in which an 8K video frame is divided into 4K video frames.

[0082] Looking at the left half of the video frame, lines 1 to 1080 (pixel group 1) are assigned to the first divided video frame, lines 1081 to 2160 (pixel group 3) are assigned to the third divided video frame, lines 2161 to 3240 (pixel group 5) are assigned to the first divided video frame, and lines 3241 to 4320 (pixel group 7) are assigned to the third divided video frame.

[0083] Similarly, if we look at the right half of the video frame, lines 1 to 1080 (pixel group 2) are assigned to the second divided video frame, lines 1081 to 2160 (pixel group 4) are assigned to the fourth divided video frame, lines 2161 to 3240 (pixel group 6) are assigned to the second divided video frame, and lines 3241 to 4320 (pixel group 8) are assigned to the fourth divided video frame.

[0084] Here, the number of horizontal pixels in each pixel group is half the number of horizontal pixels in the video frame, and the number of vertical pixels in each pixel group can be any number, but is preferably 270 lines or more (for example, 270 lines, 540 lines, or 1080 lines).

[0085] 6 is a diagram illustrating a buffer required to operate the compression processing resource in the embodiment. With reference to the diagram, an example will be described in which an 8K video frame is divided into four 4K video frames using MCD, and each of the divided 4K video frames is further divided into four (i.e., 16) 2K video frames. The figure shows an 8K video frame consisting of 4320 [px] vertically and 7680 [px] horizontally. Each pixel is assigned a number nm (n and m are natural numbers from 1 to 4). Of the numbers assigned to each pixel, n indicates which of the four 4K video frames it will be divided into. Also, m indicates which of the four 2K video frames it will be divided into. In other words, if a pixel is assigned "2-3," it indicates that it will be divided into a second 4K video frame, which will then be divided into a third 2K video frame.

[0086] Each divided 2K video frame has a one-to-one correspondence with a video compression resource, i.e., the first video compression resource corresponds to the first 2K video frame, ..., the mth video compression resource corresponds to the mth 2K video frame.

[0087] The buffering process may be performed on either the transmitting device 20 side or the receiving device 30 side. In other words, if the buffering process is performed on the transmitting device 20 side, the buffering process does not need to be performed on the receiving device 30 side, and conversely, if the buffering process is not performed on the transmitting device 20 side, the buffering process needs to be performed on the receiving device 30 side. For example, the compression unit 230 included in the transmission device 20 may perform the buffering process.

[0088] According to MCD, when processing proceeds horizontally, processing of the 16th 2K video frame begins only after processing has proceeded 1621 pixels vertically and 5761 pixels horizontally. In other words, all resources can begin operating when processing of the pixels indicated by the bold lines in the diagram is complete.

[0089] [Video transmission system operation] 7 is a flowchart showing a series of operations of the transmitting device according to the embodiment, and the series of operations of the transmitting device 20 will be described with reference to the same figure. (Step S110) The video acquisition unit 210 acquires the video stream VS captured by the imaging device 10. (Step S120) The division unit 220 divides the video frames VF of the acquired video stream VS by MCD. At this time, the division unit 220 divides the video frames VF into 2K video by dividing them once if the acquired video stream VS is 4K video, or by dividing them into four repeatedly if the acquired video stream VS is 8K video. (Step S130) The compression unit 230 compresses the divided video frames DVF divided into 2K videos. (Step S140) The transmission buffer 240 transmits the compressed video stream CVS via the network switch 27.

[0090] 8 is a flowchart showing a series of operations of the receiving device according to the embodiment, and the series of operations of the receiving device 30 will be described with reference to the same drawing. (Step S210) The receiving buffer 310 acquires the compressed video stream CVS of 2K video transmitted by the transmitting device 20. (Step S220) The decompression unit 320 decompresses the acquired compressed video stream CVS of 2K video to restore the divided video frames DVF. (Step S230) The integrator 330 integrates the divided video streams DVF into a 4K or 8K integrated video frame IVF. (Step S240) The output unit 340 outputs the integrated video frame IVF.

[0091] [Effects of video transmission system] 9 is a diagram showing the relationship between the compression ratio and PSNR according to the embodiment. The diagram shows the results of measuring the degradation in image quality using PSNR when an 8K video frame is divided into 16 2K video frames using MCD, then compressed and decompressed using JPEG XS, and the decompressed divided images are integrated to restore the 8K video frame. Figures 9(B) and 9(D) show tables showing the measurement results, and Figures 9(A) and 9(C) show tables plotting the measurement results. In order to evaluate the degree of image quality degradation, each figure also shows the results when the image was not divided, and when 2SI and SQD were used.

[0092] The examples of Figures 9(A) and 9(B) are examples when the evaluation video 1 shown in Figure 16 is used. The examples of Figures 9(C) and 9(D) are examples when the evaluation video 2 shown in Figure 16 is used. The maximum degradation in image quality for both video frames was approximately 0.2 dB, which is significantly less than that of 2SI. It can also be seen that the degradation is kept to the same level as that of SQD.

[0093] 10 is a diagram showing the amount of delay according to the embodiment. The figure shows the processing delay caused by the division process of MCD. The figure also shows the processing delay when 4K video is divided into 2K video and when 8K video is divided into 2K video. In order to evaluate the degree of the amount of delay, the results when 2SI and SQD are used are also shown.

[0094] According to MCD, the processing delay is 4.2 ms for 4K and 6.2 ms for 8K. Although the processing delay is larger than in the case of 2SI, it can be seen that the processing delay is reduced to about half of that in the case of SQD. When both image quality degradation and processing delay are taken into consideration, MCD can suppress image quality degradation to the same level as SQD, while suppressing processing delay to about half that of SQD. Therefore, MCD can suppress the image quality degradation that was a drawback of 2SI, and also improve the processing delay that was a drawback of SQD.

[0095] [Summary of the first embodiment] According to the embodiment described above, the transmitting device 20 is provided with a video acquisition unit 210 to acquire the video stream VS captured by the imaging device 10, a division unit 220 to divide 8K or 4K video frames VF into 2K video frames using MCD, a compression unit 230 to compress the divided video frames DVF, and a transmission buffer 240 to output a compressed video stream CVS. According to MCD, the divided video frame DVF divided by the dividing unit 220 has at least two or more groups of pixels located at discrete positions, where each group of pixels includes a plurality of pixels that are consecutive in both the vertical and horizontal directions of the video frame VF.

[0096] Therefore, according to this embodiment, since continuous pixel information is maintained even after division, it is possible to compress while maintaining spatial frequency components, which means that deterioration of image quality can be suppressed. Furthermore, in this embodiment, compared to SQD, which divides the video frame before division into four parts in a square grid pattern, the video frame is divided into smaller parts, so all resources start operating earlier, thereby reducing processing delays.

[0097] Furthermore, according to the embodiment described above, the division unit 220 divides a video frame VF into four divided video frames DVF, and the transmission buffer 240 outputs the divided video frames DVF as a 2K compressed video stream CVS. Therefore, according to this embodiment, the video is unified into a 2K video stream VS for transmission between the filming location 2 and the program production location 3, eliminating the need for video compression transmission devices dedicated to the 2K, 4K, and 8K formats. Therefore, the transmitting device 20 and receiving device 30 according to this embodiment can be flexibly operated even at various locations using different formats, without requiring video compression transmission devices dedicated to the 2K, 4K, and 8K formats. This allows for reduced costs and a more compact device.

[0098] Furthermore, according to the embodiment described above, the video stream VS acquired by the video acquisition unit 210 is either 8K video, 4K video, or 2K video. Furthermore, the division unit 220 divides the 8K video into four 4K videos, and then further divides it into four 2K videos. Therefore, according to this embodiment, the same algorithm can be used whether converting 8K video into 4K video or dividing 4K video into 2K video. Therefore, according to this embodiment, the algorithm can be simplified.

[0099] Furthermore, according to the embodiment described above, the division unit 220 divides the video frame VF vertically into at least two pixel groups and horizontally into at least four pixel groups, and combines the pixel groups to form the divided video frame DVF. Therefore, according to this embodiment, information about consecutive pixels can be maintained, and therefore division can be performed without losing information about spatial frequency components. In other words, degradation of image quality can be suppressed.

[0100] [Second embodiment] Next, a second embodiment will be described. The second embodiment differs from the first embodiment in that the number of dividing lines is variable. The number of dividing lines refers to the number of lines divided horizontally by the MCD, in other words, the number of pixels in the vertical direction that the pixel group has. For example, in the first embodiment described above, the dividing line is fixed at 540 lines for a 4K video frame, and at 1080 lines for an 8K video frame. According to the second embodiment, for example, the dividing line may be 135, 270, or 540 lines for a 4K video frame, or 270, 540, or 1080 lines for an 8K video frame.

[0101] 11 is a diagram illustrating an example in which a division unit according to the second embodiment of the present invention divides a 4K video frame into 2K video frames. The video division method using the MCD method according to the second embodiment will be described with reference to the same figure. An example of dividing a 4K video frame into 2K video frames by MCD in the second embodiment will be described with reference to the same figure. Fig. 11(A) shows the 4K video frame before division, and Fig. 11(B) shows the 2K video frame after division. Although a description of the case where an 8K video frame is divided into 4K video frames will be omitted, an algorithm similar to that used for dividing a 4K video frame into 2K video frames may be used.

[0102] The MCD method in the second embodiment is similar to the MCD method in the first embodiment in that it divides the video frame into left and right halves. The MCD method in the second embodiment differs from the first embodiment in that the number of dividing lines when dividing from the top down is variable. The figure shows an example of dividing every 270 lines.

[0103] As shown in the figure, the 4K video frame before division is divided into 16 pixel groups, and the 2K video frame after division is composed of 4 pixel groups each. Specifically, of the 4K video frame before division, the left half is divided into odd-numbered pixel groups, and the right half is divided into even-numbered pixel groups. Split video frame DVF1 has pixel group 1, pixel group 5, pixel group 9, and pixel group 13; split video frame DVF3 has pixel group 3, pixel group 7, pixel group 11, and pixel group 15; split video frame DVF2 has pixel group 2, pixel group 6, pixel group 10, and pixel group 14; and split video frame DVF4 has pixel group 4, pixel group 8, pixel group 12, and pixel group 16.

[0104] For example, as the number of divided lines is reduced from 540 to 270 to 135, the size of the pixel group becomes smaller. Therefore, the smaller the number of divided lines, the shorter the delay time. On the other hand, the smaller the number of divided lines, the more consecutive pixel information is lost, resulting in greater degradation of image quality.

[0105] In this way, varying the number of divided lines results in conflicting results in processing delay and image quality degradation. In the second embodiment, the number of divided lines can be varied. The number of divided lines may be determined, for example, by a user setting.

[0106] 12 is a block diagram showing an example of the functional configuration of a transmission device according to the embodiment. An example of the functional configuration of the transmission device 20A will be described with reference to the same figure. The transmission device 20A differs from the transmission device 20 in that it newly includes a divided line number acquisition unit 250 and includes a division unit 220A instead of the division unit 220.

[0107] The divided line number acquisition unit 250 acquires the divided line number DL. The divided line number DL is the number of horizontal lines in the pixel group divided by the division unit 220. The divided line number DL may be stored in a storage unit (not shown), or may be acquired based on a user operation. The divided line number DL may also be automatically determined depending on the size of the video stream VS, etc.

[0108] The division unit 220A acquires the video stream VS from the video acquisition unit 210, and acquires the number of divided lines DL from the number of divided lines acquisition unit 250. The division unit 220A divides the video frame VF of the acquired video stream VS by MCD in accordance with the acquired number of divided lines DL. Specifically, the division unit 220A divides the video frame VF horizontally for each acquired number of divided lines DL.

[0109] 13 is a diagram showing the amount of delay for each divided line according to the embodiment. The figure shows the processing delay caused by the division process of MCD according to the second embodiment. The figure also shows the processing delay when the number of divided lines DL is varied. Specifically, the figure shows the processing delay when the number of divided lines DL is set to 135 lines, 270 lines, and 540 lines for 4K video. The figure also shows the processing delay when the number of divided lines DL is set to 270 lines, 540 lines, and 1080 lines for 8K video. In the process of dividing the 4K video into 2K video after dividing the 8K video into 4K video, half the number of divided lines DL used when dividing the 8K video into 4K video is used.

[0110] In the case of 4K, the time was 1.0 [ms] when the number of divided lines DL was 135 lines, 2.1 [ms] when it was 270 lines, and 4.2 [ms] when it was 540 lines. In the case of 8K, when the number of divided lines DL is 270 lines from 8K to 4K and 135 lines from 4K to 2K, it was 1.6 [ms], when it is 540 lines from 8K to 4K and 270 lines from 4K to 2K, it was 3.1 [ms], and when it is 1080 lines from 8K to 4K and 540 lines from 4K to 2K, it was 6.2 [ms]. Although the processing delay time depends on the number of divided lines DL, it can be seen that the delay is reduced compared to SQD.

[0111] [Summary of the second embodiment] According to the embodiment described above, the transmission device 20A acquires the number of divided lines DL by including the divided line number acquisition unit 250, and the division unit 220A divides the video frame VF horizontally for each acquired number of divided lines DL. Therefore, according to this embodiment, the number of divided lines DL can be varied. Here, varying the number of divided lines DL results in conflicting results in terms of processing delay and image quality degradation. Therefore, according to this embodiment, it is possible to arbitrarily decide whether to prioritize reducing processing delay or reducing image quality degradation.

[0112] Note that all or part of the functions of each unit of the video transmission system 1 in the above-described embodiment may be realized by recording a program for realizing these functions on a computer-readable recording medium, and reading and executing the program recorded on the recording medium into a computer system. Note that the term "computer system" here includes hardware such as an OS and peripheral devices.

[0113] Furthermore, "computer-readable recording media" refers to portable media such as flexible disks, optical magnetic disks, ROMs, and CD-ROMs, as well as storage units such as hard disks built into computer systems. Furthermore, "computer-readable recording media" may also include devices that dynamically store programs for a short period of time, such as communication lines when transmitting programs over networks like the Internet or communication lines like telephone lines, or devices that store programs for a fixed period of time, such as volatile memory within computer systems that serve as servers or clients in such cases. Furthermore, the above-mentioned programs may be programs that realize some of the aforementioned functions, or may be programs that can realize the aforementioned functions in combination with programs already stored in the computer system.

[0114] Furthermore, the effects described in this specification are merely descriptive or exemplary and are not limiting. In other words, the technology according to the present disclosure may achieve other effects in addition to or in place of the above-described effects that would be apparent to those skilled in the art from the description of this specification. Furthermore, the present invention is not limited to these embodiments, and various modifications and substitutions can be made within the scope of the present invention. [Explanation of symbols]

[0115] 1. Video transmission system 2. Filming location 3 Program production site 4. Audio Transmission System 10. Imaging device 11 Sound collection device 20 Transmitting device 26 IP Gateways 27 Network Switch 30 Receiving device 36 IP Gateways 37 Network Switch 38 Program Editing System 381 Video Switching Device 382 Video editing equipment 383 Voice Synthesizer 40 Display device 210 Video acquisition unit 220 Split section 230 Compression section 240 send buffers 250 division line number acquisition section 310 Receive Buffer 320 Extension part 330 Integration Department 340 Output Section 100 Program Production System

Claims

1. a video acquisition unit that acquires a video stream captured by an imaging device; a dividing unit that divides the acquired video frames of the video stream into pixel groups each including a plurality of pixels that are consecutive in both the vertical and horizontal directions of the video frame, the dividing dividing the video frame at least in the vertical and horizontal directions, and generates divided video frames each including pixel groups that exist at at least two or more discrete positions that are not consecutive in both the vertical and horizontal directions among the divided pixel groups; a compression unit that compresses the divided video frames; a transmitter for transmitting the compressed divided video frames; A transmitting device comprising:

2. the dividing unit divides the video frame into four divided video frames; The transmission unit transmits the divided video frames as a video stream of 2K video. The transmitting device according to claim 1 .

3. The video stream is either 8K video, 4K video, or 2K video, The division unit divides the video frame into the divided video frames of 4K video when the video frame is an 8K video and then divides it into the divided video frames of 2K video when the video frame is a 4K video, and does not divide the video frame when the video frame is a 2K video.

3. The transmitting device according to claim 1 or 2.

4. The dividing unit divides the video frame vertically into at least two horizontally arranged pixel groups, divides the video frame horizontally into at least four vertically arranged pixel groups, and combines a plurality of the pixel groups to form the divided video frame. The transmitting device according to any one of claims 1 to 3.

5. a division line number acquisition unit that acquires a division line number, which is the number of horizontal lines of the pixel group divided by the division unit; The dividing unit divides the video frame horizontally for each of the acquired number of division lines. The transmitting device according to claim 4.

6. a receiving unit that receives a video stream of 2K video transmitted by a transmitting device; an expansion unit that performs expansion processing on the acquired video stream to restore divided video frames; an integration unit that integrates a plurality of pixel groups included in each of the plurality of restored divided video frames into at least two or more discrete positions to integrate the plurality of pixel groups into an integrated video frame; an output unit for outputting the synthesized video frame; Equipped with The divided video frame is a video frame included in the video stream that includes a plurality of pixel groups divided at least vertically and horizontally, and the divided pixel groups include pixel groups that exist in at least two or more discrete positions that are not consecutive in the vertical and horizontal directions. Receiving device.

7. On the computer, a video acquisition step of acquiring a video stream captured by an imaging device; a dividing step of dividing the acquired video frames of the video stream into pixel groups each including a plurality of pixels that are consecutive in both the vertical and horizontal directions of the video frame, the dividing dividing the video frame at least in the vertical and horizontal directions, and generating divided video frames each including the pixel groups that are present at at least two or more discrete positions that are not consecutive in both the vertical and horizontal directions among the divided pixel groups; a compressing step of compressing the divided video frames; a transmitting step of transmitting the compressed divided video frames; The sending program that causes the

8. On the computer, a receiving step of receiving a video stream of 2K video transmitted by a transmitting device; a decompression step of performing a decompression process on the acquired video stream to restore divided video frames; a merging step of merging a plurality of pixel groups included in each of the plurality of reconstructed divided video frames into at least two or more discrete positions to form a merged video frame; an output step of outputting the synthesized video frame; Execute The divided video frame is a video frame included in the video stream that includes a plurality of pixel groups divided at least vertically and horizontally, and the divided pixel groups include pixel groups that exist in at least two or more discrete positions that are not consecutive in the vertical and horizontal directions. Receiving program.

Images