Super-resolution wide video synchronization control system, super-resolution wide video synchronization control method, and program

Abstract

One embodiment of this super-resolution wide video synchronization control system includes a video distributor and a plurality of super-resolution synchronization control units. Each of the plurality of super-resolution synchronization control units includes a video input unit that acquires a frame image from input video, a cutting unit that cuts a sub-frame image from the frame image, an up-conversion unit that performs super-resolution processing on the sub-frame image to generate a super-resolution sub-frame image, an output buffer that stores the super-resolution sub-frame image, an output time calculation unit that calculates an output time for the super-resolution sub-frame image on the basis of a processing duration offset value for the super-resolution processing, and an output control unit that controls output of the super-resolution sub-frame image on the basis of an output schedule to which the output time has been set.

Description

Super-resolution wide-screen video synchronization control system, super-resolution wide-screen video synchronization control method, and program

[0001] The present invention relates to a super-resolution wide-screen video synchronization control system, a super-resolution wide-screen video synchronization control method, and a program.

[0002] One method for presenting high-definition wide-angle images is to extract a horizontally long area from a high-resolution input image and present it as a wide-angle image. For example, in Non-Patent Document 1, a wide-angle image with an aspect ratio of 32:9 (twice the aspect ratio of 4K image with a 16:9 aspect ratio) is generated from an 8K input image.

[0003] Methods for generating high-resolution images include, for example, upscaling by image processing using pixel interpolation techniques such as bilinear interpolation, or super-resolution processing techniques using AI (artificial intelligence) such as DNN (Deep Neural Network) or CNN (Convolutional Neural Network). One super-resolution processing technique using AI is Real-ESRGAN (Real Enhanced Super Resolution Generative Adversarial Networks).

[0004] "Successfully broadcasting an unprecedented next-generation live viewing broadcast using 8K technology in 32:9 widescreen - B-LEAGUE with overwhelming realism in ultra-high definition images," [online], [Retrieved January 16, 2024], Internet<URL:https: / / www.astrodesign.co.jp / news / news-20190124-3064.html>

[0005] Conventional methods of cropping input video and outputting wide-angle video result in a wide-angle video with lower resolution than the input video. Upscaling the cropped video is one possible method for outputting high-resolution wide-angle video. Upscaling methods using image processing impose a light processing load, but the quality of the generated image is low. In contrast, super-resolution processing using AI can generate high-definition images. However, super-resolution processing imposes a heavy processing load, and attempting to perform the processing on a single computer can result in situations where the computer freezes or other operational issues are disrupted. Therefore, a method of distributing the upscaling process across multiple computers is being considered. However, if the computers are not synchronized, a discrepancy will occur between the images when the results (images) from each computer are displayed side by side.

[0006] Fig. 1 shows an example in which a wide-angle image is divided into multiple regions and super-resolution processing of the multiple regions is distributed among multiple computers. The example shown in Fig. 1 shows a case in which a wide-angle image (hereinafter also referred to as "super-resolution wide-angle image") is generated from input video of a soccer game using super-resolution processing technology.

[0007] As shown in FIG. 1 , for example, the input video is a 4K image (resolution: 3840 x 2160). From this, four horizontally aligned regions with a resolution of 960 x 540 are cropped (cut out) to create a wide-angle image with an aspect ratio of 4x. Then, by performing super-resolution processing on each region, four horizontally aligned regions with a resolution of 3840 x 2160 are generated. By displaying the four regions horizontally, a super-resolution wide-angle image with a resolution of 15360 (i.e., four times 3840) x 2160 is generated. In this case, for example, four regions 1 to 4 corresponding to the wide-angle image are cropped from the input video. Then, the super-resolution processing of regions 1 to 4 is performed in a distributed manner by four computers. If the four computers are synchronized, combining the processed images of regions 1 to 4 as is will not result in any discrepancy in the super-resolution wide-angle image. In contrast, if the four computers are not synchronized, combining the processed images of regions 1 to 4 as is will result in discrepancies between the horizontally aligned images (regions). In the example shown in FIG. 1, the same player PL is displayed overlappingly in area 2 and area 3.

[0008] Furthermore, achieving synchronization between multiple computers on a frame-by-frame basis (e.g., exchanging information for synchronization) imposes a heavy processing load, and performing this synchronization process results in delays in the super-resolution processing.

[0009] This invention has been made in light of the above-mentioned circumstances, and its purpose is to provide a super-resolution wide video synchronization control system that synchronizes the video output from each computer in a configuration in which upscaling processing using super-resolution processing is distributed across multiple computers, without close cooperation between the computers, and with each computer operating autonomously in a loosely coupled manner.

[0010] A super-resolution wide video synchronization control system according to a first aspect of the present invention includes a video distributor that distributes input video, video input units each connected to the video distributor and acquiring frame images from the distributed input video, a cropping unit that crops out sub-frame images from the frame images, an up-conversion unit that performs super-resolution processing on the sub-frame images to generate super-resolution sub-frame images, an output buffer that stores the super-resolution sub-frame images, an output time calculation unit that calculates the output time at which the super-resolution sub-frame images are output based on a predetermined processing time offset value for the super-resolution processing, and an output control unit that controls the output of the super-resolution sub-frame images based on an output schedule in which the output times are set.

[0011] A super-resolution wide video synchronization control method according to a second aspect of the present invention includes obtaining a frame image from an input video, extracting a sub-frame image from the frame image, performing super-resolution processing on the sub-frame image to generate a super-resolution sub-frame image, calculating an output time for outputting the super-resolution sub-frame image based on a predetermined processing time offset value for the super-resolution processing, and controlling the output of the super-resolution sub-frame image based on an output schedule in which the output time is set.

[0012] A program according to a third aspect of the present invention includes instructions for causing a computer to function as the super-resolution wide-screen video synchronization control system according to the first aspect.

[0013] According to the present invention, it is possible to provide a super-resolution wide-angle video synchronization control system, a super-resolution wide-angle video synchronization control method, and a program that can distribute the super-resolution processing of wide-angle video among multiple computers and suppress discrepancies between the videos output from each computer.

[0014] FIG. 1 is a diagram showing an example in which a wide image is divided into multiple regions and super-resolution processing of the multiple regions is distributed among multiple computers. FIG. 2 is a block diagram showing an example of the functional configuration of a super-resolution wide image synchronization control system according to an embodiment. FIG. 3 is a diagram showing an example of clipping positions of four subframe images. FIG. 4 is a block diagram showing an example of the functional configuration of a super-resolution synchronization control unit included in a super-resolution wide image synchronization control system according to an embodiment. FIG. 5 is a diagram showing a specific example of an output schedule. FIG. 6 is a flowchart showing an example of a super-resolution wide image generation method according to an embodiment. FIG. 7 is a block diagram showing an example of the hardware configuration of a super-resolution synchronization control unit included in a super-resolution wide image synchronization control system according to an embodiment.

[0015] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, components having the same functions and configurations will be given the same reference numerals.

[0016] (Functional Configuration of Super-Resolution Wide Video Synchronization Control System) First, an example of the functional configuration of the super-resolution wide video synchronization control system 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the super-resolution wide video synchronization control system 1 according to an embodiment. The example shown in Fig. 2 shows a case where the super-resolution wide video synchronization control system 1 divides a wide image into four regions and performs upconversion (super-resolution processing) of the images of the four regions in a distributed (parallel) manner. Note that the number of regions that the super-resolution wide video synchronization control system 1 performs distributed processing on may be two or more.

[0017] The super-resolution wide video synchronization control system 1 is a video processing system that generates a wide video from an input video using, for example, super-resolution processing technology using AI (hereinafter also referred to as "AI super-resolution processing technology"). Hereinafter, a wide video generated using super-resolution processing technology will also be referred to as a "super-resolution wide video." Note that a wide video (wide image) in this embodiment is a video (image) that has a larger aspect ratio (longer width) than the input video.

[0018] As shown in Fig. 2, the super-resolution wide-screen video synchronization control system 1 includes a video distributor 2, multiple super-resolution synchronization control units 3, and multiple display devices 4. For example, the function of one super-resolution synchronization control unit 3 is realized by one computer. In the example shown in Fig. 2, the super-resolution wide-screen video synchronization control system 1 includes four super-resolution synchronization control units 3a, 3b, 3c, and 3d, and four display devices 4a, 4b, 4c, and 4d. Note that there may be two or more super-resolution synchronization control units 3 and display devices 4.

[0019] The video distributor 2 receives video input from an external device (hereinafter also referred to as "input video"). The video input unit 31 distributes the input video to each super-resolution synchronization control unit 3. For example, the video distributor 2 has an input port for receiving video and multiple output ports connected to each of the multiple super-resolution synchronization control units 3. Note that the input video may be video captured by a camera, video input via a network, or video data (including past video data) stored in a storage medium (not shown). For example, the input video is video captured by a single normal camera.

[0020] The multiple super-resolution synchronization control units 3 execute super-resolution processing (up-conversion) on images extracted from the input video in a distributed manner, and generate respective super-resolution processed images. The multiple super-resolution synchronization control units 3 then synchronize their output times with one another, i.e., output super-resolution sub-frame images to the corresponding display devices 4 at the same output times. Note that the output times represent, for example, absolute times at which images are output.

[0021] More specifically, the super-resolution synchronization control unit 3 acquires an image (hereinafter referred to as a "frame image FR") from the input video. Each super-resolution synchronization control unit 3 cuts out a different region from one frame image FR. That is, the cut-out positions of the regions in each super-resolution synchronization control unit 3 are different from each other. Hereinafter, an image of a region cut out from a frame image FR will be referred to as a "sub-frame image SF." In other words, each super-resolution synchronization control unit 3 cuts out a different sub-frame image SF from one frame image FR.

[0022] An example in which four super-resolution synchronization control units 3 cut out sub-frame images SF from different cut-out positions of a frame image FR will be described with reference to Fig. 3. Fig. 3 is a diagram showing an example of the cut-out positions of the four sub-frame images SF.

[0023] As shown in FIG. 3 , for example, the super-resolution synchronization control unit 3a extracts a sub-frame image SFa from the frame image FR. The super-resolution synchronization control unit 3b extracts a sub-frame image SFb from the frame image FR. The super-resolution synchronization control unit 3c extracts a sub-frame image SFc from the frame image FR. The super-resolution synchronization control unit 3d extracts a sub-frame image SFd from the frame image FR. A single wide image is constructed by horizontally combining the four sub-frame images SFa, SFb, SFc, and SFd. For example, if the frame image FR is a 4K image with a resolution of 3840 x 2160 (aspect ratio 16:9), the resolution of each of the four sub-frame images SFa, SFb, SFc, and SFd is 960 x 540 (aspect ratio 16:9). For example, as shown in FIG. 3 , the coordinates of the upper left corner of the frame image FR are (0,0) and the coordinates of the lower right corner are (3840,2160). In this case, the cutout positions of the subframe images SFa, SFb, SFc, and SFd can be set as [(upper left coordinate), width, height], which can be expressed as [(0,540), 960,540], [(960,540), 960,540], [(1920,540), 960,540], and [(2880,540), 960,540]. The aspect ratio of the wide image formed by combining the four subframe images SFa, SFb, SFc, and SFd is 64:9. Therefore, a wide image with an aspect ratio of 4x is cut out from the frame image FR. In this way, the cutout positions in each super-resolution synchronization control unit 3 are set so that they do not overlap on the frame image FR. For example, when generating a landscape wide image, multiple regions aligned horizontally are cropped.

[0024] Returning to FIG. 2 , the super-resolution synchronization control unit 3 performs super-resolution processing on the sub-frame images SF using AI super-resolution processing (hereinafter also referred to as "AI super-resolution processing"). Hereinafter, the sub-frame images SF up-converted using AI super-resolution processing technology will also be referred to as "super-resolution sub-frame images." The super-resolution synchronization control unit 3 synchronizes the output time with other super-resolution synchronization control units 3 and outputs the super-resolution sub-frame images to the corresponding display devices 4. At this time, the super-resolution synchronization control unit 3a outputs the super-resolution sub-frame images to the display device 4a. The super-resolution synchronization control unit 3b outputs the super-resolution sub-frame images to the display device 4b. The super-resolution synchronization control unit 3c outputs the super-resolution sub-frame images to the display device 4c. The super-resolution synchronization control unit 3d outputs the super-resolution sub-frame images to the display device 4d.

[0025] The system times of the multiple super-resolution synchronization control units 3 are synchronized by each referencing the same Network Time Protocol (NTP) server. Each super-resolution synchronization control unit 3 has a processing time offset value calculated in advance based on the processing time of the super-resolution processing. The processing time of the super-resolution synchronization control unit 3 depends on the hardware configuration of the super-resolution synchronization control unit 3 (computer) (e.g., a central processing unit (CPU), a graphical processing unit (GPU), bus configuration, etc.). Therefore, to reduce variations in processing time, it is preferable that each super-resolution synchronization control unit 3 (each computer) has the same hardware configuration. Each super-resolution synchronization control unit 3 calculates the output time of a super-resolution sub-frame image based on the processing time offset value. By synchronizing the output times of super-resolution sub-frame images corresponding to the same frame image FR, each super-resolution synchronization control unit 3 achieves synchronization between output images without close coordination with each other.

[0026] For example, as a preliminary preparation for synchronization control, each super-resolution synchronization control unit 3 measures the processing time required to upconvert the same sample video (the same subframe image SF). For example, the maximum processing time for one subframe image SF is set as the processing time offset value. Note that if the input video is more complex than the sample video, it is expected that the super-resolution processing will not be completed within the processing time offset value. For this reason, a processing time offset value longer than the maximum processing time (e.g., 1.2 times the maximum processing time) may be set. Alternatively, a value that takes the output frame rate into consideration may be set as the processing time offset value. More specifically, for example, if the output frame rate is 60 fps, a value that is a multiple of 16.6 ms, which is the output time interval between each output frame (e.g., 33.2 ms if the maximum processing time is 20 ms) may be set.

[0027] The display devices 4 display super-resolution subframe images input from the corresponding super-resolution synchronization control units 3. The number of display devices 4 depends on the number of super-resolution synchronization control units 3. One display device 4 is provided for one super-resolution synchronization control unit 3. In the example shown in FIG. 2 , display device 4a is connected to super-resolution synchronization control unit 3a. Display device 4b is connected to super-resolution synchronization control unit 3b. Display device 4c is connected to super-resolution synchronization control unit 3c. Display device 4d is connected to super-resolution synchronization control unit 3d. The super-resolution subframe images displayed on each of the display devices 4a, 4b, 4c, and 4d are combined to synthesize (generate) one super-resolution wide image.

[0028] (Functional configuration of super-resolution synchronization control unit) Next, an example of the functional configuration of the super-resolution synchronization control unit 3 will be described with reference to Fig. 4. Fig. 4 is a block diagram showing an example of the functional configuration of the super-resolution synchronization control unit 3. Note that the example shown in Fig. 4 shows one super-resolution synchronization control unit 3, but the other super-resolution synchronization control units 3 also have the same configuration.

[0029] As shown in FIG. 4, the super-resolution synchronization control unit 3 includes, for example, a video input unit 31, an output time calculation unit 32, a clipping unit 33, an up-conversion unit 34, an output buffer 35, an output schedule setting unit 36, and an output control unit 37.

[0030] The video input unit 31 receives input video from the video distributor 2. The video input unit 31 acquires a plurality of frame images FR from the input video. The video input unit 31 transmits the frame images FR to the cropping unit 33. The video input unit 31 also notifies the output time calculation unit 32 that the frame images FR have been acquired.

[0031] When the output time calculation unit 32 receives notification that a frame image FR has been acquired, it adds the processing time offset value to the current time (the acquisition time of the frame image FR) and calculates the scheduled output time of the super-resolution sub-frame image SR corresponding to the acquired frame image FR. The output time calculation unit 32 corrects the calculated scheduled output time to a time in output time interval increments according to the output frame rate, and calculates the output time. The output time calculation unit 32 transmits the calculated output time to the output schedule setting unit 36.

[0032] For example, if the video input unit 31 acquires a frame image FR at 19:00:00.300 and the processing time offset value is 10 ms, the output time calculation unit 32 calculates 19:00:00.400 as the scheduled output time. Furthermore, the output time calculation unit 32 corrects the scheduled output time 19:00:00.400 to 19:00:00.499, which corresponds to an output frame rate of 16.6 ms. The output time calculation unit 32 notifies the output schedule setting unit 36 ​​of the output time 19:00:00.499.

[0033] The cropping unit 33 crops (cuts out) an area corresponding to a predetermined crop position from the frame image FR to be cropped. That is, the cropping unit 33 crops out a subframe image SF from the frame image FR. The cropping unit 33 transmits the cropped subframe image SF to the upconversion unit 34. Note that typical AI super-resolution processing techniques are trained using images of a typical size (e.g., an aspect ratio of 16:9, etc.) as training data. For this reason, it is preferable that the aspect ratio and resolution of the subframe image SF be those of a typical image. For example, the aspect ratio of the subframe image SF may be 16:9 or 4:3. Furthermore, when the aspect ratio is 16:9, the resolution of the subframe image SF may be 1920 x 1080, 1280 x 720, or 960 x 540.

[0034] The video distributor 2 may have a function for cutting out the subframe images SF. In this case, the subframe images SF are input from the video distributor 2. Therefore, the cutting unit 33 is omitted. The subframe images SF acquired by the video input unit 31 are then transmitted to the upconverter 34.

[0035] The upconversion unit 34 performs upconversion of the subframe images SF using, for example, AI super-resolution processing technology, to generate super-resolution subframe images SR. The AI ​​super-resolution processing technology used by the upconversion unit 34 is not limited. For example, Real-ESRGAN is used as the AI ​​super-resolution processing technology. The upconversion unit 34 transmits the super-resolution subframe images SR to the output buffer 35. The upconversion unit 34 also transmits the frame ID of the super-resolution subframe image SR for which super-resolution processing has been completed to the output schedule setting unit 36. The frame ID is used to identify the super-resolution subframe image SR. The frame ID may be a memory address used when storing the image in the output buffer 35, or may be a management ID assigned by the upconversion unit 34 or the like.

[0036] The output buffer 35 temporarily stores the super-resolution sub-frame images SR. Any number of super-resolution sub-frame images SR may be stored in one output buffer 35. For example, the output buffer 35 is configured to be able to store two super-resolution sub-frame images. For example, the output buffer 35 stores the most recent super-resolution sub-frame image SR and the super-resolution sub-frame image SR corresponding to the immediately preceding output timing. The output buffer 35 transmits the super-resolution sub-frame images SR to the output control unit 37 under the control of the output control unit 37.

[0037] The output schedule setting section 36 sets an output schedule for the super-resolution sub-frame images SR based on the output times received from the output time calculation section 32 and the frame IDs received from the up-conversion section 34 .

[0038] A specific example of an output schedule will be described with reference to Fig. 5. Fig. 5 is a diagram showing a specific example of an output schedule. Note that the example of Fig. 5 shows a case where the output frame rate is 60 fps.

[0039] As shown in Fig. 5, for example, the output schedule is managed as a table including multiple entries (rows) for entering (storing) output times and frame IDs. For example, in the table, an entry in which the output time is blank and a frame ID is entered indicates that the super-resolution sub-frame image SR corresponding to that frame ID has already been output at the current time. An entry in which the output time and frame ID are entered indicates that the current time has not yet reached the output time for that frame ID (a state in which output is awaiting). An entry in which the output time is entered but the frame ID is not entered (unregistered) indicates that the output time has been received from the output time calculation unit 32 but the frame ID has not yet been received from the up-conversion unit 34 (i.e., the super-resolution processing has not yet been completed).

[0040] 5A shows the state of the output schedule when the output schedule setting unit 36 ​​receives an output time corresponding to frame ID 3 from the output time calculation unit 32. Frame ID 1 is entered in the first entry 1. The output time is blank because the super-resolution sub-frame image SR corresponding to frame ID 1 has already been output. Entry 2 enters an output time of 19:00:00.332 and frame ID 2. Because the current time has not yet reached 19:00:00.332, the super-resolution sub-frame image SR corresponding to frame ID 2 is waiting to be output. In this state, upon receiving an output time from the output time calculation unit 32, the output schedule setting unit 36 ​​enters an output time of 19:00:00.499 in entry 3. At this point, the output schedule setting unit 36 ​​has not yet received frame ID 3 from the upconversion unit 34. That is, the upconversion unit 34 has not yet completed generation of the super-resolution sub-frame image SR corresponding to frame ID 3 (i.e., has not yet completed super-resolution processing). Therefore, the frame ID of entry 3 is left blank.

[0041] 5B shows the state of the output schedule when the output schedule setting unit 36 ​​receives frame ID3 from the upconverter 34. After the upconverter 34 completes generation of the super-resolution sub-frame image SR corresponding to frame ID3 (after the super-resolution process is completed), it transmits frame ID3 to the output schedule setting unit 36. Upon receiving frame ID3 from the upconverter 34, the output schedule setting unit 36 ​​enters frame ID3 in entry 3.

[0042] Figure 5(c) shows the output schedule when the output of the super-resolution sub-frame image SR corresponding to frame ID2 has been completed. Information related to frame ID1 has been deleted, and information related to frames ID2 and ID3 has been moved up to the entry immediately above. Therefore, frame ID2 is entered in the first entry, entry 1. Because the super-resolution sub-frame image SR corresponding to frame ID2 has already been output, the output time is left blank. Entry 2 has an output time of 19:00:00.499 and frame ID 3 entered. Because the current time has not yet reached 19:00:00.499, the super-resolution sub-frame image SR corresponding to frame ID3 is waiting to be output. Entry 3 has an output time of 19:00:00.666 entered, and the frame ID is blank. Therefore, the super-resolution processing corresponding to frame ID4 is not yet complete.

[0043] Returning to FIG. 4, the output schedule setting unit 36 ​​transmits information about the set output schedule to the output control unit 37 .

[0044] The output control unit 37 transmits the super-resolution sub-frame images SR stored in the output buffer 35 to the corresponding display devices 4 based on the output schedule. More specifically, for example, the output control unit 37 compares the earliest output time entered in the output schedule with the current time, and when the current time reaches the output time (current time = output time), it retrieves from the output buffer 35 the super-resolution sub-frame image SR corresponding to the frame ID corresponding to that output time and outputs it to the corresponding display devices 4. On the other hand, if the frame ID corresponding to that output time is not entered in the output schedule, i.e., if the super-resolution processing is not complete, the output control unit 37 retrieves from the output buffer 35 the super-resolution sub-frame image SR corresponding to the frame ID entered in the entry where the output time is blank (output already), and outputs it to the corresponding display devices 4. In other words, the output control unit 37 re-outputs the super-resolution sub-frame image SR corresponding to the immediately previous output timing.

[0045] (Super-resolution wide video synchronization control method) Next, an example of a super-resolution wide video synchronization control method will be described with reference to Fig. 6. Fig. 6 is a flowchart showing an example of a super-resolution wide video generation method according to an embodiment. The flowchart shown in Fig. 6 shows the flow of processing corresponding to one frame image FR in one super-resolution synchronization control unit 3. When processing multiple frame images FR, the processing shown in Fig. 6 is repeatedly executed for each frame image FR.

[0046] 6 , the video input unit 31 acquires a frame image FR from an input video (S1). The video input unit 31 notifies the output time calculation unit 32 that the frame image FR has been acquired. The video input unit 31 also transmits the frame image FR to the cropping unit 33.

[0047] Next, the output time calculation unit 32 calculates the scheduled output time of the super-resolution sub-frame image SR to be generated based on the notification received from the video input unit 31 (S2). More specifically, the output time calculation unit 32 calculates the scheduled output time by, for example, adding the processing time offset value to the current time (the time when the notification was received from the video input unit 31).

[0048] Next, the output time calculation unit 32 corrects the scheduled output time based on the output frame rate and calculates the output time (S3). The output time calculation unit 32 transmits the output time to the output schedule setting unit 36. The output schedule setting unit 36 ​​enters the output time in the output schedule.

[0049] The cropping unit 33 crops an area corresponding to the super-resolution wide image from one frame image FR in accordance with the cropping setting (S4). That is, the cropping unit 33 crops a sub-frame image SF from the frame image FR. The cropping unit 33 transmits the sub-frame image SF to the up-conversion unit 34.

[0050] The upconverter 34 upconverts the subframe image SF using AI super-resolution processing (S4). For example, Real-ESRGAN is used as the AI ​​super-resolution processing. That is, the upconverter 34 generates a super-resolution subframe image SR that has been subjected to AI super-resolution processing. The upconverter 34 transmits the super-resolution subframe image SR to the output buffer 35. The upconverter 34 also transmits a frame ID to the output schedule setting unit 36. Note that the processing of steps S2 and S3 and the processing of steps S4 and S5 may be executed in parallel.

[0051] The output schedule setting unit 36 ​​writes the frame ID in the output schedule (S6). The output schedule setting unit 36 ​​transmits the updated output schedule to the output control unit 37.

[0052] When the output control unit 37 determines that the earliest output time entered in the output schedule matches the current time (S7), it checks whether a frame ID corresponding to that output time is entered in the output schedule (S8). That is, the output control unit 37 checks whether a frame ID is entered in the entry in which the earliest output time is entered.

[0053] If a frame ID is entered (S8_Yes), the output control unit 37 outputs the super-resolution sub-frame image SR corresponding to the frame ID (S9).

[0054] If no frame ID is entered (S8_No), the output control unit 37 outputs the super-resolution sub-frame image SR corresponding to the frame ID of the first entry in the output schedule whose output time is blank (S10). That is, the output control unit 37 re-outputs the previous super-resolution sub-frame image SR that has already been output.

[0055] When the output of the super-resolution sub-frame images SR is completed, the output schedule setting unit 36 ​​deletes the information (output time and frame ID) entered in the first entry of the output schedule and leaves any output times earlier than the current time blank (S11). That is, the output schedule setting unit 36 ​​leaves blank the earliest output time that matched the current time in step S7.

[0056] (Hardware configuration of super-resolution synchronization control unit) An example of the hardware configuration of the super-resolution synchronization control unit 3 will be described with reference to Fig. 7. Fig. 7 is a block diagram showing an example of the hardware configuration of the super-resolution synchronization control unit 3 included in the super-resolution wide video synchronization control system 1 according to the embodiment. Note that the example shown in Fig. 7 shows the hardware configuration of one super-resolution synchronization control unit 3, but the hardware configurations of the other super-resolution synchronization control units 3 are also the same.

[0057] 7 , the super-resolution synchronization control unit 3 is, for example, a computer. That is, the super-resolution wide video synchronization control system 1 includes multiple computers that execute super-resolution synchronization control. The super-resolution synchronization control unit 3 includes a processor 301, a ROM (Read Only Memory) 302, a RAM (Random Access Memory) 303, a storage medium 304, and an input / output interface 305.

[0058] The processor 301, ROM 302, RAM 303, storage medium 304, and input / output interface 305 are electrically connected to one another via a bus 306. The processor 301, ROM 302, RAM 303, storage medium 304, and input / output interface 305 transmit and receive data or control signals via the bus 306.

[0059] The processor 301 is configured by a general-purpose hardware processor including, for example, a CPU, a GPU, etc. The processor 301 controls the entire super-resolution synchronization control unit 3.

[0060] The ROM 302 is a non-volatile memory. For example, the ROM 302 is an erasable programmable read-only memory (EPROM). The ROM 302 is a non-transitory storage medium that stores firmware, programs, and the like. For example, the processor 301 loads firmware from the ROM 302 into the RAM 303 and executes the firmware.

[0061] The RAM 303 is a volatile memory. The RAM 303 is a dynamic random access memory (DRAM) or a static random access memory (SRAM), for example. The RAM 303 temporarily stores programs used in processing by the processor 301 and data used to execute the programs. The processor 301 executes the programs in the RAM 303 to calculate the data in the RAM 303 and store the calculation results in the RAM 303. For example, the RAM 303 stores a processing time offset value and an output schedule. For example, the RAM 303 functions as an output buffer 35. The super-resolution synchronization control unit 3 may include a volatile memory dedicated to the output buffer 35.

[0062] The storage medium 304 is configured with a nonvolatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The storage medium 304 non-temporarily stores programs executed by the processor 301 and data required for executing the programs. For example, the storage medium 304 stores a super-resolution synchronization control processing program for executing super-resolution processing and synchronization control processing. Note that the program for executing super-resolution processing and the program for executing synchronization control processing may be separate programs. The super-resolution synchronization control processing program causes the super-resolution synchronization control unit 3 to execute at least some of the functions of the components of the super-resolution synchronization control unit 3, namely, the video input unit 31, the output time calculation unit 32, the cropping unit 33, the up-conversion unit 34, the output schedule setting unit 36, and the output control unit 37. The storage medium 304 also stores data used for super-resolution synchronization control. The storage medium 304 may also store video data (input video).

[0063] The program executed by the super-resolution synchronization control unit 3 may be provided to the super-resolution synchronization control unit 3 via a readable non-transitory storage medium (not shown). Such a storage medium is called a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media include disks such as flexible disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), and magneto-optical disks (MO, etc.), as well as semiconductor memories.

[0064] The input / output interface 305 is connected to the video distributor 2, the display device 4, the input device 5, and the output device 6. The input / output interface 305 enables input of video from the video distributor 2, display of video on the display device 4, input of information from the input device 5, and output of information to the output device 6. The input / output interface 305 may also be connected to an external database or the like in which input video is stored. For example, the input / output interface 305 may be a wired interface or a wireless interface. A wired interface includes a port to which a device is connected. For example, a wireless interface has a function that satisfies communication standards such as Bluetooth (registered trademark) and WiFi (registered trademark).

[0065] The display device 4 includes a display (such as an LCD (Liquid Crystal Display), an EL (Electroluminescence) display, a projector, or a cathode ray tube.) The display device 4 can display a super-resolution wide image (super-resolution sub-frame image).

[0066] The input device 5 may include a keyboard, a mouse, a touch panel, a receiving device, a disk drive, etc. The input device 5 is not limited to these, and may include any other input device.

[0067] The output device 6 may include a transmitting device, a disk drive, etc. The output device 6 is not limited to these and may include any other output device. The input device 5 and the output device 6 may be configured as an input / output device having the functions of both the input device 5 and the output device 6.

[0068] When the super-resolution synchronization control unit 3 is started, the processor 301 executes a program in the ROM 302 and loads and starts an OS (Operating System) into the RAM 303. Under control of the OS, the processor 301 monitors input instructions, connections to external devices, and the like. Under control of the OS, the processor 301 also sets up a program area and a data area in the RAM 303. In response to an instruction input to start the super-resolution synchronization control unit 3, the processor 301 loads a super-resolution synchronization control processing program from the storage medium 304 into the program area of ​​the RAM 303, and loads data used in executing the super-resolution synchronization control processing program from the storage medium 304 into the data area of ​​the RAM 303. The processor 301 calculates data in the data area in accordance with the super-resolution synchronization control processing program and writes the calculation results to the data area. Through these operations, the processor 301, RAM 303, storage medium 304, input / output interface 305, and bus 306 work together to execute at least some of the functions of the components of the super-resolution synchronization control unit 3.

[0069] (Effects of the Embodiment) In the configuration according to the present embodiment, the super-resolution wide-angle video synchronization control system 1 includes multiple super-resolution synchronization control units 3. In the super-resolution wide-angle video synchronization control system 1, the multiple super-resolution synchronization control units 3 can be realized by multiple computers. The multiple super-resolution synchronization control units 3 can synchronize their system times by each referencing the same Network Time Protocol (NTP) server. The multiple super-resolution synchronization control units 3 can perform super-resolution processing on images extracted from input video in a distributed manner and generate respective super-resolution processed images. The multiple super-resolution synchronization control units 3 can then synchronize their output times and output respective super-resolution sub-frame images. The super-resolution synchronization control unit 3 can acquire a frame image FR from the input video. The super-resolution synchronization control unit 3 can extract a sub-frame image SF from the frame image FR. The extraction position of the sub-frame image SF in each of the multiple super-resolution synchronization control units 3 can be set to a different position. The super-resolution synchronization control unit 3 can perform super-resolution processing on the sub-frame images SF to generate super-resolution sub-frame images SR. The super-resolution synchronization control unit 3 can calculate the scheduled output time of the super-resolution sub-frame images SR based on the processing time offset value. The super-resolution synchronization control unit 3 can calculate the output time by correcting the scheduled output time based on the output frame rate. The super-resolution synchronization control unit 3 can control the output of the super-resolution sub-frame images SR based on the output time and the frame ID of the super-resolution sub-frame images SR. This allows the super-resolution wide image synchronization control system 1 to present high-resolution, high-definition wide images without any misalignment between the images. This allows the super-resolution wide image synchronization control system 1 to synchronize the images output from each computer in a configuration in which super-resolution processing is distributed across multiple computers without close coordination between the computers.

[0070] Furthermore, with the configuration according to this embodiment, the super-resolution wide video synchronization control system 1 can divide the cropped region into multiple parts, thereby making it possible to set the aspect ratio and resolution of the subframe images SF to those of general images. This makes it possible to use AI super-resolution technology that has learned using existing image data as training data when upconverting the subframe images SF.

[0071] Furthermore, with the configuration according to this embodiment, the super-resolution wide-angle video synchronization control system 1 can use video captured by a single ordinary camera as input video, making it possible to present high-resolution wide-angle video captured by switching between multiple cameras installed in different locations in a venue, for example.

[0072] Furthermore, with the configuration according to this embodiment, the super-resolution wide-screen video synchronization control system 1 can change the number of sub-frame images SF cut out for one frame image FR. For example, when live streaming super-resolution wide-screen video to multiple locations, it is possible to present super-resolution wide-screen video that corresponds to the situation of each display venue (two screens / three screens / four screens). Furthermore, the super-resolution wide-screen video synchronization control system 1 can generate super-resolution wide-screen video from past video.

[0073] Other Embodiments The present invention is not limited to the above-described embodiment, and can be implemented in various modifications without departing from the spirit and scope of the present invention.

[0074] For example, in the above embodiment, a case where a horizontally long super-resolution wide image is generated has been described, but the super-resolution wide image may also be vertically long. In this case, multiple vertically aligned regions are cut out from a frame image. Super-resolution processing is performed on each region in parallel. Then, a vertically long super-resolution wide image is generated by displaying the multiple regions aligned vertically.

[0075] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0076] 1...Super-resolution wide video synchronization control system 2...Video distributor 3, 3a, 3b, 3c, 3d...Super-resolution synchronization control section 4, 4a, 4b, 4c, 4d...Display device 5...Input device 6...Output device 31...Video input section 32...Output time calculation section 33...Cropping section 34...Up-conversion section 35...Output buffer 36...Output schedule setting section 37...Output control section 301...Processor 302...ROM 303...RAM 304...Storage medium 305...Input / output interface 306...Bus

Claims

1. A super-resolution wide video synchronization control system comprising: a video distributor that distributes input video; video input units each connected to the video distributor and configured to acquire frame images from the distributed input video; a cropping unit that crops sub-frame images from the frame images; an up-conversion unit that performs super-resolution processing on the sub-frame images to generate super-resolution sub-frame images; an output buffer that stores the super-resolution sub-frame images; an output time calculation unit that calculates an output time for outputting the super-resolution sub-frame images based on a preset processing time offset value for the super-resolution processing; and an output control unit that controls output of the super-resolution sub-frame images based on an output schedule in which the output times are set.

2. The super-resolution wide-screen video synchronization control system according to claim 1, wherein the plurality of super-resolution synchronization control units are configured by a plurality of computers.

3. The super-resolution wide video synchronization control system of claim 1, wherein when the current time and the output time match, if the output buffer does not store the super-resolution sub-frame image corresponding to the output time, the output control unit outputs a super-resolution sub-frame image stored in the output buffer that corresponds to one output timing of the output time.

4. The super-resolution wide video synchronization control system of claim 1, wherein the processing time offset value is set based on the maximum processing time taken by each of the multiple super-resolution synchronization control units to perform the super-resolution processing of the same image.

5. The super-resolution wide video synchronization control system of claim 1, wherein the output time calculation unit calculates the scheduled output time at which the up-conversion unit will finish the super-resolution processing based on the processing time offset value, and calculates the output time by correcting the scheduled output time based on the output frame rate.

6. The super-resolution wide video synchronization control system according to claim 1, wherein the positions at which each of the cutout sections of the plurality of super-resolution synchronization control sections cuts out the sub-frame images are different from one another.

7. A super-resolution wide video synchronization control method comprising: acquiring a frame image from an input video; extracting sub-frame images from the frame image; performing super-resolution processing on the sub-frame images to generate super-resolution sub-frame images; calculating an output time for outputting the super-resolution sub-frame images based on a preset processing time offset value for the super-resolution processing; and controlling the output of the super-resolution sub-frame images based on an output schedule in which the output time is set.

8. A program that causes a computer to function as the super-resolution wide-screen video synchronization control system according to any one of claims 1 to 6.

Images