Super-resolution wide video generation device, super-resolution wide video generation method, and program

Abstract

One aspect of a super-resolution wide video generation device according to the present invention includes: a video input unit that acquires a frame image from an input video; a cutout unit that cuts out a plurality of sub-frame images from the frame image; a plurality of up-conversion units that distribute and execute super-resolution processing of the respective plurality of sub-frame images to generate a plurality of super-resolution sub-frame images; a plurality of output buffers that are respectively connected to the plurality of up-conversion units and each store any one of the plurality of super-resolution sub-frame images; and an output synchronization unit that synchronizes and outputs the plurality of super-resolution sub-frame images.

Description

Super-resolution wide-angle image generating device, super-resolution wide-angle image generating method, and program

[0001] The present invention relates to a super-resolution wide-angle video generating device, a super-resolution wide-angle video generating method, and a program.

[0002] One method for presenting high-definition wide-angle images is to extract a horizontally long area from a high-resolution input image and present it as a wide-angle image. For example, in Non-Patent Document 1, a wide-angle image with an aspect ratio of 32:9 (twice the aspect ratio of 4K image with a 16:9 aspect ratio) is generated from an 8K input image.

[0003] Methods for generating high-resolution images include, for example, upscaling by image processing using pixel interpolation techniques such as bilinear interpolation, or super-resolution processing techniques using AI (artificial intelligence) such as DNN (Deep Neural Network) or CNN (Convolutional Neural Network). One super-resolution processing technique using AI is Real-ESRGAN (Real Enhanced Super Resolution Generative Adversarial Networks).

[0004] "Successfully broadcasting an unprecedented next-generation live viewing broadcast using 8K technology in 32:9 widescreen - B-LEAGUE with overwhelming realism in ultra-high definition images," [online], [Retrieved January 16, 2024], Internet<URL:https: / / www.astrodesign.co.jp / news / news-20190124-3064.html>

[0005] In conventional methods of cropping an input video and outputting a wide-angle video, a wide-angle video with a lower resolution than the input video is generated. One possible method for outputting a high-resolution wide-angle video is to upscale the cropped video. In upscaling methods using image processing, the processing is performed uniformly regardless of the image content. For this reason, upscaling methods using image processing can divide the wide-angle video into multiple regions and process the multiple regions in parallel (distributed processing). Using upscaling methods using image processing can speed up upscaling, but the quality of the generated image is low.

[0006] In contrast, super-resolution processing using AI can generate high-definition images. However, the processing time for super-resolution processing varies depending on the content of the image. For example, the processing time tends to increase as the number of objects in an image increases. Therefore, if a wide-angle image is divided into multiple regions and super-resolution processing for each region is performed in parallel, the processing time will vary depending on the region (image).

[0007] Fig. 1 shows an example in which a wide-angle image is divided into multiple regions and super-resolution processing is performed on each region in parallel. The example shown in Fig. 1 shows a case in which a wide-angle image (hereinafter also referred to as "super-resolution wide-angle image") is generated from input video of a soccer match using super-resolution processing technology.

[0008] As shown in FIG. 1 , for example, the input video is a 4K image (resolution: 3840 x 2160). From this, four horizontally aligned regions with a resolution of 960 x 540 are cropped (cut out) to create a wide-angle image with an aspect ratio of 4x. Then, by performing super-resolution processing on each region, four horizontally aligned regions with a resolution of 3840 x 2160 are generated. By displaying the four regions horizontally, a super-resolution wide-angle image with a resolution of 15360 (i.e., four times 3840) x 2160 is generated. In this case, for example, four regions 1 to 4 corresponding to the wide-angle image are cut out from the input video. Then, super-resolution processing is performed on each region in parallel. If there is no difference in the processing time of the super-resolution processing between the regions, no discrepancy will occur in the super-resolution wide-angle image even if the processed images of regions 1 to 4 are combined as is. In contrast, if there is a difference in the processing time of the super-resolution processing between the regions, combining the processed images of regions 1 to 4 as is will result in a discrepancy between the horizontally aligned images (regions). In the example shown in FIG. 1, the same player PL is displayed overlappingly in area 2 and area 3.

[0009] While it is possible to super-resolve wide images as they are, AI-based super-resolution processing technology (hereinafter also referred to as "AI super-resolution processing technology") is trained using images of a normal size (e.g., an aspect ratio of 16:9) as training data. Therefore, when super-resolving wide images as they are, training data for wide images must be prepared and training must be performed separately. Furthermore, if the output aspect ratio is changed, it becomes necessary to dynamically change the training model itself.

[0010] This invention was made with the above-mentioned circumstances in mind, and its purpose is to provide a super-resolution wide-angle image generation device that eliminates image misalignment between regions (synchronizes regions) when an image is divided into multiple regions and AI super-resolution processing is performed on each region in parallel.

[0011] A super-resolution wide image generating device according to a first aspect of the present invention includes a video input unit that acquires a frame image from an input video, a cutting unit that cuts out a plurality of sub-frame images from the frame image, a plurality of up-conversion units that perform super-resolution processing on each of the plurality of sub-frame images in a distributed manner to generate a plurality of super-resolution sub-frame images, a plurality of output buffers that are respectively connected to the plurality of up-conversion units and each store one of the plurality of super-resolution sub-frame images, and an output synchronization unit that synchronizes and outputs the plurality of super-resolution sub-frame images.

[0012] A super-resolution wide image generating method according to a second aspect of the present invention includes obtaining a frame image from an input image, extracting a plurality of sub-frame images from the frame image, performing super-resolution processing on each of the plurality of sub-frame images in a distributed manner to generate a plurality of super-resolution sub-frame images, and outputting the plurality of super-resolution sub-frame images in synchronization.

[0013] A program according to a third aspect of the present invention includes instructions for causing a computer to function as the super-resolution wide-screen image generating device according to the first aspect.

[0014] According to the present invention, it is possible to provide a super-resolution wide image generating device, a super-resolution wide image generating method, and a program that can present high-resolution, high-definition wide images without any misalignment between the images.

[0015] FIG. 1 is a diagram showing an example in which a wide image is divided into multiple regions and super-resolution processing is performed on each region in parallel. FIG. 2 is a block diagram showing an example of the functional configuration of a super-resolution wide image generation device according to the first embodiment. FIG. 3 is a diagram showing three examples of the set number of sub-frame images to be cut out in the super-resolution wide image generation device according to the first embodiment. FIG. 4 is a flowchart of a super-resolution wide image generation method according to the first embodiment. FIG. 5 is a block diagram showing an example of the hardware configuration of a super-resolution wide image generation device according to the first embodiment. FIG. 6 is a block diagram showing an example of the functional configuration of a super-resolution wide image generation device according to a second embodiment. FIG. 7 is a flowchart of a super-resolution wide image generation method according to the second embodiment. FIG. 8 is a diagram showing a specific example of the management status of super-resolution sub-frame images in an output buffer included in the super-resolution wide image generation device according to the second embodiment.

[0016] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. In the following description, components having the same functions and configurations will be given the same reference numerals.

[0017] (First embodiment) (Functional configuration of super-resolution wide image generation device) First, an example of the functional configuration of the super-resolution wide image generation device 1 will be described with reference to Fig. 2. Fig. 2 is a block diagram showing an example of the functional configuration of the super-resolution wide image generation device 1 according to the first embodiment. The example shown in Fig. 2 shows a case where the super-resolution wide image generation device 1 divides a wide image into four regions and performs up-conversion (super-resolution processing) of the images of the four regions in parallel (distributed processing). Note that the number of regions (images) that the super-resolution wide image generation device 1 processes in parallel (distributed processing) may be two or more.

[0018] The super-resolution wide video generation device 1 is, for example, a video processing system that generates a wide video from an input video using super-resolution processing technology that uses AI (hereinafter also referred to as "AI super-resolution processing technology"). Hereinafter, a wide video generated using super-resolution processing technology will also be referred to as a "super-resolution wide video." Note that a wide video (wide image) in this embodiment is a video (image) that has a larger aspect ratio (longer width) than the input video.

[0019] As shown in FIG. 2, the super-resolution wide image generating device 1 includes a wide image synchronization control unit 2 and a wide image display unit 3 .

[0020] The wide-angle video synchronization control unit 2 extracts multiple regions from one image of the input video, distributes the AI ​​super-resolution processing of each region among multiple threads, and outputs the multiple super-resolution images to the wide-angle video display unit 3 by controlling their synchronization.

[0021] The wide image display unit 3 displays a super-resolution wide image. The wide image display unit 3 includes a plurality of display devices 30. In the example shown in Fig. 2, the wide image display unit 3 includes four display devices 30a, 30b, 30c, and 30d.

[0022] Next, the internal configuration of the wide image synchronization control unit 2 will be described.

[0023] The wide-angle video synchronization control unit 2 includes, for example, a video input unit 21, a frame number assignment unit 22, a cropping unit 23, multiple upconversion units 24, multiple output buffers 25, and an output synchronization unit 26. In the example shown in FIG. 2 , the wide-angle video synchronization control unit 2 includes four upconversion units 24a, 24b, 24c, and 24d, and four output buffers 25a, 25b, 25c, and 25d. For example, the upconversion unit 24a and the output buffer 25a form one thread that executes AI super-resolution processing. Similarly, the upconversion unit 24b and the output buffer 25b form one thread that executes AI super-resolution processing. The upconversion unit 24c and the output buffer 25c form one thread that executes AI super-resolution processing. The upconversion unit 24d and the output buffer 25d form one thread that executes AI super-resolution processing. That is, the wide image synchronization control unit 2 includes four threads that execute the AI ​​super-resolution processing.

[0024] The video input unit 21 receives video input from an external device (hereinafter also referred to as "input video"). The video input unit 21 acquires multiple frame images FR from the input video. The input video may be video captured by a camera, video input via a network, or video data (including past video data) stored in a storage medium (not shown). For example, the input video is video captured by a single normal type camera. The video input unit 21 transmits the frame images FR to the frame number assignment unit 22.

[0025] The frame number assignment unit 22 assigns a frame number FN to the frame image FR. After assigning the frame number FN to the frame image FR, the frame number assignment unit 22 increments the frame number FN. In this way, when the frame number assignment unit 22 receives multiple frame images FR, it can assign consecutive frame numbers FN that are different from one another to each frame image FR. For example, the frame number assignment unit 22 can assign frame numbers FN1 to FN5, in order, to five frame images FR received consecutively from the video input unit 21. The number of digits in the frame number FN can be set arbitrarily. The frame number assignment unit 22 transmits the frame images FR to which the frame numbers FN have been assigned to the cropping unit 23.

[0026] The cropping unit 23 crops (cuts out) multiple regions corresponding to a wide image from the frame image FR to be cropped. The number of crop settings, i.e., the number of regions, can be set arbitrarily. The number of crop settings is based on the aspect ratio of the wide image. For example, a user of the super-resolution wide image generation device 1 can arbitrarily change the number of crop settings, i.e., the aspect ratio of the wide image, for the same input image. For example, the user may change the number of crop settings depending on the number of display devices 30. Each region is set so that it does not overlap in the frame image FR. For example, when generating a horizontally long wide image, multiple regions aligned horizontally are cropped. Hereinafter, images of regions cropped out from the frame image FR will be referred to as "subframe images SF." In other words, the cropping unit 23 crops multiple subframe images SF from one frame image FR. The multiple subframe images SF are combined to generate (combine) a wide image. It is preferable that the aspect ratio and resolution of the subframe images SF be those of a general image. For example, the aspect ratio of the subframe image SF may be 16:9 or 4:3. Furthermore, when the aspect ratio of the subframe image SF is 16:9, the resolution may be 1920 x 1080, 1280 x 720, or 960 x 540.

[0027] Three examples of the set number of sub-frame images SF to be cut out will be described with reference to Fig. 3. Fig. 3 is a diagram showing three examples of the set number of sub-frame images SF to be cut out.

[0028] As shown in FIG. 3 , when the cutout setting number is 2, the cutout unit 23 cuts out two subframe images SFa and SFb from one frame image FR. A single wide image is constructed by combining the two subframe images SFa and SFb horizontally. For example, if the frame image FR is a 4K image with a resolution of 3840 x 2160 (aspect ratio of 16:9), the resolution of each of the two subframe images SFa and SFb is 1920 x 1080 (aspect ratio of 16:9). The aspect ratio of the wide image formed by combining the two subframe images SFa and SFb is 32:9. Therefore, a wide image with an aspect ratio of double is cut out from the frame image FR.

[0029] When the cutout setting number is 3, the cutout unit 23 cuts out three subframe images SFa, SFb, and SFc from one frame image FR. A single wide image is constructed by horizontally combining the three subframe images SFa, SFb, and SFc. For example, if the frame image FR is a 4K image with a resolution of 3840 x 2160 (aspect ratio of 16:9), the resolution of each of the three subframe images SFa, SFb, and SFc is 1280 x 720 (aspect ratio of 16:9). The aspect ratio of the wide image formed by combining the three subframe images SFa, SFb, and SFc is 42:9. Therefore, a wide image with an aspect ratio of 3x is cut out from the frame image FR.

[0030] When the crop setting is 4, the cropping unit 23 crops four subframe images SFa, SFb, SFc, and SFd from one frame image FR. A single widescreen image is constructed by horizontally combining the four subframe images SFa, SFb, SFc, and SFd. For example, if the frame image FR is a 4K image with a resolution of 3840 x 2160 (aspect ratio of 16:9), the resolution of each of the four subframe images SFa, SFb, SFc, and SFd is 960 x 540 (aspect ratio of 16:9). For example, as shown in FIG. 3, the coordinates of the upper left corner of the frame image FR are (0, 0), and the coordinates of the lower right corner are (3840, 2160). In this case, the cropping positions of the subframe images SFa, SFb, SFc, and SFd can be set as [(top left coordinate), width, height], which can be expressed as [(0,540), 960, 540], [(960, 540), 960, 540], [(1920, 540), 960, 540], and [(2880, 540), 960, 540]. The aspect ratio of the wide image formed by combining the four subframe images SFa, SFb, SFc, and SFd is 64:9. Therefore, a wide image with an aspect ratio of 4x is cropped from the frame image FR. In this way, each cropping position is set so that it does not overlap on the frame image FR. For example, when generating a landscape-oriented wide image, multiple regions aligned horizontally are cropped.

[0031] Returning to FIG. 2 , the cropping unit 23, for example, crops four subframe images SF from one frame image FR. Each subframe image SF is assigned the frame number FN of the frame image FR from which it was cropped. That is, the four subframe images are assigned the same frame number FN. The cropping unit 23 transmits the four subframe images SF assigned the same frame number FN to the four upconversion units 24a, 24b, 24c, and 24d, respectively. In other words, the cropping unit 23 transmits the four subframe images SF assigned the same frame number FN to the four threads, respectively.

[0032] The multiple upconversion units 24 perform distributed upconversion of different subframe images SF assigned the same frame number, for example, using AI super-resolution processing technology. Hereinafter, subframe images SF upconverted using AI super-resolution processing technology will also be referred to as "super-resolution subframe images SR." The AI ​​super-resolution processing technology used by the upconversion units 24 is not limited. For example, Real-ESRGAN is used as the AI ​​super-resolution processing technology. The number of upconversion units 24 can be set arbitrarily. The number of upconversion units 24 depends on the number of subframe images SF that the cropping unit 23 crops from one frame image FR, i.e., the crop setting number. For example, if the crop setting number is three, three upconversion units 24 are provided.

[0033] The up-converter 24a transmits the super-resolution sub-frame image SR with the frame number FN assigned to the output buffer 25a. Similarly, the up-converter 24b transmits the super-resolution sub-frame image SR with the frame number FN assigned to the output buffer 25b. The up-converter 24c transmits the super-resolution sub-frame image SR with the frame number FN assigned to the output buffer 25c. The up-converter 24d transmits the super-resolution sub-frame image SR with the frame number FN assigned to the output buffer 25d.

[0034] The output buffer 25 temporarily stores the super-resolution sub-frame images SR and the corresponding frame numbers FN. The number of output buffers 25 is arbitrary. The number of output buffers 25 depends on the number of up-conversion units 24. One output buffer 25 is provided for one up-conversion unit 24. The number of super-resolution sub-frame images SR and the corresponding frame numbers FN stored in one output buffer 25 is arbitrary. In this embodiment, a case will be described in which the output buffer 25 is capable of storing two super-resolution sub-frame images SR and the corresponding frame numbers FN.

[0035] The output buffer 25a is connected to the upconversion unit 24a and temporarily stores the super-resolution sub-frame images SR and corresponding frame numbers FN received from the upconversion unit 24a. Similarly, the output buffer 25b is connected to the upconversion unit 24b and temporarily stores the super-resolution sub-frame images SR and corresponding frame numbers FN received from the upconversion unit 24b. The output buffer 25c is connected to the upconversion unit 24c and temporarily stores the super-resolution sub-frame images SR and corresponding frame numbers FN received from the upconversion unit 24c. The output buffer 25d is connected to the upconversion unit 24d and temporarily stores the super-resolution sub-frame images SR and corresponding frame numbers FN received from the upconversion unit 24d. Each of the output buffers 25a, 25b, 25c, and 25d transmits the super-resolution sub-frame images SR to the output synchronization unit 26.

[0036] The output synchronization unit 26 synchronizes and controls the multiple super-resolution sub-frame images SR to be output to the wide video display unit 3. The output synchronization unit 26 identifies the super-resolution sub-frame images SR processed by each thread by frame number FN. The output synchronization unit 26 references the frame number FN and outputs the multiple super-resolution sub-frame images SR corresponding to the same frame number FN to the wide video display unit 3 in a synchronized manner.

[0037] More specifically, for example, the output synchronization unit 26 checks whether a super-resolution sub-frame image SR corresponding to the frame number FN to be output (hereinafter also referred to as "output frame number FN") exists in each of the four output buffers 25a, 25b, 25c, and 25d at the output timing according to the output frame rate.

[0038] If a corresponding super-resolution sub-frame image SR is present in each of output buffers 25a, 25b, 25c, and 25d, output synchronizer 26 outputs that super-resolution sub-frame image SR to wide video display unit 3. For example, output synchronizer 26 outputs the super-resolution sub-frame image SR stored in output buffer 25a to display device 30a. Similarly, output synchronizer 26 outputs the super-resolution sub-frame image SR stored in output buffer 25b to display device 30b. Output synchronizer 26 outputs the super-resolution sub-frame image SR stored in output buffer 25c to display device 30c. Output synchronizer 26 outputs the super-resolution sub-frame image SR stored in output buffer 25d to display device 30d.

[0039] If a corresponding super-resolution sub-frame image SR is not present in at least one of the output buffers 25a, 25b, 25c, and 25d, i.e., if upconversion of a sub-frame image SF assigned an output frame number FN has not been completed in at least one thread, the output synchronization unit 26 synchronizes and outputs the super-resolution sub-frame image SR of the previous frame number FN-1 stored in each output buffer 25. More specifically, for example, if there is an output buffer 25 that does not contain a super-resolution sub-frame image SR corresponding to the output frame number FN, the output synchronization unit 26 outputs to the display devices 30a, 30b, 30c, and 30d the super-resolution sub-frame image SR of the frame number previous to output frame number FN (output frame number FN-1) stored in the output buffers 25a, 25b, 25c, and 25d, respectively.

[0040] The difference in processing time between the upconverters 24 is within an output time interval determined from the output frame rate (for example, when the output frame rate is 60 fps, the difference in processing time is within 16.6 ms). For example, if the difference in processing time is twice or more the output time interval, it is preferable to take measures such as lowering the output frame rate or changing the upconversion (i.e., AI super-resolution processing) performed by the upconverter 24 to a method with a shorter processing time.

[0041] The display devices 30 display super-resolution sub-frame images SR. The wide video display unit 3 may include any number of display devices 30. For example, the number of display devices 30 depends on the number of up-conversion units 24 (output buffers 25). One display device 30 is provided for one up-conversion unit 24 (output buffer 25). In the example shown in FIG. 2 , the super-resolution sub-frame images SR displayed on the four display devices 30a, 30b, 30c, and 30d are combined to synthesize (generate) one super-resolution wide image (super-resolution wide video).

[0042] In this embodiment, the output synchronization unit 26 individually outputs the super-resolution sub-frame images SR stored in each output buffer 25 to each display device 30, but this is not limiting. For example, the output synchronization unit 26 may combine multiple super-resolution sub-frame images SR to synthesize (combine) a super-resolution wide image, and then output the super-resolution wide image to the wide video display unit 3. In this case, a function (not shown) within the wide video synchronization control unit 2 may handle the distribution of the images to each display device 30 (i.e., the output ports corresponding to each display device 30) in the wide video synchronization control unit 2.

[0043] (Super-resolution wide-angle video generation method) Next, an example of a super-resolution wide-angle video generation method will be described with reference to Fig. 4. Fig. 4 is a flowchart of the super-resolution wide-angle video generation method according to the first embodiment. Note that the flowchart shown in Fig. 4 shows the flow of processing corresponding to one frame image FR. When processing multiple frame images FR, the processing shown in Fig. 4 is repeatedly executed for each frame image FR.

[0044] 4, the video input unit 21 acquires a frame image FR from an input video (S1), and transmits the frame image FR to the frame number assigning unit 22.

[0045] Next, the frame number assigning unit 22 assigns a frame number FN to the frame image FR. After assigning the frame number FN, the frame number assigning unit 22 increments the frame number FN (S2). This allows the frame number assigning unit 22 to assign the next frame number FN to the next frame image FR.

[0046] The cropping unit 23 crops multiple regions corresponding to a super-resolution wide image from one frame image FR in accordance with the cropping settings (S3). That is, the cropping unit 23 crops multiple subframe images SF from one frame image FR. Each of the multiple subframe images SF is assigned the frame number FN of the frame image FR from which it was cropped. The cropping unit 23 transmits the multiple subframe images SF assigned the same frame number FN to multiple upconversion units 24 (multiple threads). In other words, the cropping unit 23 transmits the multiple subframe images SF assigned the same frame number FN to the multiple threads. For example, if the cropping setting number is four, the cropping unit 23 transmits four different subframe images SF assigned the same frame number FN to the four upconversion units 24a, 24b, 24c, and 24d, respectively.

[0047] Each upconverter 24 upconverts the subframe image SF using AI super-resolution processing (S4). For example, Real-ESRGAN is used as the AI ​​super-resolution processing. That is, each upconverter 24 generates a super-resolution subframe image SR that has been subjected to AI super-resolution processing. The super-resolution subframe image SR is also assigned the same frame number FN as the subframe image SF before upconversion. Note that the processing time for the AI ​​super-resolution processing in each upconverter 24 may vary depending on the content of the subframe image SF to be processed.

[0048] Each upconverter 24 transmits the super-resolution sub-frame image SR and the frame number FN to the corresponding output buffer 25 (S5). That is, each upconverter 24 transmits the upconverted sub-frame image SF and the corresponding frame number FN to the corresponding output buffer 25. For example, the upconverters 24a, 24b, 24c, and 24d transmit different super-resolution sub-frame images SR assigned the same frame number FN to the output buffers 25a, 25b, 25c, and 25d, respectively.

[0049] The output synchronization unit 26 checks whether or not a super-resolution sub-frame image SR corresponding to the output frame number FN exists in each output buffer 25 (S6).

[0050] If a super-resolution sub-frame image SR corresponding to output frame number FN is present in each output buffer 25 (S6_Yes), the output synchronization unit 26 synchronizes and outputs the multiple super-resolution sub-frame images SR corresponding to output frame number FN to each of the multiple display devices 30 (S7). Each display device 30 displays the synchronized super-resolution sub-frame image SR. The super-resolution sub-frame images SR displayed on each display device 30 are combined to generate (combine) a single super-resolution wide video.

[0051] If there is an output buffer that does not contain a super-resolution sub-frame image SR corresponding to the output frame number FN (S6_No), the output synchronization unit 26 synchronizes and controls the super-resolution sub-frame image SR corresponding to the output frame number FN-1 stored in each output buffer 25 and outputs it to each of the multiple display devices 30 (S8).

[0052] Next, the output synchronization unit 26 deletes the super-resolution sub-frame image SR corresponding to the output frame number FN-1 from each output buffer 25 (S9).

[0053] Next, the output synchronization unit 26 increments the output frame number FN (S10).

[0054] (Hardware Configuration) An example of the hardware configuration of the super-resolution wide-image video generation device 1 will be described with reference to Fig. 5. Fig. 5 is a block diagram showing an example of the hardware configuration of the super-resolution wide-image video generation device 1.

[0055] 5, the super-resolution wide-screen video generation device 1 includes a wide-screen video synchronization control unit 2, a plurality of display devices 30, an input device 31, and an output device 32. In the example shown in Fig. 5, the super-resolution wide-screen video generation device 1 includes four display devices 30a, 30b, 30c, and 30d.

[0056] The wide-angle video synchronization control unit 2 is, for example, a computer, and includes a processor 2001, a ROM (Read Only Memory) 2002, a RAM (Random Access Memory) 2003, a storage medium 2004, and an input / output interface 2005.

[0057] The processor 2001, ROM 2002, RAM 2003, storage medium 2004, and input / output interface 2005 are electrically connected to one another via a bus 2006. The processor 2001, ROM 2002, RAM 2003, storage medium 2004, and input / output interface 2005 transmit and receive data or control signals via the bus 2006.

[0058] The processor 2001 is configured by a general-purpose hardware processor including, for example, a CPU (Central Processing Unit), a GPU (Graphical Processing Unit), etc. The processor 2001 controls the entire wide video synchronization control unit 2.

[0059] The ROM 2002 is a non-volatile memory. For example, the ROM 2002 is an erasable programmable read-only memory (EPROM). The ROM 2002 is a non-transitory storage medium that stores firmware, programs, and the like. For example, the processor 2001 loads firmware from the ROM 2002 into the RAM 2003 and executes the firmware.

[0060] The RAM 2003 is a volatile memory. The RAM 2003 is a dynamic random access memory (DRAM) or a static random access memory (SRAM). The RAM 2003 temporarily stores programs used in processing by the processor 2001 and data used to execute the programs. The processor 2001 executes the programs in the RAM 2003 to perform calculations on the data in the RAM 2003 and store the calculation results in the RAM 2003. For example, the RAM 2003 functions as the output buffer 25. Note that the wide video synchronization control unit 2 may include a volatile memory dedicated to the output buffer 25.

[0061] The storage medium 2004 is configured with a nonvolatile memory such as a hard disk drive (HDD) or a solid state drive (SSD). The storage medium 2004 non-temporarily stores programs executed by the processor 2001 and data required for executing the programs. For example, the storage medium 2004 stores a synchronization control processing program for executing synchronization control processing. The synchronization control processing program causes the wide video synchronization control unit 2 to execute at least some of the functions of the components of the wide video synchronization control unit 2, namely, the video input unit 21, the frame number assignment unit 22, the cropping unit 23, the upconversion unit 24, and the output synchronization unit 26. The storage medium 2004 also stores data used for synchronization control. The storage medium 2004 may also store video data (input video).

[0062] The program executed by the wide video synchronization control unit 2 may be provided to the wide video synchronization control unit 2 via a readable non-transitory storage medium (not shown). Such a storage medium is called a non-transitory computer-readable storage medium. Non-transitory computer-readable storage media include disks such as flexible disks, optical disks (CD-ROM, CD-R, DVD-ROM, DVD-R, etc.), and magneto-optical disks (MO, etc.), as well as semiconductor memories.

[0063] The input / output interface 2005 is connected to a plurality of display devices 30, an input device 31, and an output device 32. The input / output interface 2005 enables display of images on each display device 30, input of information from the input device 31, and output of information to the output device 32. The input / output interface 2005 may also be connected to an imaging device that captures input images, an external database in which the input images are stored, or the like. For example, the input / output interface 2005 may be a wired interface or a wireless interface. A wired interface includes a port to which a device is connected, for example. A wireless interface, for example, has a function that satisfies communication standards such as Bluetooth (registered trademark) and WiFi (registered trademark).

[0064] The display device 30 includes a display (such as an LCD (Liquid Crystal Display), an EL (Electroluminescence) display, a projector, or a cathode ray tube.) The display device 30 can display super-resolution wide images.

[0065] The input device 31 may include a keyboard, a mouse, a touch panel, a receiving device, a disk drive, etc. The input device 31 is not limited to these, and may include any other input device.

[0066] The output device 32 may include a transmitting device, a disk drive, etc. The output device 32 is not limited to these and may include any other output device. The input device 31 and the output device 32 may be configured as an input / output device having the functions of both the input device 31 and the output device 32.

[0067] When the super-resolution wide-screen video generation device 1 (wide-screen video synchronization control unit 2) is started, the processor 2001 executes a program in the ROM 2002 and loads and starts an OS (Operating System) into the RAM 2003. Under control of the OS, the processor 2001 monitors input instructions, connections to external devices, and the like. Under control of the OS, the processor 2001 also sets up a program area and a data area in the RAM 2003. In response to an instruction input to start the super-resolution wide-screen video generation device 1, the processor 2001 loads a synchronization control processing program from the storage medium 2004 into the program area of ​​the RAM 2003, and loads data used in executing the synchronization control processing program from the storage medium 2004 into the data area of ​​the RAM 2003. The processor 2001 calculates data in the data area in accordance with the synchronization control processing program and writes the calculation results to the data area. Through such operations, the processor 2001 , RAM 2003 , storage medium 2004 , input / output interface 2005 and bus 2006 work together to perform at least some of the functions of the components of the super-resolution wide-screen video generation device 1 .

[0068] (Effects of First Embodiment) With the configuration according to this embodiment, the super-resolution wide-angle video generation device 1 can acquire a frame image FR from an input video. The super-resolution wide-angle video generation device 1 can assign consecutive frame numbers FN to multiple frame images FR. The super-resolution wide-angle video generation device 1 can extract multiple regions (sub-frame images SF) from a single frame image FR. The super-resolution wide-angle video generation device 1 can perform up-conversion of multiple sub-frame images SF assigned the same frame number FN in parallel. In other words, the super-resolution wide-angle video generation device 1 can extract multiple regions from a single input video image and distribute the AI ​​super-resolution processing of each region across multiple threads. The super-resolution wide-angle video generation device 1 can synchronize and present multiple super-resolution sub-frame images SR (up-converted sub-frame images SF) assigned the same frame number FN. This allows the super-resolution wide-angle video generation device 1 to present high-resolution, high-definition wide-angle video without any misalignment between the images.

[0069] Furthermore, with the configuration according to this embodiment, if the processing result for the corresponding output frame number FN is not stored in the output buffer 25 at the output timing according to the frame rate, i.e., if the generation of the super-resolution sub-frame image SR is not complete, the super-resolution wide image generation device 1 can present multiple super-resolution sub-frame images SR assigned the previous output frame number FN-1 in a synchronized manner, thereby enabling the super-resolution wide image generation device 1 to reduce misalignment between images.

[0070] Furthermore, with the configuration according to this embodiment, the super-resolution wide image generation device 1 can divide the cropped region into multiple parts, thereby making it possible to set the aspect ratio and resolution of the subframe images SF to those of general images. This makes it possible to use AI super-resolution technology that has learned using existing image data as training data when upconverting the subframe images SF.

[0071] Furthermore, with the configuration according to this embodiment, the super-resolution wide-angle video generation device 1 can use video captured by a single ordinary camera as input video, making it possible to present high-resolution wide-angle video by switching between multiple cameras installed in different locations in a venue, for example.

[0072] Furthermore, with the configuration according to this embodiment, the super-resolution wide-screen video generation device 1 can change the number of sub-frame images SF cut out for one frame image FR. For example, when live streaming super-resolution wide-screen video to multiple locations, it is possible to present super-resolution wide-screen video that corresponds to the situation of each display venue (two screens / three screens / four screens). Furthermore, the super-resolution wide-screen video generation device 1 can generate super-resolution wide-screen video from past video.

[0073] (Second embodiment) Next, a second embodiment will be described. In the second embodiment, a functional configuration and synchronization control process of the super-resolution wide image generation device 1 that differs from those of the first embodiment will be described. The super-resolution wide image generation device 1 according to the second embodiment can also handle cases where the frame rate of the input image and the frame rate of the output super-resolution wide image are different. The following description will focus on the differences from the first embodiment.

[0074] (Functional Configuration of Super-Resolution Wide Video Generation Device) First, an example of the functional configuration of the super-resolution wide video generation device 1 will be described with reference to Fig. 6. Fig. 6 is a block diagram showing an example of the functional configuration of the super-resolution wide video generation device 1 according to the second embodiment. The example shown in Fig. 6 shows a case where the super-resolution wide video generation device 1 divides a wide image into four regions (images) and performs upconversion (super-resolution processing) of each image in parallel (distributed). Note that the number of regions (images) that the super-resolution wide video generation device 1 processes in parallel may be two or more.

[0075] As shown in FIG. 6, the super-resolution wide-screen video generation device 1 of this embodiment has a configuration in which the frame number assignment unit 22 is omitted from the super-resolution wide-screen video generation device 1 of the first embodiment described with reference to FIG.

[0076] The internal configuration of the wide image synchronization control unit 2 will now be described.

[0077] The wide video synchronization control unit 2 includes, for example, a video input unit 21, a cropping unit 23, a plurality of up-conversion units 24, a plurality of output buffers 25, and an output synchronization unit 26. In the example shown in Fig. 6, the wide video synchronization control unit 2 includes four up-conversion units 24a, 24b, 24c, and 24d, and four output buffers 25a, 25b, 25c, and 25d.

[0078] The video input unit 21 has the same functions as in the first embodiment. The video input unit 21 transmits a frame image FR to the cropping unit 23.

[0079] As in the first embodiment, the cropping unit 23 crops (cuts out) multiple regions corresponding to a wide image from the frame image FR to be cropped. In the example shown in FIG. 6 , the cropping unit 23 crops four subframe images SF from one frame image FR. Note that frame numbers FN are not assigned to the subframe images SF in this embodiment. The cropping unit 23 transmits the four subframe images SF cropped from one frame image FR to four upconversion units 24a, 24b, 24c, and 24d, respectively. In other words, the cropping unit 23 transmits the four subframe images SF to four threads, respectively.

[0080] As in the first embodiment, the upconversion unit 24 performs upconversion of the subframe images SF using, for example, AI super-resolution processing technology. The upconversion unit 24 generates super-resolution subframe images SR. Note that the super-resolution subframe images SR are not assigned frame numbers FN.

[0081] The up-converter 24a transmits the super-resolution sub-frame image SR to the output buffer 25a. Similarly, the up-converter 24b transmits the super-resolution sub-frame image SR to the output buffer 25b. The up-converter 24c transmits the super-resolution sub-frame image SR to the output buffer 25c. The up-converter 24d transmits the super-resolution sub-frame image SR to the output buffer 25d.

[0082] In this embodiment, the output buffer 25 temporarily stores, for example, two super-resolution sub-frame images SR. The number of super-resolution sub-frame images SR that the output buffer 25 can store can be set to any number. For example, the output buffer 25 outputs the stored super-resolution sub-frame images SR in a last-in-first-out (LIFO) format. In other words, the most recent super-resolution sub-frame image SR input to the output buffer 25 is the one to be output.

[0083] The output buffer 25a temporarily stores the super-resolution sub-frame image SR received from the up-conversion unit 24a. Similarly, the output buffer 25b temporarily stores the super-resolution sub-frame image SR received from the up-conversion unit 24b. The output buffer 25c temporarily stores the super-resolution sub-frame image SR received from the up-conversion unit 24c. The output buffer 25d temporarily stores the super-resolution sub-frame image SR received from the up-conversion unit 24d.

[0084] Each of the output buffers 25a, 25b, 25c, and 25d outputs the super-resolution sub-frame image SR to the output synchronization unit 26 in the LIFO format under the synchronization control of the output synchronization unit 26.

[0085] The output synchronization unit 26 synchronizes and controls the multiple super-resolution sub-frame images SR to be output to the wide video display unit 3. In this embodiment, the output synchronization unit 26 outputs the super-resolution sub-frame image SR at the head of each output buffer 25 to the corresponding display device 30 at an output timing according to the output frame rate. For example, the head of the output buffer 25 refers to the address position of the register selected as the output target when outputting the super-resolution sub-frame image SR from the output buffer 25.

[0086] Unlike the first embodiment, the output synchronization unit 26 outputs the super-resolution sub-frame images SR at output timings that are in accordance with the output frame rate, rather than through synchronization control based on the frame number FN. When the frame rate of the input video and the frame rate of the output super-resolution wide video are the same, the output of the super-resolution sub-frame images SR to the output buffer 25 and the output by the output synchronization unit 26 to the display device 30 will also be the same as the frame rate of the input video. As a result, synchronization between the input video and the output video (super-resolution wide video) is possible without using the method of managing and comparing frame numbers FN as described in the first embodiment.

[0087] The output synchronization unit 26 controls the output buffer 25 to copy the output super-resolution sub-frame images SR and store them in the output buffer 25. At this time, if the number of super-resolution sub-frame images SR stored in the output buffer 25 exceeds the maximum number that can be stored (for example, two), the output synchronization unit 26 deletes the oldest super-resolution sub-frame image SR.

[0088] The other configurations are the same as those of the first embodiment.

[0089] (Super-resolution wide-angle video generation method) Next, an example of a super-resolution wide-angle video generation method will be described with reference to Fig. 7. Fig. 7 is a flowchart of a super-resolution wide-angle video generation method according to a second embodiment. The flowchart shown in Fig. 7 shows the flow of processing for one frame image FR. When processing multiple frame images FR, the processing shown in Fig. 7 is repeatedly executed for each frame image FR.

[0090] 7, the video input unit 21 acquires a frame image FR from an input video (S21), and transmits the frame image FR to the cropping unit 23.

[0091] The cropping unit 23 crops multiple regions corresponding to a super-resolution wide image from one frame image FR in accordance with the cropping settings (S22). That is, the cropping unit 23 crops multiple subframe images SF from one frame image FR. The cropping unit 23 transmits the multiple subframe images SF to multiple threads, respectively. For example, if the cropping setting number is four, the cropping unit 23 transmits the four subframe images SF cropped from one frame image to the four upconversion units 24a, 24b, 24c, and 24d, respectively.

[0092] Each upconverter 24 upconverts the subframe image SF using AI super-resolution processing (S23). For example, Real-ESRGAN is used as the AI ​​super-resolution processing. That is, each upconverter 24 generates a subframe image SF that has been subjected to AI super-resolution processing as a super-resolution subframe image SR.

[0093] Each upconverter 24 transmits the super-resolution sub-frame image SR to the corresponding output buffer 25 (S24). That is, each upconverter 24 transmits the upconverted sub-frame image SF to the corresponding output buffer 25. For example, the upconverters 24a, 24b, 24c, and 24d transmit the super-resolution sub-frame image SR to the output buffers 25a, 25b, 25c, and 25d, respectively. The received super-resolution sub-frame image SR is stored at the top of the output buffers 25a, 25b, 25c, and 25d.

[0094] The output synchronization unit 26 synchronously controls the super-resolution sub-frame images SR at the head of each output buffer 25 and outputs them to each of the plurality of display devices 30 at the output timing according to the output frame rate (S25).

[0095] The output synchronization unit 26 copies the super-resolution sub-frame image SR output from the output buffer 25 and stores it in the output buffer 25 (S26). The copied super-resolution sub-frame image SR is stored (shifted) to a position other than the head of the output buffer 25.

[0096] At this time, each output buffer 25 checks the number of super-resolution sub-frame images SR stored therein (S27).

[0097] If the number of super-resolution sub-frame images SR in the output buffer 25 exceeds the setting due to the storage of the copied super-resolution sub-frame images SR (S27_No), for example, if the number of super-resolution sub-frame images SR in the output buffer 25 becomes three, the oldest super-resolution sub-frame image SR in the output buffer 25 is deleted.

[0098] Even if the copied super-resolution sub-frame images SR are saved, if the number of super-resolution sub-frame images SR in the output buffer 25 is less than the set value (S27_Yes), for example, if there are two or fewer super-resolution sub-frame images SR in the output buffer 25, the super-resolution sub-frame images SR are not deleted in the output buffer 25.

[0099] (Specific example of how super-resolution sub-frame images are managed in the output buffer) Next, with reference to FIG. 8 , a specific example of how super-resolution sub-frame images SR are managed in the output buffer 25 will be described. FIG. 8 is a diagram showing a specific example of how super-resolution sub-frame images SR are managed in the output buffer 25. The example shown in FIG. 8 shows a case where the frame rate of the input video and the frame rate of the output video are the same. In FIG. 8 , the processing time indicates the processing time of the AI ​​super-resolution processing in the up-conversion unit 24. The output interval indicates the output time interval at which the output synchronization unit 26 outputs the super-resolution sub-frame images SR. The output timings of frames 1 to 3 indicate the output timings of the three output frames 1 to 3 corresponding to the output frame rate.

[0100] First, we will explain the case where the processing time is equal to or shorter than the output interval. In this case, the AI ​​super-resolution processing corresponding to the output frame is completed by the output timing. Therefore, the super-resolution sub-frame image SR to be output next is stored at the beginning of the output buffer 25. At this time, a super-resolution sub-frame image SR corresponding to the previous output frame is stored at a position other than the beginning of the output buffer 25, so that it can be output if the AI ​​super-resolution processing is delayed. The output synchronization unit 26 outputs the super-resolution sub-frame image SR corresponding to the output frame at the output timing of the output frame.

[0101] As shown in the upper part of FIG. 8 , more specifically, by the time frame 1 is output, a super-resolution sub-frame image SR1 corresponding to output frame 1 is stored at the top of the output buffer 25. The output synchronization unit 26 outputs the super-resolution sub-frame image SR1 at the top of the output buffer 25 in synchronization with the output timing of frame 1. Next, by the time frame 2 is output, a super-resolution sub-frame image SR2 corresponding to output frame 2 is stored at the top of the output buffer 25. The super-resolution sub-frame image SR1 corresponding to the previous output frame 1 is stored at a position other than the top of the output buffer 25. The output synchronization unit 26 outputs the super-resolution sub-frame image SR2 at the top of the output buffer 25 in synchronization with the output timing of frame 2. Next, by the time frame 3 is output, a super-resolution sub-frame image SR3 corresponding to output frame 3 is stored at the top of the output buffer 25. The super-resolution sub-frame image SR2 corresponding to the previous output frame 2 is stored at a position other than the top of the output buffer 25. The oldest super-resolution sub-frame image SR1 is deleted because the number of super-resolution sub-frame images SR that can be stored in the output buffer 25 has been exceeded. The output synchronization unit 26 outputs the super-resolution sub-frame image SR3 at the top of the output buffer 25 in synchronization with the output timing of frame 3.

[0102] Next, we will explain what happens when the processing time is longer than the output interval. If the AI ​​super-resolution processing in the upconversion unit 24 is not completed in time for the output timing, the super-resolution sub-frame image SR corresponding to the previous output frame is stored at the top of the output buffer 25. The output synchronization unit 26 outputs the previous super-resolution sub-frame image SR at the top of the output buffer 25. Thereafter, when the AI ​​super-resolution processing is completed, the corresponding super-resolution sub-frame image SR is stored at the top of the output buffer 25.

[0103] As shown in the middle of Figure 8, for example, if the AI ​​super-resolution processing has not been completed by the output timing of frame 2, the super-resolution sub-frame image SR1 corresponding to the immediately previous output frame 1 is stored at the top of the output buffer 25. The output synchronization unit 26 outputs the super-resolution sub-frame image SR1 at the top of the output buffer 25 in synchronization with the output timing of frame 2. Thereafter, when the AI ​​super-resolution processing corresponding to output frame 2 is completed, the super-resolution sub-frame image SR2 is stored at the top of the output buffer 25. The super-resolution sub-frame image SR1 corresponding to the immediately previous output frame 1 is stored at a position other than the top of the output buffer 25.

[0104] As shown in the lower part of Figure 8, if the AI ​​super-resolution processing is not completed even at the output timing of frame 3 following the output timing of frame 2, the super-resolution sub-frame image SR2 corresponding to the immediately previous output frame 2 is stored at the top of the output buffer 25. The output synchronization unit 26 outputs the super-resolution sub-frame image SR2 at the top of the output buffer 25 in synchronization with the output timing of frame 3. Thereafter, when the AI ​​super-resolution processing corresponding to output frame 3 is completed, the super-resolution sub-frame image SR3 is stored at the top of the output buffer 25. At this time, the super-resolution sub-frame image SR2 corresponding to the immediately previous output frame 2 is stored at a position other than the top of the output buffer 25. Because the number of super-resolution sub-frame images SR that can be stored in the output buffer 25 exceeds the capacity of the output buffer 25, the oldest super-resolution sub-frame image SR1 is deleted.

[0105] The difference in processing time between the upconverters 24 is within an output time interval (e.g., 16.6 ms for 60 fps) determined from the output frame rate. For example, if the difference in processing time is twice or more the output time interval, it is preferable to take measures such as lowering the input frame rate and the output frame rate or changing the AI ​​super-resolution processing performed by the upconverter 24 to a method with a shorter processing time.

[0106] (Effects of the Second Embodiment) With the configuration according to this embodiment, the super-resolution wide-angle video generation device 1 can acquire a frame image FR from an input video. The super-resolution wide-angle video generation device 1 can extract multiple regions (sub-frame images SF) from a single frame image FR. The super-resolution wide-angle video generation device 1 can perform upconversion of multiple sub-frame images SF extracted from a single frame image FR in parallel. In other words, the super-resolution wide-angle video generation device 1 can extract multiple regions from a single input video and distribute the AI ​​super-resolution processing of each region across multiple threads. The super-resolution wide-angle video generation device 1 can synchronize and present multiple super-resolution sub-frame images SR (up-converted sub-frame images SF). This allows the super-resolution wide-angle video generation device 1 to present high-resolution, high-definition wide-angle video without any misalignment between the images.

[0107] Furthermore, with the configuration according to this embodiment, the super-resolution wide image generation device 1 can divide the cropped region into multiple parts, thereby making it possible to set the aspect ratio and resolution of the subframe images SF to those of general images. This makes it possible to use AI super-resolution technology that has learned using existing image data as training data when upconverting the subframe images SF.

[0108] Furthermore, with the configuration according to this embodiment, the super-resolution wide-angle video generation device 1 can use video captured by a single ordinary camera as input video, making it possible to present high-resolution wide-angle video by switching between multiple cameras installed in different locations in a venue, for example.

[0109] Furthermore, with the configuration according to this embodiment, the super-resolution wide-screen video generation device 1 can change the number of sub-frame images SF to be cut out for one frame image FR. For example, when live streaming super-resolution wide-screen video to multiple locations, it is possible to present a wide-screen video display that suits the situation at each display venue (two screens, three screens, or four screens). Furthermore, the super-resolution wide-screen video generation device 1 can generate super-resolution wide-screen video from past video.

[0110] Other Embodiments The present invention is not limited to the above-described embodiment, and can be implemented in various modifications without departing from the spirit and scope of the present invention.

[0111] For example, in the above embodiment, a case where a horizontally long super-resolution wide image is generated has been described, but the super-resolution wide image may also be vertically long. In this case, multiple vertically aligned regions are cut out from a frame image. Super-resolution processing is performed on each region in parallel. Then, a vertically long super-resolution wide image is generated by displaying the multiple regions aligned vertically.

[0112] In short, this invention is not limited to the above-described embodiments, and in the implementation stage, the components can be modified and embodied without departing from the spirit of the invention. Furthermore, various inventions can be formed by appropriately combining multiple components disclosed in the above-described embodiments. For example, some components may be omitted from all the components shown in the embodiments. Furthermore, components from different embodiments may be appropriately combined.

[0113] 1... Super-resolution wide image generating device 2... Wide image synchronization control unit 3... Wide image display unit 21... Image input unit 22... Frame number assignment unit 23... Cut-out unit 24, 24a, 24b, 24c, 24d... Up-conversion unit 25, 25a, 25b, 25c, 25d... Output buffer 26... Output synchronization unit 30, 30a, 30b, 30c, 30d... Display device 31... Input device 32... Output device 2001... Processor 2002... ROM 2003... RAM 2004... Storage medium 2005... Input / output interface 2006... Bus

Claims

1. A super-resolution wide-angle video generation device comprising: a video input unit that acquires a frame image from an input video; a cropping unit that crops out a plurality of sub-frame images from the frame image; a plurality of up-conversion units that execute super-resolution processing on each of the plurality of sub-frame images in a distributed manner to generate a plurality of super-resolution sub-frame images; a plurality of output buffers that are respectively connected to the plurality of up-conversion units and each store one of the plurality of super-resolution sub-frame images; and an output synchronization unit that synchronizes and outputs the plurality of super-resolution sub-frame images.

2. The super-resolution wide image generating device of claim 1, further comprising a frame number assigning unit that assigns different frame numbers to the plurality of frame images, and the output synchronization unit synchronizes and outputs the plurality of super-resolution sub-frame images corresponding to the frame numbers.

3. The super-resolution wide image generating device of claim 2, wherein, at an output timing based on the frame number, if at least one of the plurality of output buffers does not store any one of the plurality of super-resolution sub-frame images corresponding to the frame number, the output synchronization unit outputs a super-resolution sub-frame image corresponding to the frame number immediately preceding the frame number stored in each of the plurality of output buffers.

4. The super-resolution wide image generating device according to claim 3, wherein after output based on the frame number, the super-resolution sub-frame image corresponding to the frame number immediately preceding the frame number stored in each of the plurality of output buffers is deleted.

5. The super-resolution wide-angle image generating device according to claim 1, wherein the cutout positions of the plurality of sub-frame images are different from each other.

6. The super-resolution wide image generating device of claim 1, wherein each of the plurality of output buffers is managed in a LIFO (Last-In-First-Out) format, and is configured such that when a first super-resolution sub-frame image based on a first frame image and a second super-resolution sub-frame image based on a second frame image acquired after the first frame image are stored in the output buffer, and a third super-resolution sub-frame image based on a third frame image acquired after the second frame image is input, the third super-resolution sub-frame image is output and the first super-resolution sub-frame image is deleted.

7. A method for generating a super-resolution wide image, comprising: acquiring a frame image from an input image; extracting a plurality of sub-frame images from the frame image; performing super-resolution processing on each of the plurality of sub-frame images in a distributed manner to generate a plurality of super-resolution sub-frame images; and outputting the plurality of super-resolution sub-frame images in synchronization.

8. A program including instructions for causing a computer to function as the super-resolution wide-screen image generating device according to any one of claims 1 to 6.

Images