Computer program product, content video playing device, content video playing method and content video data generating device

Abstract

To provide a content video playback program, a content video playback device, a content video playback method, a content video data generation program, and a content video data generation device capable of playing back a video of a suspended matter in the atmosphere such as a cloud, fog, a halo, a thermal, a gas, or the like at high quality with few computer resources, a video playback processing section that plays back compressed video data generates video data at high reproducibility by only performing a simple arithmetic processing of moving raster cell image data by a raster vector and performing an interpolation operation processing of luminance.

Description

Technical Field

[0001] This invention relates to a video content playback program, video content playback device, video content playback method, video content data generation program, and video content data generation device, which are particularly suitable for video content suspended in the atmosphere such as clouds, fog, haze, heat, and gases. Background Technology

[0002] The applicant develops and publishes game programs and game services. Many game service providers, represented by the applicant, offer the following online game services: personal computers and smartphones connect to game servers that provide game content via the Internet, and receive a wide variety of game content through these game servers.

[0003] One aspect of the games offered by such game service providers is their multi-functional content playback service. This service integrates not only elements of action games where game characters (hereinafter referred to as "characters") move according to user input, but also elements of novel games, animated videos, and web browsers where the story unfolds through dialogue between characters. Furthermore, in some game content, an imagined world is presented as the character's activity space.

[0004] In one of the game services provided by the applicant, animated videos are used to display flowing clouds as one of the means of representing the imagined world.

[0005] The inventors believe that, as a means of representing the imagined world within a game, depicting clouds with high quality, realism, and a sense of presence is extremely important. No matter how beautifully drawn the characters are, if the background is of caricature or graffiti quality, it will fail to draw players into the game's world. Beautiful characters paired with beautiful backgrounds are essential for players to identify with the world's concept and become immersed in it.

[0006] The technique of beautifully depicting flowing clouds has a long history.

[0007] Rendering clouds requires calculating the scattering of sunlight within the cloud. Research on rendering this scattered light has been ongoing since the 1980s. For example, basic algorithms have been developed, including a first-order scattering approximation (Non-Patent Document 1), volumetric rendering considering multi-order scattering (Non-Patent Document 2), and a global illumination model for clouds (Non-Patent Document 3). In the 1990s, this was applied to real-time rendering using GPUs (Graphics Processing Units). In the 2010s, methods for real-time rendering of clouds as volumetric data were established using OpenVDB (https: / / www.openvdb.org), developed by DreamWorks Animation. However, rendering volumetric data is computationally expensive, and a method for efficiently and effectively rendering the wide-view clouds required for the game service provided by the applicant has not yet been established (Non-Patent Document 4).

[0008] Patent document 1 discloses the following technique: calculating the velocity vector corresponding to each pixel in a motion image based on two images that are different in time, and obtaining the velocity field vector by arranging the velocity vectors into a coordinate array.

[0009] Patent Document 2 discloses a technique for estimating model parameters of a moving object. The model parameters are estimated by an objective function of a mathematical model that exhibits characteristics of wave phenomena involving both continuous and discontinuous motion. This mathematical model is formed by combining a wave generation equation representing the time-series variation of image brightness with an optical flow model, which serves as a motion estimation model in image processing. The objective function is set to include constraints based on wave-based dispersion relations.

[0010] Existing technical documents

[0011] Patent documents

[0012] Patent Document 1: Japanese Patent Application Publication No. 2002-074369

[0013] Patent Document 2: Japanese Patent No. 6196597

[0014] Non-patent literature

[0015] Non-Patent Document 1: Blinn, J.: Light Reflection Functions for Simulation of Clouds and Dusty Surfaces, SIGGRAPH 1982, pp.21-29 (1982), Internet <https: / / ohiostate.pressbooks.pub / app / uploads / sites / 45 / 201710 / blinn-dusty.pdf>

[0016] Non-Patent Document 2: Kajiya, J. and Von Herzen, B.: Ray Tracing Volume Densities, SIGGRAPH 1984, pp.165-174 (1984), Internet <https: / / www.researchgate.net / profile / Brian_Von_Herzen3 / publication / 242588930_Ray_tracing_volume_densities_computer_graphics_18 / links / 00b7d53931e71dfc04000000 / Ray-tracing-volume-densities-computer-graphics-18.pdf>

[0017] Non-Patent Document 3: Nishita, T., Dobashi, Y. and Nakamae, E.: Display of Clouds Taking into Account Multiple Anisotropic Scattering and Sky Light, SIGGRAPH 1996, pp.379-386 (1996).

[0018] Non-Patent Document 4: The Extraordinary Attention to the "Cloud" Expression in "Granblue Fantasy Project Re:LINK" [GCC’17], Famitsu.com, Kadokawa Game Linkage Co., Ltd., Internet <https: / / www.famitsu.com / news / 201703 / 22129373.html> Summary of the Invention

[0019] Problems to be Solved by the Invention

[0020] In particular, from a computer's perspective, the computing power of a smartphone's CPU and GPU, as well as the storage capacity of its memory and storage devices, are limited resources. Recent game content tends to consume a large amount of computer resources, and the computer resources available for rendering background clouds are limited. The inventors considered playing high-quality cloud videos with as few computer resources as possible.

[0021] The present invention was made in view of the above-mentioned problems, and its object is to provide a content video playback program, content video playback device, content video playback method, content video data generation program, and content video data generation device that can play high-quality videos of clouds, fog, haze, heat, gases, and other suspended particles in the atmosphere with minimal computer resources.

[0022] Solution for solving the problem

[0023] To address the aforementioned problems, the present invention provides a content video playback program for enabling a computer to function as a content video playback device. This content video playback device comprises a video playback processing unit, a main game content generation processing unit, and an image compositing processing unit. The video playback processing unit extracts compressed video from a compressed video file and outputs video data. The main game content generation processing unit reads various game data from a game data set and generates various videos based on operations performed by the operation unit. The image compositing processing unit combines the video data generated by the main game content generation processing unit with the video data generated by the video playback processing unit and outputs the result to a display unit.

[0024] The content video playback program of the present invention is a program for implementing the functions shown in (a) and (b) below.

[0025] (a) Content data extraction and processing function: The data extraction and processing unit extracts from the compressed video file to extract the current frame image data as the reference time point, the next frame image data after a specified time from the reference time point, and the raster vector group as a collection of raster lists that enumerate raster vectors. The raster vectors represent the amount and direction of movement of the raster unit image data, which is subdivided from the current frame image data, along the coordinate direction within the screen at the time point of the next frame image data.

[0026] (b) Interpolation processing function: The interpolation processing unit performs frame interpolation between the current frame image data and the next frame image data based on the current frame image data, the next frame image data, the raster vector group, and the raster list, and outputs video data.

[0027] The effects of the invention

[0028] According to the present invention, a content video playback program, a content video playback device, a content video playback method, a content video data generation program, and a content video data generation device are provided that can play high-quality videos of atmospheric suspended particles such as clouds, fog, haze, heat, and gases with minimal computer resources.

[0029] The problems, structures, and effects other than those described above will become clear through the following description of the implementation methods. Attached Figure Description

[0030] Figure 1 This is a schematic diagram illustrating a content playback device and its display screen according to an embodiment of the present invention.

[0031] Figure 2 This is a block diagram showing the hardware structure of a content playback device.

[0032] Figure 3 This is a block diagram illustrating the software functions of a content playback device.

[0033] Figure 4 This is a block diagram illustrating the internal functions of the cloud video playback processing unit.

[0034] Figure 5 It is a schematic diagram showing the state of the raster and raster cells applied to the current frame image data.

[0035] Figure 6 This is a schematic diagram illustrating an example of a format for compressed cloud video files.

[0036] Figure 7 This is a block diagram illustrating the internal functions of the interpolation processing unit.

[0037] Figure 8 It is a summary diagram illustrating the raster relationship between the current frame image data and the next frame image data.

[0038] Figure 9 This is a summary diagram illustrating frame interpolation.

[0039] Figure 10 This is a block diagram illustrating the hardware structure of a device for generating compressed cloud video files.

[0040] Figure 11 This is a block diagram illustrating the software functions of the device for generating compressed cloud video files.

[0041] Figure 12 This is a flowchart illustrating the operation of the device for generating compressed cloud video files.

[0042] Figure 13 It is a summary diagram illustrating the relationship between the upper-level raster vector and the lower-level raster vector. Detailed Implementation

[0043] [Content playback device 101]

[0044] Figure 1 This is a schematic diagram showing the content playback device 101a and content playback device 101b according to embodiments of the present invention.

[0045] exist Figure 1 In this context, content playback device 101a is a known personal computer, and content playback device 101b is a known smartphone.

[0046] A personal computer reads the game program and functions as a content playback device 101a. The content playback device 101a includes a keyboard 103a and a display unit 102a such as an LCD monitor. Users operate the keyboard 103a and pointing devices such as a mouse on the content playback device 101a to enjoy the game.

[0047] The smartphone reads the game program and functions as a content playback device 101b. The user operates the touch panel display 102b of the content playback device 101b to enjoy the game.

[0048] Content playback devices 101a and 101b differ in their built-in CPU specifications, but they are capable of playing the same game content. Furthermore, without distinguishing between content playback devices 101a and 101b, they will be referred to hereafter as content playback device 101. Additionally, without distinguishing between the display unit 102a of content playback device 101a and the touch panel display 102b of content playback device 101b as display functions, they will be referred to as display unit 102.

[0049] A video P104 of flowing clouds from game content, played by the game program executed by the content playback device 101, is displayed on the display unit 102. By playing this cloud video, the imagined world within the game content can be effectively represented. For example, Granblue Fantasy (registered trademark), which provides services for the game content, can be cited as an example.

[0050] Figure 2 This is a block diagram showing the hardware structure of the content playback device 101b. As is well known, the computer content playback device 101b includes a CPU 201, ROM 202, RAM 203, display unit 204, operation unit 205, wide area wireless communication unit 206, wireless LAN interface 207, and non-volatile storage device 208 connected to the bus 209.

[0051] The content playback device 101b is a smartphone, so the operation unit 205 is an electrostatic touch panel, and the display unit 204 and the operation unit 205 constitute the touch panel display 102b.

[0052] The non-volatile storage device 208 stores a game program that enables the computer to function as a content playback device 101.

[0053] In the case where the content playback device 101 is a content playback device 101a composed of a personal computer, the operation unit 205 is a keyboard 103a, a mouse, etc. Moreover, in the content playback device 101a, the wide area wireless communication unit 206 is not required, and instead a NIC (Network Interface Card) (not shown) is connected to the bus 209.

[0054] As explained above, aside from minor differences, the main information processing functions of content playback devices 101a and 101b are almost identical. Therefore, they can perform functions equally. Figure 3 The software features will be explained later.

[0055] Figure 3 This is a block diagram illustrating the software functions of the content playback device 101.

[0056] The main game content generation and processing unit 301 is the core of the game engine. The main game content generation and processing unit 301 reads the game data group 302, accepts the operation from the operation unit 205, and generates various videos and still images.

[0057] The cloud video playback processing unit 303, which is a video playback processing unit, has the cloud video generation function involved in this invention, reads the compressed cloud video file 304 as compressed video data, and generates cloud video data 305.

[0058] The image compositing processing unit 306 combines the video data generated by the main game content generation processing unit 301 with the cloud video data 305 generated by the cloud video playback processing unit 303 and outputs it to the display unit 204.

[0059] Here, cloud video data is merely an example of an implementation method, and the present invention is not limited to playing cloud video data.

[0060] Figure 4 This is a block diagram illustrating the internal functions of the cloud video playback processing unit 303.

[0061] The compressed cloud video file 304 is read by the data extraction and processing unit 401.

[0062] The data extraction and processing unit 401 outputs the current frame image data 402, the next frame image data 403, the raster vector group 404, and the raster list 405 from the compressed cloud video file 304. Furthermore, the data extraction and processing unit 401 extracts and outputs the current frame image data 402, the next frame image data 403, and the raster vector group 404 from the main body (payload data) of the compressed cloud video file 304. However, for the raster list 405, the data extraction and processing unit 401 calculates and outputs the raster list 405 based on the header of the compressed cloud video file 304.

[0063] The current frame image data 402 is the image data at the reference time point.

[0064] The next frame image data 403 is the image data at a time point after a specified time has elapsed from the reference time point.

[0065] Raster vector group 404 is a collection (list or array variable) of raster vectors. A raster vector is a vector representing the amount and direction of movement of a raster cell image data subdivided from the current frame image data 402 along the coordinate direction within the frame at the time point of the next frame image data 403.

[0066] Raster list 405 is a collection (list or array variable) of raster cells. Raster list 405 is calculated based on the size of the frame image data and the raster information contained in the header information of the compressed cloud video file 304, which will be described later.

[0067] Although Figure 4 The explanation is only halfway through, but the grid list 405 and grid cell will be explained here with reference to the attached diagram.

[0068] Figure 5 This is a schematic diagram showing the state of the raster G501 and raster unit C502 applied to the current frame image data 402.

[0069] Figure 5 This illustrates the state of displaying the current frame image data 402 at a given time point on a specified display device. For this still image, a grid G501 (cell) with 5 rows and 8 columns is formed. Figure 5 In the middle, vertical and horizontal dashed lines are used to represent the grid G501.

[0070] also, Figure 5 The 5-row, 8-column grid G501 shown is an example for illustration; in actual devices, grid G501 is constructed with more rows and columns.

[0071] Grid cell C502 refers to a specific element in grid list 405. That is, the address of grid cell C502 is output, using the grid cell number relative to grid list 405 as the argument. Figure 5 In this context, each mass that makes up a grid is equivalent to a grid cell C502.

[0072] Here, the grid cell number indicates the feature number of a specific grid cell C502 in the grid list 405. That is, the grid cell number is assigned to the grid cell C502 horizontally from the top left to the bottom right.

[0073] exist Figure 5 In the grid, the top-left grid cell is assigned grid number 1. The grid cell to the right of grid cell number 1 is assigned grid number 2, its right-hand neighbor is assigned grid number 3, its right-hand neighbor is assigned grid number 4, and so on, until the rightmost grid cell is assigned grid number 8. Grid cell number 8 is the rightmost cell in the first row of the grid cell, so the leftmost grid cell in the next row is assigned grid number 9. This process continues, assigning grid numbers from the top-left grid cell to the bottom-right grid cell, with the bottom-rightmost grid cell assigned grid number 40.

[0074] A raster cell address refers to the address information of a specific raster cell in raster list 405. Here, the address information is represented by a combination of the pixel's x-axis number and y-axis number on the frame image. A raster cell can be determined based on the address information (x1, y1) of the top-left pixel and the address information (x2, y2) of the bottom-right pixel.

[0075] Therefore, when the grid cell number is assigned as an independent variable to the grid list 405, the address information (x1, y1) of the upper left pixel and the address information (x2, y2) of the lower right pixel of the grid cell corresponding to that grid cell number can be obtained.

[0076] Furthermore, in this embodiment, the grid list 405 is described as an array variable in the program, but it is also possible to install the grid list 405 as a function in the program.

[0077] Return to Figure 4 Continuing with the explanation of the function blocks.

[0078] The interpolation processing unit 406 performs frame interpolation operations between the current frame image data 402 and the next frame image data 403 based on the current frame image data 402, the next frame image data 403, the raster vector group 404, and the raster list 405, and outputs cloud video data 305.

[0079] Before explaining the operation of the interpolation processing unit 406, refer to Figure 6 This explains the 304 format (compressed video data structure) for compressed cloud video files.

[0080] Figure 6 This is a schematic diagram illustrating an example of the format of a compressed cloud video file 304.

[0081] The compressed cloud video file 304 consists of a header D601 and a body D602 (payload data).

[0082] The header D601 contains

[0083] This file indicates that it is a compressed cloud video file, and the identification information is D603.

[0084] The data format (jpg, png, tiff, etc.) of the frame images stored in the main body D602, as well as image specification information such as pixel size of the frame images stored in the main body D602, are shown in D604.

[0085] Grid cell size information D605 (vertical and horizontal pixel dimensions)

[0086] wait.

[0087] Here, Figure 4 The number of features in the raster vector group 404 and the number of features in the raster list 405 can be derived by dividing the pixel size of the frame image data by the size of the raster cell. Therefore, the data extraction processing unit 401 performs this division operation, thereby determining the number of features based on the raster data. Figure 6 The header D601 shown is used to deduce the number of vector features and the number of raster cells.

[0088] Frame image data D606 consists of image data where all pixels are identical.

[0089] If the first frame image data D606a is the current frame image data 402, then the second frame image data D606b becomes the next frame image data 403.

[0090] If the second frame image data D606b is the current frame image data 402, then the third frame image data D606c becomes the next frame image data 403.

[0091] Raster vector group 404 is a list that enumerates raster vectors. Raster vectors constitute the elements of raster vector group 404.

[0092] A raster vector is information representing the vector information in which the image of a certain raster cell in the current frame image data 402 moves to a specified position in the next frame image data 403. It consists of x-axis and y-axis components in pixels.

[0093] exist Figure 6 In the image data, raster vector groups 404a, 404b, etc. are appended after frame image data D606.

[0094] In the main body D602, such as

[0095] The separator is D607 (i.e., delimiter or separator, etc.).

[0096] Image data of frame 1, D606a

[0097] Separator D607

[0098] Grid Vector Group 1 404a

[0099] Separator D607

[0100] Image data of frame 2, D606b

[0101] Separator D607

[0102] Grid vector group 2, 404b

[0103] …

[0104] In this way, frame image data D606 and raster vector group 404 are inserted alternately with the delimiter D607 in between.

[0105] In addition, Figure 6 In this process, the compressed cloud video file 304 and video data are presented as files, but the video data can also be configured into a stream and processed by appropriately inserting header D601 at specified time intervals.

[0106] Another approach considered was to forgo the delimiter D607 and instead describe the address to each frame's image data and the size of each frame's image data in the header D601. However, without using the delimiter D607, it is difficult to process video data in streaming form.

[0107] Return to Figure 4 The content data extraction and processing function will be explained based on the operation of the data extraction and processing unit 401.

[0108] The data extraction processing unit 401 outputs the current frame image data 402, the next frame image data 403, and the raster vector group 404 located between the current frame image data 402 and the next frame image data 403 to the interpolation processing unit 406.

[0109] Furthermore, the raster list 405 contains information with fixed values ​​that are independent of frame movement and can be derived from the header D601 of the compressed cloud video file 304.

[0110] Figure 7 This is a block diagram showing the internal functions (interpolation operation processing function) of the interpolation operation processing unit 406.

[0111] The grid determination processing unit 701 retrieves the address of the nth grid cell sequentially from the beginning of the grid list 405.

[0112] The grid cell number specified by the grid determination processing unit 701 is input to the vector determination processing unit 702. The vector determination processing unit 702 retrieves the grid vector with the nth grid cell number specified by the grid determination processing unit 701 from the grid vector group 404.

[0113] On the other hand, the raster cell address output by the raster determination processing unit 701 is input to the current frame raster cell extraction processing unit 703. The current frame raster cell extraction processing unit 703, which provides the current frame raster cell extraction processing function, generates current raster cell image data 704 from the current frame image data 402, which is obtained by extracting the part specified by the raster cell address. The current raster cell image data 704 represents image data in units of raster cells C502, which are obtained by subdividing the current frame image data 402 based on the raster G501.

[0114] The raster cell address output by the raster determination processing unit 701 and the raster vector output by the vector determination processing unit 702 are input to the next frame cell extraction processing unit 705. The next frame cell extraction processing unit 705, which provides the next frame cell extraction processing function, generates next frame image data 706 from the next frame image data 403 by moving the raster cell address according to the raster vector and extracting the part specified by the moved raster cell address.

[0115] As explained above, the raster determination processing unit 701 implements the raster determination processing function in the content video playback program through the computer, and the vector determination processing unit 702 implements the vector determination processing function.

[0116] Furthermore, the current raster cell image data 704 is image data cropped based on raster G501, but the next cell image data 706 is image data cropped based on the cell address after the raster address has been moved according to the raster vector. Therefore, although the next cell image data 706 has the same size and shape as the current raster cell image data 704, it is not cropped based on raster G501, and therefore the term "raster" is not used for it.

[0117] The current raster unit image data 704 and the next unit image data 706 are input into the interpolation unit image data production unit 707.

[0118] The interpolation unit image data production unit 707 produces an interpolation unit image data group 708 of an amount corresponding to the number of frames between the current raster unit image data 704 and the next unit image data 706.

[0119] For example, if the raster unit image data obtained by photographing clouds is captured at 20-second intervals, then in the case of normal video playback at a frame rate of 30fps, 30 × 20 - 2 = 598 interpolated frame image data need to be produced. The 2 images subtracted are the amounts corresponding to the current raster unit image data 704 and the next unit image data 706.

[0120] Therefore, the interpolation unit image data generation unit 707 generates the interpolated frame image data through interpolation operations. Specifically, it performs smoothing between a pixel in the current raster unit image data 704 and a pixel at the same address in the next unit image data 706.

[0121] For example, if the red brightness of a pixel in the current raster unit image data 704 is set to 100 and the red brightness of a pixel at the same address in the next unit image data 706 is set to 200, then the change from 100 to 200 is divided into 600 frames. This causes the brightness to increase by approximately 1 every 6 frames starting from 100.

[0122] When the brightness of a certain color component of a certain pixel in the current raster cell image data 704 is set to y1,

[0123] Furthermore, the brightness of the same color component of the pixel at the same address in the next unit image data 706 is set to y2.

[0124] And when the number of interpolation frames is set to m,

[0125] The brightness y of the pixels in the x-th frame can be derived using the following linear equation.

[0126] y = {(y2-y1) / m} * x + y1

[0127] After the brightness y of the pixels of the interpolated frame is approximated to an integer value through the above interpolation operation, it is output as the interpolation unit image data group 708.

[0128] Furthermore, by reducing the number of interpolated frames, a fast-forward playback effect can be achieved. The content playback device involved in the embodiments of the present invention is game content; therefore, by playing the video at a speed faster than the typical movement speed of a cloud, the performance effect of the imagined world provided by the game content can be enhanced.

[0129] The interpolation unit image data production unit 707 performs the above-described calculation process according to each raster unit number, thereby generating an interpolation unit image data group 708 for all raster units.

[0130] The interpolation unit configuration processing unit 709, which provides interpolation unit configuration processing functions, reads the raster cell address from the raster list 405 and reads the raster vector from the raster vector group 404. Then, the interpolation unit configuration processing unit 709 configures the interpolation unit image data group 708 on the interpolation frame area formed in the temporary memory 710 of RAM 203.

[0131] However, if interpolated raster cell image data is simply configured on the interpolated frame region formed in the temporary memory 710, gaps or overlaps may occur between adjacent interpolated raster cell image data. Therefore, a cell boundary interpolation processing unit 711 that provides cell boundary interpolation processing function performs smoothing, average value interpolation, and other processing at the boundaries between these interpolated raster cell image data to make the gaps less noticeable.

[0132] Thus, the interpolated frame image data group 712 is completed.

[0133] When the image data merging processing unit 713 inserts the interpolated frame image data group 712 between the current frame image data 402 and the next frame image data 403, the cloud video data 305 is completed.

[0134] Figure 8 A and Figure 8 B is a schematic diagram illustrating the relationship between the current raster unit image data 704 and the next unit image data 706 in the current frame image data 402 and the next frame image data 403.

[0135] exist Figure 8 In A, the first raster unit C704a, which is the current raster unit image data 704 in the current frame image data 402, is (in Figure 8 A is described as "1") and moves to the first unit C706a, which is the next unit of image data 706 (in Figure 8 A is described as "1'" in the text.

[0136] Similarly, the second grid cell C704b (in) Figure 8 (A is "2") moves to the second unit C706b (in) Figure 8 A is "2'", the third grid cell C704c (in Figure 8 Move the value in A (which is "3") to the third unit C706c (in Figure 8 A is "3'", the fourth grid cell C704d (in Figure 8 Move A (which is "4") to the fourth unit C706d (in Figure 8 (A is "4'").

[0137] When Figure 8 When raster cells 1, 2, 3, and 4 in the current frame image data 402 of A overlap with cells 1', 2', 3', and 4' in the next frame image data 403, they become... Figure 8 The configuration relationship is shown in B.

[0138] Cell 1' is a region of the same size as grid cell 1, and is located within... Figure 10 When creating video data as described later, raster unit 1 is used, and the region most likely to be the moving destination of raster unit 1 is determined from the area near the corresponding region of raster unit 1 in the next frame image using methods such as template matching. That is, when creating video data, the raster vector is determined based on the difference between raster unit 1 and unit 1'.

[0139] Furthermore, the raster cell image data of raster cell 1 only needs to be sufficiently similar to the cell image data of cell 1', but they do not need to be completely identical.

[0140] Figure 9 This is a summary diagram illustrating frame interpolation.

[0141] exist Figure 9 The image shows the case where the first grid cell C704a of the current frame image data 402 changes to the first cell C706a ​​of the next frame image data 403 after 20 seconds.

[0142] During the movement from a grid cell in the current frame image data 402 to a cell in the next frame image, the interpolation unit image data production unit 707, which provides interpolation unit image data production functions, performs interpolation calculations for each pixel.

[0143] exist Figure 9The image shows a state where an interpolation unit C903 is formed between the first grid unit C704a of the current frame image data 402 and the first unit C706a ​​of the next frame image data 403 by the interpolation unit image data production unit 707. At this time, the characteristic pattern P901 existing in the first grid unit C704a and the characteristic pattern P902 existing in the first unit C706a ​​are mixed at the interpolation unit C903 formed by the interpolation operation performed by the interpolation unit image data production unit 707, resulting in the formation of pattern P903.

[0144] The cloud video playback processing unit 303 and, in particular, the interpolation processing unit 406 in the aforementioned content playback device 101 utilize the well-known morphing technique. Morphing is a sophisticated interpolation process that modifies two well-defined image data, but for clouds with indistinct outlines, even if the existing morphing technique is directly applied, smooth video data cannot be generated.

[0145] The inventors noticed that when generating videos of clouds, the movement of clouds could be represented simply by moving a portion of the image. This technique does not mimic known distortions and does not involve partial image distortion, thus minimizing the computational load required for video data generation.

[0146] The cloud video playback processing unit 303 in the content playback device 101 according to the embodiments of the present invention can generate video data of suspended particles in the atmosphere such as clouds, fog, haze, hot air, and gas with high fidelity by simply performing simple calculations such as moving the grid unit image data according to the grid vector and performing brightness interpolation calculations.

[0147] [Compressed Cloud Video File Generation Device 1001]

[0148] The above describes the format of the new compressed cloud video file 304 and the playback device 101 for playing the content of the compressed cloud video file 304.

[0149] Subsequently, refer to Figures 10-13 This describes a compressed cloud video file generating apparatus 1001 used to encode a large amount of still image data obtained by taking pictures of clouds at specified time intervals to generate a compressed cloud video file 304.

[0150] Figure 10 This is a block diagram showing the hardware structure of the compressed cloud video file generation device 1001.

[0151] The compressed cloud video file generation device 1001, which is a computer known as a general personal computer, includes a CPU 1002, ROM 1003, RAM 1004, display unit 1005, operation unit 1006, NIC 1007 (network interface card), serial interface 1008 such as USB, and non-volatile storage device 1009 connected to the bus 1010.

[0152] The non-volatile storage device 1009 contains a program for enabling the computer to operate as a compressed cloud video file generation device 1001.

[0153] Reading still image files from the cloud via a serial interface and / or network, etc.

[0154] Figure 11 This is a block diagram illustrating the software functions of the compressed cloud video file generation device 1001.

[0155] The current frame image data 402 and the next frame image data 403 are read by the mosaic processing unit 1101.

[0156] The mosaic processing unit 1101 generates current frame mosaic image data 1102 and next frame mosaic image data 1103, which are reduced in resolution after being processed by a known average calculation.

[0157] The current frame mosaic image data 1102 and the next frame mosaic image data 1103 are read by the optical flow processing unit 1104.

[0158] The optical flow processing unit 1104 provides optical flow processing functions, and the grid size determination processing unit 1105 provides grid size determination processing functions.

[0159] The optical flow processing unit 1104, for a portion of image data extracted according to the grid size indicated by the grid size determination processing unit 1105, determines the region most likely to be the destination of the movement of the portion of image data from the next frame of mosaic image data 1103 using methods such as template matching. Then, it calculates the grid size vector as the movement vector.

[0160] The optical flow processing unit 1104 performs the processing on a portion of the image data of all gratings to generate a grating size vector group 1106.

[0161] The mosaic size determination processing unit 1107 cooperates with the grid size determination processing unit 1105 to determine the mosaic size corresponding to the grid size for the mosaic processing unit 1101 and the optical flow calculation processing unit 1104, and controls the optical flow calculation processing unit 1104.

[0162] When mosaic processing is applied to image data, the similarity between image data can be determined on a per-mosaic basis. Therefore, mosaic processing reduces the computational load of the optical flow processing unit 1104. Furthermore, it reduces the error of the vector obtained through computation. The effect of reducing this vector error is even greater than the effect obtained through mosaic processing.

[0163] First, the mosaic size determination processing unit 1107 assigns a large mosaic size to the mosaic processing unit 1101. The mosaic processing unit 1101 performs mosaic processing according to the size assigned by the mosaic size determination processing unit 1107, generating current frame mosaic image data 1102 and next frame mosaic image data 1103 with coarse mosaic size. That is, the mosaic processing unit 1101 implements the mosaic processing function in the content video playback program through a computer.

[0164] Then, the grid size determination processing unit 1105 assigns a grid size to the optical flow calculation processing unit 1104 that corresponds to the grid size assigned by the mosaic size determination processing unit 1107 to the mosaic processing unit 1101. The optical flow calculation processing unit 1104 performs optical flow calculation processing according to the grid size assigned by the grid size determination processing unit 1105 and generates a coarse grid size vector group 1106.

[0165] That is, the grid size determination processing unit 1105 implements the mosaic size determination processing function in the content video playback program through a computer.

[0166] Next, the mosaic size determination processing unit 1107 divides the previous mosaic size into four equal parts, dividing it in half vertically and in half horizontally. Then, it assigns the subdivided mosaic size to the mosaic processing unit 1101. The mosaic processing unit 1101, as before, performs mosaic processing according to the size specified by the mosaic size determination processing unit 1107, and generates current frame mosaic image data 1102 and next frame mosaic image data 1103, whose mosaic size is smaller than the mosaic size of the previously created image data.

[0167] Then, the grid size determination processing unit 1105 divides the previous grid size into four equal parts, dividing it in half vertically and in half horizontally. The grid of this subdivided grid size is then assigned to the optical flow calculation processing unit 1104. The optical flow calculation processing unit 1104 performs optical flow calculation processing according to the grid size specified by the grid size determination processing unit 1105, and generates a grid size vector group 1106 whose grid size is smaller than the previously created grid size vector group 1106.

[0168] Similarly, the mosaic size determination processing unit 1107 and the grid size determination processing unit 1105 reduce the mosaic size and grid size step by step, respectively, while the mosaic processing unit 1101 and the optical flow calculation processing unit 1104 work to generate a grid size vector group 1106 corresponding to the grid size.

[0169] The grid size determination processing unit 1105 manages the number of times a series of repetitive calculations are performed.

[0170] The raster size vector group 1106 generated by the repeated processing of the mosaic processing unit 1101 and the optical flow processing unit 1104 described above is input into the vector optimization processing unit 1108.

[0171] The vector optimization processing unit 1108, which provides vector optimization processing, detects errors in the raster size vectors contained in the raster size vector group 1106 and replaces the erroneous vectors with the raster size vectors of the higher level. Then, finally, the vector optimization processing unit 1108 performs weighted summation processing on the vectors of multiple raster sizes to calculate the raster vector group 404 of the raster size to be used in the compressed cloud video file 304.

[0172] Furthermore, this embodiment simply illustrates a permutation using a higher-level vector, but vector optimization processing is not necessarily limited to permutations using higher-level vectors. Corrections using even higher-level vectors or weighted summation can also be employed.

[0173] In addition, the data reconstruction processing unit 1109, which provides data reconstruction processing functions, reads the raster vector group 404, the current frame image data 402, and the next frame image data 403, and generates a compressed cloud video file 304.

[0174] Specifically, the data reconstruction processing unit 1109 generates a compressed cloud video file 304 by sandwiching a raster vector group 404 between the current frame image data 402 and the next frame image data 403 with a separator D607 and attaching a header D601.

[0175] Figure 12 This is a flowchart illustrating the operation of the cloud video file generation device 1001.

[0176] When processing begins (S1101), the grid size determination processing unit 1105 initializes the count variable i to 0 (S1102). This count variable i is a number corresponding to the grid size level.

[0177] Next, the grid size determination processing unit 1105 sets an initial value for the number of pixels in the grid. Furthermore, the mosaic size determination processing unit 1107 sets the mosaic size in the mosaic processing (S1103).

[0178] This process is then repeated.

[0179] First, the mosaic processing unit 1101 performs mosaic processing on the current frame image data 402 and the next frame image data 403 according to the mosaic size determined by the mosaic size determination processing unit 1107, and outputs the current frame mosaic image data 1102 and the next frame mosaic image data 1103 (S1104).

[0180] Next, the optical flow processing unit 1104 performs optical flow processing on the current frame mosaic image data 1102 and the next frame mosaic image data 1103 based on the grid size determined by the grid size determination processing unit 1105 to calculate the grid size vector (S1105).

[0181] Next, the grid size determination processing unit 1105 checks whether a parent grid size vector exists at that time point (S1106). In the initial processing, no parent grid size vector exists ("No" in S1106), so the count variable i is incremented by 1 (S1107). Then, the grid size determination processing unit 1105 checks whether the count variable i has reached its maximum value. max The above (S1108). In the initial processing, the count variable i only increased to 1 ("No" in S1108), so the grid size determination processing unit 1105 divided the grid size into four parts to create the lower-level grid, and determined the mosaic size to be smaller than the size of the most recent processing (S1109). Then, the processing from step S1104 is repeated.

[0182] In step S1106, if there is a parent grid size vector (S1106 "Yes"), the grid size determination processing unit 1105 instructs the vector optimization processing unit 1108 to perform processing.

[0183] First, the vector optimization processing unit 1108 initializes the count variable j to 0 (S1110). This count variable j is a number assigned based on the position of the grid. Next, the vector optimization processing unit 1108 checks whether the angle formed by the grid size vector located at the j-th grid and its superior grid size vector is above a threshold (S1111).

[0184] In step S1111, if the angle formed with the grid size vector located above the upper level is above a threshold ("Yes" in S1111), it means that the grid size vector calculated in step S1105 is oriented in a direction that is significantly different from the upper-level grid size vector. Therefore, the grid size vector is considered an invalid vector that failed to detect, and the grid size vector is covered by the upper-level grid size vector (S1112).

[0185] In step S1111, if the angle formed with the grid size vector located above is less than a threshold (S1111 "No"), nothing is done.

[0186] Regardless of whether step S1112 or step S1111 is "No", the vector optimization processing unit 1108 increments the count variable j by 1 (S1113).

[0187] Next, the vector optimization processing unit 1108 checks whether the j-th grid size vector does not exist (S1114).

[0188] In step S1114, if there is a grid size vector j (S1114 "No"), the vector optimization processing unit 1108 repeats the processing from step S1111 again.

[0189] In step S1114, if there is no grid size vector j (S1114 "Yes"), the vector optimization processing unit 1108 temporarily terminates the processing and hands over the control of the processing to the grid size determination processing unit 1105. The grid size determination processing unit 1105 accepts the handover and repeats the processing from step S1107.

[0190] Additionally, in step S1108, if the count variable i has reached its maximum value i max If "yes" is indicated in S1108, then the raster size determination processing unit 1105 instructs the data reconstruction processing unit 1109 to perform processing.

[0191] The data reconstruction processing unit 1109 generates a compressed cloud video file 304 by sandwiching the raster vector group 404 between the current frame image data 402 and the next frame image data 403 with a separator D607 and attaching a header D601. This completes a series of processes (S1116).

[0192] Figure 13 It is a summary diagram illustrating the relationship between the upper-level raster vector and the lower-level raster vector.

[0193] Figure 13 A represents two grid cells in the top-level grid.

[0194] Figure 13 B represents the second level of the grid from the top level, belonging to... Figure 13 A shows 2 grid cells and 8 grid cells.

[0195] Figure 13 C represents the third level of the grid from the top level. Figure 13 The two grid cells shown in A and Figure 13B represents 8 grid cells or 32 grid cells.

[0196] for Figure 13 The top-level grid cell C1301 of A is calculated by the optical flow processing unit 1104, which calculates the top-level grid size vector V1311 of the top-level grid cell C1301 and associates the top-level grid cell C1301 with the top-level grid size vector V1311. Dividing the top-level grid cell C1301 into four equal parts, both vertically and horizontally, results in... Figure 13 The second-highest grid cell of B, C1302, C1303, C1304, and C1305. These grid cells C1302, C1303, C1304, and C1305 are subordinate to the highest-highest grid cell C1301.

[0197] Dividing grid cell C1302 into a total of four equal parts by dividing it into two equal parts vertically and two equal parts horizontally results in the following: Figure 13 The third level above C are grid cells C1306, C1307, C1308, and C1309. These grid cells C1306, C1307, C1308, and C1309 are equivalent to the lower levels of grid cell C1302.

[0198] That is, from the hierarchical relationship between the grid units in the top-level grid unit C1301, grid units C1302, C1303, C1304 and C1305, and grid units C1306, C1307, C1308 and C1309, it can be seen that there is a hierarchical structure.

[0199] exist Figure 13 In C, the directions of grid size vectors V1316, V1317, and V1319 are roughly similar to those of their parent grid size vector V1312. However, only grid size vector V1318 faces a direction significantly different from that of its parent vector V1312. For grid size vectors whose directions are completely different from those of their parent vectors, the parent vector is used to cover them. Figure 13 In case C, Figure 12 In step S1112, the grid size vector V1318 is covered by the grid size vector V1312.

[0200] The angle formed by vectors is calculated using the inner product.

[0201] Exceptionally, when the value of the inner product is less than 0, that is, when the angle formed by the vectors is greater than 90°, the value of the inner product can be used directly for evaluation.

[0202] The embodiments of the present invention described above can have the variations shown below.

[0203] (1) In the above embodiments, the grid applied to the frame image data is defined as a square or rectangular shape, but the shape of the grid is not necessarily a square or rectangular shape. For example, for virtual hemispherical image data obtained by taking pictures in all directions, such as planetariums, in addition to using polygons such as equilateral triangles, quadrilaterals, and hexagons to form the grid, a radial shape such as the meridians / parallels of a globe can also be used.

[0204] When using a grid with a non-uniform rectangular shape, the coordinates of each intersection point of the grid and the grid cell number need to be included in the header D601.

[0205] (2) The optical flow processing unit 1104 calculates the raster size vector as a motion vector by referring to the next frame mosaic image data 1103 for a portion of the image data extracted from the current frame mosaic image data 1102. In other words, it performs calculation processing for calculating the positive motion vector in the time series.

[0206] As a method to make the grid size vector calculated by the optical flow processing unit 1104 more accurate, it is conceivable to also calculate the reverse movement vector in the time series, and calculate the average value of the forward and reverse movement vectors to set the final grid size vector.

[0207] In other words, for a portion of image data extracted from the next frame of mosaic image data 1103, a reverse motion vector in the time series is calculated with reference to the current frame of mosaic image data 1102. Then, its average value with the forward motion vector is calculated and set as the raster size vector.

[0208] (3) Furthermore, it can also be configured to perform template matching operation processing in the optical flow operation processing unit 1104, and when referring to the next frame mosaic image data 1103 for a portion of the image data extracted from the current frame mosaic image data 1102, it also detects the scaling of the portion of the image data.

[0209] In this case, when the input raster cell number is used as the independent variable, the raster vector group 404, which is an array variable, outputs not only the x-component and y-component of the raster vector, but also the z-component of the raster vector representing the magnification of the raster cell image. Then, the vector determination processing unit 702 outputs a raster vector containing the raster cell image magnification, the current frame raster cell extraction processing unit 703 outputs the current raster cell image data 704 according to the raster vector containing the raster cell image magnification, and the next frame cell extraction processing unit 705 outputs the next cell image data 706 according to the raster vector containing the raster cell image magnification.

[0210] In embodiments of the present invention, a content playback device 101, a content video playback program for implementing the device, a compressed cloud video file generation device 1001, a compressed cloud video file generation program for implementing the device, and a format for a compressed cloud video file 304 are disclosed.

[0211] Compared to previous video data formats, compressed cloud video files 304, which consist of still image data in units of a few seconds to tens of seconds and raster vector groups 404 representing the direction and distance of movement of raster unit image data groups between still image data, can overwhelmingly reduce the amount of data.

[0212] The cloud video playback processing unit 303, which plays the compressed cloud video file 304, can play video data of suspended particles in the atmosphere such as clouds, fog, haze, heat, and gas with high fidelity by simply performing simple calculations such as moving the raster unit image data according to the raster vector and performing brightness interpolation.

[0213] The compressed cloud video file generating device 1001, which generates the compressed cloud video file 304, changes the raster from large raster units to small raster units when generating the raster vector, and performs optical flow calculations simultaneously. At this time, the lower-level raster vectors that form angles significantly different from the higher-level raster vectors are corrected to match the higher-level raster vectors, thereby minimizing flaws in the compressed cloud video file 304.

[0214] The above description focuses on playing compressed cloud video files as an embodiment of the present invention. However, compressed cloud video files are only one embodiment, and the video playback program of the present invention is not limited to video playback programs for compressed cloud video files.

[0215] The video playback program of the present invention, without departing from the spirit of the invention as set forth in the claims, also includes a wide range of variations and applications for playing other compressed video files besides compressed cloud video files.

[0216] Explanation of reference numerals in the attached figures

[0217] 101: Content playback device; 102: Display unit; 201: CPU; 202: ROM; 203: RAM; 204: Display unit; 205: Operation unit; 206: Wide-area wireless communication unit; 207: Wireless LAN interface; 208: Non-volatile storage device; 209: Bus; 301: Main game content generation and processing unit; 302: Game data set; 303: Cloud video playback processing unit; 304: Compressed cloud video file; 305: Cloud video data 306: Image Composition Processing Unit; 401: Data Extraction Processing Unit; 402: Current Frame Image Data; 403: Next Frame Image Data; 404: Raster Vector Group; 405: Raster List; 406: Interpolation Calculation Processing Unit; 701: Raster Determination Processing Unit; 702: Vector Determination Processing Unit; 703: Current Frame Raster Unit Extraction Processing Unit; 704: Current Raster Unit Image Data; 705: Next Frame Unit Extraction Processing Unit; 706: Next Unit Image Data 707: Interpolation unit image data production unit; 708: Interpolation unit image data group; 709: Interpolation unit configuration processing unit; 710: Temporary memory; 711: Unit boundary interpolation processing unit; 712: Interpolation frame image data group; 713: Image data merging processing unit; 1001: Compressed cloud video file generation device; 1002: CPU; 1003: ROM; 1004: RAM; 1005: Display unit; 1006: Operation unit; 1007: NIC; 1008: Serial interface; 1009: Non-volatile storage device; 1010: Bus; 1101: Mosaic processing unit; 1102: Current frame mosaic image data; 1103: Next frame mosaic image data; 1104: Optical flow processing unit; 1105: Grid size determination processing unit; 1106: Grid size vector group; 1107: Mosaic size determination processing unit; 1108: Vector optimization processing unit; 1109: Data reconstruction processing unit.

Claims

1. A computer program product, comprising a content video playback program, wherein the content video playback program is used to cause a computer to perform content data extraction processing steps and interpolation calculation processing steps, wherein, The computer is a content video playback device composed of a video playback processing unit, a main game content generation processing unit, and an image compositing processing unit. The video playback processing unit extracts compressed video from compressed video files and outputs video data. The main game content generation processing unit reads various game data from a game data set and generates various videos based on operations performed by the operation unit. The image compositing processing unit combines the video data generated by the main game content generation processing unit with the video data generated by the video playback processing unit and outputs the result to the display unit. The content data extraction and processing steps are as follows: The data extraction and processing unit extracts from the compressed video file the current frame image data (serving as a reference time point), the next frame image data after a predetermined time elapsed from the reference time point, and a raster vector group, which is a collection of raster lists listing raster vectors. The raster vectors represent the amount and direction of movement of the raster unit image data (subdivided from the current frame image data) along the coordinate direction within the frame at the time point of the next frame image data. The interpolation processing step consists of the following steps: the interpolation processing unit performs frame interpolation between the current frame image data and the next frame image data based on the current frame image data, the next frame image data, the raster vector group, and the raster list, and outputs video data.

2. The computer program product according to claim 1, wherein the content video playback program causes the computer to further perform the following steps: The grid determination process involves the grid determination unit retrieving grid cell addresses sequentially from the beginning of the grid list. The vector determination processing step involves retrieving, from the grid vector group, the grid vector with the grid cell number specified by the grid determination processing unit. The current frame raster unit extraction processing step involves extracting current raster unit image data from the current frame image data through the current frame raster unit extraction processing unit. This current raster unit image data is obtained by cropping the part specified by the raster unit address. The next frame unit extraction processing step extracts the next unit image data from the next frame image data through the next frame unit extraction processing unit. The next unit image data is obtained by moving the grid unit address according to the grid vector and extracting the part specified by the moved grid unit address. The interpolation unit image data production step involves creating an interpolation unit image data group, corresponding to a set number of frames, between the current raster unit image data and the next unit image data, using the interpolation unit image data production unit. The interpolation unit configuration processing step involves configuring the interpolation unit image data group produced by the interpolation unit image data production unit on a temporary memory through the interpolation unit configuration processing unit; and The unit boundary interpolation processing step involves the unit performing smoothing or average interpolation at the boundaries between interpolation unit image data groups of interpolation unit image data groups configured by the interpolation unit configuration processing unit for the interpolation frame image, in order to make the gaps inconspicuous.

3. A content video playback device, comprising a video playback processing unit, a main game content generation processing unit, and an image compositing processing unit, wherein the video playback processing unit extracts compressed video from a compressed video file and outputs video data; the main game content generation processing unit reads various game data from a game data set and generates various videos based on operations performed by an operation unit; and the image compositing processing unit combines the video data generated by the main game content generation processing unit with the video data generated by the video playback processing unit and outputs the results to a display unit. The video playback processing unit includes: The data extraction and processing unit extracts data from the compressed video file to output the current frame image data as a reference time point, the next frame image data after a predetermined time elapsed from the reference time point, and a raster vector group as a collection of raster lists, wherein the raster vectors represent the amount and direction of movement of the raster unit image data, which is a subdivision of the current frame image data, along the coordinate direction within the frame at the time point of the next frame image data; and The interpolation processing unit performs frame interpolation operations between the current frame image data and the next frame image data based on the current frame image data, the next frame image data, the raster vector group, and the raster list, and outputs video data.

4. The content video playback device according to claim 3, wherein, The interpolation processing unit includes: The grid determination processing unit retrieves grid cell addresses sequentially from the beginning of the grid list; The vector determination processing unit extracts the grid vector with the grid cell number specified by the grid determination processing unit from the grid vector group; The current frame raster unit extraction processing unit extracts current raster unit image data from the current frame image data. This current raster unit image data is obtained by cropping the part specified by the raster unit address. The next frame unit extraction processing unit extracts the next unit image data from the next frame image data. The next unit image data is obtained by moving the grid unit address according to the grid vector and extracting the part specified by the moved grid unit address. The interpolation unit image data production unit produces an interpolation unit image data set of a quantity corresponding to a set number of frames that exists between the current raster unit image data and the next unit image data. An interpolation unit configuration processing unit configures the interpolation unit image data set produced by the interpolation unit image data production unit on a temporary memory; and The unit boundary interpolation processing unit performs processing at the boundary between interpolation unit image data of a group of interpolation unit image data corresponding to the number of frames configured by the interpolation unit configuration processing unit to make the gaps inconspicuous by smoothing or averaging interpolation.

5. A method for playing content video, which uses a content video playback device, wherein, The content video playback device comprises a video playback processing unit, a main game content generation processing unit, and an image compositing processing unit. The video playback processing unit extracts compressed video from a compressed video file and outputs video data. The main game content generation processing unit reads various game data from a game data set and generates various videos based on operations performed by the operation unit. The image compositing processing unit combines the video data generated by the main game content generation processing unit with the video data generated by the video playback processing unit and outputs it to the display unit. The content video playback method includes the following steps: The data extraction and processing steps involve extracting data from the compressed video file to output the current frame image data as a reference time point, the next frame image data after a specified time elapsed from the reference time point, and a raster vector group, which is a collection of raster lists listing raster vectors. The raster vectors represent the amount and direction of movement of the raster unit image data, which is a subdivision of the current frame image data, along the coordinate direction within the frame at the time point of the next frame image data. The interpolation processing step involves performing frame interpolation between the current frame image data and the next frame image data based on the current frame image data, the next frame image data, the raster vector group, and the raster list, and then outputting video data.

6. The content video playback method according to claim 5 further includes the following steps: The grid determination process involves the grid determination unit retrieving grid cell addresses sequentially from the beginning of the grid list. The vector determination processing step extracts the grid vector with the grid cell number specified by the grid determination processing unit from the grid vector group; The current frame raster unit extraction processing step extracts the current raster unit image data from the current frame image data. This current raster unit image data is obtained by cropping the part specified by the raster unit address. The next frame unit extraction processing step extracts the next unit image data from the next frame image data. This next unit image data is obtained by moving the raster unit address according to the raster vector and extracting the part specified by the moved raster unit address. The interpolation unit image data production step involves creating an interpolation unit image data group, corresponding to a set number of frames, between the current raster unit image data and the next unit image data, using the interpolation unit image data production unit. The interpolation unit configuration processing step involves configuring the interpolation unit image data group produced by the interpolation unit image data production unit on a temporary memory through the interpolation unit configuration processing unit; and The unit boundary interpolation processing step involves performing smoothing or average interpolation on the boundaries between interpolation unit image data groups configured by the interpolation unit configuration processing unit for the interpolation frame image, corresponding to the number of frames, to make the gaps inconspicuous.

7. A computer program product, comprising a content video data generation program, the content video data generation program being used to cause a computer to perform the following steps: The mosaic processing step involves generating current frame mosaic image data and next frame mosaic image data by the mosaic processing unit based on the current frame image data obtained from the image data at the reference time point extracted from the compressed video file and the next frame image data after a specified time from the reference time point. The mosaic size determination processing step involves subdividing the mosaic size of the frame mosaic image data by the mosaic size determination processing unit, and assigning the subdivided mosaic size to the mosaic processing unit. The grid size determination processing step involves subdividing the grid size of the frame mosaic image data by the grid size determination processing unit, and assigning the subdivided grid size to the optical flow calculation processing unit. The optical flow calculation processing step involves the optical flow calculation processing unit performing optical flow calculation processing according to the grid size specified by the grid size determination processing unit, generating a grid size vector group whose grid size is smaller than the grid size of the previously created grid size vector group; The vector optimization processing step involves detecting errors in the raster size vectors within the raster size vector group, replacing erroneous vectors with higher-level raster size vectors, and performing a weighted summation of multiple raster size vectors to calculate the raster vector group of the raster size to be used in the compressed video file; and The data reconstruction processing step involves reading the raster vector group, the current frame image data, and the next frame image data through the data reconstruction processing unit and generating a compressed video file.

8. A content video data generation device, comprising: The mosaic processing unit generates current frame mosaic image data and next frame mosaic image data based on current frame image data obtained from image data extracted from a compressed video file as a reference time point, and next frame image data after a predetermined time from the reference time point. The mosaic size determination processing unit subdivides the mosaic size of the frame mosaic image data and assigns the subdivided mosaic size to the mosaic processing unit. The grid size determination processing unit subdivides the grid size of the frame mosaic image data and assigns the subdivided grid size to the optical flow calculation processing unit; The optical flow processing unit performs optical flow processing according to the grid size specified by the grid size determination processing unit, and generates a grid size vector group whose grid size is smaller than the grid size of the previously created grid size vector group; The vector optimization processing unit detects errors in the raster size vectors contained in the raster size vector group, replaces the erroneous vectors with higher-level raster size vectors, and performs weighted summation on the vectors of multiple raster sizes to calculate the raster vector group of the raster sizes to be used in the compressed video file; and The data reconstruction processing unit reads the raster vector group, the current frame image data, and the next frame image data and generates a compressed video file.

Images