Image generation device, image generation method, and program

Abstract

To provide a technique for generating a video to create an illusion of continuous motion by adding an arbitrary number of images on two original images on the basis of motion detection properties in the human visual system.SOLUTION: A video generating device generates a video for making an observer who views a presentation when a unit video for presentation is repeatedly and continuously presented perceive a viewed object as if it were moving continuously. The video generating device includes an additional image generating unit that generates N additional images using a first original image, which is a certain image, and a second original image, which is a different image from the first original image by assuming N as an integer equal to or greater than 1 and a presentation video generating unit for generating a set of N+2 images as a presentation unit video, consisting of the first original image, the second original image, and the N additional images. The additional image generating unit generates N additional images so that the kinetic energies per unit time when presented are as equal as possible between two adjacent images when presented in N+2 images included in the unit image for presentation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to a technology for generating images that create an illusion, and in particular to a technology for generating images that, when a plurality of images are repeated in the same order to produce a continuous image, give the observer the perception that a visual object is moving continuously in the image. [Background technology]

[0002] Non-Patent Document 1 describes that two consecutive images are obtained as a first frame image and a second frame image from a video including a moving object, the contrast polarity of the pixel values ​​of the first frame image and the second frame image are inverted to generate a third frame image and a fourth frame image, and the first frame image to the fourth frame image are repeated in order to present a continuous video, so that an observer who sees the video perceives an object included in the video as if it continues to move in one direction. Patent Document 1 also describes that even if the contrast polarity of only the image information in the high spatial frequency band of the above-mentioned first frame image and the second frame image is inverted to generate a third frame image and a fourth frame image, an observer who sees the video perceives an object included in the video as if it continues to move in one direction. The illusion described in Non-Patent Document 1 and Patent Document 1 is hereinafter referred to as "Four-stroke apparent motion".

[0003] Non-Patent Document 2 describes that two consecutive images are obtained as a first frame image and a second frame image from a video including a moving object, an image in which all pixels are gray is prepared as a third frame image, and the first frame image to the third frame image are repeated in order to present a continuous video, so that an observer who sees the video perceives an object included in the video as if it were moving continuously in one direction. The illusion described in Non-Patent Document 2 is hereinafter referred to as "two-stroke apparent motion" as described in Non-Patent Document 2.

[0004] The illusion described in Non-Patent Document 1, Patent Document 1, and Non-Patent Document 2, i.e., the illusion in which a continuous image is presented by repeating three or more frame images in sequence, and an observer viewing the image perceives the visual objects contained in the image as continuously moving, will be referred to below for convenience as the "continuous motion illusion." [Prior art documents] [Patent documents]

[0005] [Patent Document 1] JP 2018-180753 A [Non-patent literature]

[0006] [Non-Patent Document 1] Anstis, SM and Rogers, BJ, "Illusory Continuous Motion from Oscillating Positive-Negative Patterns: Implications for Motion Perception," Perception, Vol.15, pp.627-640, 1986. [Non-Patent Document 2] Mather, G. and Challinor, KL, "Psychophysical properties of two-stroke apparent motion," Journal of Vision, 9(1):28, pp.1-6, 2009. Summary of the Invention [Problem to be solved by the invention]

[0007] There are two known continuous motion illusions: four-stroke apparent motion and two-stroke apparent motion, and no other methods are known for generating images that induce a continuous motion illusion. It is known that four-stroke apparent motion and two-stroke apparent motion are illusions based on the motion detection characteristics of the human visual system, but it is not known how to generate images that induce a continuous motion illusion based on the motion detection characteristics of the human visual system when generating images that are repeated in sequence with an arbitrary number of frames. In other words, the method of generating the third and fourth frame images that are added to induce four-stroke apparent motion is completely different from the method of generating the third frame image that is added to induce two-stroke apparent motion, and no technology is known for generating images that induce a continuous motion illusion by adding an arbitrary number of frames of images in the same framework to an original two-frame image.

[0008] The objective of the present invention is to provide a technology for generating images that create the illusion of continuous motion by adding any number of images to two original images, based on the motion detection characteristics of the human visual system. [Means for solving the problem]

[0009] One aspect of the present invention is an image generating device that generates an image that, when a presentation unit image is presented repeatedly and continuously, causes an observer viewing the presentation to perceive a visual object as moving continuously, the image generating device including: an additional image generating unit that generates N additional images using a first original image, where N is an integer greater than or equal to 1, and a second original image that is different from the first original image; and a presentation image generating unit that generates a set of N+2 images, the first original image, the second original image, and the N additional images, as the presentation unit image, wherein the additional image generating unit generates the N additional images such that, among the N+2 images included in the presentation unit image, the kinetic energy per unit time when presented between two adjacent images when presented is as equal as possible. Effect of the Invention

[0010] According to the present invention, images for creating a continuous motion illusion by adding any number of images to two original images can be generated based on the motion detection characteristics of the human visual system. [Brief description of the drawings]

[0011] [Figure 1] FIG. 1 is a block diagram illustrating a functional configuration of an image generating device according to an embodiment. [Diagram 2] FIG. 2 is a flowchart showing the process of the image generating device of the embodiment. [Diagram 3] FIG. 3 is a diagram illustrating an example of a functional configuration of a computer that realizes the image generating device according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] To produce the illusion of continuous motion, it is necessary to present a continuous image by repeating images of three or more frames in sequence. For example, when using images of three frames from the first to the third frame, the smallest unit is an image set of three images in the order of the first frame, the second frame, and the third frame, in which this smallest unit of image set is repeated to present a continuous image set as an image. Hereinafter, this entire image set presented is called the "continuous image for presentation," and the smallest unit of image set is called the "unit image for presentation." In other words, an image in which the unit images for presentation are repeated in succession is the continuous image for presentation.

[0013] In each of the presentation unit image for generating four-stroke apparent motion, the presentation unit image for generating two-stroke apparent motion, and the presentation unit image of the present invention, a presentation unit image is generated using two images, a first image which is a certain image, and a second image which is an image different from the first image. The first image and the second image are images from which the presentation unit image is generated, and therefore the first image and the second image are hereinafter referred to as "original images". A typical example of the first original image is an image including a certain object, and a typical example of the second original image is an image equivalent to an image in which the certain object is given a motion with respect to the first original image. Note that "an image equivalent to an image in which the certain object is given a motion with respect to the first original image" is a convenient expression, and "an image equivalent to an image in which the certain object is given a motion with respect to the first original image" properly means an image in which the certain object is at a position moved with respect to its position in the first original image, and other objects other than the certain object are at substantially the same positions as in the first original image. For example, the first original image is an image in which an object is arranged at a first position in a certain background image, and the second original image is an image in which the object is arranged at a second position adjacent to the first position in the certain background image. The adjacent positions are positions within a range in which movement is perceived when the first original image and the second original image are presented in succession over time. Note that the object and other objects may have different shapes, patterns, or sizes in the first original image and the second original image, as long as they are not perceived by the observer. The term "approximately the same position" refers to a difference that is not perceived by the observer even if it is actually slightly different, and is perceived by the observer as being the same position.

[0014] A presentation unit image for generating four-stroke apparent motion is obtained by adding two images to two original images. A presentation unit image for generating two-stroke apparent motion is obtained by adding one image to two original images. A presentation unit image of the present invention is obtained by adding any number of images to two original images. These added images are hereinafter referred to as "additional images". That is, among the images included in the presentation unit image, images other than the original images are additional images. Specifically, a presentation unit image for generating four-stroke apparent motion is composed of two original images and two additional images, an image of a first frame of the presentation unit image is a first original image, an image of a second frame of the presentation unit image is a second original image, an image of a third frame of the presentation unit image is a first additional image, and an image of a fourth frame of the presentation unit image is a second additional image. Moreover, the presentation unit video for generating two-stroke apparent motion is composed of two original images and one additional image, the image of the first frame of the presentation unit video is the first original image, the image of the second frame of the presentation unit video is the second original image, and the image of the third frame of the presentation unit video is the additional image. Moreover, the presentation unit video of the present invention is composed of two original images and any number of additional images, for example, the image of the first frame of the presentation unit video is the first original image, the image of the second frame of the presentation unit video is the second original image, and each of the images of the third frame and after of the presentation unit video is an additional image. Hereinafter, the number of additional images included in the presentation unit video of the present invention is defined as N. N is an integer of 1 or more.

[0015] For the sake of simplicity, it is assumed that all images are of the same size. In other words, in the following description, it is assumed that all images are two-dimensional images with pixels arranged in both the vertical and horizontal directions, and that the number of vertical pixels in all images is equal, and the number of horizontal pixels in all images is equal. In addition, it is assumed that the frame rate of the presented video is fixed, that is, the time interval at which the images are presented is constant, up to the second embodiment.

[0016] <Summary of the Invention> Both four-stroke apparent motion and two-stroke apparent motion are effective image generation methods in light of human motion detection characteristics when a specific number of additional images are provided, but the procedures for generating the images are completely different, and they are not generally usable when generating an arbitrary number of additional images. In contrast, in the present invention, additional images are generated by a procedure that is independent of the number of additional images, taking into consideration the optimization of the motion detection characteristics of the human visual system, that is, so as to be as appropriate as possible for the motion detection characteristics of the human visual system. Specifically, in the present invention, in an image generation process that generates N additional images using two original images to generate a unit image for presentation using N+2 images, N additional images are generated so that the kinetic energy between two adjacent images is as equal as possible when presented.

[0017] For example, when N=1, there are three pairs of two adjacent images that are presented: a pair of an image of the first frame and an image of the second frame, a pair of an image of the second frame and an image of the third frame, and a pair of an image of the third frame and an image of the first frame. Therefore, when N=1, one additional image is generated so that the three kinetic energies, namely, the kinetic energy between the image of the first frame and the image of the second frame, the kinetic energy between the image of the second frame and the image of the third frame, and the kinetic energy between the image of the third frame and the image of the first frame, are as equal as possible.

[0018] For example, when N=2, pairs of two adjacent images when presented are four pairs: a pair of an image of the first frame and an image of the second frame, a pair of an image of the second frame and an image of the third frame, a pair of an image of the third frame and an image of the fourth frame, and a pair of an image of the fourth frame and an image of the first frame. Therefore, when N=2, two additional images are generated so that the four kinetic energies, namely, the kinetic energy between the image of the first frame and the image of the second frame, the kinetic energy between the image of the second frame and the image of the third frame, the kinetic energy between the image of the third frame and the image of the fourth frame, and the kinetic energy between the image of the fourth frame and the image of the first frame, are as equal as possible.

[0019] For example, in the present invention, an image set of N+2 images consisting of two original images and initial tentative images of N additional images is set as an initial image set to be updated, and N additional images are generated by repeating an update process to update the tentative images of the N additional images included in the image set to be updated so as to satisfy a predetermined condition to obtain a new image set to be updated. The predetermined condition is a condition that the kinetic energy between N+2 pairs of images, including N+1 pairs of two images adjacent in order in the image set to be updated and one pair of the N+2th image and the 1st image in the image set to be updated, is as equal as possible. In the present invention, as a specific example of an update process that satisfies this condition, a process is used in which the sum of the absolute values ​​of the differences in kinetic energy between the N+2 pairs of images is set as a loss, and the pixel values ​​of the provisional images of the additional images are updated so as to reduce the loss. Of course, any update process may be used as long as it satisfies the above-mentioned condition, and a process of updating the pixel values ​​of the provisional images of the additional images so as to reduce a loss different from the above-mentioned specific example may be used. The initial provisional image of the additional image may be any image, but for example, a two-dimensional noise image caused by white noise or Gaussian noise may be used as the initial provisional image of each additional image.

[0020] First Embodiment As illustrated in FIG. 1, the image generating device 100 of the first embodiment has an original image acquisition unit 110, an additional image generation unit 120, and a presentation image generation unit 130. As illustrated in FIG. 2, the image generating device 100 performs processes of steps S110, S120, and S130. The image generating device 100 generates N additional images using two images, a first original image that is a certain image, and a second original image that is an image different from the first original image, and generates a presentation unit image of N+2 images. The image generating device 100 is an image generating device that generates an image for making an observer who sees the presentation perceive an object as if it is moving continuously when the presentation unit images are repeatedly presented in succession.

[0021] [Original image acquisition unit 110] The original image acquisition unit 110 acquires and outputs a first original image, which is a certain image, and a second original image, which is an image different from the first original image (step S110). The first original image and the second original image output by the original image acquisition unit 110 are input to the added image generation unit 120 and the presentation image generation unit 130.

[0022] [[First example of original image acquisition unit 110]] When a video including a movement of a certain object is input to the image generating device 100 as an input image, the input image input to the image generating device 100 is input to the original image acquiring unit 110. For example, the original image acquiring unit 110 acquires and outputs images of two consecutive frames from the input image as a first original image and a second original image. When the difference in the position of the certain object is not visually recognized in the images of two consecutive frames in the input image, the original image acquiring unit 110 may acquire and output images of two frames with one or more frames between them from the input image as a first original image and a second original image. That is, the original image acquiring unit 110 acquires and outputs one frame including a certain object from the input image as a first original image, and acquires and outputs an image of a frame other than the one frame from the input image, which is equivalent to an image in which the certain object has been given a movement with respect to the first original image, as a second original image (step S110A).

[0023] [[Second example of original image acquisition unit 110]] When an image including a certain object is input to the image generating device 100 as an input image, the original image acquisition unit 110 outputs the input image as a first original image, and also synthesizes an image in which the position of the certain object included in the input image has been moved to another nearby position using a known image synthesis technique, and outputs the result as a second original image (step S110B).

[0024] [[Third example of original image acquisition unit 110]] When an image of a certain object and an image of a certain background are input to the image generating device 100 as input images, the original image acquiring unit 110 synthesizes an image in which the certain object is placed at a first position in the background image and outputs it as a first original image, and also synthesizes an image in which the certain object is placed at a second position close to the first position in the background image and outputs it as a second original image (step S110C).

[0025] The original image acquisition unit 110 in the first to third examples generates a first original image that is an image including a certain object, and a second original image that is an image corresponding to an image in which the certain object is given a movement with respect to the first original image. That is, in the first original image and the second original image generated by the original image acquisition unit 110 in the first to third examples, a certain object is in a different position, and other objects are in approximately the same position. However, this is a typical example, and in the first original image and the second original image, multiple objects may be in different positions, and other objects may be in approximately the same position. In addition, the original image acquisition unit 110 may obtain a first original image that is a certain image and a second original image that is an image different from the first original image, without considering which objects are in different positions and which objects are in approximately the same position in the first original image and the second original image. In short, the original image acquisition unit 110 may acquire two different images as a first original image and a second original image.

[0026] When a first original image, which is a certain image, and a second original image, which is an image different from the first original image, are input to the image generating device 100 as two input images, the original image acquiring unit 110 may output the two input images input to the image generating device 100 as the first original image and the second original image as they are. In this case, the image generating device 100 may not include the original image acquiring unit 110, the image generating device 100 may not perform step S110, and the two input images input to the image generating device 100 may be input to the additional image generating unit 120 and the presentation image generating unit 130 as the first original image and the second original image as they are.

[0027] [Additional image generation unit 120] The additional image generating unit 120 receives the first and second original images output by the original image acquiring unit 110, or the first and second original images, which are two images input to the video generating device 100. The first and second original images are different from each other, and an example of the first original image is an image including a certain object, and an example of the second original image is an image corresponding to an image in which the object is given motion with respect to the first original image. The additional image generating unit 120 generates and outputs N additional images using the first and second original images (step S120). At that time, the additional image generating unit 120 generates N additional images such that the kinetic energy between two adjacent images is as equal as possible when presented in the N+2 images included in the presentation unit video generated by the presentation video generating unit 130 described later. In generating additional images, the order of images when presented, i.e., the order of images included in the presentation unit images generated by the presentation image generating unit 130 described later, must be taken into consideration, so details of the operation of the additional image generating unit 120 will be explained after the explanation of the presentation image generating unit 130. The N additional images output by the additional image generating unit 120 are input to the presentation image generating unit 130.

[0028] The value of N, which is the number of additional images to be generated, may be stored in advance in the image generating device 100, or may be input to the additional image generating unit 120 from outside the image generating device 100. When the value of N is stored in advance in the image generating device 100, a storage unit (not shown) may be provided in the image generating device 100, and the value of N may be stored in the storage unit in advance. When the value of N is input to the additional image generating unit 120 from outside the image generating device 100, an input unit (not shown), such as a keyboard, numeric keypad, or touch panel, may be provided in the image generating device 100, and the input unit may receive an input of the value of N by a user. The value of the number of additional images to be generated input by the user to the input unit, i.e., the value received by the input unit, may be output to the additional image generating unit 120 as the value of N, and the value of N may be input to the additional image generating unit 120 as shown by the dashed line in FIG. 1.

[0029] Furthermore, the value of N, which is the number of additional images to be generated, may be a single value or multiple different values. When the value of N is multiple different values ​​(N1, N2,...), the additional image generating unit 120 may generate and output an additional image for each value of N (i.e., each of N1, N2,...), and the presentation image generating unit 130 described later may perform processing for each value of N (i.e., each of N1, N2,...).

[0030] [Presentation video generation unit 130] The presentation image generating unit 130 receives the first and second original images output by the original image acquiring unit 110, or the first and second original images, which are two images input to the image generating device 100, and the N additional images output by the additional image generating unit 120. The presentation image generating unit 130 generates an image set in which the input N+2 images are connected in the order in which they are presented as a minimum unit, and outputs the generated image set of the minimum unit as a presentation unit image, or outputs an image set in which the minimum unit image set (i.e., presentation unit image) is repeated multiple times and connected as a presentation continuous image (step S130). The presentation unit image or presentation continuous image output by the presentation image generating unit 130 is output from the image generating device 100 and input to an image presenting device, a recording device for recording images to be input to the image presenting device, or the like.

[0031] If n is an integer between 1 and N, for example, the presentation image generation unit 130 generates a minimum unit image set consisting of N+2 frames by treating the first original image as the image of the first frame of the minimum unit image set, the second original image as the image of the second frame of the minimum unit image set, and the nth image of the N additional images as the image of the n+2 frame of the minimum unit image set, and sequentially concatenating the images of the first frame to the N+2 frame.

[0032] For example, in the case of N=1, the presentation image generating unit 130 generates a minimum unit image set by sequentially concatenating the images of the first frame to the third frame of the minimum unit image set with the first original image as the image of the first frame of the minimum unit image set, the second original image as the image of the second frame of the minimum unit image set, and one additional image as the image of the third frame of the minimum unit image set. Also, for example, in the case of N=2, the presentation image generating unit 130 generates a minimum unit image set by sequentially concatenating the images of the first frame to the fourth frame of the minimum unit image set with the first original image as the image of the first frame of the minimum unit image set, the second original image as the image of the second frame of the minimum unit image set, the first image of the two additional images as the image of the third frame of the minimum unit image set, and the second image of the two additional images as the image of the fourth frame of the minimum unit image set.

[0033] [Video presentation] The image presentation device is a device that presents an image so that an observer can view it, and includes, for example, a display device that displays the image and a projector that projects the image. The image presentation device receives the unit image for presentation or the continuous image for presentation output by the image generation device 100. When the unit image for presentation is input, the image presentation device presents an image in which the input unit image for presentation is repeated and connected multiple times, and when the continuous image for presentation is input, the image presentation device presents the input continuous image for presentation as is. In other words, the image presentation device presents an image set in which the minimum unit image set described above is repeated and connected multiple times.

[0034] Therefore, the order of images presented in the minimum unit image set consisting of the above-mentioned N+2 frames (i.e., unit video for presentation) is the image of the first frame, the image of the second frame, the image of the nth frame where n is from 1 to N, the image of the first frame, the image of the second frame, the image of the nth frame where n is from 1 to N, the image of the first frame, the image of the second frame, ... In other words, the pairs of two images that are adjacent when presented in the N+2 images included in the minimum unit image set (i.e., unit video for presentation) are N+2 pairs in total, consisting of N+1 pairs of two images that are adjacent in order in the minimum unit image set and one pair of the N+2th image and the first image in the minimum unit image set.

[0035] [Details of the additional image generating unit 120] As described above, the additional image generating unit 120 generates N additional images so that the kinetic energy between two adjacent images is as equal as possible when presented in the N+2 images included in the presentation unit image generated by the presentation image generating unit 130. Also, as described above, the presentation unit image generated by the presentation image generating unit 130 is, for example, a minimum unit image set consisting of N+2 frames in which the first original image is the image of the first frame of the minimum unit image set, the second original image is the image of the second frame of the minimum unit image set, and the nth image of the N additional images is the image of the n+2 frame of the minimum unit image set, and the image of the first frame to the image of the N+2 frame are sequentially connected. Therefore, in this example, the additional image generating unit 120 generates additional images so that the kinetic energy between the two images constituting each pair is as equal as possible for N+2 pairs, including N+1 pairs of images of two adjacent frames from the image of the first frame to the image of the N+2 frame, and one pair of the image of the N+2 frame and the image of the first frame. A specific example of the operation of the additional image generating unit 120 when N=2 and when N=3 is as follows.

[0036] [Specific example of operation of the additional image generating unit 120 when N=2] When N=2, that is, when the image generating device 100 generates a presentation unit image made up of four frame images, the additional image generating unit 120 performs, for example, the following steps S120-A1 to S120-A3.

[0037] The added image generating unit 120 first sets the first original image acquired by the original image acquiring unit 110 as the image of the first frame, the second original image acquired by the original image acquiring unit 110 as the image of the second frame, a predetermined initial image as the provisional image of the third frame, and the predetermined initial image as the provisional image of the fourth frame (step S120-A1). The predetermined initial image is an arbitrary image, for example, a two-dimensional noise image due to white noise or Gaussian noise.

[0038] The additional image generating unit 120 then repeats a process of updating pixel values ​​of the provisional image of the third frame and pixel values ​​of the provisional image of the fourth frame a predetermined number of times so as to satisfy a condition that the four kinetic energies, namely, the kinetic energy between the image of the first frame and the image of the second frame, the kinetic energy between the image of the second frame and the provisional image of the third frame, the kinetic energy between the provisional image of the third frame and the provisional image of the fourth frame, and the kinetic energy between the provisional image of the fourth frame and the image of the first frame, are as equal as possible (step S120-A2). Each update process of step S120-A2 is, for example, the process of steps S120-A21 and S120-A22 below. Note that in step S120-A2, the additional image generating unit 120 does not necessarily have to end the update process a predetermined number of times, and may repeat the update process until the predetermined end condition is satisfied.

[0039] Step S120-A21: The additional image generation unit 120 calculates each of the kinetic energies between frames. Specifically, the additional image generation unit 120 calculates the kinetic energy between the image of the first frame and the image of the second frame, the kinetic energy between the image of the second frame and the tentative image of the third frame, the kinetic energy between the tentative image of the third frame and the tentative image of the fourth frame, and the kinetic energy between the tentative image of the fourth frame and the image of the first frame.

[0040] Step S120-A22: The additional image generating unit 120 performs an optimization process to update pixel values ​​of the provisional image of the third frame and pixel values ​​of the provisional image of the fourth frame so as to reduce the loss based on the loss that makes the kinetic energy between frames as equal as possible obtained in step S120-A21. The additional image generating unit 120 may perform any known optimization process, and may perform, for example, an optimization process using the algorithm described in Reference 1. Reference 1: Kingma, D. and Ba, J. "Adam: A Method for Stochastic Optimization," Proceedings of the 3rd International Conference for Learning Representations, San Diego, 2015.

[0041] The additional image generation unit 120 then outputs the tentative image of the third frame and the tentative image of the fourth frame finally obtained in step S120-A2, i.e., the tentative image of the third frame and the tentative image of the fourth frame after optimization, as an additional image of the third frame and an additional image of the fourth frame, respectively (step S120-A3).

[0042] The additional image generating unit 120 performs the process of step S120-A21 for calculating the kinetic energy between two adjacent frames when presented, for example, by the following steps A and B. Step A: First, an image sequence in which two adjacent images are linked in the time direction when presented is convolved with each filter from a group of multiple different three-dimensional filters defined by a set of four parameters, namely, spatial frequency f, spatial orientation o, temporal frequency τ, and phase φ, to obtain each convolution result. Each filter is stored in advance in a storage unit (not shown) of the added image generating unit 120. Since the image sequence to be subjected to the convolution process includes images for two frames, each filter may be a filter whose time range is limited to two frames.

[0043] In the additional image generating unit 120, as a three-dimensional filter group that satisfies these requirements, for example, a filter generated using the two-dimensional steerable filter described in Reference 2 is used. Specifically, the additional image generating unit 120 uses two two-dimensional steerable filters ψ' f,o,φ , ψ' f,o,φ+2πτ The three-dimensional filter ψ f,o,τ,φis used as the phase φ. In a storage unit (not shown) of the added image generating unit 120, the three-dimensional filters generated for a plurality of combinations of spatial frequency f, spatial orientation o, time frequency τ, and phase φ are stored in advance, and the added image generating unit 120 convolves each filter with the input image sequence to obtain each convolution result. As the spatial frequency f, spatial orientation o, and time frequency τ, it is sufficient to use preset values ​​so that the time-spatial frequency band and orientation contained in the image sequence to be subjected to the convolution process can be detected. As the phase φ, two types, 0o and 90o, that is, two types, 0 and π / 2, are used. The convolution results for each of f, o, τ, and φ obtained by the above-mentioned convolution process are S p,f,o,τ,φ Here, p represents each position of a pixel.

[0044] Step B: The additional image generating unit 120 calculates the sum of squares of the convolution results obtained in step A that differ only in phase, thereby obtaining the kinetic energy M p,f,o,τ That is, the added image generating unit 120 obtains the kinetic energy at each pixel for each combination of the spatial frequency f, the spatial orientation o, and the temporal frequency τ by the following formula (1).

number

[0045] The kinetic energy between two adjacent frames when presented is the combination of the kinetic energy for each pixel p, spatial frequency f, spatial orientation o, and temporal frequency τ obtained in step B. Reference 2: Simoncelli, EP and Freeman, WT, "The Steerable Pyramid: A Flexible Architecture for Multi-Scale Derivative Computation," IEEE Second International Conference on Image Processing, Washington DC, 1995.

[0046] In step S120-A22, the added image generating unit 120 may use, for example, one of two losses L1 and L2 described below as a loss for making the kinetic energy between two adjacent images as equal as possible when presented. The first loss L1 is a loss for making the sum of the absolute values ​​of the differences in kinetic energy between two adjacent images when presented smaller, and the second loss L2 is a loss for making each of the kinetic energies between two adjacent images when presented, other than the kinetic energy between two adjacent original images when presented, as close as possible to the target, with the kinetic energy between two adjacent original images when presented.

[0047] An index is assigned to each pair of two adjacent images when presented, i.e., each pair of two images for which kinetic energy was calculated in step S120-A21, and each loss is specifically described. In this description, the index of the pair of an image of the first frame and an image of the second frame (i.e., a pair of a first original image and a second original image) is i=1, the index of the pair of an image of the second frame and an image of a third frame is i=2, the index of the pair of an image of the third frame and an image of a fourth frame is i=3, and the index of the pair of an image of the fourth frame and an image of the first frame is i=4. In addition, the kinetic energy corresponding to each combination of pixel p, spatial frequency f, spatial orientation o, and temporal frequency τ of each pair i obtained in step S120-A21 is denoted as M i p,f,o,τLet F={(1,2), (2,3), (3,4)} be a set of pairs of indexes of two adjacent pairs, C be a set of all pairs of spatial frequency f, spatial orientation o, and temporal frequency τ, and X be a set of pixels included in the image. Then, the first loss L1, that is, the loss for reducing the sum of the absolute values ​​of the differences in kinetic energy between two adjacent images when presented, is as shown in the following formula (2). That is, when the loss L1 is used, in step S120-A22, the additional image generating unit 120 performs an optimization process of updating the pixel values ​​of the provisional image of the third frame and the pixel values ​​of the provisional image of the fourth frame so that the value of the loss L1 in the following formula (2) is reduced.

number

[0048] Note that the additional image generating unit 120 may perform optimization processing in step S120-A22 to update pixel values ​​of the provisional image of the third frame and pixel values ​​of the provisional image of the fourth frame so as to reduce the value of loss L1 in the following equation (2') without including the constant part of equation (2). L1 in the following equation (2') is the sum of absolute values ​​of differences in kinetic energy between two adjacent images when presented.

number

[0049] In addition, the second loss L2, i.e., the kinetic energy M 1 p,f,o,τWith the above as a goal, the loss required for each of the kinetic energies between two adjacent images when presented, other than the kinetic energy between two adjacent original images when presented, to come as close as possible to the goal is as shown in the following formula (3). That is, when the loss L2 is used, in step S120-A22, the added image generating unit 120 performs an optimization process of updating the pixel values ​​of the provisional image of the third frame and the pixel values ​​of the provisional image of the fourth frame so that the value of the loss L2 in the following formula (3) becomes smaller.

number

[0050] In addition, the additional image generation unit 120 may perform an optimization process in step S120-A22 in which the pixel values ​​of the provisional image of the third frame and the pixel values ​​of the provisional image of the fourth frame are updated so that the value of loss L2 in the following equation (3') is reduced, without including the constant part of equation (3).

number

[0051] If equation (3') is generalized to include the case where N=3 (described later), the following equation (3'') is obtained. L2 in equation (3'') is the sum of the absolute values ​​of the differences between the kinetic energy between the first original image and the second original image and each of the kinetic energies between two adjacent images when presented other than the kinetic energy between the first original image and the second original image.

number

[0052] [Specific example of operation of the additional image generating unit 120 when N=3] When N=3, that is, when the image generating device 100 generates a presentation unit image made up of five frame images, the additional image generating unit 120 performs, for example, the following steps S120-B1 to S120-B3.

[0053] The added image generating unit 120 first sets the first original image acquired by the original image acquiring unit 110 as the image of the first frame, the second original image acquired by the original image acquiring unit 110 as the image of the second frame, a predetermined initial image as the provisional image of the third frame, a predetermined initial image as the provisional image of the fourth frame, and a predetermined initial image as the provisional image of the fifth frame (step S120-B1). The predetermined initial image is any image, for example, a two-dimensional noise image due to white noise or Gaussian noise.

[0054] The additional image generating unit 120 then repeats a process of updating pixel values ​​of the provisional image of the third frame, the provisional image of the fourth frame, and the provisional image of the fifth frame a predetermined number of times so as to satisfy a condition that five kinetic energies, namely, the kinetic energy between the image of the first frame and the image of the second frame, the kinetic energy between the image of the second frame and the provisional image of the third frame, the kinetic energy between the provisional image of the third frame and the provisional image of the fourth frame, the kinetic energy between the provisional image of the fourth frame and the provisional image of the fifth frame, and the kinetic energy between the provisional image of the fifth frame and the image of the first frame, are as equal as possible (step S120-B2). Each update process of step S120-B2 is, for example, the process of steps S120-B21 and S120-B22 below. Note that in step S120-B2, the additional image generating unit 120 does not necessarily have to end the update process a predetermined number of times, and may repeat the update process until the predetermined end condition is satisfied.

[0055] Step S120-B21: The additional image generating unit 120 calculates each of the kinetic energies between frames. Specifically, the additional image generating unit 120 calculates the kinetic energy between the image of the first frame and the image of the second frame, the kinetic energy between the image of the second frame and the tentative image of the third frame, the kinetic energy between the tentative image of the third frame and the tentative image of the fourth frame, the kinetic energy between the tentative image of the fourth frame and the tentative image of the fifth frame, and the kinetic energy between the tentative image of the fifth frame and the image of the first frame.

[0056] Step S120-B22: The additional image generation unit 120 performs an optimization process to update the pixel values ​​of the provisional image of the third frame, the pixel values ​​of the provisional image of the fourth frame, and the pixel values ​​of the provisional image of the fifth frame so as to reduce the loss based on the loss that makes the kinetic energy between frames as equal as possible obtained in step S120-B21. For example, the additional image generating unit 120 may perform an optimization process to update pixel values ​​of the provisional image of the third frame, pixel values ​​of the provisional image of the fourth frame, and pixel values ​​of the provisional image of the fifth frame so as to reduce the loss, using either of the two losses L1 and L2 described above in the description of the specific example of the operation of the additional image generating unit 120 when N=2, specifically, the sum of absolute values ​​of the differences in kinetic energy between two adjacent images when presented, or the sum of absolute values ​​of the differences between the kinetic energy between the first original image and the second original image and each of the kinetic energies between two adjacent images when presented other than the kinetic energy between the first original image and the second original image, as the loss. The additional image generating unit 120 may perform any known optimization process, but may perform an optimization process using the algorithm described in Reference 1, for example.

[0057] The additional image generation unit 120 then outputs the tentative image of the third frame, the tentative image of the fourth frame, and the tentative image of the fifth frame obtained in step S120-B2, i.e., the tentative image of the third frame, the tentative image of the fourth frame, and the tentative image of the fifth frame after optimization, as the additional image of the third frame, the additional image of the fourth frame, and the additional image of the fifth frame (step S120-B3).

[0058] <Second embodiment> In the first embodiment, a specific example was described in which the first original image is the image of the first frame of the presentation unit video, the second original image is the image of the second frame of the presentation unit video, and the nth image of the N additional images is the image of the n+2th frame of the presentation unit video, but one or more additional images may be included between the first original image and the second original image in the presentation unit video. In this case, as in the first embodiment, the additional image generating unit 120 may generate N additional images such that the kinetic energy between two adjacent images is as equal as possible when presented among the N+2 images included in the presentation unit video.

[0059] For example, the presentation image generating unit 130 may generate a minimum unit image set consisting of N+2 frames as a presentation unit image by sequentially concatenating the image of the first frame to the image of the N+2 frame, where N is an integer equal to or greater than 2, with the first original image being the image of the first frame of the minimum unit image set, the first additional image being the image of the second frame of the minimum unit image set, the second original image being the image of the third frame of the minimum unit image set, and each additional image from the second onwards being the image of the n+2 frame of the minimum unit image set. In this example, the additional image generating unit 120 may generate additional images for N+2 pairs, including N+1 pairs of images of two adjacent frames from the image of the first frame to the image of the N+2 frame, and one pair of an image of the N+2 frame and an image of the first frame, as in the first embodiment, so that the kinetic energy between the two images constituting each pair is as equal as possible. As the loss in the optimization process of the additional image generating unit 120, the loss L1 described above in the first embodiment or the loss L3 described below may be used. The loss L3 is a loss that aims to ensure that the kinetic energy between two adjacent images when presented is as close as possible to the target value, which is the kinetic energy between the first and second original images divided by the number of additional images between the first and second original images plus 1.

[0060] A specific example of the loss in the optimization process performed by the additional image generation unit 120 will be described below with N=3, in which the presentation image generation unit 130 generates a minimum unit image set consisting of N+2 frames as a presentation unit image by sequentially concatenating the first frame image to the N+2th frame image, with the first original image being the image of the first frame of the minimum unit image set, the second original image being the image of the third frame of the minimum unit image set, the first additional image being the image of the second frame, the second additional image being the image of the fourth frame, and the third additional image being the image of the fifth frame, with the first frame image to the N+2th frame image being concatenated. In this example, the additional image generation unit 120 first calculates the target kinetic energy M T p,f,o,τ The image sequence obtained by temporally connecting the first frame image and the third frame image is subjected to the multiple different three-dimensional filters ψ f,o,τ,φ Each convolution result S obtained by convolving p,f,o,τ,φ Using this, it is obtained by the following equation (4).

number

number

[0061] Furthermore, more generally, where N is an integer greater than or equal to 2 and x is an integer greater than or equal to 3, the presentation image generation unit 130 may generate a minimum unit image set consisting of N+2 frames as a presentation unit image by sequentially concatenating the images of the first frame to the N+2 frames, with the first original image being the image of the first frame of the minimum unit image set, the first additional image being the image of the second frame of the minimum unit image set, ..., the x-2nd additional image being the image of the x-1th frame of the minimum unit image set, the second original image being the image of the xth frame of the minimum unit image set, the x-1th additional image being the image of the x+1th frame of the minimum unit image set, ..., the Nth additional image being the image of the N+2th frame of the minimum unit image set, and so on. In this case, the additional image generating unit 120 may generate additional images for N+2 pairs, including N+1 pairs of images of two adjacent frames from the image of the first frame to the image of the N+2th frame, and one pair of an image of the N+2th frame and an image of the first frame, so that the kinetic energy between the two images constituting each pair is as equal as possible, as in the first embodiment. As the loss in the optimization process of the additional image generating unit 120, the loss L1 or the loss L3 may be used as described above.

[0062] The additional image generating unit 120 first calculates the target kinetic energy M T p,f,o,τ , and apply each of the multiple different three-dimensional filters ψ described above in the first embodiment to an image sequence in which the image of the first frame and the image of the x-th frame are temporally linked. f,o,τ,φ Each convolution result S obtained by convolving p,f,o,τ,φ Using this, the following equation (4') is obtained.

number

number

[0063] <Third embodiment> Up to the second embodiment, it is assumed that the frame rate of the presented video is fixed, but the time intervals for the N+2 pairs, i.e., N+1 pairs of images of two adjacent frames from the image of the first frame to the image of the N+2 frame and one pair of the image of the N+2 frame and the image of the first frame, are not necessarily the same, and may be a preset time interval. In the present invention, regardless of whether the time intervals for the N+2 pairs are all the same, the additional images may be generated by taking into consideration the motion detection characteristics of the human visual system when the observer observes. Therefore, regardless of whether the time intervals for the N+2 pairs are all the same, the additional image generating unit 120 may generate N additional images so that the kinetic energy per unit time between the two adjacent images when presented is as equal as possible.

[0064] <Programs and recording media> The processing of each unit of the image generating device described above may be realized by a computer, in which case the processing contents of the functions that each device should have are described by a program. Then, by loading this program into the storage unit 1020 of the computer 1000 shown in Fig. 3 and operating the arithmetic processing unit 1010, the input unit 1030, the output unit 1040, etc., various processing functions of each of the above devices are realized on the computer.

[0065] The program describing the processing contents can be recorded on a computer-readable recording medium. The computer-readable recording medium is, for example, a non-transitory recording medium, specifically, a magnetic recording device, an optical disk, etc.

[0066] The program may be distributed, for example, by selling, transferring, lending, etc. portable recording media such as DVDs and CD-ROMs on which the program is recorded. Furthermore, the program may be distributed by storing the program in a storage device of a server computer and transferring the program from the server computer to other computers via a network.

[0067] A computer that executes such a program, for example, first stores a program recorded on a portable recording medium or a program transferred from a server computer in the auxiliary recording unit 1050, which is a non-transient storage device of the computer. Then, when executing a process, the computer reads the program stored in the auxiliary recording unit 1050, which is a non-transient storage device of the computer, into the storage unit 1020, and executes a process according to the read program. As another execution form of the program, the computer may directly read the program from the portable recording medium into the storage unit 1020 and execute a process according to the program, or may execute a process according to the received program each time a program is transferred from the server computer to the computer. Alternatively, the server computer may not transfer the program to the computer, but may execute the above-mentioned process by a so-called ASP (Application Service Provider) type service that realizes a processing function only by issuing an execution instruction and obtaining a result. Note that the program in this embodiment includes information used for processing by an electronic computer and equivalent to a program (data that is not a direct instruction to a computer but has a nature that specifies the processing of the computer, etc.).

[0068] In addition, in this embodiment, the present device is configured by executing a specific program on a computer, but at least a part of the processing contents may be realized by hardware.

[0069] Needless to say, other modifications can be made without departing from the spirit of the present invention.

Claims

1. An image generating device that generates an image for making an observer who sees the presentation perceive a visual target as moving continuously when the presentation unit images are repeatedly and continuously presented, Let N be an integer equal to or greater than 1, an additional image generating unit that generates N additional images using a first original image that is a certain image and a second original image that is an image different from the first original image; a presentation image generating unit that generates a set of N+2 images, the first original image, the second original image, and N additional images, as the presentation unit image; Including, The additional image generating unit is Generate N additional images so that the kinetic energy per unit time between two adjacent images when presented is as equal as possible among the N+2 images included in the presentation unit video. Image generation device.

2. 2. The image generating device according to claim 1, The second original image is an image in which one or more visual objects are located at positions that are not substantially the same as their positions in the first original image, and objects other than the visual objects are located at positions that are substantially the same as their positions in the first original image. Image generation device.

3. 3. The image generating device according to claim 2, The position of the one or more visual targets in the second original image is within a range in which movement is perceived when the first original image and the second original image are presented in succession over time. Image generation device.

4. 4. The image generating device according to claim 1, The additional image generating unit is A set of N+2 images including the first original image, the second original image, and the initial tentative images of the N additional images is set as an initial image set to be updated; For the image set to be updated, a sum of absolute values ​​of the difference in kinetic energy between two adjacent images when presented is set as a loss, and a process of updating pixel values ​​of the provisional images of the N additional images so as to reduce the loss is repeated until a predetermined condition is satisfied, thereby obtaining the N provisional images as the N additional images. Image generation device.

5. 4. The image generating device according to claim 1, Let n be an integer between 1 and N, inclusive. The presentation image generation unit is The first original image is a first frame image, the second original image is a second frame image, and the n-th image of the N additional images is an n+2-th frame image, and the images of the first frame to the N+2-th frame are sequentially linked to generate the unit video for presentation; The additional image generating unit is generating N additional images for N+2 pairs, including N+1 pairs of images of two adjacent frames from the image of the first frame to the image of the N+2 frame, and one pair of the image of the N+2 frame and the image of the first frame, so that the kinetic energy between the two images constituting each pair is as equal as possible; Image generation device.

6. 6. The image generating device according to claim 5, The additional image generating unit is A set of the image of the first frame, the image of the second frame, and N initial tentative images of the n+2 frame is set as an initial image set to be updated; For the image set to be updated, a sum of absolute values ​​of differences in kinetic energy between two images constituting each pair of N+2 pairs, including N+1 pairs of images of two adjacent frames from the image of the first frame to the image of the N+2 frame, and one pair of the image of the N+2 frame and the image of the first frame; or a sum of absolute values ​​of differences between kinetic energy between two images constituting each of N+1 pairs, including N pairs of images of two adjacent frames from the image of the second frame to the image of the N+2 frame, and one pair of the image of the N+2 frame and the image of the first frame, and between kinetic energy between the image of the first frame and the image of the second frame; a loss is set as the loss, and a process of updating pixel values ​​of the tentative images of the N additional images is repeated so as to reduce the loss until a predetermined condition is satisfied, thereby obtaining the N tentative images as the N additional images. Image generation device.

7. 4. The image generating device according to claim 1, Let N be an integer equal to or greater than 2, and n be an integer equal to or greater than 1 and equal to or less than N, The presentation image generation unit is The first original image is an image of a first frame, the second original image is an image of an xth frame, which is any one of the 3rd frame to the N+1th frame, and each of the N additional images is an image of each of the N frames that are neither the 1st frame nor the xth frame, and the image of the 1st frame to the image of the N+2th frame are sequentially connected to generate the unit video for presentation; The additional image generating unit is generating N additional images for N+2 pairs, including N+1 pairs of images of two adjacent frames from the image of the first frame to the image of the N+2 frame, and one pair of the image of the N+2 frame and the image of the first frame, so that the kinetic energy between the two images constituting each pair is as equal as possible; Image generation device.

8. 8. The image generating device according to claim 7, The additional image generating unit is A set of the image of the first frame, the image of the xth frame, and initial tentative images of N frames other than the first frame and the xth frame is set as an initial image set to be updated; For the image set to be updated, a sum of absolute values ​​of differences in kinetic energy between two images constituting each pair of N+2 pairs, including N+1 pairs of images of two adjacent frames from the image of the first frame to the image of the N+2 frame, and one pair of the image of the N+2 frame and the image of the first frame; or the sum of absolute values ​​of the differences between the value obtained by dividing the kinetic energy between the image of the first frame and the image of the xth frame by x-1 and the kinetic energy between the two images constituting each of the N+2 pairs; a loss is set as the loss, and a process of updating pixel values ​​of the tentative images of the N additional images is repeated so as to reduce the loss until a predetermined condition is satisfied, thereby obtaining the N tentative images as the N additional images. Image generation device.

9. An image generating method for generating an image for causing an observer who sees a presentation of unit images for presentation repeatedly and continuously to perceive a visual target as if the visual target were moving continuously, the image generating method comprising: Let N be an integer equal to or greater than 1, an additional image generating step in which the image generating device generates N additional images using a first original image that is a certain image and a second original image that is an image different from the first original image; a presentation image generating step in which the image generating device generates, as the presentation unit image, a set of N+2 images including the first original image, the second original image, and N additional images; Including, The additional image generating step includes: Generate N additional images so that the kinetic energy per unit time between two adjacent images when presented is as equal as possible among the N+2 images included in the presentation unit video. Video generation method.

10. A program for causing a computer to function as each unit of the image generating device according to any one of claims 1 to 8.

Images