Video creation device, video creation method, program, and storage medium

Abstract

To generate a high-quality video of a normal angle of view from panorama images.SOLUTION: A video generation apparatus for generating a video of a normal angle of view from a plurality of panorama images includes: a matching degree calculation unit 103 which calculates a degree of matching of camera work types of a virtual camera with respect to the panorama images; a type allocation unit 104 which determines allocation of the camera work types to the panorama images on the basis of the matching degree and appearance frequency of the camera work types; and a video encoding unit 105 which generates a video of a normal angle of view from the panorama images with the allocated camera work types.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a moving image generating device, a moving image generating method, a program, and a storage medium. [Background technology]

[0002] A technique for generating a video with a normal angle of view by creating camerawork of a virtual camera (hereinafter referred to as a virtual camera) for a panoramic image such as a spherical image has been known for some time. The camerawork is the camera parameters (position, attitude, focal length, etc.) of the virtual camera that change over time. The camerawork allows an information processing device or the like to display a video of a panoramic image viewed from various viewpoints.

[0003] Patent Document 1 discloses a technique for creating a single video from videos captured by multiple cameras.

[0004] Patent Document 2 discloses a technique for generating a moving image by tracking a subject from a panoramic image.

[0005] Patent Document 3 discloses a technique for presenting a view seen from a viewpoint moving along a predetermined route as a video by extracting a video from an omnidirectional video.

[0006] Patent Document 4 discloses a technique for extracting an important area from a single spherical image and generating a moving image by transitioning the important area.

[0007] These techniques are techniques for generating a normal video from a plurality of panoramic images by regarding the panoramic video as a collection of a plurality of panoramic still images and generating a video from the panoramic video. Summary of the Invention [Problem to be solved by the invention]

[0008] However, in conventional videos with a normal angle of view created from panoramic images, there is room for improvement in terms of camerawork quality in terms of the suitability of each panoramic image and the interrelationships between the panoramic images.

[0009] In view of the above-mentioned problems, an object of the present invention is to provide a moving image generating device capable of generating a high-quality moving image with a normal angle of view from a panoramic image. [Means for solving the problem]

[0010] In view of the above problem, the present invention provides a method for extracting a panoramic image from a plurality of panoramic images. The angle of view is narrower than the panoramic image A moving image generating device for generating a moving image, the moving image generating device being configured to generate a moving image for a plurality of panoramic images According to the importance distribution of each region of the panoramic image, a compatibility calculation unit that calculates a compatibility of a camera work type of a virtual camera; a type assignment unit that determines an assignment of a camera work type to the plurality of panoramic images based on the compatibility and an appearance frequency of each camera work type; and The angle of view is narrower than the panoramic image and a moving image encoding unit that creates moving images. Effect of the Invention

[0011] It is possible to provide a moving image generating device capable of generating high-quality moving images with a normal angle of view from a panoramic image. [Brief description of the drawings]

[0012] [Figure 1] FIG. 2 is a diagram illustrating camera parameters and camera work. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of a moving image generating device. [Diagram 3] FIG. 2 is an example of a functional block diagram illustrating functions of the moving image generating device in the form of blocks. [Figure 4] FIG. 2 is a flowchart illustrating an example of a procedure for generating camerawork by a moving image generating device (first embodiment). [Diagram 5]FIG. 13 is a diagram showing an example of a camera work type. [Figure 6] This is a visualization of an example of a mixed von Mises-Fisher distribution. [Figure 7] FIG. 11 is an example of a functional block diagram illustrating functions of a moving image generating device according to a second embodiment in the form of blocks. [Figure 8] FIG. 11 is a flowchart illustrating an example of a procedure for generating camerawork by a moving image generating device (second embodiment). [Figure 9] 13A to 13C are diagrams illustrating examples of camera work types in the second embodiment. [Figure 10] FIG. 13 is an example of a functional block diagram illustrating functions of a moving image generating device according to a third embodiment in the form of blocks. [Figure 11] FIG. 13 is a flowchart illustrating an example of a procedure for generating camerawork by a moving image generating device (third embodiment). [Figure 12] FIG. 13 is an example of a functional block diagram illustrating functions of a moving image generating device according to a fourth embodiment in the form of blocks. [Figure 13] FIG. 13 is a flowchart illustrating an example of a procedure for generating camerawork by a moving image generating device (fourth embodiment). [Figure 14] FIG. 1 is a diagram showing a schematic diagram of a camera work type and a compatibility degree output from a panoramic image in CAGN. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0013] Hereinafter, a moving image generating device and a moving image generating method performed by the moving image generating device will be described as an example of an embodiment of the present invention.

[0014] <Terminology> First, the terms used in this embodiment will be explained.

[0015] The camera parameters are information that determines the viewpoint of a virtual camera in three-dimensional space, and refer to, for example, one or more of the position, orientation, focal length, aspect ratio, lens distortion, and the like of the virtual camera.

[0016] Camerawork is a chronological arrangement of camera parameters.

[0017] A virtual camera is not an actual camera, but a virtually existing camera that generates captured images using defined camera parameters.

[0018] The degree of suitability reflects the aesthetic quality of the image and the recognizability of the target object, and is subjectively set by humans. It is preferable that the higher the value, the more desirable the image is perceived by humans.

[0019] <Outline of this embodiment> Conventional techniques are techniques for generating a normal video from a plurality of panoramic images by regarding the panoramic video as a collection of a plurality of panoramic still images and generating a video from the panoramic video.

[0020] However, because typical panoramic images do not have consecutive scenes, it is difficult for a video generation device to adopt conventional video generation techniques for panoramic videos that assume smooth changes in images between frames.

[0021] There is also a method of applying a conventional method of generating a video from a single panoramic still image to each image and stitching together the generated videos. However, this method involves processing each panoramic image independently, so there is a risk that the video will be monotonous due to a series of similar camerawork.

[0022] Therefore, in this embodiment, when creating virtual camera work for panoramic images and generating a video with a normal angle of view, the suitability for each panoramic image and the interrelationship between the panoramic images are taken into consideration to prevent similar camera work types from appearing consecutively, thereby making it possible to generate a high-quality video with a normal angle of view from panoramic images.

[0023] [First embodiment] As a first embodiment, an example will be described in which a moving image generating device adaptively assigns a plurality of fixed cameraworks to each panoramic image to generate one moving image. First, a prerequisite for explaining the processing of the moving image generating device will be explained.

[0024] A panoramic image refers to an image captured at a wide angle, and wide-angle capture means that the image is captured at a wider angle than the angle of view of the video to be output. For example, even if the angle of view is 60 degrees, if the angle of view of the video to be output is less than 60 degrees, the original image is considered to be a panoramic image in this embodiment. In addition, a 360-degree horizontal and 180-degree vertical omnidirectional image is also a panoramic image.

[0025] Assume that the internal parameters and lens distortion of the camera that took the panoramic image have been calibrated by other means, and the relative projection direction of each pixel in the panoramic image in three-dimensional space is known. In this case, the panoramic image can be reprojected onto a two-dimensional unit sphere. Formally, the projection direction of pixel i is (xi, yi, zi) ∈ R 3 (However, 2 +yi 2 +zi 2 = 1), and assign the pixel value of pixel i to position (xi, yi, zi) on the two-dimensional unit sphere. Although pixel positions are discrete, the video generator can assign pixel values ​​continuously on the two-dimensional unit sphere by interpolating using the nearest neighbor method, bilinear method, bicubic method, etc. This two-dimensional unit sphere is referred to as the "image sphere" below.

[0026] Next, we define camerawork, which refers to the time series data of the parameters of a virtual camera that converts a panoramic image into a partial perspective projection image.

[0027] FIG. 1 is a diagram for explaining camera parameters and camera work. The video generating device projects a panoramic image onto an image sphere, and obtains a partial perspective projection image by projecting this image sphere onto a shooting plane with a virtual camera. As shown in FIG. 1(a), the parameters of the virtual camera include the projection center, the optical axis direction, the optical axis rotation, and the focal length. Other parameters of the virtual camera may include the aspect ratio and the lens distortion coefficient. Typically, a natural perspective projection image without distortion can be obtained by matching the projection center with the image sphere center. When the projection center exists on the image sphere and the image sphere center exists on the optical axis, it becomes a stereo projection method, and a field of view of 180 degrees or more can be effectively captured in one flat image.

[0028] 1(b), a camerawork type is a parameter used when generating camerawork for an image. Formally, when a set of images is X, a set of parameters is Z, a set of camerawork is Y, and a mapping for camerawork generation is G:X×Z→Y, an element of parameter set Z is called a camerawork type. In the first embodiment, parameter set Z is made up of a finite number of elements, and a case is handled in which G corresponds an element of parameter set Z to one element of camerawork set Y (therefore, camerawork type does not depend on image).

[0029] <Configuration example> 2 shows an example of a hardware configuration diagram of the moving image generating device 100. As shown in FIG. 2, the moving image generating device 100 has a CPU 11, a ROM 12, a RAM 13, an input unit 14, a display unit 15, and a connection I / F 16.

[0030] The CPU 11 performs overall control of the operation of the moving image creation device 100. The CPU 11 uses a predetermined area of ​​the RAM 13 as a working area, executes various control programs stored in the ROM 12, and realizes various functions of the moving image creation device 100. Specific contents of the functions of the moving image creation device 100 will be described later.

[0031] The ROM 12 is a non-volatile memory (a non-rewritable memory) that stores programs and various setting information related to the video generating device 100. The RAM 13 is a storage means such as an SDRAM, which functions as a working area for the CPU 11 and temporarily stores programs and various data.

[0032] The input unit 14 is a peripheral device for receiving input from a user (operation of a keyboard, mouse, voice, etc.). The display unit 15 is a component or an external peripheral device for displaying various information related to the video generating device 100, and is configured, for example, as a liquid crystal display. The display unit 15 and the input unit 14 may be integrated into one unit, such as a touch panel. The connection I / F 16 is an interface for connecting to an external device. For example, the connection I / F 16 may be a general-purpose interface such as a USB, or may be a communication interface for communicating with an external device via a wired or wireless connection.

[0033] <Functions of the video generating device> Next, the functions of the video generating device 100 in this embodiment will be described with reference to FIG. 3. FIG. 3 is an example of a functional block diagram showing the functions of the video generating device 100 in blocks. The video generating device 100 has an image acquisition unit 101, an importance calculation unit 102, a compatibility calculation unit 103, a type assignment unit 104, and a video encoding unit 105. Each of these functions of the video generating device 100 is a function or means realized by the CPU 11 executing a program expanded from the ROM 12 to the RAM 13 shown in FIG. 2. Note that this program may be distributed from a server for program distribution, or may be distributed in a state stored in a computer-readable storage medium such as a USB memory or an optical storage medium.

[0034] 3 is a device that stores a panoramic image. The storage device 20 may be, for example, a USB memory, an external HDD, a NAS (Network Attached Storage), or a server, as long as it stores a panoramic image. The storage device 20 may be present inside the video creation device 100.

[0035] Since each function of the moving image generating device 100 corresponds to each step in the flowchart shown in FIG. 4, each function of the moving image generating device 100 will be described with reference to the flowchart.

[0036] 4 is an example of a flowchart showing a procedure for generating camerawork by the video generating device 100 in this embodiment. The following will be described with reference to the flowchart in FIG.

[0037] (Processing step S101) In processing step S101, the image acquisition unit 101 acquires from the storage device 20 a plurality of panoramic images to be processed.

[0038] (Processing step S102) In processing step S102, the importance of each region in the acquired panoramic image is calculated by the importance calculation unit 102. The importance calculation unit 102 preferably calculates the importance for each pixel, but may also calculate with sub-pixel accuracy or for each specific region.

[0039] Moreover, the importance is an index indicating the importance of the viewer viewing the region, and is approximately calculated by the following methods. One is a method using visual saliency detection. Visual saliency detection includes a method of stacking individual rules in a bottom-up manner, such as setting high saliency to edges and isolated points, and a method of estimating from an input image in a top-down manner using a machine learning method such as a neural network or a support vector machine. Either method can be used in this embodiment. Also, as a method other than visual saliency, a person in charge can predetermine object categories with high importance (for example, people, faces, animals, automobiles, etc.), and the importance of an area in which the object is detected by an object detection algorithm can be determined to be high. Alternatively, the importance calculation unit 102 can estimate the layout of a scene in a panoramic image, and estimate the importance of a direction in which a radial composition, a division of thirds composition, or a horizontal composition of the scene is obtained as being high. Alternatively, the importance calculation unit 102 can use the attention obtained by the attention mechanism of a neural network as the importance. An attention mechanism is a mechanism that learns the relationships between elements or the points to pay attention to, and is mainly introduced into Encoder-Decoder models intended for machine translation, image processing, etc.

[0040] It is also possible to combine the above methods of visual saliency detection, object detection, layout estimation, and attention. 2 (S 2 The importance of v is given by I(v)∈R. 2 →R is called importance distribution I. 1. Importance distributions are composed of a combination of multiple probability distributions. 2.The individual probability distributions that make up the importance distribution are referred to as element distributions.

[0041] (Processing step S103) In processing step S103, the suitability calculation unit 103 calculates the suitability of a camera work type prepared in advance from the importance distribution I of the panoramic images.

[0042] Figure 5 shows an example of a camera work type. For example, as shown in Figure 5, the camera work includes (a) The projection center is aligned with the center of the image sphere and the optical axis direction is moved horizontally from one end of the panoramic image to the other (one rotation in the case of a full-circle image); (b) The optical axis is fixed in the direction of gravity and rotated using stereographic projection. (c) The optical axis is fixed in the opposite direction to gravity and rotated using a stereographic projection. etc.

[0043] Each of these camerawork types has different characteristics: (a) is suitable for generating videos that are viewed widely horizontally, while (b) and (c) are suitable for generating videos that provide a wide, vertical, bird's-eye view.

[0044] The suitability calculation unit 103 calculates the suitability of each camera work type for each panoramic image according to the importance distribution I. As a specific method, the importance distribution I can be modeled using a mixed von Mises-Fisher distribution, and the suitability of the camera work type can be calculated from the parameters of the model. The mixed von Mises-Fisher distribution is a random variable v∈S 2 is a probability distribution with the probability density function given by equation (1).

[0045]

number

[0046] K is the number of element distributions α K ∈[0,1] is the mixture rate of the kth element distribution (Σ K K=1 α K =1) μ K ∈S 2 is the mean direction of the kth element distribution κ K ∈R(≧0) is the concentration of the kth element distribution (κ is kappa) <·,·> is the inner product (R3 (standard inner product of C(κ k )∈R is a normalization constant that satisfies equation (2).

[0047]

number

[0048] In the mixed von Mises-Fisher distribution, {α k , μ k , κ k} K k=1 is a parameter. The compatibility calculation unit 103 obtains this parameter by maximum likelihood estimation for the importance distribution I. Specifically, the compatibility calculation unit 103 obtains the parameter {α k , μ k , κ k} K k=1 Request.

[0049]

number

[0050] The method of calculating the suitability is not limited to these, and any method that can evaluate the camera work type in a manner that is in line with the distribution of importance will suffice. The suitability of the m-th camera work type for the n-th panoramic image calculated in the above manner is denoted as Dnm.

[0051] (Processing step S104) In processing step S104, the type assignment unit 104 calculates the camera work type suitability {Dnm} N,M n=1,m=1 A camerawork type is assigned to each panoramic image using the formula: where N is the number of panoramic images, and M is the number of camerawork types. If the same type of camerawork appears frequently, the overall video will lack variety, so the person in charge determines in advance the minimum number of times each type of camerawork will appear B (the frequency of appearance of each type of camerawork).

[0052] By optimizing the suitability of each image while taking into account the frequency of appearance of camerawork types, it is possible to assign an appropriate camerawork type to each image without being biased towards a specific camerawork type. For example, in this embodiment, camerawork type A is suitable for image 1 when viewed alone, but image 2 is more suitable for camerawork type A than image 1, so in order to avoid consecutive occurrences of the same camerawork type, camerawork type A is assigned to image 2, which is more suitable, and camerawork type B is assigned to image 1, resulting in an interaction. The suitability for each panoramic image and the interrelationships between panoramic images can be taken into consideration.

[0053] Then, the type assignment unit 104 determines the assignment of camera work types to panoramic images that maximizes the sum of the degrees of compatibility. The assignment is performed by a variable τ nm By introducing the above, we can formulate the problem as the following integer programming problem (equation (4)).

[0054]

number

[0055] (Processing step S105) In processing step S105, the video encoding unit 105 creates partial perspective projection videos from each panoramic image according to the camerawork specified by the set camerawork type, and encodes and outputs the videos as one video. With the above steps, a high-quality video with a normal angle of view can be generated from the panoramic images.

[0056] <Processing variations> In the following, preferred variations of the process in the first embodiment will be described.

[0057] (Processing step S103) In the parameter estimation of the mixed von Mises-Fisher distribution in the process step S103, the estimation result can be stabilized by using Bayesian estimation. k , μ k , κ k} K k=1A prior distribution is set for v∈S to obtain the posterior distribution of the parameters. This can be approximated by using the variational Bayes method. It is also possible to directly calculate the suitability of the camerawork type from the importance distribution without going through parameter estimation of the distribution. For example, in the case of horizontal movement camerawork type (a), the suitability calculation unit 103 calculates the suitability of the camerawork type v∈S 2 The importance I(v) of v is expressed as the gravity direction g∈S as shown in equation (5). 2 The degree of matching may be calculated by multiplying the weights related to the inner product with the above.

[0058]

number

[0059] In process step S103, the evaluation function is weighted by {w nm ∈R} N,M n=1,m=1 It is also possible to adjust the camerawork type to be selected by introducing the following. Specifically, the evaluation function in equation (4) is replaced by equation (6).

[0060]

number

[0061]

number

[0062] According to the first embodiment described above, by assigning a camera work type to each panoramic image based on the suitability and frequency of appearance of the camera work type, videos using various types of camera work can be generated for multiple panoramic images.

[0063] Furthermore, the video generation device incorporates the frequency of appearance of camera work types into a constraint condition or an evaluation function, and determines the allocation of camera work types to each panoramic image so as to maximize the sum of the suitability of the camera work types. This makes it possible to generate appropriate camera work for each panoramic image while suppressing bias toward specific camera work types.

[0064] In addition, by calculating the suitability of the camerawork type according to the distribution of importance in the image, it is possible to generate camerawork that is adaptive to the content of the image.

[0065] [Second embodiment] As a second embodiment, a moving image generation device that defines camerawork types by categorizing sets of parameters to be provided to a camerawork generation module (camerawork generation unit 205 described later) and generates one moving image from multiple panoramic images will be described.

[0066] In the first embodiment, the camera work is fixed to one for each camera work type, but in this embodiment, even if the camera work type is the same, different camera works are generated depending on the contents of the panoramic image.

[0067] Fig. 7 is an example of a functional block diagram showing functions of the moving image generating device 100 in the second embodiment. The moving image generating device 100 of this embodiment has a camerawork generating unit 205 in addition to the moving image generating device 100 in Fig. 3. These functions will be described with reference to Fig. 8.

[0068] Fig. 8 is an example of a flowchart showing the procedure for generating camerawork by the video production device 100. Compared to Fig. 4, step S203 is different, and step S205 has been added. The following mainly describes the differences from Fig. 4. Steps S201 and S202 may be the same as steps S101 and S102 in Fig. 4.

[0069] (Processing step S203) In processing step S203, the compatibility calculation unit 203 calculates the compatibility of the camerawork type prepared in advance from the importance distribution I of the panoramic image. The camerawork type in the second embodiment determines parameters (hereinafter, referred to as generation parameters) in the camerawork generation unit 205 that generates camerawork from the importance distribution.

[0070] For the same importance distribution, different cameraworks are generated depending on the generation parameters, which include the number of intermediate compositions, the path interpolation method, and the acceleration / deceleration rules.

[0071] 9 shows examples of camera work types in the second embodiment. The following three camera work types will be described as examples. (d) The suitability calculation unit 203 generates a path with slight variations for one intermediate composition, and moves the camera at a constant speed along the path. (e) The compatibility calculation unit 203 generates a movement path for the two intermediate compositions by linear interpolation, and moves the virtual camera along that path with acceleration and deceleration. (f) The compatibility calculation unit 203 generates a movement path for the three intermediate compositions by spline interpolation, and moves the virtual camera at a constant speed on that path.

[0072] [Table 1] Table 1 compares the characteristics of camerawork types (d) to (e).

[0073] The compatibility calculation unit 203 calculates the compatibility of these camera work types from the importance distribution. First, the compatibility calculation unit 203 models the importance distribution with a mixed von Mises-Fisher distribution, as in the processing step S103, and calculates the parameters {α k , μ k , κ k} K k=1 is obtained by maximum likelihood estimation.

[0074] Here, the number of element distributions K is set to 3, which is the maximum number of intermediate compositions for the assumed camera work type.

[0075] For camerawork type (d), the mixture ratio {α k} K k=1 The difference between the maximum value of and the average of the rest is regarded as the fitness. In other words, the more importance is concentrated in one element distribution, the higher the fitness.

[0076] For camerawork type (e), the mixture ratio {α k} K k=1 The difference between the average and the minimum value of is regarded as the fitness. In other words, the more importance is concentrated in the two element distributions, the higher the fitness.

[0077] For camerawork type (f), the mixture ratio {α k} K k=1The variance of σ 2 As exp(-σ 2 ) is the degree of conformance. In other words, the smaller the variation in the mixture ratio, the higher the degree of conformance.

[0078] The suitability of the m-th camera work for the n-th panoramic image obtained in the above manner is denoted as Am.

[0079] (Processing step S204) In processing step S204, the type assignment unit 204 calculates the camera work compatibility score {Dnm} N,M n=1,m=1 This process is similar to the process step S104 in the first embodiment.

[0080] (Processing step S205) In processing step S205, the camerawork generation unit 205 generates camerawork for each panoramic image based on the assigned camerawork type. Here, the parameters {α k , μ k , κ k} K k=1 An example in which the camerawork generation unit 205 generates camerawork using the above will be described.

[0081] That is, the camerawork generation unit 205 generates camerawork using the parameters related to the importance distribution of each region of the panoramic image and the panoramic image as inputs. The camerawork generation unit sets parameters according to the camerawork type assigned by the type assignment unit.

[0082] For the camera work type (d), the camera work generation unit 205 first calculates the mixture ratio {α k} K k=1 Select the element distribution k for which is maximized. The mean direction μ k is the optical axis direction of the virtual camera c, and the camerawork generation unit 205 calculates the concentration degree κk The angle of view γ of the virtual camera is determined from the concentration degree κ. For example, the following formula (8) can be used to calculate the angle of view γ of the virtual camera from the concentration degree κ.

[0083]

number

[0084] The camerawork generation unit 205 obtains (c1, γ1) and (c2, γ2) by applying small variations to the optical axis direction c and the angle of view γ. The camerawork generation unit 205 then linearly interpolates these two points to generate camerawork paths c(s), γ(s). In other words, the camerawork generation unit 205 generates camerawork in which the parameters of the virtual camera transition between different pairs of optical axis directions and angles of view, with the optical axis directions and angles of view being treated as pairs.

[0085]

number

[0086]

number

[0087] Intermediate composition 1 is (c1, γ1), and intermediate composition 2 is (c2, γ2). The camerawork generation unit 205 uses equation (9) to generate a path that transitions between the two intermediate compositions by linear interpolation. Next, the camerawork generation unit 205 determines s for time t. Here, a minimum jerk model is used as the acceleration / deceleration model, and it is set as in equation (11).

[0088]

number

[0089] Finally, for the camera work type (f), the camera work generation unit 205 first calculates the average direction of each element distribution {μ k} K k=1 The direction of the optical axis of the virtual camera is k} K k=1 Let the concentration {κ k} K k=1 From the virtual camera's angle of view {γ k} K k=1 Using these as intermediate compositions, the camerawork generation unit 205 generates a path of the camera parameters passing through the intermediate compositions by spline interpolation, and the virtual camera moves at a constant speed on the path.

[0090] (Processing step S206) In processing step S206, the video encoding unit 206 creates partial perspective projection videos from each panoramic image in accordance with the set camera work, and encodes and outputs the videos as one video.

[0091] <Processing variations> In the following, preferred variations of the process in the second embodiment will be described.

[0092] First, similarly to the first embodiment, it is possible to apply the variational Bayes method to parameter estimation of a mixed von Mises-Fisher distribution, directly calculate the goodness of fit from the importance distribution, and weight the evaluation function.

[0093] Furthermore, the fixed camerawork of the first embodiment may be mixed with the camerawork of the second embodiment. Various camerawork types can be assumed by combining the number of intermediate compositions, the number of intermediate compositions, the route interpolation method, and the acceleration / deceleration rules. Any camerawork type other than the above can be used (however, if there are a large number of camerawork types, inconveniences will arise, and it is preferable to apply the following third embodiment to solve these inconveniences).

[0094] According to this embodiment, it is possible to generate a variety of cameraworks for a plurality of panoramic images according to the contents of the images.

[0095] The camerawork generation unit generates camerawork using parameters related to the importance distribution of each region of the panoramic image and the panoramic image as input, thereby making it possible to generate camerawork that is adaptive to the content of the image while suppressing bias in camerawork types.

[0096] By regarding the importance distribution in a panoramic image as a mixed distribution (element distribution) of probability distributions with multiple distributions as elements, and generating camerawork according to the parameters of the element distribution, it is possible to generate camerawork that is robust and adaptive to the content of the image.

[0097] [Third embodiment] As a third embodiment, we will describe a video generation device that assigns a camera work type to each panoramic image, generates camera work, and outputs a video when the number of camera work types is very large compared to the number of input images, or in the extreme case when there are an infinite number of camera work types.

[0098] In such a case, there is a possibility that similar camera work types will be assigned to different panoramic images, resulting in a video with a series of similar camera work.

[0099] In the third embodiment, the moving image production device defines the distance between camera work types and incorporates the distance into a constraint condition or an evaluation function, thereby producing various types of camera work.

[0100] Fig. 10 is an example of a functional block diagram showing functions of the moving image generating device 100 in the third embodiment in the form of blocks. The moving image generating device 100 in this embodiment is similar to the moving image generating device 100 in Fig. 7. However, the contents of the processing of the compatibility calculation unit 303 and the type assignment unit 304 are different. These functions will be described with reference to Fig. 11.

[0101] Fig. 11 is an example of a flowchart showing the procedure for generating camerawork by the video production device 100. Fig. 11 is similar to Fig. 8, but the contents of steps S303 and S304 are different. The following mainly describes the differences from Fig. 8.

[0102] (Processing step S303) In processing step S303, the compatibility calculation unit 303 calculates the compatibility of the camerawork type from the panoramic image importance distribution I. Here, as an example, the camerawork type (f) in the second embodiment is used, and the camerawork type is divided using the video length T in equation (10) as a variable (for example, T=300 and T=500 are different camerawork types).

[0103] When classifying the camerawork types, it is considered to use the spread of the importance distribution. For example, similar to the processing step S103, the compatibility calculation unit 103 calculates the parameter {α k , μ k , κ k} K k=1 is obtained by maximum likelihood estimation. k} K k=1 The variance of these is the suitability of the camerawork type. In the following, the suitability of camerawork type m for the n-th panoramic image is denoted as Dnm (in the above example, m = T).

[0104] (Processing step S304) In processing step S304, the type assignment unit 304 assigns camerawork to each panoramic image using the calculated camerawork compatibility Dnm. Here, the type assignment unit 304 uses the distance d(m,m') between camerawork types m and m'. This distance is a constraint, and typically L2, L1, etc. can be used.

[0105] The camera type to be assigned to the nth panorama image is m. n Then, the optimization problem of Equation (12) is constructed.

[0106]

number

[0107] (Processing step S305) In processing step S305, the camerawork generating unit 305 generates camerawork for each panoramic image based on the assigned camerawork type. This processing is similar to processing step S205 in the second embodiment.

[0108] (Processing step S306) In process step S306, the video encoding unit 306 creates partial perspective projection videos from each panoramic image in accordance with the set camera work, and encodes and outputs the videos as one video.

[0109] <Processing variations> As a constraint in processing step S304, instead of using the distance between camera work types as a constraint, it is also possible to incorporate the distance into the evaluation function as shown in equation (13).

[0110]

number

[0111] In the third embodiment, the video length T is used as the camerawork type, but this is not limiting, and anything can be used as long as it can define the suitability of the camerawork type and the distance between the camerawork types. For example, the number of intermediate compositions, or the hyperparameter η that determines the angle of view conversion in equation (8), can be used as the camerawork type, or a combination of these can be used.

[0112] According to this embodiment, even if there are many camera work types, it is possible to generate diverse and high-quality camera work for multiple panoramic images. Note that high quality means high compatibility with each panoramic image and high quality in terms of correlation between the panoramic images.

[0113] The video generation device incorporates the distance between camera work types into a constraint condition or an evaluation function, and determines the allocation of camera work types to each panoramic image so as to maximize the sum of the suitability of the camera work types. This makes it possible to generate appropriate camera work for each panoramic image while suppressing bias in similar camera work types.

[0114] [Fourth embodiment] As a fourth embodiment, an example will be shown in which a Conditional Generative Adversary Network (hereinafter abbreviated as CGAN) is used to generate camerawork. In the CGAN framework, a panoramic image x and a latent variable z can be input to a generating function G to obtain camerawork y=G(x,z). Since the variation of camerawork can be obtained by the latent variable z, the latent variable z can be regarded as the camerawork type in the fourth embodiment.

[0115] In addition, in CGAN, a camerawork discriminant function D is acquired during the learning process, and the closer the generated camerawork y is to the camerawork included in the training data, the larger the value of D(x,y). Therefore, this D(x,y) is regarded as the compatibility in the fourth embodiment.

[0116] In addition, the distance between camerawork types is determined by the distance in the space of the latent variable z. In this way, even when using CGAN as a camerawork generation method, camerawork can be generated by taking into account the compatibility with each panoramic image and the interrelationships between panoramic images.

[0117] Fig. 12 is an example of a functional block diagram showing functions of the moving image generating device 100 in the fourth embodiment. The moving image generating device 100 of this embodiment is similar to the moving image generating device 100 in Fig. 10, but the arrangement of the compatibility calculation unit 403 is different. These functions will be described with reference to Fig. 13.

[0118] Fig. 13 is an example of a flowchart showing the procedure for generating camerawork by the video production device 100. Compared to Fig. 11, Fig. 13 does not have a step for calculating the degree of conformity. The following mainly describes the differences from Fig. 11.

[0119] (Processing step S403) In processing step S403, a type assignment unit 404 assigns camerawork types to each panoramic image. As in the third embodiment, the camerawork type assignment is performed by solving an optimization problem related to the compatibility and the distance between camerawork types. In the fourth embodiment, however, this optimization problem is formulated as in equation (14).

[0120]

number

[0121] The discriminant function D outputs a value of 1 if the input image and camera work pair is close to the training data, and a value close to 0 if it is far from the training data. This is regarded as the suitability of the camera work type.

[0122] Fig. 14 is a diagram showing a model of the camerawork type and the compatibility output from a panoramic image in CAGN. A panoramic image x (21) and a camerawork type z (22) are input to a generation function G (23), which outputs a camerawork y (24). In addition, a discrimination function D (25) receives the panoramic image x and the camerawork y, which outputs a compatibility score (26).

[0123] In addition, a distance d in the space of latent variables (camera work types) is set as a constraint. The type assignment unit 404 determines the assignment of camera work types to panoramic images that maximizes the sum of the goodness of fit, using the distance d as a constraint.

[0124] To solve this optimization problem, random search, hill climbing, or metaheuristics can be used. Since calculating the compatibility for a combination of a panoramic image and a camera work type in advance as in the first, second, and third embodiments is computationally expensive (if the cardinality of Z is infinite, it is impossible to calculate in the first place), the compatibility calculation unit 403 calculates D(x n ,G(x n ,z n ) is calculated.

[0125] (Processing step S404) In processing step S404, the camerawork generation unit 405 generates camerawork for each panoramic image based on the assigned camerawork type. n For G(x n ,z n ) for n = 1, 2, …, N.

[0126] (Processing step S405) In processing step S405, the video encoding unit 406 creates partial perspective projection videos from each panoramic image in accordance with the determined camera work, and encodes and outputs the videos as one video.

[0127] <Processing variations> Any configuration can be used for the generation function G and the discrimination function D. As an example of the configuration of the generation function G, it can be configured to combine the camera work type in the channel direction of the input image, process it with a convolutional neural network (CNN) to calculate features, and output the camera work, which is time-series data, using a recurrent neural network (RNN), a gated recurrent unit (GRU), a long short-term memory (LSTM), or a transformer.

[0128] The classification function D takes image data and time-series data as input. It is preferable to extract features from the image data using CNN, and from the time-series data using RNN, GRU, LSTM, or Transformer, and then combine the features before processing them using a fully connected layer.

[0129] In the above example, a Euclidean space is assumed for the set Z of camera parameter types, but this is not limited to the above. When the set Z is composed of a small number of countable elements, it is also possible to adopt a configuration in which the fitness is calculated in advance before the optimization calculation as in the first, second, and third embodiments.

[0130] According to the fourth embodiment, by using CGAN to generate videos from multiple panoramic images, CGAN can construct a camerawork generation function and a compatibility calculation function (discrimination function) in a data-driven manner, so that if a sufficient amount of training data exists, diverse and high-quality camerawork can be generated.

[0131] <Other application examples> The above describes the best mode for carrying out the present invention using an embodiment, but the present invention is not limited to such an embodiment, and various modifications and substitutions can be made within the scope that does not deviate from the gist of the present invention.

[0132] For example, this embodiment can be used for generating 3DCG videos, virtual reality systems, etc. The videos generated by this embodiment can also be applied to advertisements. In addition, by distributing videos on various EC sites, viewers can effectively view products and services.

[0133] In this embodiment, some of the processing is realized by machine learning. Machine learning is a technology for making a computer acquire a learning ability like a human being, and refers to a technology in which a computer autonomously generates an algorithm required for judgment such as data identification from learning data that is previously input, and applies this to new data to make predictions. The learning method for machine learning may be any of supervised learning, unsupervised learning, semi-supervised learning, reinforcement learning, and deep learning, or may be a combination of these learning methods. Any learning method for machine learning is acceptable.

[0134] Moreover, the configuration examples shown in Fig. 3 and the like in the above embodiment are divided according to main functions in order to facilitate understanding of the processing of the moving image generating device 100. However, the present invention is not limited by the manner in which each processing unit is divided or the names of each processing unit. The moving image generating device 100 can also be divided into more processing units according to the processing content. Also, it can be divided so that one processing unit includes more processes.

[0135] Each function of the above-described embodiments can be realized by one or more processing circuits. Here, the term "processing circuit" in this specification includes a processor programmed to execute each function by software, such as a processor implemented by an electronic circuit, and a device such as an ASIC (Application Specific Integrated Circuit), a DSP (digital signal processor), an FPGA (field programmable gate array), or a conventional circuit module designed to execute each function described above. [Explanation of symbols]

[0136] 100 Video Creation Device 101, 201, 301, 401 Image acquisition unit 102, 202, 302, 402 Importance calculation part 103, 203, 303, 403 Compatibility calculation section 104, 204, 304, 404 Type Allocation Section 105, 206, 306, 406 Video encoding unit 205, 305, 405 Camera work generation section [Prior art documents] [Patent documents]

[0137] [Patent Document 1] Patent No. 6432029 [Patent Document 2] JP 2004-241834 A [Patent Document 3] Patent No. 5861499 [Patent Document 4] JP 2018-151887 A

Claims

1. A moving image generating device that generates a moving image having a narrower angle of view than a plurality of panoramic images from the panoramic images, comprising: a suitability calculation unit that calculates a suitability of a camerawork type of a virtual camera for a plurality of panoramic images in accordance with a distribution of importance of each region of the panoramic images; a type assignment unit that determines an assignment of camera work types to the plurality of panoramic images based on the degree of suitability and the frequency of appearance of each camera work type; a video encoding unit that creates a video having a narrower angle of view than the panoramic images from the plurality of panoramic images using an assigned camerawork type; A moving image generating device comprising:

2. The video generation device described in claim 1, characterized in that the type assignment unit incorporates the frequency of occurrence of camera work types into a constraint condition or evaluation function, and determines the assignment of camera work types to the multiple panoramic images so as to maximize the sum of the suitability of the camera work types.

3. The type assignment unit is Calculating the distance between each camera work type in a space in which parameter values ​​of the camera work types are arranged on axes of the parameters constituting the camera work types; The video generation device described in claim 1, characterized in that the distance between the camera work types is incorporated into a constraint condition or evaluation function, and the assignment of camera work types to the multiple panoramic images is determined so as to maximize the sum of the suitability of the camera work types.

4. An importance calculation unit that calculates the importance of each region of the panoramic image, 4. The video generating device according to claim 1, wherein the compatibility calculation unit calculates the compatibility of a camera work type according to a distribution of importance in the panoramic image.

5. a camerawork generation unit that generates camerawork using parameters related to the importance distribution of each region of the panoramic image and the panoramic image as input; 4. The video generating device according to claim 1, wherein the camerawork generating section sets the parameters in accordance with the camerawork type assigned by the type assigning section.

6. An importance calculation unit that calculates the importance of each region of the panoramic image, The video generation device described in claim 5, characterized in that the camerawork generation unit regards the importance distribution in the panoramic image as a mixture distribution of probability distributions having multiple distributions as elements, and generates camerawork according to parameters of the mixture distribution.

7. the probability distribution is a von Mises-Fisher distribution, the camerawork generation unit determines an optical axis direction of the virtual camera along an average direction of the element distribution, and determines an angle of view of the virtual camera according to a concentration degree of the element distribution; The video generation device according to claim 6, wherein the device generates camerawork in which the parameters of the virtual camera transition between different pairs of optical axis directions and angles of view, with the optical axis directions and angles of view being paired.

8. The camerawork generation unit is realized by a conditional generative adversarial network, The latent variables in the conditional generative adversarial network are camerawork types, The video generation device according to claim 5 , wherein camerawork is generated by using the panoramic image and a camerawork type as inputs to the conditional generative adversarial network.

9. The type assignment unit incorporates the distance in the space of the latent variables into a constraint or an evaluation function; The video generation device according to claim 8, characterized in that the allocation of camera work types to the multiple panoramic images is determined so as to maximize the sum of the goodness of fit output by a discriminant function for the generated camera work.

10. A moving image generating method for generating a moving image having a narrower angle of view than a plurality of panoramic images from the panoramic images, comprising: a compatibility calculation unit calculating a compatibility of a camera work type of a virtual camera for a plurality of panoramic images according to an importance distribution of each region of the panoramic images; a type assignment unit determining an assignment of camera work types to the plurality of panoramic images based on the suitability and an appearance frequency of each camera work type; a video encoding unit generating a video having a narrower angle of view than the panoramic images from the plurality of panoramic images using an assigned camera work type; A moving image generating method comprising:

11. A moving image generating device that generates a moving image having a narrower angle of view than a plurality of panoramic images from the panoramic images, a suitability calculation unit that calculates a suitability of a camerawork type of a virtual camera for a plurality of panoramic images in accordance with a distribution of importance of each region of the panoramic images; a type assignment unit that determines an assignment of camera work types to the plurality of panoramic images based on the degree of suitability and the frequency of appearance of each camera work type; a video encoding unit that creates a video having a narrower angle of view than the panoramic images from the plurality of panoramic images using the assigned camera work type; A program to function as a

12. A computer-readable storage medium storing the program according to claim 11.

Images