Image generating device, image generating method, and program

Abstract

To display a desired virtual viewpoint image according to characteristics of a display apparatus.SOLUTION: An image generation apparatus for generating a virtual viewpoint image controls a virtual camera position so that a position of a subject in a virtual space is included in an imaging range of display means with respect to a shooting range of the virtual camera located in the virtual space, and generates the virtual viewpoint image based on the controlled virtual camera position.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image generating device, an image generating method, and a program. [Background technology]

[0002] In the field of computer graphics, an image of an object placed in a three-dimensional virtual space is captured by a virtual camera (hereinafter, referred to as a virtual camera) placed in the virtual space, and the captured image is displayed on an arbitrary display device. In the following description, an image captured by the virtual camera is referred to as a virtual viewpoint image.

[0003] Patent Document 1 discloses a method of moving a virtual camera when a person placed in a virtual space moves and the moving person is to be kept captured in a virtual viewpoint image. Here, the method of moving the virtual camera is a method that calculates virtual camera coordinates where the subjects are included in the shooting range from the positional relationship of multiple subjects to be captured in the shooting range, and moves the virtual camera to the calculated virtual camera coordinates. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-545 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, depending on the characteristics of the display device that displays the virtual viewpoint image, the desired virtual viewpoint image may not be displayed. For example, a naked-eye stereoscopic display that displays a stereoscopic image has a characteristic of having a physical imaging range in which an object can be imaged, and the imaging range may be narrower than the shooting range of the virtual camera. Therefore, even if the virtual camera is moved so that the object is included in the shooting range of the virtual camera, as in the virtual camera moving method described in Patent Document 1, if the object moves beyond the imaging range, it may not be possible to image the object because it does not fall within the imaging range. As a result, the object reflected in the virtual viewpoint image may be displayed blurred, and the desired virtual viewpoint image may not be displayed.

[0006] The present invention has been made in consideration of the above-mentioned problems, and has an object to enable a desired virtual viewpoint image to be displayed in accordance with the characteristics of a display device. [Means for solving the problem]

[0007] In order to achieve the above object, an image generating device according to one aspect of the present invention includes: a control means for controlling a position of a virtual camera, which is a virtual camera located in a virtual space, so that an image forming range of a display means for a shooting range of the virtual camera includes a position of a subject in the virtual space; a generating means for generating a virtual viewpoint image based on the position of the virtual camera controlled by the control means; Equipped with 、 The control means a depth calculation means for calculating a depth of the subject in a direction of an optical axis of the virtual camera based on a position of the virtual camera and a position of the subject; a comparison means for comparing the imaging range with the depth of the subject, If the comparison by the comparison means indicates that the depth of the subject is not included in the imaging range, the position of the virtual camera is controlled so that the depth of the subject is included in the imaging range. It is characterized by the above. Effect of the Invention

[0008] According to the present invention, a desired virtual viewpoint image can be displayed in accordance with the characteristics of the display device. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram for explaining the functional configuration of a virtual visual point image generating system according to a first embodiment. [Diagram 2] FIG. 2 is a diagram showing an example of the arrangement of a plurality of image capturing devices according to the first and second embodiments. [Diagram 3] FIG. 1 is a block diagram for explaining a hardware configuration of a virtual viewpoint image generating system according to the first and second embodiments. [Figure 4] FIG. 2 is a diagram for explaining an imaging range of a display device. [Diagram 5] FIG. 2 is a diagram for explaining the functional configuration of an operable area control unit according to the first embodiment. [Figure 6] 3 is a flowchart illustrating a process flow according to the first embodiment. [Figure 7] FIG. 3 is an explanatory diagram of a control process according to the first embodiment. [Figure 8] FIG. 11 is a block diagram for explaining the functional configuration of a virtual visual point image generating system according to a second embodiment. [Figure 9] FIG. 11 is a diagram for explaining the functional configuration of an operation area control unit according to the second embodiment. [Figure 10] 10A to 10C are diagrams for explaining an example of calculation by a depth calculation unit according to the second embodiment. [Figure 11] 10 is a flowchart illustrating a process flow according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the attached drawings. Note that the following embodiments do not limit the invention according to the claims. Although the embodiments describe a number of features, not all of these features are essential to the invention, and the features may be combined in any manner. Furthermore, in the attached drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] [Embodiment 1] In this embodiment, a process of controlling the position of the virtual camera so that an arbitrary subject is included in the imaging range of the display device will be described.

[0012] In the following explanation, the entire virtual viewpoint image generation system of embodiment 1 will be described with reference to Figure 1, and an example of the arrangement of an imaging device when a subject in a virtual space is generated based on an image captured by an actual imaging device will be described with reference to Figure 2.

[0013] Then, the hardware configuration of the virtual viewpoint image generation system is described with reference to Fig. 3, and the imaging range of the display device is described with reference to Fig. 4. Next, the functional configuration related to the process of controlling the imaging range so that an arbitrary subject is included in the imaging range, which is a feature of the first embodiment of the present invention, is described with reference to Fig. 5, and the flow of the control process is described with reference to Fig. 6.

[0014] (Explanation of the virtual viewpoint image generation system 100) 1 is a block diagram for explaining the functional configuration of a virtual viewpoint image generation system 100 according to the first embodiment. The virtual viewpoint image generation system 100 is a system that images a subject in a virtual space as a virtual viewpoint image representing a view from a virtual viewpoint specified by a user. The virtual viewpoint image generation system 100 includes an operation information input unit 101 and a virtual viewpoint image generation device 102.

[0015] (Operation information input unit 101) An operation input of the virtual viewpoint is performed via an operation information input unit 101, and a virtual viewpoint image is generated by a virtual viewpoint image generating device 102. Note that the virtual viewpoint image in the first embodiment is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user, and for example, an image corresponding to a viewpoint selected by the user from a plurality of viewpoint candidates provided in advance is also included in the virtual viewpoint image. Also, although the first embodiment will be described mainly with respect to a case where the virtual viewpoint image is a moving image, the present invention is not limited to this example, and the virtual viewpoint image may be a still image.

[0016] The operation information input unit 101 has a function of receiving a user's instruction, processing the input information, and transmitting the processed information to the virtual viewpoint image generating device 102. The transmitted information includes operation instruction information for a virtual camera for operating the virtual viewpoint, physical information for identifying a physical imaging range of a display device (not shown), and subject information to be imaged. Here, the user's instruction is inputted, for example, by an external device such as a mouse or a keyboard connected to the operation information input unit 101.

[0017] The virtual camera operation instruction information is a parameter value that specifies the amount of movement of the virtual camera in the virtual space. The parameter value is, for example, a vector value that indicates the XYZ directions in a three-dimensional space. The parameter value is assumed to be specified by a user, and can be specified by, for example, a controller that operates three axes in the pan, tilt, and roll directions, such as a joystick connected to the operation information input unit 101. Note that, other than a joystick, a device that can be operated by a user, such as a mouse or a keyboard, may also be used.

[0018] In addition, the virtual camera operation instruction information may be parameter values ​​that can operate the angle of view, which is the size of the field of view of the virtual viewpoint, in addition to the three axes in the XYZ directions, and may be parameter values ​​that can operate multiple virtual cameras rather than a single virtual camera.

[0019] The physical information for identifying the physical imaging range of the display device is, for example, information for identifying an individual device, such as a model number of the display device (not shown).

[0020] As a method of acquiring the physical information, for example, the physical information can be acquired by manually selecting a corresponding model number from a list of model numbers held in advance by the operation information input unit 101, but in addition, a configuration in which the operation information input unit 101 automatically recognizes an unintended display device connected to the virtual viewpoint image generation system 100 and selects a corresponding model number from the list is also possible. Here, the display device includes, for example, a naked eye stereoscopic display.

[0021] The subject information to be imaged is, for example, identification information (ID) assigned to each subject in the virtual space. The operation information input unit 101 has a table in which subjects and identification information (ID) are previously associated with each other, and the identification information (ID) associated with an arbitrary subject is selected by clicking the arbitrary subject with a mouse on the monitor of a display device (not shown) capable of displaying a virtual viewpoint image. In addition to selecting an arbitrary subject on the monitor with a mouse, it is also possible to select a corresponding subject from an image by image processing based on an arbitrary threshold value. For example, a rugby player holding a ball may be extracted from an image as a subject and selected. Also, a subject that is not a subject to be imaged may be selected.

[0022] (Virtual viewpoint image generating device 102) The virtual viewpoint image generating device 102 determines the viewpoint of a virtual camera such that a subject in a virtual space is included in the imaging range of the display device, and generates a virtual viewpoint image, based on each piece of information transmitted from the operation information input unit 101. The information transmitted from the operation information input unit 101 is processed by the display characteristic identification unit 103, the virtual camera coordinate acquisition unit 104, and the subject coordinate acquisition unit 105, respectively.

[0023] The display characteristic identification unit 103 identifies the position and size of the imaging range relative to the shooting range of the virtual camera from the physical information identifying each device received from the operation information input unit 101. The position and size of the imaging range are associated with physical information indicating each device, such as a model number, and the virtual viewpoint image generation device 102 has a table (not shown) that associates the position and size of the imaging range relative to the shooting range with the physical information. The display characteristic identification unit 103 can identify the position and size of the imaging range associated with the physical information by referring to the table based on the physical information of the display device received from the operation information input unit 101. The display characteristic identification unit 103 inputs the position and size of the identified imaging range to the operable area control unit 106.

[0024] The virtual camera coordinate acquisition unit 104 acquires the position coordinates (position information) of the virtual camera in the virtual space from the operation instruction information received from the operation information input unit 101. Here, the operation instruction information is a parameter value, for example, a vector value, that specifies the amount of movement of the virtual camera in the virtual space, and acquires the coordinates obtained by moving the current virtual camera position by the vector value as the position coordinates (position information) of the next virtual camera that reflects the operation instruction. The virtual camera coordinate acquisition unit 104 inputs the acquired position coordinates to the operationable area control unit 106.

[0025] The subject coordinate acquisition unit 105 acquires the position coordinates of the subject in the virtual space from the subject information received from the operation information input unit 101. The virtual viewpoint image generation device 102 has a table (not shown) that associates the subject information with the position coordinates of the subject in the virtual space, and the subject coordinate acquisition unit 105 can acquire the position coordinates of the subject by referring to the table. The subject coordinate acquisition unit 105 inputs the acquired position coordinates to the operationable area control unit 106.

[0026] The operable area control unit 106 controls the viewpoint of the virtual camera (the position of the virtual camera) so that the subject in the virtual space is included within the imaging range of the display device, based on the information received from the display characteristic identification unit 103, the virtual camera coordinate acquisition unit 104, and the subject coordinate acquisition unit 105. The operable area control unit 106 transmits the controlled position of the virtual camera to the virtual camera position setting unit 107, which actually sets the position of the virtual camera based on the information of the position of the virtual camera controlled by the operable area control unit 106. Thereafter, the virtual viewpoint image generation unit 108 generates a virtual viewpoint image based on the viewpoint of the virtual camera.

[0027] Note that a plurality of operation information input units 101 may be connected to one virtual viewpoint image generating device 102, and a plurality of virtual viewpoint image generating devices 102 may be connected to one operation information input unit 101. When connecting the virtual viewpoint image generating devices 102 and the operation information input units 101 in a one-to-many or many-to-one manner, the connection method may be a wired communication cable, a wireless communication cable, or a configuration via a relay device such as a router or a network hub.

[0028] (Example of arrangement of multiple imaging devices) The subject in the virtual space may be a subject generated by computer graphics, or may be generated based on images actually captured by multiple imaging devices. Fig. 2 is a diagram showing an example of the arrangement of multiple imaging devices when capturing images using multiple imaging devices.

[0029] For example, a plurality of imaging devices 210a to 210n are set at different positions surrounding a stadium 211 where a sport such as soccer or rugby is played, and capture images in synchronization, thereby acquiring a plurality of captured images by the plurality of imaging devices 210a to 210n. Next, from the acquired plurality of captured images, a foreground image in which a foreground region corresponding to a predetermined object such as a person or a ball is extracted, and a background image in which a background region other than the foreground region is extracted are acquired.

[0030] Then, a foreground model representing the three-dimensional shape of a predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring a background model representing the three-dimensional shape of a background such as a stadium is generated based on the background image. Then, the texture data is mapped onto the foreground model and background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information, thereby generating a subject in a virtual space. However, the method of generating a subject is not limited to this, and various methods can be used, such as a method of generating a subject by projective transformation of a captured image without using a three-dimensional model.

[0031] The multiple imaging devices 210a-210n do not have to be installed all around the imaging area, and may be installed only in a part of the imaging area depending on the installation location and other factors. The number of imaging devices 210a-210n is not limited to the example shown in the figure, and for example, when the imaging area is a soccer stadium, about 30 imaging devices 210a-210n may be installed around the stadium 211. Imaging devices with different functions, such as a telephoto camera and a wide-angle camera, may also be installed.

[0032] (Hardware configuration of the virtual viewpoint image generation system 100) FIG. 3 is a block diagram for explaining the hardware configuration of the virtual viewpoint image generating system 100. As shown in FIG.

[0033] A central processing unit (CPU) 320 executes processing according to an embodiment of the present invention using programs and data stored in a random access memory (RAM) 321 and a read only memory (ROM) 322 .

[0034] The CPU 320 performs overall operation control of the virtual viewpoint image generation system 100. Note that the operation information input unit 101 or the virtual viewpoint image generation device 102 may have one or more dedicated hardware different from the CPU 320, and the dedicated hardware may execute at least a part of the processing by the CPU 320. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor), for example.

[0035] The ROM 321 holds programs and data. The RAM 322 has a work area for temporarily storing programs and data read from the ROM 321. The RAM 322 also provides a work area used by the CPU 320 when it executes each process.

[0036] The operation input unit 323 is, for example, a mouse, a keyboard, or a three-axis controller such as a joystick, and acquires information operated by a user. The acquired information is transmitted to the operation information input unit 101. Note that the operation input unit 323 is not limited to these.

[0037] The external interface 324 may transmit information to an external monitor via an image output port such as a High Definition Multimedia Interface (HDMI (registered trademark)) or an SDI (serial digital interface), etc. Also, image data may be transmitted via Ethernet (registered trademark) or the like.

[0038] (Imaging range of display device 403) Fig. 4 is a diagram for explaining the imaging range of a display device. Fig. 4(a) is a simulated diagram of a display of a display device 403 with a person as the subject. Subject 405 drawn with a dotted line indicates that the subject is displayed in a blurred state without being imaged, and subject 404 drawn with a solid line indicates that the subject is displayed in an imaged state. The shooting conditions at this time will be explained with reference to Figs. 4(b) and 4(c).

[0039] FIG. 4(b) is a schematic diagram of a scene in which the virtual camera 400 captures a virtual viewpoint image to be displayed on the display device 403, and FIG. 4(c) is a diagram of FIG. 4(b) seen from the side. An object 404 is located in front of the position of the virtual camera 400, and an object 405 is located behind the position of the virtual camera 400. A display range 401 indicates a shooting range that can be displayed on the display device 403 from the shooting range at the viewpoint of the virtual camera 400, and only the object within the frame is displayed on the display device 403. An imaging range 402 indicates a range in which the object is imaged from the display range 401, and only the object within the frame is imaged on the display device 403. Since the object 404 is located within the imaging range 402, an image is formed on the display device 403. On the other hand, since the object 405 is located outside the imaging range 402, an image is not formed on the display device 403, and is displayed, for example, in a blurred state.

[0040] The shooting range, display range 401, and imaging range 402 have the relationship shown in FIG. 4(b) due to the characteristics of the naked eye stereoscopic display assumed as the display device in this embodiment. The naked eye stereoscopic display is a display that allows three-dimensional (3D) images to be viewed with the naked eye without using devices such as three-dimensional (3D) glasses. For example, in the case of a naked eye stereoscopic display using a lens array, the optical characteristics of the lens array arranged on the surface of the display are used to provide an imaging surface of the lens array on a surface in the depth direction within the display where the subject is displayed. This is the imaging range 402. Due to this principle, it is difficult to make the entire depth direction of the display an imaging surface, and the imaging range 402 is narrower than the display range 401.

[0041] Furthermore, since the imaging surface is determined by a lens array physically provided on the display surface, the position and size of the imaging range 402 relative to the display range 401 are unique settings for each display (display device). Therefore, for example, it is not possible to enlarge only the imaging range 402 relative to the display range 401. To image a subject 405 located outside the imaging range 402, for example, it is necessary to move the position of the virtual camera 400 away from the subject and increase the relative size of the imaging range 402 relative to the subject.

[0042] Therefore, the operable area control unit 106 shown in FIG. 3 determines whether a specific subject is included within the imaging range 402, and if the subject is not included within the imaging range 402, controls the position of the virtual camera 400 so that the position of the subject in the virtual space is included within the imaging range 402.

[0043] (Configuration of the operable area control unit 106) 5 is a block diagram for explaining the functional configuration of the operation area control unit 106 according to the first embodiment of the present invention. The operation area control unit 106 has, as its functional configuration, an imaging range calculation unit 500, a depth calculation unit 501, a comparison unit 502, a coordinate calculation unit 503, and a coordinate transmission unit 504.

[0044] First, imaging range calculation unit 500 obtains the shooting range of virtual camera 400 based on the position coordinates of virtual camera 400 input from virtual camera coordinate acquisition unit 104. Next, imaging range calculation unit 500 calculates imaging range 402 at the position coordinates of virtual camera 400 from a table of the position and size of the imaging range relative to the shooting range of virtual camera 400 identified by display characteristic identification unit 103. That is, imaging range calculation unit 500 calculates the imaging range of the display device at the current viewpoint of virtual camera 400 (position of virtual camera 400) based on instruction information for operating the virtual camera input from virtual camera coordinate acquisition unit 104 and physical information for identifying the imaging range of the display device input from display characteristic identification unit 103.

[0045] Depth calculation unit 501 obtains position coordinates between a first position with respect to virtual camera 400 and a second position farther from virtual camera 400 than the first position among the intersections of optical axis 700 of virtual camera 400 and the subject, and calculates the distance based on these position coordinates as the depth of the subject. For example, based on the position information of virtual camera 400 and the coordinate position information of the subject, depth calculation unit 501 obtains position coordinates between a nearest part (first position) that is closest to virtual camera 400 and a farthest part (second position) that is farthest from virtual camera 400 among the intersections of the optical axis direction of virtual camera 400 and the subject, for the subject selected by subject coordinate acquisition unit 105, and calculates the distance between the two points as the depth of the subject.

[0046] A comparison unit 502 compares the imaging range 402 calculated by the imaging range calculation unit 500 with the subject depth calculated by the depth calculation unit 501 to determine whether the subject is included within the imaging range 402.

[0047] When the comparison result of the comparison unit 502 determines that the depth of the subject is not included within the imaging range 402 (depth of the subject>imaging range 402), the coordinate calculation unit 503 calculates the position of the virtual camera 400 by correcting the position coordinates of the virtual camera 400 so that the subject is included within the imaging range 402.

[0048] On the other hand, when the comparison result of the comparison unit 502 determines that the depth of the subject is included within the imaging range 402 (depth of the subject ≦ imaging range 402), the coordinate transmission unit 504 updates the position coordinates of the virtual camera 400 based on the acquired position coordinates of the virtual camera 400 without correcting the position coordinates of the virtual camera 400 by the coordinate calculation unit 503.

[0049] A coordinate transmitting unit 504 sets the position coordinates of the virtual camera 400 in the virtual camera position setting unit 107 based on the information from the comparing unit 502 or the position information of the virtual camera 400 corrected by the coordinate calculating unit 503 .

[0050] (Processing of embodiment 1) Fig. 6 is a flowchart for explaining the flow of processing according to the first embodiment, and processing for controlling the position of virtual camera 400 so that an arbitrary subject is included in the imaging range, which is a feature of the first embodiment of the present invention, will be explained using the flowchart in Fig. 6. This control processing is executed by CPU 320.

[0051] First, in step S 601 , physical information of the display device 403 connected to the operation information input unit 101 is referred to, and the operation information input unit 101 transmits the physical information of the display device 403 to the display characteristic identification unit 103 .

[0052] In the following step S602, the display characteristics identification unit 103 identifies the position and size of the imaging range relative to the shooting range of the virtual camera from the physical information indicating the characteristics of each display device (display device) received from the operation information input unit 101 by referring to a table stored therein.

[0053] In step S603, the user selects a subject to be imaged on a monitor (not shown), and the operation information input unit 101 acquires, for example, an ID of the selected subject by referring to a table that holds the ID, and transmits the acquired ID to the subject coordinate acquisition unit 105 as subject information of the subject to be imaged.

[0054] In step S604, the subject coordinate acquisition unit 105 acquires the position coordinates of the subject in the virtual space associated with the received subject information (ID) by referring to the table stored therein. Then, the subject coordinate acquisition unit 105 transmits the acquired position coordinates of the subject to the operation area control unit 106.

[0055] Note that the present invention is not limited to the case where a subject to be imaged is selected and the position of virtual camera 400 is controlled so that the subject is included in the imaging range, and it is also possible to select a subject that is not to be imaged. In this case, in the subsequent steps, control is performed to correct the position of virtual camera 400 so that the subject that is not to be imaged is not included in the imaging range.

[0056] In step S605, the user operates the position of virtual camera 400. Operation information input unit 101 transmits virtual camera operation instruction information based on the user's operation (for example, a movement vector specifying the amount of movement of virtual camera 400) to virtual camera coordinate acquisition unit 104.

[0057] In step S606, the virtual camera coordinate acquisition unit 104 acquires the position coordinates of the virtual camera 400 in the virtual space based on the operation instruction information (movement vector) received from the operation information input unit 101. Thereafter, the virtual camera coordinate acquisition unit 104 transmits the acquired position coordinates of the virtual camera 400 to the operationable area control unit 106.

[0058] In step S607, the imaging range calculation unit 500 calculates the display range 401 (imaging range) that can be displayed on the display device 403 based on the position coordinates of the virtual camera 400 received in step S606.

[0059] In step S608, imaging range calculation unit 500 calculates imaging range 402 at the current position coordinates of virtual camera 400 from a table of the position and size of the imaging range with respect to the shooting range of virtual camera 400. Then, imaging range calculation unit 500 transmits the calculated imaging range 402 to comparison unit 502.

[0060] In step S609, the depth calculation unit 501 calculates the optical axis 700 at the position coordinates of the virtual camera 400 received in step S606.

[0061] In step S610, depth calculation unit 501 calculates depth 701 of the subject located at the position coordinates calculated in step S604 in the direction of optical axis 700 of virtual camera 400. When there are multiple subjects to be imaged, for example, as shown in FIG. 7B, the distance between subject 404 closest to virtual camera 400 and subject 405 farthest from virtual camera 400 is found, and the distance between the two points is calculated as depth 701.

[0062] In step S611, the comparison unit 502 judges whether or not the depth 701 calculated by the depth calculation unit 501 in step S610 is included within the imaging range 402 calculated by the imaging range calculation unit 500 in step S608. If the result of the judgment in step S611 is that the depth 701 of the subject is included within the imaging range 402 (S611-Yes), the process proceeds to step S612. In step S612, the coordinate transmission unit 504 transmits the position coordinates of the virtual camera 400 acquired in step S606 to the virtual camera position setting unit 107. Here, the coordinate transmission unit 504 maintains the position of the viewpoint of the virtual camera 400 based on the position coordinates of the virtual camera 400 acquired in step S606 (updates the position without correction), without correcting the position coordinates of the virtual camera 400 by the coordinate calculation unit 503.

[0063] On the other hand, if the result of the determination in S611 is that the depth 701 of the subject is not included within the imaging range 402 (S611-No), the process proceeds to step S613. In step S613, the coordinate calculation unit 503 corrects the position coordinates of the virtual camera 400 so that the depth 701 calculated in step S610 is included within the imaging range 402, and updates the position of the virtual camera using the corrected position coordinates. For example, as shown in FIG. 7C, by correcting the position coordinates of the virtual camera 400 in a direction away from the subject along the optical axis 700 (the direction of the arrow 702), the relative size of the imaging range 402 with respect to the subject can be increased. This makes it possible to control the position of the virtual camera so that the depth 701 of the subject is included within the expanded imaging range 402. By such processing, it becomes possible to change from a state in which the image of subject 405, which is the object of imaging, cannot be formed (as shown by the dashed line) as in Fig. 7(a) before the position coordinates are corrected to a state in which the image can be formed (as shown by the solid line) as in Fig. 7(d) after the position coordinates are corrected, so that the subject can be included within imaging range 402. Note that the control for correcting the position coordinates of virtual camera 400 is assumed to be corrected along the direction of optical axis 700 so as not to shift the viewpoint, but the correction may be made in a direction deviating from optical axis 700.

[0064] As described above, by controlling the process of including an arbitrary subject in the imaging range of the display device, it is possible to generate a virtual viewpoint image without manually checking whether or not the subject is included in the imaging range. As a result, when displaying a virtual viewpoint image on a display device such as a naked eye stereoscopic display, it is possible to continue imaging even if the subject moves, and it is possible to continue imaging an arbitrary subject even in a scene in which the subject moves continuously, such as in dancing or sports.

[0065] [Embodiment 2] In this embodiment, a control process will be described in which the extent to which each subject is included in the imaging range of the display device is arbitrarily specified.

[0066] In the first embodiment, a method for controlling the position coordinates of the virtual camera 400 so that the subject to be imaged is completely included within the imaging range 402 has been described. However, a video production method that intentionally does not image the subject is also possible. For example, in video production such as dramas and movies, a representation method is sometimes used in which the depth of field of the camera is narrowed and other subjects are blurred to a certain extent in order to make a target subject stand out. In the second embodiment, it is possible to accommodate such a representation method by making it possible to set a threshold value for the degree to which an image is to be formed for each subject.

[0067] In the following explanation, the entire virtual viewpoint image generation system of the second embodiment will be described with reference to FIG. 8, the functional configuration related to the process of specifying the range in which the subject is included in the imaging range, which is a feature of the second embodiment of the present invention, will be described with reference to FIG. 9 and FIG. 10, and the flow of the process will be described with reference to FIG. 11.

[0068] The same components as those in the first embodiment are designated by the same numbers, and the description thereof will be omitted here.

[0069] (Explanation of the virtual viewpoint image generation system 800) 8 is a block diagram for explaining the functional configuration of a virtual visual point image generating system 800 according to embodiment 2. The virtual visual point image generating system 800 includes an operation information input unit 801 and a virtual visual point image generating device 802.

[0070] (Operation information input unit 801) The operation information input unit 801 transmits to the virtual viewpoint image generating device 802 operation instruction information for the virtual camera for operating the virtual viewpoint, physical information for identifying the physical imaging range of the display device, information on the subjects to be imaged, and threshold information for each subject.

[0071] The threshold information for each subject is a threshold that arbitrarily specifies the extent (proportion) of the subject's inclusion in the imaging range of the display device, and is additionally set when the user selects the subject as the imaging target. The threshold information is set as a percentage, for example, 50% or 100%, and the depth 701 is corrected based on the set threshold information. Note that the method of setting the threshold information may be a setting indicating a threshold level, such as "high" priority or "low" priority, other than a percentage.

[0072] The virtual viewpoint image generating device 802 generates a virtual viewpoint image by controlling the position of the virtual camera so that the subject in the virtual space is included in the imaging range of the display device based on each piece of information transmitted from the operation information input unit 801. The virtual viewpoint image generating device 802 differs from the virtual viewpoint image generating device 102 described in the first embodiment in that it has a threshold setting unit 803 and an operable area control unit 804 as functional components.

[0073] The threshold setting unit 803 identifies threshold information transmitted from the operation information input unit 801, which indicates the percentage of each subject to be included in the imaging range 402. For example, the threshold (percentage) when the entire subject is included in the imaging range 402 is set to 100%. When the threshold information transmitted from the operation information input unit 801 is transmitted with a priority of "high / low", the threshold setting unit 803 modifies the threshold to a threshold set as a high priority (e.g., 70%) or a threshold set as a low priority (e.g., 30%).

[0074] Operable area control unit 804 controls the position of the virtual camera so that the subject is included within the imaging range at a ratio equal to the specified threshold, based on the information received from display characteristic identification unit 103, virtual camera coordinate acquisition unit 104, subject coordinate acquisition unit 105, and threshold setting unit 803. Operable area control unit 804 transmits the controlled position of the virtual camera to virtual camera position setting unit 107, and virtual camera position setting unit 107 actually sets the position of the virtual camera based on the information on the position of the virtual camera controlled by operable area control unit 804.

[0075] (Configuration of the operable area control unit 804) 9 is a block diagram for explaining the functional configuration of an operable area control unit 804 according to the second embodiment of the present invention. The operable area control unit 804 has a functional configuration in which a depth calculation unit 900 is changed from the functional configuration of the operable area control unit 106 in the first embodiment.

[0076] The depth calculation unit 900 calculates the depth based on the threshold information set for each object acquired from the threshold setting unit 803, the position information of the virtual camera 400, and the coordinate position information of the object. The depth calculation unit 900 transmits the calculated depth to the comparison unit 502. The comparison unit 502 compares the imaging range 402 calculated by the imaging range calculation unit 500 with the depth of the object calculated by the depth calculation unit 900, and determines whether the imaging range 402 includes the depth of the object based on the threshold information. The result of the comparison by the comparison unit 502 is transmitted to the coordinate calculation unit 503 or the coordinate transmission unit 504, and thereafter, the position coordinates of the virtual camera 400 are transmitted to the virtual camera position setting unit 107 through the same process as in the first embodiment.

[0077] (Explanation of Depth Calculation Unit 900) 10 is a diagram for explaining an example of calculation of depth by depth calculation unit 900 according to embodiment 2. Subject 404 is located in front of the position of virtual camera 400, and subject 405 is located behind the position of virtual camera 400. In this example, the threshold set for subject 404 is 100%, and the threshold set for subject 405 is 50%. In this case, depth 1002 is calculated as the distance from the closest position of subject 404 to virtual camera 400 to a position that is half the depth 1001 of subject 405 itself (corresponding to the set threshold: 50%).

[0078] (Processing of embodiment 2) Fig. 11 is a flowchart for explaining the flow of processing according to the second embodiment, and the flowchart in Fig. 11 will be used to explain processing for controlling the position of virtual camera 400 based on a threshold value for arbitrarily specifying the extent to which each subject is to be included in the imaging range of the display device, which is a feature of the second embodiment of the present invention. This processing is executed by CPU 320. Processing similar to that in the processing flowchart (Fig. 6) explained in the first embodiment is given the same step numbers, and explanations will be omitted.

[0079] First, in step S1101, the user sets threshold information for each subject to be imaged selected in step S603. The set threshold information is transmitted from the operation information input unit 801 to the threshold setting unit 803.

[0080] In step S1102, the depth calculation unit 900 calculates the depth of the subject based on the coordinate position information of the subject (S604), the position information of the virtual camera 400 (S606), and the threshold information (S1101) set for each subject obtained from the threshold setting unit 803. The depth calculation unit 900 transmits the calculated depth 1102 to the comparison unit 502.

[0081] The subsequent control processing is the same as the control processing according to the first embodiment. If the result of the determination in step S611 is that the depth of the subject based on the threshold information set for each subject is included within the imaging range 402 (S611-Yes), the process proceeds to step S612. In step S612, the coordinate transmission unit 504 transmits the position coordinates of the virtual camera 400 acquired in step S606 to the virtual camera position setting unit 107. Here, the coordinate transmission unit 504 updates the position coordinates of the virtual camera 400 based on the position coordinates of the virtual camera 400 acquired in step S606, without correcting the position coordinates of the virtual camera 400 by the coordinate calculation unit 503.

[0082] On the other hand, if the result of the determination in S611 is that the depth of the subject based on the threshold information set for each subject is not included within imaging range 402 (S611-No), the process proceeds to step S613. In step S613, coordinate calculation unit 503 corrects the position coordinates of virtual camera 400 and controls the position of virtual camera 400 so that depth 701 calculated in step S610 is included within imaging range 402.

[0083] In the embodiment 1 described above, an example was described in which, in order to include the depth 701 of the subject in the imaging range 402, the position coordinates of the virtual camera 400 are corrected in a direction away from the subject along the optical axis 700 (in the direction of the arrow 702), as shown in Figure 7(c), thereby increasing the relative size of the imaging range 402 with respect to the subject.

[0084] In the second embodiment, in addition to the processing example described in FIG. 7(c), for example, when it is necessary to reduce the relative size of the imaging range for the subject in order to include the depth of the subject for which a threshold is set in the imaging range, it is also possible to modify the position coordinates of virtual camera 400 in a direction that brings it closer to the subject along optical axis 700 (the opposite direction of arrow 702).

[0085] As described above, when displaying a virtual viewpoint image on a display device such as a naked eye stereoscopic display, it is possible to continue to image the subject even if the subject moves, and further, by processing to arbitrarily specify the range to be included in the imaging range of each subject based on threshold information set for each subject for the imaging range of the display device, it becomes possible to generate a virtual viewpoint image in which only a specific subject is imaged and other subjects are blurred to a certain extent.

[0086] (Other embodiments) The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions.

[0087] The invention is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0088] 100: Virtual viewpoint image generation system; 101: operation information input unit, 102: virtual viewpoint image generating device, 103: display characteristic identification unit, 104: virtual camera coordinate acquisition unit, 105: Subject coordinate acquisition unit, 106: Operable area control unit, 107: Virtual camera position setting unit, 400: Virtual camera, 402: Imaging range, 403: Display device

Claims

1. a control means for controlling a position of the virtual camera so that an image forming range of the display means for a shooting range of the virtual camera, which is a virtual camera located in a virtual space, includes a position of a subject in the virtual space; a generating means for generating a virtual viewpoint image based on the position of the virtual camera controlled by the control means; Equipped with The control means a depth calculation means for calculating a depth of the subject in a direction of an optical axis of the virtual camera based on a position of the virtual camera and a position of the subject; a comparison means for comparing the imaging range with the depth of the subject, An image generating device characterized in that, when a comparison by said comparison means shows that the depth of the subject is not included in the imaging range, the position of the virtual camera is controlled so that the depth of the subject is included in the imaging range.

2. an input means for inputting operation instruction information for operating the virtual camera, physical information for identifying an imaging range of the display means, and subject information for identifying a selected subject; a camera coordinate acquisition means for acquiring a position of the virtual camera in the virtual space based on the operation instruction information; an identification means for identifying a position and a size of the imaging range relative to a shooting range of the virtual camera based on the physical information; a subject coordinate acquisition means for acquiring a position of the subject in the virtual space based on the subject information; The image generating device of claim 1 further comprising:

3. The control means 3. The image generating device according to claim 1, further comprising an imaging range calculation means for calculating a shooting range based on a position of the virtual camera, and calculating the imaging range at the position of the virtual camera from a table that holds the position and size of the imaging range for the shooting range.

4. 4. The image generating device according to claim 1, wherein the control means controls the position of the virtual camera in a direction away from the subject when the depth of the subject is not included in the imaging range.

5. 5. The image generating apparatus according to claim 4, wherein the control means controls the position of the virtual camera along an optical axis of the virtual camera in a direction away from the subject.

6. 4. The image generating device according to claim 1, wherein the control means maintains the position of the virtual camera when the comparison by the comparison means indicates that the depth of the subject is included in the imaging range.

7. The image generating device according to any one of claims 1 to 6, characterized in that the depth calculation means determines position coordinates between a first position with respect to the virtual camera and a second position, which is farther from the virtual camera than the first position, among the intersections of the optical axis of the virtual camera and the subject, and calculates a distance based on the position coordinates as the depth of the subject.

8. a threshold setting unit that sets a ratio of the depth of the subject to the imaging range, the depth calculation means calculates a depth of the subject based on a position of the subject, a position of the virtual camera, and the ratio; 8. The image generating device according to claim 1, wherein the control means controls a position of the virtual camera so that the depth of the subject for which the ratio is set is included in the imaging range.

9. 9. The image generating apparatus according to claim 8, wherein, when there are a plurality of subjects, the threshold setting means sets the ratio for each of the plurality of subjects.

10. 10. The image generating apparatus according to claim 8, wherein the depth calculation means corrects the depth of the subject based on the ratio set by the threshold setting means.

11. 11. The image generating device according to claim 1, wherein the display means includes a naked eye stereoscopic display.

12. 1. An image generation method for an image generation device that generates a virtual viewpoint image, comprising: controlling a position of the virtual camera such that an image-forming range of a display means for a shooting range of the virtual camera, which is a virtual camera located in a virtual space, includes a position of a subject in the virtual space; generating the virtual viewpoint image based on the controlled position of the virtual camera; Controlling the position of the virtual camera includes: calculating a depth of the subject in a direction of an optical axis of the virtual camera based on a position of the virtual camera and a position of the subject; Comparing the imaging range with the depth of the subject; An image generating method characterized by including, when the comparison shows that the depth of the subject is not included in the imaging range, controlling the position of the virtual camera so that the depth of the subject is included in the imaging range.

13. A program for causing a computer to function as each of the means of the image generating apparatus according to any one of claims 1 to 11.

Images