Information processor, information processing method, and program

By generating a transformed imaging space with simplified structures, the system addresses the challenge of setting natural camera paths in complex virtual environments, improving user interaction and operational ease.

JP2024158729APending Publication Date: 2024-11-08CANON KK
View PDF 1 Cites 0 Cited by

Patent Information

Application Number
JP2023074205
Authority / Receiving Office
JP · JP
Patent Type
Applications
Current Assignee / Owner
Filing Date
2023-04-28
Publication Date
2024-11-08

AI Technical Summary

Technical Problem

Existing technologies face difficulties in setting a natural camera path for virtual cameras in virtual spaces with complex structures, requiring complex operations to avoid unnatural movements and maintain a consistent distance from changing surfaces.

Method used

The system generates a transformed imaging space with a simplified structure by altering the shape of complex structures, allowing users to operate virtual cameras more naturally by converting camera parameters using coordinate transformations.

Benefits of technology

This approach simplifies the operation of virtual cameras in complex environments by enabling natural camera paths with easier user interactions, reducing the complexity of controlling the camera's position and orientation.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure 2024158729000001_ABST
    Figure 2024158729000001_ABST
Patent Text Reader

Abstract

To solve a problem that it is difficult for a virtual camera user to acquire a desired camera path with a simple operation when a shape of a structure becomes complicated.SOLUTION: An information processor 103 receives a user operation of a virtual camera corresponding to a first virtual viewpoint image generated on the basis of first three-dimensional shape data corresponding to a structure while the first virtual viewpoint image is being displayed on a user display unit 402a, and a camera operation conversion unit 507 generates, on the basis of the user operation, a camera parameter indicating a position and an attitude of the virtual camera corresponding to a second virtual viewpoint image generated on the basis of second three-dimensional shape data indicating a shape of the structure, the second three-dimensional shape data indicating a shape different from the shape indicated by the first three-dimensional shape data.SELECTED DRAWING: Figure 5
Need to check novelty before this filing date? Find Prior Art

Description

[Technical field]

[0001] The present disclosure relates to techniques for generating virtual viewpoint images. [Background technology]

[0002] There is a technology that generates an image (virtual viewpoint image) that represents a view from a virtual viewpoint (virtual camera) specified by a user's operation by installing multiple imaging devices at different positions and capturing images synchronously, and using the multiple captured images obtained by the capturing. When generating a virtual viewpoint image, the user operates a joystick or the like to continuously set the position and attitude of a new virtual viewpoint in a time series while referring to a generated virtual viewpoint image corresponding to a virtual camera capturing a desired object (subject), for example. The movement path of the virtual camera set in this way is generally called a "camera path." The user can move the virtual camera around the entire three-dimensional space of the imaging target, but when, for example, a person as an object runs on a structure with a complex shape, it is necessary to take care not to move the virtual camera unnaturally in relation to the structure. Patent Document 1 discloses a technology that corrects the moving position of the virtual camera so that it is outside the obstacle when the moving position of the virtual camera overlaps with an obstacle (no-entry area) such as a wall of a building. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2012-063958 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the shape of a structure becomes complicated, it is difficult for a user to obtain a desired camera path with simple operations. For example, when capturing an image of a bicycle race and generating a virtual viewpoint image, the bicycle race track has a runway (course) with a unique shape called a bank, and the bicycle racers race along this bank. Therefore, in order for the user operating the virtual camera to capture an appropriate image of the racers while they are racing, it is necessary to perform operations that take into account the shape of the bank.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to improve the operability of a virtual camera in a virtual space that includes structures with complex shapes. [Means for solving the problem]

[0006] The information processing device according to the present disclosure includes a reception unit that receives a user operation of a virtual camera corresponding to a first virtual viewpoint image generated based on first three-dimensional shape data corresponding to a structure while the first virtual viewpoint image is being displayed; a generating means for generating, based on the user operation, camera parameters indicating a position and an attitude of a virtual camera corresponding to a second virtual viewpoint image generated based on second three-dimensional shape data indicating a shape of the structure, the second three-dimensional shape data indicating a shape different from the shape indicated by the first three-dimensional shape data; has. Effect of the Invention

[0007] According to the present disclosure, it is possible to improve the operability of a virtual camera in a virtual space including structures with complex shapes. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a configuration diagram of an image processing system 10. [Diagram 2] FIG. 2 is a diagram showing a world coordinate system and a camera coordinate system in a virtual space. [Diagram 3] FIG. 2 is a diagram illustrating a hardware configuration of an image processing device 102. [Figure 4] FIG. 1 is a diagram illustrating an input / output unit included in an information processing device 103. [Diagram 5] FIG. 2 is a diagram illustrating a functional configuration of an image processing device 102. [Figure 6] FIG. 13 is a diagram illustrating a modified background model. [Figure 7] FIG. 11 is a diagram illustrating background model information. [Figure 8] FIG. 11 is a diagram illustrating deformation information. [Figure 9] 1A and 1B are diagrams illustrating a background model and a modified background model. [Figure 10] 11A and 11B are diagrams illustrating transformation of the position and orientation of a foreground model. [Figure 11] 11 is a flowchart illustrating a process for generating a virtual viewpoint image using a transformed imaging space. [Figure 12] 11A and 11B are diagrams illustrating an example of the operation of a virtual camera in a transformed imaging space. [Figure 13] FIG. 13 is a diagram illustrating conversion of camera parameters of a virtual camera. [Figure 14] FIG. 13 is a diagram showing an example of a UI screen for setting a transformed imaging space. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the present embodiment will be described with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the configurations shown in the drawings. Also, not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention.

[0010] <Check the assignment> For example, consider the case where a virtual viewpoint image is generated for soccer played on a large, flat field. The user can set a camera path in the imaging space with a high degree of freedom, for example, starting to follow a player from a position where the player is captured from behind, and going around to a position where the player is captured from the front when the ball is kicked toward the goal. Here, the "imaging space" means a virtual three-dimensional space (virtual space) expressed by computer graphics or the like, which corresponds to a three-dimensional space in which multiple imaging devices actually capture images. In contrast, consider the case where a virtual viewpoint image is generated for a track bicycle race, for example. In track bicycle races, athletes run on a circuit course that is generally called a "bank," and is made up of a curved section with a maximum inclination of 30 degrees to 45 degrees and a flat straight section that are continuously connected. In this case, even with the technologies of Patent Documents 1 and 2 described above, it was difficult to set a natural camera path. For example, suppose that the technology of Patent Document 1 is applied to set a camera path that runs parallel to the athletes running around the bank. In this case, when the athlete enters the curved section, the inclination angle of the track surface gradually increases with the movement of the virtual viewpoint running alongside the athlete, and the position of the virtual viewpoint is corrected every time the virtual camera position overlaps with the no-entry area. The camera path obtained in this way causes the virtual viewpoint to draw a zigzag trajectory up and down as the athlete moves, and a virtual viewpoint image that is not tolerable to viewing is not obtained. Furthermore, when the athlete leaves the curved section, the inclination angle of the track surface gradually becomes gentler as the athlete moves, and the virtual camera position does not overlap with the no-entry area, so the position of the virtual viewpoint is not corrected at all. In the camera path obtained in this way, the distance between the virtual viewpoint and the track surface gradually deviates. To avoid this, delicate operations are required to maintain the virtual viewpoint at a constant height from the track surface, whose height gradually changes, which places a heavy burden on the user.

[0011] As described above, in the conventional technology, when an object moves on a structure having a complex three-dimensional shape such as a bank, for example, complex virtual viewpoint operations are required to set a natural camera path without creating an awkward feeling. The technology disclosed herein has been made in consideration of such points, and its purpose is to enable a natural camera path to be set with simpler operations.

[0012] <System configuration> FIG. 1 is a diagram showing the overall configuration of an image processing system.

[0013] The image processing system 10 includes an imaging system 101, an image processing device 102, and an information processing device 103. The image processing system 10 is capable of generating a virtual viewpoint image.

[0014] In the imaging system 101, multiple imaging devices are installed at different positions surrounding an imaging area, and capture images in a time-synchronized manner. Multiple images captured synchronously from multiple viewpoints are transmitted to an image processing device 102. The imaging area may be an imaging studio where imaging is performed to generate a virtual viewpoint image, a stadium where a sports competition is held, or a stage where a performance is performed. In this embodiment, multiple imaging devices are installed at different positions surrounding a velodrome, and capture images in a time-synchronized manner.

[0015] The image processing device 102 generates a virtual viewpoint image seen from a virtual camera (virtual viewpoint) based on multiple images captured synchronously from multiple viewpoints. The virtual camera is a virtual camera that can move freely within an imaging space. The virtual camera is expressed by camera parameters determined by the information processing device 103, which will be described later. Unless otherwise specified, the term "image" in the following description will be described as including the concepts of both moving images and still images. In other words, the image processing system 10 can process both still images and moving images.

[0016] The image processing device 102 extracts the subject as the foreground from the multiple captured images sent from the imaging system 101, and generates a foreground model from the extracted foreground image. One method of extracting the foreground is to use background difference information. Specifically, a state in which the foreground does not exist is captured in advance as a background image, and the difference between the image in which the foreground exists and the background image is calculated. If the difference value is greater than a threshold value, the pixel position is determined to be the foreground. There are various other methods for extracting the foreground, such as a method using image features related to the subject or machine learning, but in this embodiment, any method for extracting the foreground is used. The foreground model may be generated by a volume intersection method or may be generated using depth data obtained from stereo image processing. In this embodiment, there is no limitation on the method of generating the foreground model. An imaging space in which the foreground model and the background model are arranged is generated, and a virtual viewpoint image viewed from a virtual camera is generated. In this embodiment, the background model is data indicating structures such as a studio set and a field of a stadium that have been captured in advance. As a method for generating a virtual viewpoint image, for example, Model-Based Rendering (MBR) can be used. This process allows a virtual viewpoint image seen from the position and orientation of a virtual camera to be generated. However, the method for generating a virtual viewpoint image is not limited to this.

[0017] The information processing device 103 controls the virtual camera and determines camera parameters that represent the viewpoint of the virtual camera. The camera parameters of the virtual camera include parameters for specifying the position, attitude, focal length, zoom, and time. The position of the virtual camera specified by the camera parameters is indicated by three-dimensional coordinates with a predetermined position as the origin. The position specified by the camera parameters of the virtual camera is indicated by parameters (x, y, z) of three axes, i.e., the x-axis, the y-axis, and the z-axis. The attitude of the virtual camera specified by the camera parameters is composed of parameters (rx, ry, rz) of three axes, i.e., pan, tilt, and roll. The camera parameters may include parameters that define other elements, or may not include all of the above-mentioned parameters. In this embodiment, the operator of the virtual camera is referred to as a user, and the generated virtual viewpoint image is viewed by a viewer.

[0018] The information processing device 103 transmits the determined camera parameters of the virtual camera to the image processing device 102. Next, the image processing device 102 generates a virtual viewpoint image based on the received camera parameters, and transmits the generated virtual viewpoint image to the information processing device 103.

[0019] <Explanation of imaging space> In this embodiment, two imaging spaces are generated: a main imaging space (first virtual space) including a three-dimensional shape generated from multiple images acquired by multiple imaging devices capturing reality, and a transformed imaging space (second virtual space) for facilitating operation of a virtual camera in the main imaging space. By operating the virtual camera in the transformed imaging space having a structure generated by changing the shape of a structure with a complex three-dimensional shape such as a bank into a simpler shape, it becomes possible to set a natural camera path with simpler operation.

[0020] 2 is a diagram showing a world coordinate system and a camera coordinate system in the main imaging space. Note that the transformed imaging space also has a world coordinate system and a camera coordinate system similar to the world coordinate system and the camera coordinate system in the main imaging space described below.

[0021] FIG. 2(a) is a diagram showing a world coordinate system (x, y, z) in the main imaging space. The world coordinate system is used to represent the camera parameters of the virtual camera and the positions of the foreground model and background model. In this embodiment, a bicycle race is imaged, a three-dimensional shape of the subject shown in the image is generated, and a virtual viewpoint image is generated using the generated three-dimensional shape. The subject is a tangible object that exists in the imaging space, such as a field 201 of a stadium and players 202. In the world coordinate system, the center of the field 201 is set as the origin (0, 0, 0). The x-axis is set in the short side direction of the field 201, the y-axis is set in the long side direction of the field 201, and the z-axis is set in the vertical direction to the field 201. Note that the method of setting the world coordinate system is not limited to this.

[0022] 2(b) is a diagram showing the camera coordinate system (X, Y, Z) of the virtual camera. The camera coordinate system is expressed as a Cartesian coordinate system (X, Y, Z) of three axes, an X-axis 212, a Y-axis 213, and a Z-axis 214. The X-axis 212, the Y-axis 213, and the Z-axis 214 are axes in the left-right direction, the front-back direction, and the height direction of the virtual camera, respectively. In addition, tilt 215, roll 216, and pan 217, which represent the attitude of the virtual camera, are expressed as angles with the X-axis 212, the Y-axis 213, and the Z-axis 214 as rotation axes, respectively.

[0023] <Hardware configuration> 3 is a diagram for explaining the hardware configuration of the image processing device 102. Note that the hardware configuration of the information processing device 103 is also similar to the configuration of the image processing device 102 described below. The image processing device 102 has a CPU 301, a ROM 302, a RAM 303, an auxiliary storage device 304, a display unit 305, an operation unit 306, a communication I / F 307, and a bus 308.

[0024] 1 by controlling the entire image processing device 102 using computer programs and data stored in the ROM 302 and the RAM 303. The image processing device 102 may have one or more pieces of dedicated hardware different from the CPU 301, and at least a part of the processing by the CPU 301 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0025] The ROM 302 stores programs that do not require modification.

[0026] The RAM 303 temporarily stores programs and data supplied from the auxiliary storage device 304, and data supplied from the outside via the communication I / F 307, and the like.

[0027] The auxiliary storage device 304 is configured, for example, by a hard disk drive, and stores various data such as image data and audio data.

[0028] The display unit 305 is configured with, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) and the like for the user to operate the image processing device 102 .

[0029] The operation unit 306 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by a user to input various instructions to the CPU 301. The CPU 301 operates as a display control unit that controls the display unit 305, and an operation control unit that controls the operation unit 306.

[0030] The communication I / F 307 is used for communication with an external device of the image processing device 102. For example, when the image processing device 102 is connected to the external device by wire, a communication cable is connected to the communication I / F 307. When the image processing device 102 has a function of wirelessly communicating with the external device, the communication I / F 307 includes an antenna.

[0031] The bus 308 connects each part of the image processing device 102 and transmits information.

[0032] In this embodiment, the display unit 305 and the operation unit 306 are assumed to exist inside the image processing device 102, but at least one of the display unit 305 and the operation unit 306 may exist outside the image processing device 102 as a separate device.

[0033] FIG. 4 is a diagram illustrating an input / output unit included in the information processing device 103. As shown in FIG.

[0034] The input / output unit has an input unit 401 and a display unit 402 for controlling the virtual camera and for making settings related to the imaging space.

[0035] The input unit 401 has a joystick 411a, a joystick 411b, a seesaw switch 412, and a group of buttons 414. A user (an operator who operates the virtual camera) operates them to change the camera parameters of the virtual camera. The sticks 411a and 411b each have an operation axis with three degrees of freedom, and the position of the virtual camera is operated by the stick 411a, and the attitude of the virtual camera is operated by the stick 411b. In addition, the focal length and zoom value of the virtual camera are changed by tilting the seesaw switch 412 to the plus side or the minus side. In addition, the input unit 401 has an input unit (not shown) for setting the virtual space. This input unit is, for example, a keypad or a mouse, and performs settings for conversion of the virtual space and switching of modes by operations such as key input and icon clicking. The camera parameters of the virtual camera are calculated based on the amount of operation received by the input unit 401.

[0036] The display unit 402 is composed of multiple display units, and has a user display unit 402a where the user operates the virtual camera and a viewing display unit 402b where the viewer views a virtual viewpoint image. It displays the virtual viewpoint image generated by the image processing device 102, the state of the virtual camera, and an image for generating a transformed virtual space. For example, the user display unit 402a is a display, and the viewing display unit 402b is a smartphone or a tablet.

[0037] 4 is merely an example, and the present invention does not limit the configuration of the input / output unit. For example, a configuration in which a user uses both the user display unit 402a and the viewing display unit 402b may be used.

[0038] <Functional configuration> FIG. 5 is a diagram illustrating the functional configuration of the image processing device 102.

[0039] The image processing device 102 receives a plurality of captured images captured synchronously from the imaging system 101 and camera parameters from the information processing device 103, and generates a virtual viewpoint image. Each function installed in the image processing device 102 will be described in order.

[0040] 5(a) is a block diagram showing an example of the functional configuration of the image processing device 102. The image processing device 102 includes a background model information management unit 501, a coordinate conversion value calculation unit 502, a foreground model information acquisition unit 503, an imaging space conversion unit 504, a converted image generation unit 506, a camera operation conversion unit 507, and a virtual viewpoint image generation unit 508.

[0041] The background model information management unit 501 manages main background model information. The main background model information is a group of data on three-dimensional positions of the main background model, which defines the three-dimensional shape and line information in the world coordinate system (x, y, z). The line information is information representing the shape and position of lines drawn on the background model, such as the outer stripe lines of a bank, the lane lines of an athletics track, and the starting line. The acquired main background model information is transmitted to the coordinate transformation value calculation unit 502, the imaging space transformation unit 504, and the virtual viewpoint image generation unit 508. Note that both the main background model information of the main background model in the main imaging space and the transformed background model information of a transformed background model in a transformed imaging space described later may be managed.

[0042] The coordinate transformation value calculation unit 502 receives the main background model information from the background model information management unit 501, and acquires the transformation information held in the RAM or the auxiliary storage device. The transformation information is information for generating a transformed background model by transforming the main background model into a certain shape. The transformed background model is generated by transforming the main background model based on the transformation information. Furthermore, the coordinate transformation value calculation unit 502 calculates the correspondence between the three-dimensional positions in the imaging space where each is arranged by comparing the reference line of the main background model with the reference line of the transformed background model. More specifically, the three-dimensional position in the transformed imaging space of the transformed background model that corresponds to the three-dimensional position in the main imaging space where the main background model is arranged is determined, and the three-dimensional positions are recorded in correspondence with each other. In this embodiment, the correspondence between the three-dimensional positions in the transformed imaging space and all three-dimensional positions in the space surrounding the main background model by a bounding box is recorded. This correspondence is recorded as a coordinate transformation value. The transformed background model information and the coordinate transformation value indicating the three-dimensional shape data of the generated transformed background model are transmitted to the imaging space conversion unit 504.

[0043] The foreground model information acquisition unit 503 acquires foreground model information from a foreground model of a subject generated based on multiple images received from the imaging system 101 and parameters such as the position and orientation of each imaging device. The foreground model information includes shape information indicating the shape of the foreground model, position information and orientation information of the subject, etc. The acquired foreground model information is transmitted to the imaging space conversion unit 504 and the virtual viewpoint image generation unit 508.

[0044] The imaging space conversion unit 504 acquires background model information from the background model information management unit 501 and foreground model information from the foreground model information acquisition unit 503. Then, based on the transformed background model information and coordinate transformation values ​​acquired from the coordinate transformation value calculation unit 502, a transformed imaging space is generated in which the transformed background model and the foreground model are arranged. The generated transformed imaging space information is transmitted to a transformed image generation unit 506.

[0045] The virtual camera operation unit 505 acquires input information from the input unit 401 input by the user, and acquires camera parameters such as the position and posture of the virtual camera in each frame. The camera parameters are transmitted to the converted image generation unit 506 and the camera operation conversion unit 507. In this embodiment, it is assumed that while a virtual viewpoint image generated by the converted image generation unit 506 is displayed on the user display unit 402a, the user operates the virtual camera operation unit 505 with reference to the virtual viewpoint image. Note that the user can select whether to operate the virtual camera in the main imaging space or in the converted imaging space by an operation to be described later with reference to FIG. 14.

[0046] The converted image generating unit 506 acquires the camera parameters from the virtual camera operating unit 505, and generates a virtual viewpoint image seen from the position and orientation of the virtual camera in the converted imaging space generated by the imaging space converting unit 504. The generated virtual viewpoint image is transmitted to the information processing device 103 and displayed on the user display unit 402a.

[0047] The camera operation conversion unit 507 converts the camera parameters acquired from the virtual camera operation unit 505 with respect to the main imaging space, based on the coordinate conversion values ​​acquired from the coordinate conversion value calculation unit 502. The converted camera parameters are transmitted to a virtual viewpoint image generation unit 508.

[0048] The virtual viewpoint image generator 508 acquires the camera parameters converted from the converted imaging space to the main imaging space by the camera operation converter 507. Then, it generates a virtual viewpoint image seen from the position and attitude of the virtual camera in the main imaging space. The generated virtual viewpoint image is transmitted to the information processing device 103 and displayed on the viewing display unit 402b.

[0049] FIG. 5B is a block diagram showing an example of the functional configuration of the imaging space conversion unit 504.

[0050] The imaging space conversion unit 504 includes a foreground model coordinate conversion unit 509 and a conversion space generation unit 510 .

[0051] Foreground model coordinate transformation unit 509 acquires foreground model information from foreground model information acquisition unit 503 and coordinate transformation values ​​from coordinate transformation value calculation unit 502. Then, it transforms the position and orientation of the foreground model based on the coordinate transformation values ​​acquired from coordinate transformation value calculation unit 502. The transformed foreground model information is transmitted to transformation space generation unit 513.

[0052] The transformed space generation unit 509 acquires the transformed background model information and the transformed foreground model information from the coordinate transformation value calculation unit 502 and the foreground model coordinate transformation unit 512. Then, the transformed imaging space including the foreground model and the transformed background model is generated. The generated transformed imaging space information is transmitted to the transformed image generation unit 506.

[0053] FIG. 6 is a diagram for explaining the deformation of the main background model. An example of the deformation of the main background model is shown in FIG. 6, in which a bank 601 having a slope with a varying inclination, a curved portion, and a straight portion is deformed into a flat straight line. FIG. 6(a) and (b) are perspective views of the main background model and the deformed background model, respectively. FIG. 6(c) and (d) are cross-sectional views of the main background model and the deformed background model in a plane that passes through the start line 603 and is parallel to the zx plane. In the main background model, the heights (z-axis direction) of the outer belt line 602 and the start line 603 change depending on the position according to the inclination of the bank 601, but in the deformed background model, the heights of the outer belt line 602 and the start line 603 are constant (z=0) regardless of the position. In addition, the outer belt line 602 of the main background model does not have an end and is composed of a curved portion and a straight portion, but in the deformed background model, it is a straight line with the start line 603 as an end. When the user operates the virtual camera to capture a foreground model arranged in the main imaging space, it is necessary to change the position and attitude of the virtual camera according to the slope, curved part, and straight part of the main background model whose inclination changes. Specifically, it is necessary to adjust five axes of the virtual camera, namely, the X-axis, Y-axis, Z-axis, pan, and tilt, for each frame, which makes it difficult to operate the virtual camera. In contrast, when the virtual camera is operated to capture a foreground model arranged in a deformed background model, the foreground model moves on a straight line, so that the foreground model can be captured by inputting only the Y-axis. Therefore, the user can easily operate the virtual camera with respect to the generated deformed background model. Note that it is only necessary to make it easy for the user to operate the virtual camera, and the deformed background model is not limited to a straight line. When the control of the virtual camera is limited to a circular orbit, the deformed background model is deformed into a circle. In this way, the shape of the deformed background model may be determined by the operation content on the input unit 401 and the calculation method of the camera parameters associated with the operation content, that is, the control method of the virtual camera.

[0054] FIG. 7 is a diagram illustrating the main background model information.

[0055] The main background model information is a data group of three-dimensional positions in which three-dimensional shape data and line information of the main background model are defined in a world coordinate system (x, y, z). The three-dimensional shape data includes information on each surface constituting the main background model. The position of the three-dimensional shape is expressed in a world coordinate system (x, y, z) with the center of the field 201 as the origin. The surface constituting the background model is not limited to a flat surface, but may be a curved surface such as a curved portion of the bank 601. In addition, one surface may be divided into multiple surfaces, such as by dividing the surface into the inside and outside of the outer belt line 602. The main background model information includes information on lines (line information) that represent the characteristics of the competition and the main background model. The lines that represent the characteristics of the competition and the main background model are, for example, lines that are some kind of reference line for the competition, such as the outer belt line 602, the start line 603, and the 30m line 701 of a bicycle race. The line information is expressed in a world coordinate system (x, y, z) with the center of the field 201 as the origin. Note that the line information is not necessarily limited to real lines, and may include imaginary lines set by the user.

[0056] 8 shows an example of a table for managing transformation information. The transformation information is made up of a transformation background model ID, reference information, and transformed shape information.

[0057] A unique ID is set for each transformed background model that is generated through transformation.

[0058] The reference information stores information on lines and surfaces that are the reference for calculating coordinate transformation values ​​for transforming a background model into a predetermined shape, among the line information and three-dimensional shape data included in the main background model information. The reference information is composed of a reference line, a start line, and a reference surface. The reference line is a line that serves as a guide for transforming the main background model. The calculation of the coordinate transformation value is performed by first calculating the coordinate transformation value of the reference line, and then calculating the coordinate transformation value of each three-dimensional position that constitutes the main background model with reference to the reference line. The start line is a line for specifying the starting point of the transformation. For example, when transforming a background model that does not have an end such as a circle into a shape that has an end such as a straight line, the transformation is performed so that the line specified as the start line becomes the end. The reference surface is a surface that serves as the reference for the transformation, and specifies the amount of transformation in the height direction.

[0059] The deformed shape information specifies the shape into which the main background model is deformed. The deformed shape information is composed of shape information and line length ratio. The shape information is composed of reference line transformation and reference surface transformation, and specifies the shape of the deformed background model, such as a flat straight line. The line length ratio specifies the expansion or reduction of the length of the lead line. For example, when a bank with a circumference of 400m is transformed into a straight line shape, if the line length ratio is 1, the length of the straight line will be the same as before transformation, at 400m. If the line length ratio is 0.5, the length of the straight line will be reduced to 200m. The line length ratio can be specified separately for the tangential and normal directions.

[0060] The configuration of the transformed background model information management table is not limited to this. For example, the transformed background model information may be included. In this case, the speed of the foreground model included in the foreground model information may be acquired, and the foreground model may be converted so that the speed of the foreground model is constant in the transformed imaging space.

[0061] FIG. 9 is a diagram for explaining the main background model and the transformed background model.

[0062] 9(a) and (b) are diagrams showing the main background model and the transformed background model, respectively. In the example of FIG. 8, "Bank_A Velodrome" is registered in the transformed background model ID. As a result, the main background model serving as the reference for transformation is designated as the bank 601, and the background model information of the bank 601 is referred to in the calculation of the coordinate transformation value. In addition, "Outer belt line" is registered in the reference line of the reference information, "Start line" in the start line, and "Bank top surface" in the reference surface. The reference line "Outer belt line", the start line "Start line", and the reference surface "Bank top surface" respectively indicate the outer belt line 602, the start line 603, and the bank top surface 901. The bank top surface 901 is the shaded surface in FIG. 9. In addition, the main background model is transformed with the start line 603 as its end. Meanwhile, in the shape information of the transformed shape information, "Straight line" is registered in the reference line transformation, and "z'=0" is registered in the reference surface transformation. At this time, prime (') indicates coordinates in the transformed imaging space, and "z' = 0" indicates that the three-dimensional position is transformed (on the XY plane) so that the z' coordinate of the "bank top surface" set as the reference plane is always 0. "Straight line" indicates that the bank 601 is transformed into a straight line. In other words, the bank 601 is transformed into a flat straight line. At this time, in a competition such as a bicycle race in which the same place is circled, a transformed background model may be generated in which the courses for the circuit in the main background model are connected in a straight line. In this way, when the player 202 enters the next circuit, the player 202 moves seamlessly without transitioning to a distant position (from the end to the start of the transformed background model). Meanwhile, the line length ratios are set to "0.8" and "1.0" in the tangential direction and normal direction, respectively. Since the line length ratio in the tangential direction is 0.8, the line is transformed into a straight line whose length is 0.8 times the circumference of the outer band line 602 in FIG. 9(a). Since the line length ratio in the normal direction is 1.0, the width of the bank 601 is the same before and after the deformation.

[0063] FIG. 10 is a diagram for explaining the transformation of the position and orientation of the foreground model.

[0064] 10(a) and (b) respectively show the main imaging space and the transformed imaging space, and the foreground model in FIG. 10(a) is transformed in three-dimensional position and orientation to the foreground model in FIG. 10(b).

[0065] The position and orientation of the foreground model in the transformed imaging space are transformed using coordinate transformation values ​​at each three-dimensional position in the main imaging space. In the example of FIG. 10, the three-dimensional position (xi, yi, zi) on the bank 601 in the main imaging space is transformed to (x′i, y′i, z′i) in the transformed imaging space. At this time, the player 202 located at the three-dimensional position (xi, yi, zi) is placed (transformed) to a position (x′i, y′i, z′i) in the transformed imaging space. The player's orientation is transformed so that the angle α between the line 1002 connecting the three-dimensional position on the bank 601 where the player 202 is located and the center point 1001 of the player and the reference plane 1003 becomes equal in the main imaging space and the transformed imaging space. At this time, in the example of FIG. 10, the reference plane is the top surface of the bank. The three-dimensional position of the foreground model is the center position of the coordinates where the foreground model and the background model are in contact. The center of the foreground model is approximated by the center of a bounding box surrounding the foreground model. However, the approximation method of the three-dimensional position on the background model where the foreground model is located and the center of the foreground model is not limited to this. When the imaging space is transformed based on the transformation information shown in the example of FIG. 8, the bank 601 having a slope with a variable inclination and a curved portion is transformed into a flat straight line, and the foreground model such as the player 202 is placed based on the transformation. As a result, a transformed imaging space is generated in which the player runs straight on the straight bank. The user operates the position and attitude of the virtual camera in the transformed imaging space, and the result of the virtual camera operation is converted to the position and attitude of the virtual camera in the main imaging space by the camera operation conversion unit 507. The conversion of the virtual camera operation will be described later.

[0066] FIG. 11 is a flowchart showing the process of the image processing device 102.

[0067] In step S1101, the background model information management unit 501 acquires main background model information from the RAM or auxiliary storage device of the image processing apparatus 102.

[0068] In step S1102, the coordinate transformation value calculation unit 502 receives the main background model information from the background model information management unit 501, and generates a transformed background model based on the transformation information stored in the RAM or auxiliary storage device. Then, the coordinate transformation value is calculated from the generated transformed background model and the main background model.

[0069] In step S1103, the foreground model information acquisition unit 503 acquires foreground model information generated based on a plurality of images received from the imaging system 101 and parameters such as the positions and orientations of the imaging devices.

[0070] In step S1104, the user uses the input unit 401 to select whether or not to capture a virtual viewpoint image by operating the virtual camera in the converted imaging space. Hereinafter, the mode in which the virtual camera is operated in the converted imaging space to capture a virtual viewpoint image is referred to as a converted imaging space mode. If the converted imaging space mode is ON, the process proceeds to step S1105, and if it is OFF, the process proceeds to step S1111.

[0071] In step S1105, the deformed background model information is received from the coordinate transformation value calculation unit 502, and the foreground model information is received from the foreground model information acquisition unit 503. Then, a transformed imaging space in which the deformed background model and the foreground model are arranged is generated.

[0072] In step S1106, the virtual camera operation unit 505 acquires input information input to the input unit by the user received from the input unit 401, and acquires camera parameters indicating the position and orientation of the virtual camera in each frame.

[0073] In step S1107, the converted image generation unit 506 acquires the deformed background model information and the foreground model information from the imaging space conversion unit 504, and the camera parameters of the virtual camera from the virtual camera operation unit 505. Then, in the converted imaging space, a virtual viewpoint image is generated that captures the deformed background model and the foreground model as viewed from the position and orientation of the virtual camera.

[0074] In step S1108, the camera operation conversion unit 507 converts the camera parameters acquired from the virtual camera operation unit 505 into the coordinate system of the main imaging space based on the coordinate conversion values ​​acquired from the coordinate conversion value calculation unit 502.

[0075] In step S1109, the virtual viewpoint image generation unit 508 acquires the main background model information and foreground model information from the background model information management unit 501 and the foreground model information acquisition unit 503. Also, the virtual viewpoint image generation unit 508 acquires camera parameters converted for the main imaging space from the camera operation conversion unit 507. Then, in the main imaging space, a virtual viewpoint image is generated that captures the main background model and foreground model as viewed from the position and orientation of the virtual camera.

[0076] In step S1110, it is determined whether or not an instruction to end the work has been given for the information processing device 103. If an instruction to end the work has not been given, the process returns to step S1103 and the flow is repeated, and if an instruction to end the work has been given, the flow ends.

[0077] In step S1111, the virtual camera operation unit 505 acquires input information corresponding to the user's input received from the input unit 401, and acquires camera parameters indicating the position and orientation of the virtual camera in each frame. As a result, when the converted imaging space mode is OFF, the user operates the virtual camera in the main imaging space to capture a virtual viewpoint image.

[0078] FIG. 12 is a diagram illustrating an example of the operation of the virtual camera in the transformed imaging space.

[0079] Fig. 12(a) is a plan view of the transformed imaging space, and Fig. 12(b) is a plan view of the main imaging space. In the example of Fig. 12, a bank 601 having a slope with a varying inclination and a curved portion is transformed into a flat straight line, and a foreground model of a player 202 is placed on the transformed background model. When the transformed imaging space mode is ON, the user operates the virtual camera in the transformed imaging space, and the operation result of the virtual camera is reflected in the main imaging space. Based on the coordinate transformation value calculated by the coordinate transformation value calculation unit 502, the camera parameters in the transformed imaging space are transformed into camera parameters in the main imaging space, thereby transforming the virtual camera operation.

[0080] In Fig. 12(a) and (b), the position of the virtual camera 1201 corresponds to the trajectory 1202. That is, when the user operates the virtual camera in the transformed imaging space and moves it from the position 1201a to the position 1201b shown in Fig. 12(a), the virtual camera 1201 moves in the main imaging space from the position 1203a to the position 1203b shown in Fig. 12(b). For example, when the virtual camera is translated linearly along the linear bank 601 in the transformed imaging space as shown in Fig. 12(a), the virtual camera is converted to a movement along the curve of the bank 601 in the main imaging space as shown in Fig. 12(b). When the user performs an operation to move the virtual camera along the shape of the bank in the main imaging space, it is necessary to perform an operation to move the position of the virtual camera while manipulating the attitude of the virtual camera in accordance with the slope or curved portion of the bank whose inclination changes, which is cumbersome. On the other hand, by manipulating the position and orientation of the virtual camera in a transformed imaging space having a deformed background model in which the bank is transformed into a simple shape, and reflecting the results of the manipulation in the main imaging space, the user can achieve complex camera movements with simple operations.

[0081] FIG. 13 is a diagram for explaining conversion of camera parameters of a virtual camera.

[0082] 13(a) to 13(d) show a transformed background model obtained by transforming the main background model so that a bank 601 having a slope with a varying inclination angle and a curved portion becomes a flat straight line. The camera parameters of the virtual camera in the transformed imaging space are transformed into the camera parameters of the virtual camera in the main imaging space using the coordinate transformation values ​​at each three-dimensional position of the background model information.

[0083] FIG. 13(a) is a plan view of the transformed imaging space, and FIG. 13(b) is a plan view of the main imaging space. The transformation of the camera parameters of the virtual camera is specified using the normal of the reference line. The normal of the reference line is a line on the reference plane 1003 shown in FIG. 10 or on a plane parallel to the reference plane 1003. For example, when the virtual camera exists on the normal 1302 of the reference line 1301 shown in FIG. 13(a), it is transformed so that it exists on the normal 1302 of the reference line 1301 shown in FIG. 13(b) in the transformed imaging space. At this time, the normal 1302 shown in FIG. 13(b) is transformed into the normal 1302 shown in FIG. 13(a) by the transformation of the background model.

[0084] The camera parameters of the position of the virtual camera are converted based on a vector 1304 having an intersection 1303 of a reference line 1301 and a normal line 1302 as a start point and an end point at the position coordinates of the virtual camera 1201. In a three-dimensional orthogonal coordinate system (right-handed system) having the intersection 1303 as the origin and the tangent direction and normal direction of the reference line 1301 passing through the intersection 1303 as two axes, the camera parameters are converted so that the value of the vector 1304 becomes equal between the main imaging space and the transformed imaging space.

[0085] The camera parameters of the virtual camera are converted so that the angle between the normal 1302 and the optical axis of the virtual camera is equal between the main imaging space and the transformed imaging space. Figures 13(a) and 13(b) are diagrams for explaining the conversion of camera parameters in terms of the angle of the pan axis, and the angle θ of the pan axis between the normal 1302 and the optical axis of the virtual camera is equal between the main imaging space and the transformed imaging space.

[0086] FIG. 13(c) is a cross-sectional view of the converted imaging space, and FIG. 13(d) is a cross-sectional view of the main imaging space. In the main imaging space and the converted imaging space, the distance h between the virtual camera 1201 and the normal line 1302 of the reference line, and the angle of the tilt axis between the normal line 1302 and the optical axis of the virtual camera are equal. FIG. 13(c) and (d) are diagrams for explaining the conversion of camera parameters at the tilt axis angle, and the bank 601 with the inclination angle φ shown in FIG. 13(d) is transformed into a flat shape (inclination angle 0) shown in FIG. 13(d). The inclination angle φ is the angle between the normal line 1302 and the horizontal plane of the ground. When the camera parameters of the virtual camera 1201 in the converted imaging space shown in FIG. 13(c) are converted into the camera parameters of the virtual camera in the main imaging space shown in FIG. 13(d), the angle of the tilt axis of the virtual camera changes by φ (points downward).

[0087] FIG. 14 is a diagram showing an example of a UI screen for setting the converted imaging space.

[0088] The setting of the transformed imaging space is realized by the user operating the input unit 401 shown in Fig. 4. The user display unit 402a has a screen for performing the setting operation of the transformed imaging space shown in Fig. 14(a) and (b), and the user edits and registers the deformation information while operating the screen displayed on the user display unit 402a.

[0089] Fig. 14(a) is a screen for editing and registering the transformation information, and Fig. 14(b) is a screen for editing background model information to be registered in the transformation information. Fig. 14(b) displays a background model existing in the imaging space. The user inputs a transformed background model ID 1402, reference information 1403, and transformed shape information 1406, and sets the transformed imaging space mode 1401 to ON or OFF.

[0090] The transformed background model ID 1402 is input by selecting background model 1421 from the editing screen shown in Fig. 14(b) or by selecting from the pull-down menu shown in Fig. 14(a). When selecting from the editing screen shown in Fig. 14(b), the background model to be registered in the transformed background model ID is selected by clicking or the like.

[0091] The user sets a reference line 1301, a start line 1405, and a reference plane 1003 as reference information 1403. The reference line 1301 and the start line 1405 are input by selecting a line 1422 on the background model from the editing screen shown in Fig. 14(b) or by selecting from the pull-down menu shown in Fig. 14(a). The reference plane 1003 is input by selecting a surface 1423 on the background model from the editing screen shown in Fig. 14(b) or by selecting from the pull-down menu shown in Fig. 14(a). When selecting from the editing screen shown in Fig. 14(b), the line or surface to be registered in the transformed background model ID is selected by clicking or the like.

[0092] The deformed shape information 1406 is composed of shape information 1407 and a line length ratio 1410. The user sets reference line transformation 1408 and reference plane transformation 1409 as the shape information 1407, and a tangent direction 1411 and a normal direction 1412 as the line length ratio 1410. The input of the reference line transformation 1408 and the reference plane transformation 1409 is selected from the pull-down menu shown in Fig. 14(a). The input of the tangent direction 1411 and the normal direction 1412 is performed by inputting numerical values ​​by operating the keyboard or the like.

[0093] A transformed imaging space mode 1401 is set by selecting either ON or OFF. When it is ON, the user operates the virtual camera in the transformed imaging space. When it is OFF, the user operates the virtual camera in the main imaging space.

[0094] The set transformation information is stored in the RAM or auxiliary storage device when the user presses the registration button shown in FIG.

[0095] In this embodiment, a main background model, which is the field of a competition, is transformed into a transformed background model having a simple shape, and a transformed imaging space is generated in which the transformed background model and a foreground model are arranged. The user operates the position and attitude of the virtual camera in the transformed imaging space. The camera parameters of the virtual camera in the transformed imaging space are converted into the camera parameters of the virtual camera in the main imaging space based on coordinate transformation values ​​generated based on the main background model and the transformed background model. Therefore, the user only needs to operate the virtual camera in accordance with the movement of the background model and foreground model having a simple shape, and can easily operate the complicated and difficult camera movement.

[0096] In this embodiment, a main imaging space including a main background model generated from a plurality of captured images and a transformed imaging space for facilitating the operation of a virtual camera in the main imaging space are generated, but the present invention is not limited to this. For example, a main background model generated from a plurality of captured images and a transformed background model may be arranged side by side in the main imaging space. In this case, a coordinate transformation value is calculated based on the positional relationship between the main background model and the transformed background model, and another virtual camera is generated by changing the camera parameters of the virtual camera based on the accepted user operation based on the coordinate transformation value.

[0097] In this embodiment, the transformed imaging space is generated by the image processing device 102, but the present invention is not limited to this and may be generated by, for example, the information processing device 103. In this case, the coordinate transformation value calculation unit 502, the imaging space conversion unit 504, the transformed image generation unit 506, and the camera operation conversion unit 507 are included in the information processing device 103. Therefore, regardless of whether the transformed imaging space mode is ON or OFF, the camera parameters of the virtual camera in the main imaging space are transmitted from the information processing device 103 to the image processing device 102.

[0098] <Other embodiments> The present invention can also be realized by supplying a program for implementing the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0099] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0100] (Configuration 1) A reception means for receiving a user operation of a virtual camera corresponding to a first virtual viewpoint image generated based on first three-dimensional shape data corresponding to a structure while the first virtual viewpoint image is being displayed; a generating means for generating, based on the user operation, camera parameters indicating a position and an attitude of a virtual camera corresponding to a second virtual viewpoint image generated based on second three-dimensional shape data indicating a shape of the structure, the second three-dimensional shape data indicating a shape different from the shape indicated by the first three-dimensional shape data; 13. An information processing device comprising:

[0101] (Configuration 2) The information processing device according to configuration 1, wherein the second three-dimensional shape data is generated by transforming the first three-dimensional shape data.

[0102] (Configuration 3) Further, the device has an acquisition means for acquiring deformation information based on a user operation, 3. The information processing apparatus according to configuration 2, wherein the second three-dimensional shape data is generated by transforming the second three-dimensional shape data based on the transformation information.

[0103] (Configuration 4) The information processing apparatus according to configuration 3, wherein the transformation information includes information for designating a shape of the second three-dimensional shape data.

[0104] (Configuration 5) The first three-dimensional shape data has a course including straight lines and curved lines, 3. The information processing apparatus according to configuration 2, wherein the second three-dimensional shape data has a course obtained by transforming the course of the first three-dimensional shape data into a straight line or a circle.

[0105] (Configuration 6) The information processing device according to configuration 5, wherein the course of the second three-dimensional shape data is located on an XY plane.

[0106] (Configuration 7) The information processing device according to configuration 1, wherein the second virtual viewpoint image is output to a device different from that to which the first virtual viewpoint image is output.

[0107] (Configuration 8) The information processing device according to configuration 1, wherein the second virtual viewpoint image is generated based on a plurality of captured images acquired by capturing images using a plurality of imaging devices.

[0108] (Configuration 9) The information processing device according to configuration 1, further comprising output means for outputting the generated camera parameters to a device for generating the second virtual viewpoint image.

[0109] (Method) A receiving step of receiving a user operation of a virtual camera corresponding to a first virtual viewpoint image generated based on first three-dimensional shape data corresponding to a structure while the first virtual viewpoint image is being displayed; a generating step of generating, based on the user operation, camera parameters indicating a position and an attitude of a virtual camera corresponding to a second virtual viewpoint image generated based on second three-dimensional shape data indicating a shape of the structure, the second three-dimensional shape data indicating a shape different from the shape indicated by the first three-dimensional shape data; 13. An information processing method comprising:

[0110] (Program) A program for causing a computer to execute each of the means of the information processing device according to any one of configurations 1 to 9. [Explanation of symbols]

[0111] 402a User display unit 505 Virtual camera operation unit 507 Camera operation converter

Claims

1. a reception means for receiving a user operation of a virtual camera corresponding to a first virtual viewpoint image generated based on first three-dimensional shape data corresponding to a structure while the first virtual viewpoint image is being displayed; a generating means for generating, based on the user operation, camera parameters indicating a position and an attitude of a virtual camera corresponding to a second virtual viewpoint image generated based on second three-dimensional shape data indicating a shape of the structure, the second three-dimensional shape data indicating a shape different from the shape indicated by the first three-dimensional shape data; 13. An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the second three-dimensional shape data is generated by modifying the first three-dimensional shape data.

3. The image forming apparatus further includes an acquisition unit for acquiring deformation information based on a user operation, 3 . The information processing apparatus according to claim 2 , wherein the second three-dimensional shape data is generated by transforming the second three-dimensional shape data based on the transformation information.

4. 4. The information processing apparatus according to claim 3, wherein the transformation information includes information for designating a shape of the second three-dimensional shape data.

5. the first three-dimensional shape data has a course including straight portions and curved portions; 3. The information processing apparatus according to claim 2, wherein the second three-dimensional shape data has a course obtained by transforming the course of the first three-dimensional shape data into a straight line or a circle.

6. 6. The information processing apparatus according to claim 5, wherein the course of the second three-dimensional shape data is located on an XY plane.

7. The information processing apparatus according to claim 1 , wherein the second virtual viewpoint image is output to a device different from that to which the first virtual viewpoint image is output.

8. The information processing apparatus according to claim 1 , wherein the second virtual viewpoint image is generated based on a plurality of captured images acquired by capturing images using a plurality of imaging devices.

9. 2. The information processing apparatus according to claim 1, further comprising output means for outputting the generated camera parameters to a device for generating the second virtual viewpoint image.

10. a receiving step of receiving a user operation of a virtual camera corresponding to a first virtual viewpoint image generated based on first three-dimensional shape data corresponding to a structure while the first virtual viewpoint image is being displayed; a generating step of generating, based on the user operation, camera parameters indicating a position and an attitude of a virtual camera corresponding to a second virtual viewpoint image generated based on second three-dimensional shape data indicating a shape of the structure, the second three-dimensional shape data indicating a shape different from the shape indicated by the first three-dimensional shape data; 13. An information processing method comprising:

11. A program for causing a computer to execute each of the means of the information processing apparatus according to any one of claims 1 to 9.

Citation Information

Patent Citations

  • Game program and game device

    JP2012063958A