IMAGE PROCESSING APPARATUS, IMAGE PROCESSING SYSTEM, IMAGE PROCESSING METHOD, AND PROGRAM
The image processing system synthesizes objects from multiple perspectives to create a composite virtual viewpoint image, addressing the challenge of positioning objects arbitrarily within captured images, thereby enhancing the realism and immersion in events.
Patent Information
- Application Number
- JP2020168045
- Authority / Receiving Office
- JP · JP
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2020-10-02
- Publication Date
- 2024-11-12
- Estimated Expiration
- 2040-10-02
AI Technical Summary
Existing methods do not enable the synthesis of an object in another imaging space at an arbitrary position within the imaging space of a captured image.
An image processing system that includes multiple imaging devices and control units to generate a virtual viewpoint image by aligning and synthesizing images from different perspectives, allowing an object to be positioned arbitrarily within the captured image space.
Enables the generation of a composite virtual viewpoint image that allows users to view objects from various angles, enhancing the sense of reality and immersion in events like sports games.
Smart Images

Figure 0007581002000016 
Figure 0007581002000017 
Figure 0007581002000018
Abstract
Description
[Technical field]
[0001] The technology disclosed herein relates to generation of data based on captured images. [Background technology]
[0002] There is a method of synthesizing an image of an object that is not included in an actually captured image by using computer graphics technology.
[0003] Patent Document 1 describes a technique for synthesizing graphics of other players from archived data onto footage of an actual match. [Prior art documents] [Patent documents]
[0004] [Patent Document 1] JP 2002-101400 A Summary of the Invention [Problem to be solved by the invention]
[0005] However, Patent Document 1 does not disclose a method for generating an image in which an object in another imaging space is synthesized at an arbitrary position in an imaging space of a captured image.
[0006] The technology disclosed herein aims to generate an image in which an object in an arbitrary position in an imaging space is synthesized with an object in another imaging space. [Means for solving the problem]
[0007] The image processing device of the present disclosure includes a first acquisition means for acquiring line of sight information of a plurality of imaging devices included in a second imaging device group different from the first imaging device group, the line of sight information being determined based on information for identifying a virtual viewpoint corresponding to a virtual viewpoint image generated based on a first group of images obtained by imaging from a plurality of directions by a first imaging device group, a second acquisition means for acquiring a second group of images based on an object being imaged from a plurality of directions by the second imaging device group, and a generation means for generating a virtual viewpoint image in which the object is present at a predetermined position in an imaging space imaged by the first imaging device group, based on the line of sight information acquired by the first acquisition means and the second group of images acquired by the second acquisition means; and the line of sight information acquired by the first acquisition means is line of sight information of a plurality of imaging devices included in the second imaging device group, which is determined based on a reference point in an imaging space captured by the second imaging device group, a position of a display unit that displays a virtual viewpoint image generated based on the first image group, information for identifying a virtual viewpoint in the virtual viewpoint image displayed on the display unit, and the predetermined position. It is characterized by: Effect of the Invention
[0008] According to the technology of the present disclosure, it is possible to generate an image in which an object from another imaging space is synthesized at an arbitrary position in an imaging space. [Brief description of the drawings]
[0009] [Figure 1] FIG. 1 is a diagram illustrating the configuration of an image processing system. [Diagram 2] FIG. 2 is a block diagram illustrating the functional configuration of the image processing system. [Diagram 3] FIG. 2 is a diagram for explaining the hardware configuration of an image processing apparatus. [Figure 4] 6 is a flowchart for explaining an imaging process by the first imaging device. [Diagram 5] 10 is a flowchart illustrating an imaging process by a second imaging device. [Figure 6] 11 is a flowchart illustrating a position alignment process. [Figure 7] FIG. 4 is a diagram for explaining a translation vector. [Figure 8] FIG. 4 is a diagram for explaining the angle of a rotation matrix. [Figure 9] FIG. 11 is a diagram for explaining a position alignment process. [Figure 10]11 is a flowchart illustrating a synthetic virtual viewpoint image generation process. [Figure 11] 5 is a flowchart illustrating a control process of the ball launching device. [Figure 12] A diagram explaining the direction of the ball. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS
[0010] Hereinafter, the technology of the present disclosure will be described in detail based on the embodiments with reference to the accompanying drawings. The following embodiments are merely examples of concrete implementations of the technology of the present disclosure, and should not be interpreted as limiting the technical scope of the present disclosure. The technology of the present disclosure can be implemented in various forms without departing from its technical idea or main features.
[0011] <Embodiment 1> [System Configuration] FIG. 1 is a diagram showing a configuration of an image processing system 1 of the present embodiment for generating a virtual viewpoint image by synthesizing three-dimensional models of objects in different imaging spaces. A virtual viewpoint image is an image seen from a virtual viewpoint, which is a viewpoint of a non-existent camera different from the installation position and imaging direction of an actually installed camera, and is also called a free viewpoint image or an arbitrary viewpoint image. According to the technology for generating a virtual viewpoint image, for example, a highlight scene of a soccer or basketball game can be viewed from various angles, so that a user can be given a high sense of realism compared to a normal image. The virtual viewpoint image may be a moving image or a still image. In the present embodiment, the virtual viewpoint image will be described as a moving image.
[0012] For convenience of explanation, in this embodiment, the virtual viewpoint may be replaced with a virtual camera (virtual camera) as a concept representing the virtual viewpoint. In this case, the position of the virtual camera corresponds to the position of the virtual viewpoint, and the line of sight direction from the virtual viewpoint corresponds to the posture of the virtual camera. Also, the virtual viewpoint image corresponds to a captured image obtained by virtually capturing an image by the virtual camera.
[0013] The image processing system 1 of this embodiment includes a first imaging device 110, a first control device 100, a second imaging device 210, a second control device 200, a virtual viewpoint image display device 230, an image processing device 300, a composite image display device 410, and an operation device 420. The devices are connected to each other via, for example, a network 231, and various data are transmitted and received via IP communication or the like.
[0014] The first imaging device 110 is a plurality of imaging devices (a group of imaging devices) that capture the first imaging space 120, and is, for example, a plurality of digital cameras that can capture captured images such as still images and moving images. The first imaging space 120 is an imaging space that includes a first object. For example, the first imaging space 120 is a field of a stadium where a soccer match is played, and the first object is a soccer player 122. When the first imaging space 120 is a stadium field as shown in FIG. 1, the first imaging device 110 is disposed around the stadium field and captures images of the field from various angles in a time-synchronized manner. As a result, captured images from multiple viewpoints are obtained. In this specification, the first object is described as being a soccer player.
[0015] The first control device 100 is a control device that controls the first imaging device 110 to cause the first imaging device 110 to capture an image of the first imaging space 120 in time synchronization.
[0016] The second imaging device 210 is a plurality of imaging devices that capture the second imaging space 220, and is, for example, a plurality of digital cameras that can capture images such as still images and moving images. The second imaging space 220 is, for example, a studio installed in a stadium. As shown in FIG. 1, a second object exists in the second imaging space 220, and the second imaging device 210 is arranged around the studio and captures the second object from various angles in a time-synchronized manner. As a result, captured images from multiple viewpoints are obtained. In this specification, the user 222 is described as the second object. Note that the second imaging space 220 may be a studio installed in a different location from the stadium in which the first imaging device 110 is installed. Also, the second imaging space 220 does not have to be a studio, and may be, for example, a field of a stadium other than the stadium in which the first imaging device 110 is installed.
[0017] The second control device 200 is a control device that controls the second imaging device 210 to time-synchronize the second imaging device 210 with the second imaging space 220. The function of the second control device 200 will be described in detail later.
[0018] The virtual viewpoint image display device 230 is a display device for displaying a virtual viewpoint image generated by the image processing device 300 based on the captured image of the first imaging space 120. A display unit for displaying a screen in the virtual viewpoint image display device 230 is installed in a position near the second imaging space 220 and at a position where the user 222 can view the screen. The vicinity of the second imaging space 220 may include the second imaging space 220. In other words, the virtual viewpoint image display device 230 may be installed in the second imaging space 220 as long as the user 222 can view the screen. The virtual viewpoint image display device 230 shown in the figures of this specification refers to the display unit of the virtual viewpoint image display device 230.
[0019] The image processing device 300 is a device that is composed of, for example, a plurality of servers and a database. The image processing device 300 acquires a plurality of captured images obtained by the first imaging device 110 capturing images in a time-synchronized manner, and generates a virtual viewpoint image in which the first imaging space 120 is viewed from a virtual viewpoint. The image processing device 300 is a device that generates a virtual viewpoint image by synthesizing a three-dimensional model of the user 222 generated based on an image captured by the second control device 200 with a three-dimensional model generated based on an image captured by the first imaging device 110. The virtual viewpoint image obtained as a result of this synthesis is referred to as a synthetic virtual viewpoint image. The image processing device 300 in FIG. 1 is an example that is installed in a relay van 310 parked near a stadium. Details of the functions of the image processing device 300 will be described later. The image processing device 300 may not be installed in the relay van 310, but may be installed in another location.
[0020] The operation device 420 is an operation device for instructing the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint when the image processing device 300 generates a synthetic virtual viewpoint image. The operator 244 who operates the operation device 420 may be the manager of the image processing system 1 or a spectator 243 visiting the stadium.
[0021] The synthetic image display device 410 is a display device for displaying the synthetic virtual viewpoint image generated by the image processing device 300.
[0022] In this way, the image processing system 1 is a system for providing an attraction that generates and displays a composite virtual viewpoint image by synthesizing an image of a game being played in a stadium viewed from a virtual viewpoint with an image of a user 222 who is a participant of the attraction. The user 222 is, for example, a spectator who visits the stadium. This attraction allows the user 222 to relive the play of the players 122 on the field.
[0023] For this reason, Fig. 1 shows an example in which the second imaging device 210, the second control device 200, and the virtual viewpoint image display device 230 are arranged in a studio in a booth installed in a stadium to provide an attraction. Similarly, Fig. 1 shows an example in which the composite image display device 410 is installed in the stadium. In this way, according to the image processing system 1, spectators 243 who visit the stadium can enjoy the composite virtual viewpoint image in addition to the actual game held in the stadium.
[0024] [Function configuration] Fig. 2 is a diagram showing the functional configuration of the image processing system 1. The functions of each device constituting the image processing system 1 will be described with reference to Fig. 2.
[0025] The first control device 100 has a first imaging control unit 101. The first imaging control unit 101 controls the first imaging device 110 to cause the first imaging device 110 to capture an image of the first imaging space 120 in time synchronization.
[0026] The second control device 200 includes a second imaging control unit 201 , an imaging space matching unit 202 , a display control unit 205 , and a reception unit 206 .
[0027] The second imaging control unit 201 performs control to cause the second imaging device 210 to time-synchronize and capture an image of the second imaging space 220. Then, the second imaging control unit 201 acquires a plurality of captured images (a group of captured images) obtained as a result.
[0028] The imaging space alignment unit 202 includes a time alignment unit 203 and a position alignment unit 204. The time alignment unit 203 performs a process of adjusting the time associated with the captured image obtained by imaging by the second imaging device 210 to the time of the virtual viewpoint image generated based on the captured image of the first imaging device 110 for each frame. The position alignment unit 204 performs a process of changing the camera parameters that define the position and orientation of the second imaging device 210 for each frame based on information on the virtual viewpoint of the virtual viewpoint image displayed on the virtual viewpoint image display device 230, etc. The details of the process of the imaging space alignment unit 202 will be described later.
[0029] The display control unit 205 acquires a virtual viewpoint image generated by the image processing device 300, and performs control to display the acquired virtual viewpoint image on the virtual viewpoint image display device 230. The display control unit 205 may be included in the functions of the image processing device 300, and may perform control to display the virtual viewpoint image generated by the display control unit of the image processing device 300 on the virtual viewpoint image display device 230. The reception unit 206 receives an operation of an operation unit (not shown) of the second control device 200 by a user of a second object, etc.
[0030] The image processing device 300 has an object detection unit 301, a first captured image management unit 302, a virtual viewpoint image generation unit 303, a second captured image management unit 304, a composite virtual viewpoint image generation unit 305, a display control unit 306, and a virtual viewpoint control unit 307. Furthermore, the image processing device 300 has a ball launch control unit 308.
[0031] The object detection section 301 detects objects such as players and the ball in the first imaging space 120 from the captured image obtained by the first imaging device 110 capturing images. In this embodiment, an image pyramid is created from each received image, and an object is detected using SVM or the like, but the method of detecting an object is not limited to this method.
[0032] The first captured image management unit 302 stores and manages a plurality of captured images (a group of captured images) obtained by the first imaging device 110 capturing images and the results of object detection by the object detection unit 301.
[0033] The virtual viewpoint image generating unit 303 generates a virtual viewpoint image in which the first imaging space 120 is viewed from a virtual viewpoint, based on a captured image obtained by imaging by the first imaging device 110. A method for generating the virtual viewpoint image will be described later.
[0034] The second captured image management section 304 acquires captured images obtained by imaging using the second imaging device 210 and information relating to the captured images from the second control device 200, and stores and manages the acquired data.
[0035] The synthetic virtual viewpoint image generating unit 305 synthesizes a three-dimensional model of the user 222 at an arbitrary position in the first imaging space, and generates a synthetic virtual viewpoint image which is an image of the synthesized first imaging space viewed from a virtual viewpoint of the user 222. The details of the processing of the synthetic virtual viewpoint image generating unit 305 will be described later.
[0036] The display control unit 306 controls the display of the synthetic virtual viewpoint image generated by the synthetic virtual viewpoint image generation unit 305 on the synthetic image display device 410. The virtual viewpoint control unit 307 accepts an operation of the operation device 420 by an operator to specify a virtual viewpoint when generating the synthetic virtual viewpoint image.
[0037] The ball launch control unit 308 performs control such as calculating the time when the ball launching device 1206 (see FIG. 12) connected to the image processing system 1 launches the ball and instructing the ball launching device 1206 to launch the ball. The ball launching device 1206 and the ball launch control unit 308 will be described in embodiment 2. In this embodiment, the ball launch control unit 308 may be omitted.
[0038] [Hardware configuration] The hardware configuration of the image processing device 300 will be described with reference to Fig. 3. The hardware configurations of the first control device 100 and the second control device 200 are also similar to the configuration of the image processing device 300 described below. The image processing device 300 has a CPU 311, a ROM 312, a RAM 313, an auxiliary storage device 314, a display unit 315, an operation unit 316, a communication I / F 317, and a bus 318.
[0039] The CPU 311 controls the entire image processing device 300 using computer programs and data stored in the ROM 312 and the RAM 313. The image processing device 300 may have one or more dedicated hardware pieces different from the CPU 311, and at least a part of the processing by the CPU 311 may be executed by the dedicated hardware pieces. Examples of the dedicated hardware pieces include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The ROM 312 stores programs that do not require modification. The RAM 313 temporarily stores programs and data supplied from the auxiliary storage device 314, and data supplied from the outside via the communication I / F 317. The auxiliary storage device 314 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data.
[0040] The display unit 315 is composed of, for example, a liquid crystal display, an LED, etc., and displays a GUI (Graphical User Interface) etc. for the user to operate the image processing device 300. The operation unit 316 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations by the user and inputs various instructions to the CPU 311. The CPU 311 operates as a display control unit that controls the display unit 315 and an operation control unit that controls the operation unit 316.
[0041] The communication I / F 317 is used for communication with an external device of the image processing device 300. For example, when the image processing device 300 is connected to an external device by wire, a communication cable is connected to the communication I / F 317. When the image processing device 300 has a function of wirelessly communicating with an external device, the communication I / F 317 includes an antenna. The bus 318 connects each unit of the image processing device 300 to transmit information.
[0042] In this embodiment, the display unit 315 and the operation unit 316 are assumed to exist inside the image processing device 300, but at least one of the display unit 315 and the operation unit 316 may exist outside the image processing device 300 as a separate device.
[0043] The functions of the devices constituting the image processing system 1 of FIG. 1, which have been described using FIG. 2, are realized by the CPU of each device executing a predetermined program, but are not limited to this. Other hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) for accelerating calculations may also be used. That is, the functional units of the devices constituting the image processing system 1 of this embodiment in FIG. 2 may be realized by cooperation between software and hardware such as a dedicated IC, or some or all of the functions may be realized only by hardware.
[0044] [Regarding imaging processing by the first imaging device] Fig. 4 is a sequence diagram for explaining a process for retaining an object included in a frame of a moving image captured by the first imaging device 110 and a time code of the frame. The process in each device shown in the sequence diagram of Fig. 3 is performed by the CPU of each device expanding program code stored in ROM into RAM and executing it. In addition, some or all of the functions of the steps in Fig. 4 may be realized by hardware such as ASIC or electronic circuitry of each device. Note that the symbol "S" in the explanation of each process indicates a step in the flow, and the same applies to the following sequence diagrams and flowcharts.
[0045] In S401, the first imaging control unit 101 executes calibration to estimate actual camera parameters indicating the actual positions and orientations of the respective imaging devices constituting the first imaging device 110. For example, the calibration is executed as follows.
[0046] A large number of markers on which calibration patterns are printed are placed in the first imaging space 120. The first imaging device 110 captures these markers, and the positions of the calibration patterns included in the captured image obtained as a result of capturing the markers are associated with each other. The first imaging control unit 101 calculates real camera parameters indicating the actual positions and orientations of the imaging devices constituting the first imaging device 110 using a perspective projection model based on the associated positions. The position t of the imaging device is expressed by a vector shown in Equation 1, for example.
[0047]
number
[0048] The orientation R of the imaging device is expressed, for example, by a rotation matrix shown in Equation 2.
[0049]
number
[0050] In S402, the first imaging control unit 101 transmits to the image processing device 300 the actual camera parameters of the first imaging device 110 estimated in S401.
[0051] In S403, the first captured image management unit 302 of the image processing device 300 stores the actual camera parameters of the first image capturing device 110 in a storage unit such as the ROM 312 or the auxiliary storage device 314.
[0052] The next steps S404 to S408 are processes in which the first imaging device 110 captures one frame, and an object detected from the resulting frame and the time code of the frame are stored.
[0053] In S404, the first imaging control unit 101 instructs the first imaging device 110 to capture one frame in synchronization with the time. The first imaging device 110 captures, for example, a soccer match being held in a stadium.
[0054] In S405, the first imaging control unit 101 acquires image data of frames from multiple viewpoints by acquiring image data for one frame from each of the multiple imaging devices constituting the first imaging device 110 obtained as a result of the process of S404. Then, the first imaging control unit 101 transmits the image data of the frames from multiple viewpoints and the time codes of the frames to the image processing device 300.
[0055] In S406, the image processing device 300 receives the image data of one frame from multiple viewpoints and the time code transmitted in S405.
[0056] In S407, the object detection unit 301 detects an object included in the image represented by the image data received in S406. For example, the object detection unit 301 creates an image pyramid from each received image and detects an object using a method such as SVM.
[0057] In S408, the first captured image management unit 302 stores the received image data of one frame from multiple viewpoints, the time code of the frame, and the positions of each object detected from the frame in association with each other.
[0058] In S409, the first imaging control unit 101 determines whether to cause the first imaging device 110 to continue imaging. If imaging is to be ended, the process of this flowchart ends. If imaging is to be continued, the process returns to S404, and the first imaging device 110 is instructed to capture one frame again. The method of determination in this step is, for example, to determine that imaging is to be ended when an instruction to end imaging is received from the administrator of the image processing system 1 or the like.
[0059] [Regarding imaging processing by the second imaging device] FIG. 5 is a flowchart illustrating a process of causing the second imaging device to capture an image of the second imaging space in accordance with a virtual viewpoint image generated by the image processing device 300.
[0060] In S501, the second imaging control unit 201 executes calibration to estimate real camera parameters indicating the actual position, orientation, etc. of each imaging device constituting the second imaging device 210. Then, the second imaging control unit 201 acquires the real camera parameters of each imaging device constituting the second imaging device 210. The calibration method may be the same as the method of S401.
[0061] In S502, the second imaging control unit 201 stores the actual camera parameters of the second imaging device 210.
[0062] Next, a scene to be re-experienced by the second object is specified via an operation unit (not shown) of the second control device 200. In S503, the reception unit 206 receives an operation of the operation unit (not shown) of the second control device 200 and acquires information on the scene to be re-experienced. Specifically, time codes of the start time and end time of the scene to be re-experienced are specified. In addition, an object to be re-experienced is specified from among the objects detected by the object detection unit 301. For example, when a shooting scene in a soccer game is to be re-experienced, a time code of 10 seconds before the shooting time is specified as the start time, and a time code of 10 seconds after the goal is scored is specified as the end time. 10 seconds is an example, and any time may be set. In addition, if the player 122 is the player who shoots in that scene, the player 122 is specified as the object to be re-experienced. In this embodiment, it is assumed that the player 122 is specified as the player to be re-experienced. Note that the administrator may store several scenes in association with times, and a desired scene may be specified from among the stored scenes.
[0063] Next, the playback speed of the scene to be re-experienced by the second object is specified via an operation unit (not shown) of the second control device 200. In S504, the reception unit 206 receives an operation of the operation unit (not shown) of the second control device 200 and acquires the playback speed of the scene to be re-experienced. When the playback speed is specified as a constant speed, the user 222 can re-experience the specified scene with the movements of the player 122 at the actual speed. On the other hand, when the playback speed is specified as a slow speed, the user 222 can more easily reproduce the play of the player 122. The scene and the playback speed are specified, for example, by the user 222 operating the operation unit (not shown) of the second control device 200.
[0064] Then, the reception unit 206 transmits the specified scene, object, and playback speed to the image processing device 300.
[0065] In S505, the virtual viewpoint image generating unit 303 receives and holds the specified scene, object, and playback speed.
[0066] In the next steps S506 to S509, a process of generating one frame of a virtual viewpoint image from frames of a moving image based on imaging by the first imaging device 110 is performed.
[0067] In S506, the virtual viewpoint image generating unit 303 determines the time to be associated with one frame of the virtual viewpoint image to be generated in the subsequent step S509. In this embodiment, the time is described as a time code. The time code is time information expressed in the format of [hour:minute:second.frame]. In the first processing of this step in the loop processing, the time code of the start time of the specified scene is determined as the time of the virtual viewpoint image. In the second and subsequent processing of this step, the time is determined according to the set playback speed of the scene.
[0068] For example, a case where a virtual viewpoint image with a frame rate of 60 fps (frames per second) is generated will be described. In this case, when the playback speed of the scene is specified as constant speed, the time code is advanced by one frame every 1 / 60 seconds, and the time code is determined as the time of the virtual viewpoint image. When the playback speed of the scene is specified as double slow playback, the time code is advanced by one frame every 1 / 30 seconds.
[0069] In S507, the virtual viewpoint image generating unit 303 sets a virtual viewpoint for generating a virtual viewpoint image, which is an image of the first imaging space 120 viewed from a virtual viewpoint. Specifically, virtual viewpoint parameters representing the position of the virtual viewpoint, an attitude indicating the line of sight from the virtual viewpoint, and the like are set. The virtual viewpoint parameters are parameters similar to the real camera parameters. However, unlike the real camera parameters, the virtual camera parameters allow the viewpoint to be set at a position where the first imaging device 110 does not actually exist.
[0070] The virtual viewpoint image generated based on the virtual viewpoint determined in this step is an image for the user 222 to view, and the user 222 replays the play of the player 122 while viewing the virtual viewpoint image. For this reason, the virtual viewpoint set in this step is set to, for example, a viewpoint in which the designated player 122 is viewed from a third person's viewpoint. In this case, if the designated scene is a goal scene, the line of sight from the virtual viewpoint is set to the direction of the goal, which makes it easier for the user to replay the play. Alternatively, the line of sight from the virtual viewpoint may be set to the same line of sight as that of the designated player 122. In this case, the position of the virtual viewpoint may be set to the position of the face of the designated player 122. When the virtual viewpoint is set in this way, the degree to which the user 222 relives the experience can be increased.
[0071] The setting of the virtual viewpoint in this step may be determined according to a specified scene or a specified object by the virtual viewpoint image generating unit 303. Alternatively, the operator of the image processing system 1 may specify a virtual viewpoint, and the virtual viewpoint image generating unit 303 may set the specified virtual viewpoint.
[0072] In S508, the virtual viewpoint image generating unit 303 generates one frame of a three-dimensional model (also referred to as three-dimensional shape data) indicating the three-dimensional shape of an object in the first imaging space 120. The three-dimensional model is generated, for example, as follows.
[0073] First, from the frames of the moving image obtained by capturing the first captured space 120 managed by the first captured image management unit 302, the frames corresponding to the time determined in S506 are obtained for each of the frames captured by the respective capturing devices constituting the first imaging device 110.
[0074] The area of an object is detected from one frame of the multiple viewpoints acquired. The object detection process may use a method similar to that of the object detection unit 301, or may be based on the difference with the background image. Then, a silhouette image showing the shape of the object and a texture image showing the texture of the object cut out from the captured image are generated.
[0075] Then, a three-dimensional model of the object is generated by, for example, a volume intersection method using the silhouette image. At this time, since the real camera parameters of the first imaging device 110 are required, the real camera parameters managed by the first captured image management unit 302 are used. The real camera parameters of the first imaging device 110 that captured the first imaging space are not changed.
[0076] In S509, the virtual viewpoint image generating unit 303 renders the three-dimensional model of the object in the first imaging space 120 generated in S508 so as to be an image seen from the virtual viewpoint set in S507. Specifically, the three-dimensional model seen from the virtual viewpoint is colored using the texture image at that time. In this way, one frame of the virtual viewpoint image is generated.
[0077] In S510, the virtual viewpoint image generating unit 303 transmits the generated virtual viewpoint image for one frame, the time determined in S506, and the virtual viewpoint parameters of the virtual viewpoint set in S507 to the second control device 200. Furthermore, the virtual viewpoint image generating unit 303 transmits the position of a designated object in the generated virtual viewpoint image to the second control device 200. In this embodiment, the position of the player 122 is transmitted.
[0078] The next steps S511 to S515 are to perform processing for adjusting the frame time based on the image captured by the second image capturing device 210 and the camera parameters of the second image capturing device 210 to the generated virtual viewpoint image.
[0079] In S511, the second control device 200 receives a virtual viewpoint image, a time code, virtual viewpoint parameters, and a position of a specified object.
[0080] In S512, the display control unit 205 superimposes the received time code on the received virtual viewpoint image, and displays the virtual viewpoint image with the superimposed time code on the virtual viewpoint image display device 230. The user 222 can enjoy the fun of imitating the play of the player 122 by, for example, moving in accordance with the movement of the player 122 while watching the displayed virtual viewpoint image.
[0081] In S513, the time synchronizing unit 203 determines a time to be associated with a frame obtained in S515 by imaging by the second imaging device 210. The same time code as the time code superimposed and displayed on the virtual viewpoint image on the virtual viewpoint image display device 230 is then determined as the time to be associated with a frame obtained in S515 by imaging by the second imaging device 210.
[0082] The time code displayed superimposed on the virtual viewpoint image on the virtual viewpoint image display device 230 is the time code determined immediately before in S506 and received in S511. As a result, the time code of the frame obtained by the second imaging device 210 capturing an image in S515 is determined to be the same as the time code of the virtual viewpoint image currently displayed on the virtual viewpoint image display device 230. By matching the time of the captured image of the second imaging space to the time of the virtual viewpoint image of the first imaging space 120 in this way, even when the game nature is enhanced by playing the scene in reverse, the user 222 can easily imitate the movements of the player 122 while watching the virtual viewpoint image.
[0083] In S514, the alignment unit 204 performs alignment processing on the camera parameters of the second imaging device 210. This alignment processing changes the actual camera parameters indicating the position and orientation of the second imaging device 210 according to the line of sight direction from the virtual viewpoint, etc. A specific processing procedure will be described later. The changed camera parameters of the second imaging device 210 after alignment (referred to as changed camera parameters) are temporarily stored separately from the actual camera parameters before alignment.
[0084] In S515, the second imaging control unit 201 synchronizes the imaging devices constituting the second imaging device 210 with time and causes them to capture the second imaging space 220. As a result, one frame of image data of multiple viewpoints including the user 222 is obtained.
[0085] In S516, the second imaging control unit 201 transmits image data indicating the frames of the multiple viewpoints acquired in S515 and the time codes of the frames determined in S513 to the image processing device 300. Furthermore, the second imaging control unit 201 transmits the changed camera parameters of the imaging devices constituting the second imaging device 210 obtained as a result of S514 to the image processing device 300. After being transmitted to the image processing device 300, the changed camera parameters are deleted from the storage unit of the second control device 200.
[0086] In S517, the second captured image management unit 304 of the image processing device 300 receives one frame of image data from multiple viewpoints, the time code of the frame, and the changed camera parameters of the second imaging device 210.
[0087] In S518, the second captured image management unit 304 stores the received image data for one frame from multiple viewpoints, the time code of the frame, and the changed camera parameters of the second image capturing device 210 in association with each other.
[0088] In S519, the second imaging control unit 201 determines whether or not to end imaging of the second imaging space. If the time set in S506 is equal to the end time of the scene specified in S503, it determines that imaging is to end. If imaging is to end, the process ends, and if not, the process returns to S506 and repeats the process.
[0089] When the second imaging device 210 is a single imaging device, the second imaging control unit 201 may be configured to control the second imaging device 210 to capture an image so as to follow the user 222.
[0090] [Regarding the alignment process] 6 is a flowchart for explaining the processing procedure of the positioning unit 204 in S514. In the positioning process, the camera parameters of the second imaging device 210 used to generate a three-dimensional model are changed based on line-of-sight information of a virtual viewpoint, etc. In this embodiment, as shown in FIG. 9(b), the camera parameters of the imaging devices constituting the second imaging device 210 installed on the circumference of a circle centered on the origin of the coordinate system of the second imaging space 220 are changed. Note that the second imaging device 210 does not have to be on the circumference, and may be installed in accordance with the space of the second imaging space 220.
[0091] In S601, the positioning unit 204 sets a reference point in the second imaging space 220. For example, the center of the second imaging space 220 is set as the reference point. In this case, the reference point does not change throughout the imaging of the second imaging space. Alternatively, for example, the position of the second object (user 222) may be set as the reference point. In this case, the reference point moves as the second object moves.
[0092] In S602, the positioning unit 204 calculates a translation vector Δt for moving the reference point in the second imaging space 220 so as to be equal to the position of the first object (player 122) which is a specified object in the first imaging space. The position p1 of the first object (player 122) in the coordinate system of the first imaging space 120 is expressed by Equation 3. Note that the position of the first object (player 122) is the position in the frame received in S511. Note that the reference point in the second imaging space 220 may be adjusted to match a specified specific position other than a player in the first imaging space.
[0093]
number
[0094] Moreover, the position p2 of the reference point in the coordinate system of the second imaging space 220 is expressed by Equation 4.
[0095]
number
[0096] The translation vector Δt is expressed by Equation 5.
[0097]
number
[0098] Fig. 7 is a diagram for explaining the processing of this step. Fig. 7(a) is a diagram showing a coordinate system in the first imaging space 120, and a point in Fig. 7(a) indicates the position of the specified player 122. Fig. 7(b) is a diagram showing a coordinate system in the second imaging space 220. A reference point 701 is the reference point set in S601. A translation vector Δt is expressed as shown in Fig. 7(c) from Equation 5.
[0099] In S603, the positioning unit 204 acquires the line of sight direction from the virtual viewpoint set in order to generate the virtual viewpoint image displayed on the virtual viewpoint image display device 230. That is, the line of sight direction v from the virtual viewpoint is obtained from the virtual viewpoint parameters set in S507. 20 The virtual camera pose R is expressed by Equation 6.
[0100]
number
[0101] View direction from the virtual viewpoint v 20 is expressed by Equation 7.
[0102]
number
[0103] In S604, the position alignment unit 204 calculates the direction from the reference point in the second imaging space 220 set in S601 to the virtual viewpoint image display device 230. The position p3 of the virtual viewpoint image display device 230 in the coordinate system of the second imaging space 220 is expressed by Equation 8.
[0104]
number
[0105] A direction v from a reference point in the second imaging space 220 to the virtual viewpoint image display device 230 10 is calculated using Equation 9.
[0106]
number
[0107] In S605, the positioning unit 204 calculates a direction v from the reference point to the virtual viewpoint image display device 230 in the xy plane. 10 is the line of sight from the virtual viewpoint v 20 First, a rotation matrix is calculated to rotate the image in a direction v from a reference point in the second imaging space 220 to the virtual viewpoint image display device 230 in the xy plane. 10 and the line of sight from the virtual viewpoint v 20 The angle θ between the reference point and the virtual viewpoint image display device 230 is calculated. 10 If the vector obtained by projecting onto the xy plane is v1, then v1 is expressed by Equation 10.
[0108]
number
[0109] View direction from the virtual viewpoint v 20 If the vector obtained by projecting onto the xy plane is v2, then v2 is expressed by Equation 11.
[0110]
number
[0111] Therefore, θ can be calculated using Equation 12.
[0112]
number
[0113] In the xy plane, the direction v from the reference point to the virtual viewpoint image display device 230 10 is the line of sight from the virtual viewpoint v 20 The rotation matrix ΔR that rotates the image to be equal to the image is expressed by Equation 13.
[0114]
number
[0115] 8A and 8B are diagrams for explaining the processing of this step. FIG. 8A is a diagram showing a coordinate system in the first imaging space 120, and a point in the diagram shows the position of a player 122, which is a specified object. Also, a virtual camera 801 shows the position of the virtual viewpoint set in S507 and the line of sight direction from the virtual viewpoint, and the line of sight direction v from the virtual viewpoint is shown in FIG. 20 is represented by the orientation of the virtual camera 801. FIG. 8(b) is a diagram showing a coordinate system in the second imaging space 220, and the reference point 701 is the reference point set in S601. 10 represents the direction from the reference point 701 to the virtual viewpoint image display device 230. As shown in FIG. 8(c), θ is the line of sight direction v from the virtual viewpoint on the xy plane. 20 and a direction v from the reference point 701 to the virtual viewpoint image display device 230. 10 This is the angle between them.
[0116] In S606, the alignment unit 204 uses the rotation matrix ΔR to change the actual camera parameters representing the orientation of the imaging devices constituting the second imaging device 210 so as to rotate the orientation of the imaging devices by θ, thereby obtaining changed camera parameters. The orientation R' of each imaging device constituting the second imaging device 210 after the change is obtained by Equation 14. R in Equation 14 is the orientation of each imaging device constituting the second imaging device 210 before alignment. This makes it possible to align the orientation for each frame.
[0117]
number
[0118] In S607, the alignment unit 204 uses the calculated rotation matrix ΔR and translation vector Δt to change the actual camera parameters indicating the positions of the imaging devices constituting the second imaging device 210, and obtains changed camera parameters. The position t' of each imaging device constituting the second imaging device 210 after movement is obtained by Equation 15. t in Equation 15 is the position of each imaging device constituting the second imaging device 210 before alignment in this step. By using the changed camera parameters indicating the positions after movement, alignment with the first imaging space is possible for each frame.
[0119]
number
[0120] The changed camera parameters are stored separately from the actual camera parameters obtained as a result of the calibration in S501. The changed camera parameters of the second imaging device 210 indicating the attitude after rotation obtained as a result of the processing in S606 and the position after movement obtained as a result of the processing in S607 are temporarily stored in the second control device 200 as described above, and are transmitted to the image processing device 300 in S516. The actual camera parameters obtained as a result of the calibration in S501 and held in the second imaging control unit 201 are stored without being changed even after the completion of this flowchart.
[0121] Fig. 9 is a diagram for explaining the position and posture of the imaging device indicated by the changed camera parameters after position adjustment. Fig. 9(a) is a diagram showing the first imaging space 120. A virtual camera 801 indicates the position of the virtual viewpoint set in S507 and the line of sight direction from the virtual viewpoint, and a point indicates the position of a player 122. If a reference point is set at the origin of the coordinate system of the second imaging space 220 in S601, a translation vector Δt is expressed as a vector from the origin to the player 122 as shown in Fig. 9(a).
[0122] Fig. 9(b) is a diagram showing the second imaging space 220 and the actual positions and orientations of the imaging devices constituting the second imaging device 210 installed in the second imaging space 220. That is, the positions and orientations of the imaging devices in Fig. 9(b) are diagrams showing the positions and orientations of the imaging devices indicated by the actual camera parameters of the second imaging device 210. The results of the processing in this flowchart will be explained with a focus on the position and orientation (direction) of camera A among the imaging devices constituting the second imaging device 210.
[0123] FIG. 9(c) is a diagram showing the orientation of the imaging device indicated by the changed camera parameters obtained as a result of changing the real camera parameters in S606. The dotted arrow indicates the orientation of the imaging device indicated by the camera parameters before the processing of S606 is performed, and the solid arrow indicates the orientation of the imaging device indicated by the camera parameters after the processing of S606 is performed. As shown in FIG. 9(c), as a result of the processing of S606, the real camera parameters are changed so that the orientation of the imaging device rotates by θ. FIG. 9(d) shows the position of the imaging device indicated by the changed camera parameters obtained as a result of changing the real camera parameters in S607. As shown in FIG. 9(c), the real camera parameters are changed so that the position of the imaging device rotates by θ around the origin and moves in parallel by Δt.
[0124] [About the generation process of synthetic virtual viewpoint images] FIG. 10 is a flowchart for explaining a process in which the image processing device 300 generates a synthetic virtual viewpoint image and displays the synthetic virtual viewpoint image on the synthetic image display device 410.
[0125] First, the operator 244 specifies a scene by operating the operation device 420. The scenes that the operator 244 can specify are the scenes specified in S503. In S1001, the virtual viewpoint control unit 307 acquires information indicating the scene specified by the operator 244 via the operation device 420.
[0126] Next, the operator 244 operates the operation device 420 to specify the playback speed of the scene. The playback speeds that the operator can specify include normal speed playback, slow playback, etc. In S1002, the virtual viewpoint control unit 307 acquires information indicating the playback speed specified by the operator 244 via the operation device 420.
[0127] The processing of S1003 to S1008 is processing for generating a synthetic virtual viewpoint image for one frame of a specified scene and displaying the synthetic virtual viewpoint image.
[0128] The operator 244 operates the operation device 420 to specify the position of the virtual viewpoint and the line of sight from the virtual viewpoint. At the start of the synthetic virtual viewpoint image, a default virtual viewpoint is set according to the specified scene. The operator 244 can operate the virtual viewpoint by moving it forward, backward, up, down, left and right from the default state, changing its orientation such as panning and tilting, zooming, etc. In S1003, the virtual viewpoint control unit 307 acquires information on the operation of the operation device 420 by the operator 244, and sets virtual viewpoint parameters that indicate the position of the virtual viewpoint and the line of sight from the virtual viewpoint.
[0129] In S1004, the virtual viewpoint control unit 307 updates the time for generating a synthetic virtual viewpoint image according to the specified playback speed. The time is specifically a time code. In the first processing of this step in the loop processing, the time code at the start of the specified scene is determined. In the second and subsequent processing of this step, the time is updated according to the playback speed of the specified scene. The method of update is the same as the method described in S506.
[0130] In S1005, the synthetic virtual viewpoint image generating unit 305 acquires the virtual viewpoint parameters set in S1003 and the time set in S1004. Then, the synthetic virtual viewpoint image generating unit 305 generates a three-dimensional model of an object in the first imaging space 120 corresponding to the time set in S1004. The method of generation is the same as that in S508, and therefore the description is omitted. Alternatively, if a three-dimensional model of an object in the first imaging space 120 generated by the process of the flowchart in Fig. 5 is stored, a three-dimensional model may not be generated in this step, and a three-dimensional model corresponding to the time set in S1004 may be acquired.
[0131] In S1006, the synthetic virtual viewpoint image generating unit 305 generates a three-dimensional model of the user 222 (second object) corresponding to the time set in S1005. For example, the three-dimensional model of the user 222 is generated as follows. First, from the captured images based on the capture of the second imaging device 210 managed by the second captured image management unit 304, image data of one frame of each imaging device corresponding to the time set in S1004 is acquired. For each acquired one frame, an object area is detected. Then, a three-dimensional model of the second object is generated using a visual volume intersection method. At this time, the camera parameters of the second imaging device 210 that captured the user 222 are required, and among the changed camera parameters managed by the second captured image management unit 304, the changed camera parameters corresponding to the time set in S1004 are used. That is, the changed camera parameters have different values for each frame.
[0132] Through this step, a three-dimensional model of the user 222 is generated so that the reference point set in S601 is located at the position of the player 122. For example, if the reference point is set as the position of the user 222, a three-dimensional model of the user 222 is generated so that the user 222 is located at the position of the player 122.
[0133] In S1007, the synthetic virtual viewpoint image generating unit 305 synthesizes the generated three-dimensional model so that the three-dimensional model of the object in the first imaging space 120 and the three-dimensional model of the user 222 are placed in the same three-dimensional space.
[0134] The changed camera parameters of the second imaging device 210 that captured the user 222 are aligned with the virtual viewpoint image of the first imaging space 120. Thus, in S1006, a three-dimensional model of the user 222 is generated so that the user 222 exists at a position based on the player 122, who is a specified object in the first imaging space 120. By matching the three-dimensional model of the object in the first imaging space 120 with the three-dimensional model of the second object, a three-dimensional model in which the user 222 exists at a specified position in the first imaging space 120 together with the object in the first imaging space 120 can be obtained.
[0135] Then, the synthetic virtual viewpoint image generating unit 305 generates one frame of a synthetic virtual viewpoint image by rendering the synthetic three-dimensional model so that it becomes an image seen from the virtual viewpoint set in S1003. The background is rendered using the background of the first imaging space 120. Note that, when rendering the synthetic virtual viewpoint image, a process of suppressing the presence of the player 122 may be performed. For example, the three-dimensional model of the player 122 may be deleted. Alternatively, the three-dimensional model of the player 122 may be rendered so as to be transparently displayed.
[0136] In S1008, the synthetic virtual viewpoint image generation unit 305 outputs the generated synthetic virtual viewpoint image for one frame and its time to the display control unit 306. The display control unit 306 controls the display of the synthetic virtual viewpoint image for one frame on the synthetic image display device 410, so that the synthetic virtual viewpoint image for one frame is displayed on the synthetic image display device 410 so that it can be viewed by the spectators 243. Therefore, the synthetic virtual viewpoint image can provide the spectators 243, who are viewers of the synthetic virtual viewpoint image, with the enjoyment of being able to check the degree to which the user 222 reproduces the play of the player 122.
[0137] In S1009, the display control unit 306 determines whether to end the display of the synthetic virtual viewpoint image. If the received time is equal to the end time of the specified scene, it determines that the display is to end. If the display is to end, the process ends, and if not, the process returns to S1003 and the display continues.
[0138] As described above, in this embodiment, the camera parameters, which are line-of-sight information indicating the position and attitude of the second imaging device that images the user, are changed according to the virtual viewpoint image displayed on the virtual viewpoint image display device 230. Then, by generating a three-dimensional model of the user 222 using the changed camera parameters, it is possible to generate a composite virtual viewpoint image in which the user 222 is composited at an arbitrary position in the first imaging space 120 with an arbitrary orientation.
[0139] This change of camera parameters is performed for each frame. By changing the camera parameters for each frame, even if the point of interest (for example, the position of the player 122) differs for each frame, it is possible to generate a synthetic virtual viewpoint image in which the user 222 is appropriately synthesized at a position based on the point of interest. Therefore, it is possible to synthesize the user not only in cases where the position of a designated object is fixed, such as a batter in baseball, but also in dynamic scenes where the position and direction of a designated object change, such as a shooting scene in soccer.
[0140] Furthermore, when the reference point in the second imaging space 220 is the position of the user 222, the camera parameters of the second imaging device 210 are changed according to the direction from the user's position to the position of the virtual viewpoint image display device 230. This makes it possible to generate a composite virtual viewpoint image by compositing, for example, the user 222 who is watching the virtual viewpoint image and imitating the shooting action of the player 122 at an appropriate position and direction.
[0141] The functions of each device constituting the image processing system 1 in this embodiment can also be realized by a computer (or a CPU or MPU) of the device constituting the image processing system 1 reading and executing program code stored in a storage medium. A storage medium on which program code of software that realizes the functions of each device constituting the image processing system 1 in this embodiment is recorded may be supplied to the device constituting the image processing system 1. In this case, the program code itself read from the storage medium realizes the functions of this embodiment, and the storage medium storing the program code constitutes this embodiment. Examples of storage media for supplying the program code include flexible disks, hard disks, optical disks, magneto-optical disks, CD-ROMs, CD-Rs, magnetic tapes, non-volatile memory cards, and ROMs.
[0142] In addition, the functions of each device constituting the image processing system 1 may be realized by an OS (operating system) running on the computer performing some or all of the actual processing based on instructions of the program code read by the computer.
[0143] The program code read from the storage medium may be written to a memory provided on a function expansion board inserted into a computer or a function expansion unit connected to the computer, and the functions of the devices constituting the image processing system 1 in this embodiment may be realized by a CPU or the like provided on the function expansion board or function expansion unit performing part or all of the actual processing based on instructions of the program code.
[0144] It is not necessary to change the line-of-sight information of the multiple imaging devices included in the second imaging device group. For example, the coordinate system of the imaging space captured by the first imaging device group and the coordinate system of the imaging space captured by the second imaging device group may be adjusted. For example, when the center of the second imaging space is set as the reference point, a three-dimensional model of an object generated based on an image captured by the second imaging device group can be arranged at a predetermined position and orientation in the imaging space captured by the first imaging device group using coordinate transformation. In other words, the center of the second imaging space is set as the reference point and the position of the display unit is fixed regardless of the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint, so that the coefficient of the coordinate transformation can be uniquely determined. Therefore, by acquiring the coefficient of the coordinate transformation in advance, the processing load can be reduced and the processing speed can be improved.
[0145] <Embodiment 2> In this embodiment, a method for a user to relive the play of the first object with a more realistic feeling using a ball launching device installed in the second imaging space will be described. The present embodiment will be described focusing on the differences from the first embodiment. The parts not specifically mentioned have the same configuration and processing as the first embodiment.
[0146] The scene assumed in this embodiment is a scene in a soccer game where a ball is passed to a player 122 and the player 122 shoots the ball. In this embodiment, a method of reproducing the pass in the scene by having a ball launching device 1206 actually launch the ball will be described.
[0147] 11 is a flowchart for explaining the processing procedure of the ball launch control unit 308. The processing of this flowchart is started after a scene is specified in S503 and before the second imaging device 210 starts imaging the user. In the processing of this flowchart, the scene in which the player 122 shoots the ball passed to him is assumed to be specified in S503.
[0148] In S1101, the ball launch control unit 308 acquires the time code of the time when the player 122 receives the ball that has been passed to him in the scene specified in S503. For example, the time when the ball is received may be specified by the user.
[0149] In S1102, the object detection unit 301 calculates the direction in which the ball moved in the time code acquired in S1101. Specifically, the object detection unit 301 detects the position of the ball one frame before the acquired time code and the position of the ball in the frame of the acquired time code. Then, the object detection unit 301 calculates the direction in which the ball moved from the difference between the respective positions.
[0150] Fig. 12 is a diagram for explaining the processing of this flowchart. Fig. 12(a) is a diagram showing a first imaging space 120, and as a result of the processing of this step, a moving direction 1202 of a ball 1201 is calculated.
[0151] In S1103, the ball launch control unit 308 converts the direction in which the ball moves calculated in S1102 into a direction in the second imaging space 220. In this embodiment, the description will be given assuming that the line of sight direction from the virtual viewpoint for generating a virtual viewpoint image to be displayed on the virtual viewpoint image display device 230 is determined by the virtual viewpoint image generation unit 303 based on the frame.
[0152] The ball launch control unit 308 acquires the line of sight direction from the virtual viewpoint at the time code acquired in S1101, and further calculates the rotation matrix ΔR at that time code by the method explained in S605, and calculates the inverse matrix ΔR -1 The orientation of the ball in the second imaging space 220 is calculated by Equation 16. Orientation in the second shooting space = ΔR -1 The direction in which the ball moves in the first imaging space (Formula 16)
[0153] 12B is a diagram showing the second imaging space 220. The direction 1202 in which the ball moves in the first imaging space 120 shown in FIG. 12A is converted into a direction 1204 as a result of the processing in this step. The direction 1204 is a direction V from the reference point 701 to the virtual viewpoint image display device 230. 10 is the direction in which the ball moves when the virtual camera is facing in the same direction.
[0154] In S1104, the ball launch control unit 308 determines a position such that the direction from the ball launch position 1203 of the ball launching device 1206 to the kick position in the second imaging space 220 becomes equal to the direction 1204 calculated in S1103. Then, the ball launching device 1206 is moved so that the ball launch position of the ball launching device 1206 becomes the determined position. For example, FIG. 12(b) is an example in which the center of the second imaging space 220 is the kick position 1205. FIG. 12(b) shows a state in which the ball launching device 1206 is moved so that the direction from the ball launch position 1203 to the kick position 1205 becomes equal to the direction 1204 calculated in S1103. When the positions to which the ball launching device 1206 can be moved in order to launch the ball are limited, the ball launching device 1206 is moved to a position where the direction 1204 calculated in S1103 is closest to the direction from the ball launching position 1203 to the kick position 1205.
[0155] Next, the speed of the ball to be shot is specified by the user 222 via an operation unit (not shown) of the second control device 200. For example, the user 222 specifies one of several speeds, such as high speed, medium speed, and low speed. In S1105, the reception unit 206 receives the speed specified by the user, and the ball shot control unit 308 acquires the speed specified by the user.
[0156] In S1106, the ball launch control unit 308 calculates a time code indicating the time when the ball is launched. For example, the distance from the ball launch position 1203 to the kick position 1205 is divided by the ball speed designated by the user to obtain the time it takes for the ball to travel from the ball launch position 1203 to the kick position 1205. Then, the time obtained by subtracting the previously obtained time from the time when the player 122 receives the ball is calculated as the time code indicating the time when the ball is launched.
[0157] Thereafter, by proceeding with the processing of the flowchart in FIG. 5, the ball launch control unit 308 acquires the time code of the virtual viewpoint image displayed on the virtual viewpoint image display device 230 in S1107.
[0158] In S1108, the ball launch control unit 308 judges whether or not the acquired time code is equal to the time code of the time when the ball is launched. If they are equal, the process proceeds to S1109. If they are not equal, the process returns to S1107.
[0159] In S1109, the ball launch control unit 308 causes the ball launching device 1206 to launch the ball toward the kick position 1205 at a specified speed.
[0160] As described above, according to this embodiment, by using the ball launching device, the user can re-experience playing with the first object with a greater sense of realism.
[0161] <Other embodiments> In the above-described embodiment, the functions of the first control device 100, the second control device 200, and the image processing device 300 have been described as being possessed by each device. Alternatively, the functions of the first control device 100, the second control device 200, and the image processing device 300 shown in Fig. 2 may be realized by one or two devices. For example, the image processing device 300 may have the functions of the first control device 100 and the second control device 200.
[0162] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions. [Explanation of symbols]
[0163] 300 Image processing device 302 First captured image management unit 304 Second captured image management unit 305 Synthetic Virtual Viewpoint Image Generation Unit 230 Virtual viewpoint image display device
Claims
1. a first acquisition means for acquiring line-of-sight information of a plurality of imaging devices included in a second imaging device group different from the first imaging device group, the line-of-sight information being determined based on information for identifying a virtual viewpoint corresponding to a virtual viewpoint image generated based on a first image group obtained by imaging from a plurality of directions using a first imaging device group; a second acquisition means for acquiring a second group of images based on an object being imaged from a plurality of directions by the second group of imaging devices; a generating means for generating a virtual viewpoint image in a case where the object is present at a predetermined position in an image capture space captured by the first image capture device group, based on line of sight information acquired by the first acquisition means and a second image group acquired by the second acquisition means, The line-of-sight information acquired by the first acquisition means is line-of-sight information of a plurality of imaging devices included in the second imaging device group, which is determined based on a reference point in an imaging space captured by the second imaging device group, a position of a display unit that displays a virtual viewpoint image generated based on the first image group, information for identifying a virtual viewpoint in the virtual viewpoint image displayed on the display unit, and the predetermined position.
13. An image processing device comprising:
2. a third acquisition means for acquiring a position of a specified object from an object included in the first image group; The predetermined position is a position based on the position of the designated object acquired by the third acquisition means.
2. The image processing device according to claim 1,
3. The line of sight information acquired by the first acquisition means is line-of-sight information related to the second imaging device group determined based on a parameter for aligning the reference point with the position of the specified object and a parameter for aligning a direction from the reference point to the display unit with a line-of-sight direction from a virtual viewpoint in a virtual viewpoint image displayed on the display unit.
3. The image processing device according to claim 2.
4. The reference point is a position of an object imaged by the second imaging device group.
4. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing apparatus.
5. The display unit is installed at a position visible from an object imaged by the second imaging device group.
5. The image processing device according to claim 1,
6. A detection means for detecting an object included in the first set of images is further provided. The specified object is an object specified from the detected objects.
4. The image processing device according to claim 2, wherein the first and second inputs are input to the image processing apparatus.
7. The generating means performs processing to suppress the shape of the specified object.
7. The image processing device according to claim 2, 3 or 6.
8. The first acquisition means acquires, as time information to be associated with the second image group, time information associated with a virtual viewpoint image generated based on the first image group.
8. The image processing device according to claim 1,
9. The present invention further includes a control means for controlling a ball launching device installed in the vicinity of an object imaged by the second imaging device group.
9. The image processing device according to claim 1,
10. The control means determines a direction in which the ball is launched by the ball launching device, based on a parameter for matching a direction from a reference point in an imaging space of the second imaging device group to a display unit that displays the virtual viewpoint image generated based on the first image group, with a line of sight direction from a virtual viewpoint in the virtual viewpoint image generated based on the first image group, and based on a direction in which a ball moves in the virtual viewpoint image.
10. The image processing device according to claim 9,
11. a change unit that changes line-of-sight information related to the second imaging device group based on at least information that defines how the virtual viewpoint image displayed on the display unit looks from a virtual viewpoint when the second imaging device group captures an image, The first acquisition means acquires line-of-sight information regarding the second imaging device group changed by the change means.
6. The image processing device according to claim 1,
12. a fourth acquisition means for acquiring line-of-sight information relating to the first image group and the first image capturing device group; and a second generation means for generating three-dimensional shape data of an object included in the first image group based on the first image group and line-of-sight information of the first image capturing device group, and generating a virtual viewpoint image to be displayed on the display unit.
6. The image processing device according to claim 1,
13. The generating means includes: synthesizing three-dimensional shape data of an object included in the first image group with three-dimensional shape data of an object included in the second image group generated based on the second image group acquired by the second acquisition means and line-of-sight information acquired by the second acquisition means, and generating a virtual viewpoint image based on the three-dimensional shape data obtained as a result of the synthesis; The image processing device according to claim 12 .
14. The display device further includes a display control unit that controls the virtual viewpoint image generated by the generating unit to be displayed on a second display unit different from the display unit.
6. The image processing device according to claim 1,
15. The second imaging device group is arranged so as to be located on a circumference of an imaging space of the second imaging device group.
15. The image processing device according to claim 1,
16. An image processing device according to any one of claims 1 to 15, a display control unit that controls display of a virtual viewpoint image generated based on the first image group on a display unit located at a position where an object imaged by the second image capturing device group can be seen; 1. An image processing system comprising:
17. a first acquisition step of acquiring line-of-sight information of a plurality of imaging devices included in a second imaging device group different from the first imaging device group, the line-of-sight information being determined based on information for identifying a virtual viewpoint corresponding to a virtual viewpoint image generated based on a first image group obtained by imaging from a plurality of directions by a first imaging device group; a second acquisition step of acquiring a second group of images based on the object being imaged from a plurality of directions by the second group of imaging devices; a generating step of generating a virtual viewpoint image in a case where the object is present at a predetermined position in an image capture space captured by the first image capture device group, based on line of sight information acquired by the first acquisition step and a second image group acquired by the second acquisition step, The line-of-sight information acquired by the first acquisition step is line-of-sight information of a plurality of imaging devices included in the second imaging device group, which is determined based on a reference point in an imaging space captured by the second imaging device group, a position of a display unit that displays a virtual viewpoint image generated based on the first image group, information for identifying a virtual viewpoint in the virtual viewpoint image displayed on the display unit, and the predetermined position.
13. An image processing method comprising:
18. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 15.
Citation Information
Patent Citations
Method for fetching other element to actual game and conducting game
JP2002101400A
Image processing device, image processing method, and program
JP2020086983A
