Information processing device, information processing method and program

Abstract

To solve a problem that it is difficult to set a virtual camera in an appropriate position and orientation for a fast-moving subject.SOLUTION: An information processing device identifies a first speed, which is a movement speed of a subject within a virtual viewpoint image being displayed, and controls the movement speed of the subject to a second speed slower than the first speed when the first speed exceeds a threshold.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device that controls the position and orientation of a virtual camera used to generate a virtual viewpoint video. [Background technology]

[0002] There is a technology for generating a virtual viewpoint image using a plurality of captured images obtained by arranging a plurality of imaging devices to surround an imaging area and capturing images of a subject in synchronization. With this technology, a virtual camera (virtual camera) can be placed at any position in a virtual space containing a 3D model of the subject generated from the plurality of captured images, thereby generating a virtual viewpoint image showing the view from the virtual camera. In addition, a virtual viewpoint video can be generated by generating a plurality of frames of virtual viewpoint images.

[0003] In Patent Document 1, an operator sets the position and orientation of a virtual camera in multiple frames and generates information indicating the trajectory of the virtual camera. At that time, it is described that the moving speed of a subject in a virtual viewpoint video, which is displayed as a reference when the operator sets the position and orientation of the virtual camera, can be changed based on a user operation. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2020-10301 Summary of the Invention [Problem to be solved by the invention]

[0005] For operators who set the position and orientation of a virtual camera, it has been difficult to set the virtual camera in an appropriate position and orientation for a fast-moving subject. For example, a system for generating virtual viewpoint images can be installed at a sports venue to generate virtual viewpoint images of various sports in real time. In such cases, the subject may move at high speed during decisive sports scenes, such as an attack in volleyball or a smash in tennis. In particular, when the subject's movement speed increases instantaneously, it has been difficult to set the virtual camera in an appropriate position and orientation to match the subject's movement.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and aims to make it easier to set up a virtual camera that captures a subject that moves at high speed. [Means for solving the problem]

[0007] In order to solve the above problems, an information processing device according to the present disclosure has the following configuration: That is, the information processing device generates information indicating a trajectory of a virtual camera for generating a virtual viewpoint video, and includes: an identification means for identifying a first speed that is a moving speed of a subject included in the virtual viewpoint video being displayed; and a control means for controlling the moving speed of the subject to a second speed that is slower than the first speed when the first speed is greater than a threshold value. [Effects of the Invention]

[0008] According to the present disclosure, it becomes easy to specify a virtual camera that captures an object moving at high speed. [Brief explanation of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating a functional configuration of a virtual viewpoint video generation device 100 according to a first embodiment. [Figure 2] FIG. 2 is a diagram illustrating a functional configuration of a virtual camera control unit 106 according to the first embodiment. [Figure 3] 1 is a diagram illustrating a hardware configuration of a virtual viewpoint image generating device 100 according to a first embodiment. [Figure 4] 10 is a flowchart showing a process performed by the playback speed determination unit 201 to determine the playback speed. [Figure 5] FIG. 10 is a diagram showing a UI of a generation speed determination setting unit 202. [Figure 6] FIG. 10 is a diagram showing the change in playback speed between default mode and continuous mode. [Figure 7] FIG. 7 is a diagram illustrating a functional configuration of a virtual viewpoint video generation device 700 according to a second embodiment. [Figure 8] FIG. 10 is a diagram illustrating a functional configuration of a virtual camera control unit 706 according to a second embodiment. [Figure 9] 10 is a flowchart showing a process for determining whether a subject is captured by a virtual camera and determining a playback speed. DETAILED DESCRIPTION OF THE INVENTION

[0010] According to a preferred embodiment of the present invention, an information processing device generates information indicating a trajectory of a virtual camera for generating a virtual viewpoint video. The information processing device also includes a specifying means for specifying a first speed, which is the moving speed of a subject in the virtual viewpoint video displayed when an operator sets the virtual camera. The information processing device also includes a control means for controlling the moving speed of the subject to a second speed slower than the first speed when the first speed is greater than a threshold. The displayed virtual viewpoint video here refers to a virtual viewpoint image corresponding to a currently displayed frame in a virtual viewpoint video composed of multiple virtual viewpoint images corresponding to multiple frames. The moving speed of the subject in the virtual viewpoint video is calculated from the position of the subject included in the virtual viewpoint image corresponding to the currently displayed frame and the position of the subject included in the virtual viewpoint image corresponding to the frame immediately preceding the currently displayed frame. The displacement of the subject's position between frames is calculated from the position of the subject in the currently displayed frame and the position of the subject in the previous frame, and the time corresponding to the interframe period is calculated from the first time, which is the capture time of the currently displayed frame, and the second time, which is the capture time of the previous frame. The moving speed of the subject is then calculated by dividing the displacement of the subject's position between frames by the time corresponding to the interframe period. Therefore, the moving speed of the subject can also be said to be the amount of change in the position of the subject per unit time.

[0011] According to this aspect, when the movement speed of the subject increases, the movement speed of the subject is automatically slowed down, so that it is possible to suppress momentary increases in the movement speed of the subject in the virtual viewpoint image that the operator uses as a reference for setting the virtual camera, thereby enabling the operator to easily set the virtual camera.

[0012] In the above embodiment, since the second speed is slow, it is generally interpreted as slow playback. On the other hand, since the moving speed of the virtual camera can be controlled separately, for example, the moving speed of the subject can be slowed down while the moving speed of the virtual camera remains unchanged.

[0013] Furthermore, the moving speed of the subject remains at the first speed when the first speed is equal to or less than the threshold value. In other words, when the first speed is equal to or less than the threshold value, the information processing device does not control the moving speed of the subject to the second speed.

[0014] Furthermore, the position of the subject used to determine the movement speed of the subject may be a three-dimensional position in the virtual space. In this case, the movement speed of the subject can be set regardless of the movement speed of the virtual camera operated by the operator. Furthermore, the position of the subject used to determine the movement speed of the subject may be the position of the subject on the virtual viewpoint image. In this case, the movement speed of the subject can be set from the three-dimensional position of the subject in the virtual space and the position and orientation of the virtual camera.

[0015] The virtual viewpoint video is composed of a plurality of frames of virtual viewpoint images.

[0016] Furthermore, the information indicating the trajectory of the virtual camera is made up of information indicating the position and orientation of the virtual camera in each of the plurality of frames.

[0017] Furthermore, when the first speed is greater than a threshold value, the information processing device controls the display means to display information indicating that the movement speed of the subject is being controlled to the second speed. For example, when the first speed is greater than the threshold value, an icon indicating that the movement speed of the subject is being controlled is superimposed on a virtual viewpoint video that an operator views as a reference when setting the position and attitude of the virtual camera. Alternatively, the information processing device may indicate that the movement speed of the subject is being controlled by highlighting the screen edge of the virtual viewpoint video being displayed in a specific color.

[0018] This allows the operator to understand that the moving speed of the subject is being controlled.

[0019] Furthermore, the information processing device may specify a third speed, which is a moving speed of a virtual camera corresponding to the virtual viewpoint video being displayed, and, if the first speed is greater than a threshold value, may perform control to change the moving speed of the virtual camera to a fourth speed that is slower than the third speed. That is, in addition to the control to slow down the moving speed of the subject, the information processing device may also perform control to slow down the moving speed of the virtual camera being operated by the operator.

[0020] This aspect prevents the subject from moving at high speed in the virtual viewpoint image due to the operator being unable to respond when the subject's movement speed is automatically changed and moving the virtual camera too quickly. Alternatively, when the subject's movement speed is automatically changed, the virtual camera's movement speed can be slowed down accordingly, making it easier to set the position and orientation of the virtual camera.

[0021] The virtual viewpoint video is generated based on a plurality of captured images.

[0022] Furthermore, the information processing device can set the threshold value based on a user operation.

[0023] This aspect allows the threshold value for the control to slow down the moving speed of the subject to be changed in accordance with the user's preference or the subject being photographed. Note that the threshold value may be set in advance for each subject being photographed.

[0024] Furthermore, the information processing device can set the second speed based on a user operation. That is, the information processing device controls the subject's movement speed to be slowed down when the subject's movement speed is greater than a threshold, and the user can set the speed to which the subject should be slowed down.

[0025] This aspect allows the user to determine to what speed the subject's movement speed should be slowed down in accordance with their preferences.

[0026] According to another preferred embodiment of this embodiment, a program causes a computer to execute the above-described method for controlling a mobile object. In this case, the computer may be installed inside the mobile object, or may be installed outside the mobile object and be capable of communicating with the mobile object. By executing this program, the computer preferably functions as the above-described information processing device.

[0027] <Example> The present invention will be described in detail below based on preferred embodiments thereof with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the configurations shown in the drawings.

[0028] The image processing system generates a virtual viewpoint image representing a scene from a specified virtual viewpoint based on multiple images captured by multiple imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint image, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by a user. For example, the virtual viewpoint image also includes an image corresponding to a viewpoint selected by a user from multiple candidates. Furthermore, this embodiment will mainly describe a case where the virtual viewpoint is specified by a user operation, but the virtual viewpoint may also be specified automatically based on the results of image analysis, etc. Furthermore, this embodiment will mainly describe a case where the virtual viewpoint image is a video, but the virtual viewpoint image may also be a still image.

[0029] The viewpoint information used to generate a virtual viewpoint image is information indicating the position and orientation (line of sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set serving as viewpoint information may include a parameter indicating the size of the field of view (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets corresponding to multiple frames constituting a moving image of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of multiple consecutive time points.

[0030] The image processing system has multiple imaging devices that capture images of an imaging area from multiple directions. The imaging area may be, for example, a stadium where sports such as soccer or karate are held, or a stage where a concert or play is held. The multiple imaging devices are installed at different positions surrounding the imaging area and capture images synchronously. Note that the multiple imaging devices do not need to be installed around the entire periphery of the imaging area; depending on installation space restrictions, they may be installed only around a portion of the periphery of the imaging area. Furthermore, the number of imaging devices is not limited to the example shown in the figure. For example, if the imaging area is a soccer stadium, approximately 30 imaging devices may be installed around the stadium. Furthermore, imaging devices with different functions, such as telephoto cameras and wide-angle cameras, may be installed.

[0031] In this embodiment, the multiple image capturing devices are cameras each having an independent housing and capable of capturing images from a single viewpoint. However, this is not limiting, and two or more image capturing devices may be configured in the same housing. For example, a single camera equipped with multiple lens groups and multiple sensors and capable of capturing images from multiple viewpoints may be installed as the multiple image capturing devices.

[0032] A virtual viewpoint image is generated, for example, by the following method. First, multiple images (multiple viewpoint images) are acquired by capturing images from different directions using multiple imaging devices. Next, a foreground image in which a foreground region corresponding to a predetermined object, such as a person or a ball, is extracted, and a background image in which a background region other than the foreground region is extracted are acquired from the multiple viewpoint images. Furthermore, a foreground model representing the three-dimensional shape of the predetermined object and texture data for coloring the foreground model are generated based on the foreground image, and texture data for coloring a background model representing the three-dimensional shape of a background, such as a stadium, is generated based on the background image. The texture data is then mapped to the foreground model and background model, and rendering is performed according to the virtual viewpoint indicated by the viewpoint information, thereby generating a virtual viewpoint image. However, the method for generating a virtual viewpoint image is not limited to this, and various methods can be used, such as a method of generating a virtual viewpoint image by projective transformation of captured images without using a three-dimensional model.

[0033] A foreground image is an image in which an object region (foreground region) is extracted from an image captured by an imaging device. An object extracted as a foreground region is a dynamic object (moving body) that moves (its absolute position and shape can change) when images are captured from the same direction in a time series. Examples of objects include players, referees, and other people on the field where a sport is being played, such as a ball in a ball game, or singers, musicians, performers, and presenters in a concert or entertainment event.

[0034] A background image is an image of at least a region (background region) different from the foreground object. Specifically, a background image is an image in which the foreground object has been removed from the captured image. Furthermore, the background refers to an imaged object that remains stationary or nearly stationary when images are captured from the same direction in chronological order. Examples of such imaged objects include a stage for a concert, a stadium where an event such as a sport is held, a structure such as a goal used in a ball game, or a field. However, the background is at least a region different from the foreground object, and the imaged object may include other objects in addition to the object and background.

[0035] A virtual camera is a virtual camera that is different from the multiple imaging devices actually installed around the imaging area, and is a concept for conveniently explaining a virtual viewpoint related to the generation of a virtual viewpoint image. That is, a virtual viewpoint image can be considered to be an image captured from a virtual viewpoint set in a virtual space associated with the imaging area. The position and orientation of the viewpoint in the virtual image capture can be expressed as the position and orientation of the virtual camera. In other words, a virtual viewpoint image can be considered to be an image simulating an image captured by a camera assuming that the camera exists at the position of the virtual viewpoint set in space. In this embodiment, the content of the transition of the virtual viewpoint over time is referred to as a virtual camera path. However, the concept of a virtual camera is not required to realize the configuration of this embodiment. That is, it is sufficient if at least information representing a specific position and information representing a direction in space are set, and a virtual viewpoint image is generated according to the set information.

[0036] Example 1 1 is a diagram illustrating a functional configuration of a virtual viewpoint image generation device 100 (information processing device) according to a first embodiment of the present disclosure. The virtual viewpoint image generation device 100 is an image processing device including a shape estimation unit 101, a storage unit 102, an image generation unit 103, a display unit 104, a movement speed calculation unit 105, and a virtual camera control unit 106. The virtual viewpoint image generation device 100 is connected to an image capture device 10a, an image capture device 10b to an image capture device 10z, and a controller 110. The controller 110 may be incorporated into the virtual viewpoint image generation device 100.

[0037] A plurality of imaging devices 10a, 10b, ..., 10z are arranged to surround the imaging area. The number of imaging devices 10a, 10b, ..., 10z does not limit the number of imaging devices, and 100 or more imaging devices may be used. Hereinafter, imaging devices 10a, 10b, ..., 10z will be collectively referred to as imaging device 10. The imaging devices 10 acquire multiple captured images by capturing images synchronously. The imaging area may be, for example, a stadium where sports such as volleyball or tennis are played, or a stage where musical performances or plays are held. The imaging devices 10 do not need to be installed all around the imaging area; they may be installed only in one direction of the imaging area due to installation space limitations. Furthermore, imaging devices 10 with different functions, such as telephoto cameras and wide-angle cameras, may be installed together.

[0038] Shape estimation unit 101 distinguishes between foreground and background regions in multiple captured images acquired by imaging device 10, generates multiple silhouette images, and estimates the three-dimensional shape of the subject using the multiple silhouette images to generate a foreground model. Furthermore, shape estimation unit 101 generates texture data for coloring the foreground model based on the foreground and background regions of the silhouette images, and stores the generated foreground model and texture data in storage unit 102 for each frame.

[0039] The foreground region is the region of the captured image containing the subject for which the user desires to generate a foreground model, while the background region is the region of the captured image other than the foreground region. A silhouette image is a black-and-white image in which the subject is filled in. A silhouette image contains information about the subject's shape by representing the inside and outside of the subject's outline with binary information for each pixel. Here, if the subject for which a foreground model is to be generated is a moving object whose position and shape may change when images are captured from the same direction over time, the foreground and background regions can be separated using background subtraction processing. Even if the subject is not moving, machine learning such as a convolutional neural network (CNN) can be used to recognize and separate the foreground and background regions. In this way, a method appropriate for each object can be selected to separate the foreground and background regions. Examples of objects for which a foreground model can be generated include people such as players and referees on a stadium during a competition, sports equipment such as a ball or net, and singers, performers, actors, and emcees in a musical performance or play. The objects included in the background region include, for example, buildings such as the roof and seats of a stadium or stage where a performance or play is performed, structures such as goals used in ball games, and the floor of a stadium. The foreground model is the three-dimensional shape of the object estimated from the foreground region of the silhouette image, and a volume intersection method is used to generate the foreground model. Note that the shape of the foreground model may be estimated using any technology for acquiring three-dimensional shapes, such as a ToF (Time of Flight) camera or a stereo camera.

[0040] Data for generating a virtual viewpoint image is called material data. For example, material data includes a foreground model and texture data generated based on a captured image. The type of material data is not limited as long as it is data for generating a virtual viewpoint image. For example, the material data may include camera parameters that represent the imaging conditions of the imaging device 10. The image generation unit 103 generates a virtual viewpoint image by mapping the texture data to the foreground model stored in the storage unit 102 and rendering it according to viewpoint information (described later) input by the virtual camera control unit 106.

[0041] The display unit 104 displays the virtual viewpoint image generated by the image generation unit 103. The display unit 104 may be any device that can electrically display the virtual viewpoint image, and examples of the device include a display, a smartphone, and a head-mounted display.

[0042] Movement speed calculation unit 105 calculates the movement speed of the subject from the material data stored in storage unit 102, and transmits the movement speed of the subject to virtual camera control unit 106. For example, the movement speed of the subject is calculated as follows: in virtual space, the center point of a bounding box surrounding a foreground model, which is material data, is set as the coordinates of the subject, and by tracking the foreground model, the movement distance of the subject between frames is calculated, and this is divided by the time between frames to obtain the movement speed of the subject.

[0043] The movement speed of the subject does not have to include all directions, and only a component in a specific direction may be extracted as the movement speed. Furthermore, when multiple subjects are captured, the movement speed of each subject can be calculated as the subject for determining the playback speed, and the subject with the fastest movement speed can be selected and transmitted to the virtual camera control unit 106. Furthermore, a subject can be identified from the captured image using image recognition technology, and the subject whose movement speed is transmitted to the virtual camera control unit 106 can be set. Furthermore, a subject can be selected by combining image recognition and the movement speed of the subject. Furthermore, the movement speeds of multiple subjects can be transmitted to the virtual camera control unit 106.

[0044] The range including the subject whose movement speed is to be acquired does not have to be the entire imaging area of ​​the virtual viewpoint image. That is, the movement speed may be acquired only from the subject that is at a certain height or above within the imaging area or from the subject that is within a range delimited by a frame line.

[0045] The virtual camera control unit 106 sets viewpoint information to be used in generating a virtual viewpoint image and transmits the viewpoint information to the image generation unit 103 so that the virtual viewpoint image is generated from that viewpoint. Here, the viewpoint information is information indicating the position and orientation of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including parameters indicating the three-dimensional position in virtual space of the x-axis, y-axis, and z-axis, and the orientation in virtual space of the pan direction, tilt direction, and roll direction. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set as viewpoint information may include a parameter indicating the size of the field of view (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have multiple parameter sets. For example, the viewpoint information may have multiple parameter sets corresponding to multiple frames constituting a video of a virtual viewpoint image, respectively, and may be information indicating the position and orientation of the virtual viewpoint at each of multiple consecutive time points. Furthermore, the viewpoint information may be automatically designated based on the results of image analysis, etc.

[0046] Furthermore, the virtual camera control unit 106 can set viewpoint information based on input from the controller 110 by the operator. For example, if the controller 110 is a controller that uses two three-axis joysticks as a set, one joystick is assigned the movement directions of the x-axis, y-axis, and z-axis, and the other joystick is assigned the inclination directions of pan, tilt, and roll. This allows simultaneous operation of six axes. Note that virtual viewpoint images are not limited to images for which the user arbitrarily specifies the viewpoint and angle of view; for example, images corresponding to a viewpoint selected by the user from multiple options are also included in virtual viewpoint images.

[0047] Furthermore, the virtual camera control unit 106 receives the moving speed of the subject from the moving speed calculation unit 105, and is able to determine the playback speed of the virtual viewpoint video based on the moving speed of the subject. The detailed configuration is shown in FIG.

[0048] Furthermore, the moving speed calculation unit 105 and the virtual camera control unit 106 may not be included in the virtual viewpoint image generation device 100, but may be included in another information processing device.

[0049] 2 is a diagram showing the virtual camera control unit 106. The virtual camera control unit 106 includes a playback speed determination unit 201, a playback speed setting unit 202, a viewpoint information determination unit 203, and a virtual camera information transmission unit 204.

[0050] The playback speed determination unit 201 receives the movement speed of the subject calculated by the movement speed calculation unit 105 and determines the playback speed of the virtual viewpoint video that the operator views as a reference when operating the virtual camera, based on the setting of the playback speed setting unit 202. The virtual viewpoint video that the operator views as a reference is generated from the position and orientation of the virtual camera that the operator is operating. Note that there is a time lag between when the operator operates the virtual camera and changes the position and orientation of the virtual camera, and when a virtual viewpoint image generated using the position and orientation of the virtual camera is displayed; however, in this embodiment, the time lag is assumed to be very short. In other words, the operator can operate the virtual camera while viewing the virtual viewpoint video as seen from the virtual camera that he or she is operating.

[0051] The playback speed setting unit 202 sets how to slow down the playback speed, and how much to slow down the playback speed depending on how fast the subject is moving. These can be set, for example, in each of the items 501 to 503 on the playback speed setting screen 500 in FIG.

[0052] A viewpoint information determination unit 203 determines viewpoint information such as the position of the virtual camera, the shooting direction, the angle of view, and the time code based on information input by the operator through the controller 110 .

[0053] The virtual camera information transmission unit 204 acquires the playback speed of the virtual viewpoint video when the virtual camera is operated from the playback speed determination unit 201, and acquires the position of the virtual camera from the viewpoint information determination unit 203. It transmits information to the image generation unit 103 so that the image generation unit 103 generates a virtual viewpoint image using the acquired playback speed information and virtual camera position information.

[0054] In this embodiment, the playback speed refers to the playback speed of the virtual viewpoint video displayed when operating the virtual camera. Here, a method for changing the playback speed of the virtual viewpoint video will be described. The movement speed of the foreground model in the virtual space can be slowed by slowing down the update frequency of the foreground model placed in the virtual space. For example, when the playback speed is 100% and the virtual viewpoint video is generated at 60 fps, the foreground model is updated at 60 fps. In contrast, when the playback speed is 50%, the movement speed of the foreground model can be slowed by updating the foreground model at 30 fps when the virtual viewpoint video is generated at 60 fps. This makes the movement speed of the foreground model appear slower, which is perceived as a slower playback speed of the virtual viewpoint video. In this embodiment, slowing down the movement speed of the foreground model as described above is referred to as slowing down the playback speed of the virtual viewpoint video. In this case, the update frequency of the virtual camera may be slowed down to slow down the movement speed of the virtual camera in the virtual space. The frame rate of the virtual viewpoint video may be slowed without changing the movement speed of the foreground model. For example, if the playback speed is 100% and the virtual viewpoint video is generated at 60 fps, the playback speed may be set to 50% and the virtual viewpoint video may be generated at 30 fps.

[0055] 3 is a diagram illustrating a hardware configuration of a virtual viewpoint image generation device 100 according to Example 1. The virtual viewpoint image generation device 100 includes a CPU 311, a ROM 312, a RAM 313, an auxiliary storage device 314, a display unit 315, an operation unit 316, a communication I / F 317, and a bus 318.

[0056] The CPU 311 controls the entire virtual viewpoint image generation device 100 using computer programs and data stored in the ROM 312 and RAM 313, thereby realizing each function of the virtual viewpoint image generation device 100. Note that the virtual viewpoint image generation device 100 may have one or more dedicated hardware components different from the CPU 311, and at least a portion of the processing by the CPU 311 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). The ROM 312 stores programs that do not require modification. The RAM 313 temporarily stores programs and data supplied from the auxiliary storage device 314, as well as data supplied from the outside via the communication I / F 317. The auxiliary storage device 314 is configured, for example, by a hard disk drive or the like, and stores various data such as image data and audio data.

[0057] The display unit 315 is configured with, for example, a liquid crystal display, an LED (Light Emitting Diode), or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the virtual viewpoint image generation device 100. The operation unit 316 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and receives operations by the user to input various instructions to the CPU 311. The CPU 311 operates as a display control unit that controls the display unit 315 and an operation control unit that controls the operation unit 316.

[0058] The communication I / F 317 is used for communication between the virtual viewpoint image generation device 100 and an external device. For example, if the virtual viewpoint image generation device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 317. If the virtual viewpoint image generation device 100 has a function for wireless communication with an external device, the communication I / F 317 is equipped with an antenna. The bus 318 connects each unit of the virtual viewpoint image generation device 100 to transmit information.

[0059] In this embodiment, the display unit 315 and the operation unit 316 are assumed to exist inside the virtual viewpoint image generation device 100, but at least one of the display unit 315 and the operation unit 316 may exist as a separate device outside the virtual viewpoint image generation device 100. Note that the display unit 315 and the operation unit 316 may also exist as a separate device outside the virtual viewpoint image generation device 100.

[0060] 4 is a flowchart illustrating the processing performed by the playback speed determination unit 201 to determine the playback speed. The processing in steps S401 to S404 is performed when the playback speed designation method 501 specified by the playback speed setting unit 202 is not manual. The processing shown in FIG. 4 is repeated and is executed for each frame.

[0061] In step S401, the playback speed determination unit 201 acquires the moving speed of the subject calculated by the moving speed calculation unit 105.

[0062] In step S402, it is determined whether the moving speed of the subject exceeds a threshold value based on the playback speed designation method 501 and the value of the threshold value 502 designated by the playback speed setting unit 202. If the moving speed is equal to or less than the threshold value, the process of step S403 is executed, and if it is greater than the threshold value, the process of step S404 is executed.

[0063] In step S403, the playback speed is not changed and the playback speed determination process for the frame is terminated.

[0064] In step S404, the playback speed is changed based on the playback speed 503 setting in the playback speed designation method 501, and the playback speed determination process is terminated.

[0065] Note that slowing the playback speed may increase the difference between the time the subject is captured and the time the virtual viewpoint image is generated (hereinafter referred to as an increase in delay). For example, in the case of live streaming, in which a virtual viewpoint image generated by capturing a subject is immediately distributed, the difference between the time the subject was captured and the viewer's viewing time becomes large. As a result, the real-time sense of presence given to the viewer in the live streaming may be diminished, potentially reducing the appeal of live streaming. Therefore, when the delay becomes large, control may be added to reduce the delay by speeding up the playback speed if the subject's movement speed returns to below a threshold. Alternatively, control may be added to return the playback speed to normal playback speed when the delay becomes large, preventing the delay from increasing any further.

[0066] 5, a playback speed setting screen 500, which is a UI (User Interface) for inputting settings in the playback speed setting unit 202, will be described. The playback speed setting screen 500 includes a playback speed designation method 501, a threshold value 502, and a playback speed 503 to be set.

[0067] The playback speed can be specified in at least three modes: default, continuous, and manual, as playback speed specification method 501. In default and continuous modes, the values ​​of threshold 502 and playback speed 503 to be set can be specified based on the operator's settings.

[0068] The threshold value 502 is a threshold value for determining whether or not the moving speed of the subject calculated by the moving speed calculation unit 105 exceeds the reference for slowing down the playback speed.

[0069] A playback speed 503 is a value that sets the playback speed in a process of slowing down the playback speed when the moving speed of the subject exceeds a threshold value 502 .

[0070] When the playback speed designation method 501 is in the default mode, if the moving speed of the subject exceeds the threshold value input in the threshold value 502, the playback speed is immediately changed to the playback speed 503 that is set.

[0071] When playback speed specification method 501 is in continuous mode, two thresholds can be specified in threshold 502. When the smaller of the two thresholds is exceeded, the playback speed starts to slow down, and the faster the subject's movement speed becomes, the slower the playback speed becomes. When the subject's movement speed exceeds the larger threshold entered in threshold 502, the playback speed is set to the value entered in playback speed 503.

[0072] When the playback speed designation method 501 is in manual mode, the playback speed is not adjusted using the threshold value 502, and is always set to the value designated by the playback speed setting 503.

[0073] In addition, if the subject that the operator wishes to photograph with the virtual camera is different from the subject whose speed the movement speed calculation unit 105 acquires as a reference for changing the playback speed, manual may be selected in the playback speed designation method 501 to disable the function of automatically changing the playback speed.

[0074] 6 will be used to explain how to slow down the playback speed in the two modes, default and continuous, specified by playback speed specification method 501. Default mode playback speed change 600a shows the change in playback speed in default mode, and continuous mode playback speed change 600b shows the change in playback speed in continuous mode, with the vertical axis representing playback speed 601 and the horizontal axis representing subject movement speed 602. Default mode playback speed change 600a has threshold 603, and continuous mode playback speed change 600b has thresholds 604 and 605, and thresholds 603 to 605 can be set using threshold 502.

[0075] In the default mode playback speed change 600a, if the subject's moving speed is equal to or less than the threshold 603, the playback speed is 100%, but if the subject's moving speed exceeds the threshold 603, the playback speed is immediately set to the value set in the playback speed 503. In the graph of the default mode playback speed change 600a, the playback speed is switched to 25%.

[0076] In the continuous mode playback speed change 600b, if the subject's moving speed is equal to or less than the threshold 604, the playback speed is 100%, but if it exceeds the threshold 604, the playback speed begins to decrease, and the faster the subject's moving speed becomes, the more the playback speed decreases proportionally. If the subject's moving speed exceeds the threshold 605, the playback speed becomes constant thereafter, and becomes the value set in the set playback speed 503. In the graph of the default mode playback speed change 600a, it has switched to 15%.

[0077] As described above, this embodiment makes it possible to change the playback speed for operating the virtual camera, and simplifies the operation of photographing a fast-moving subject with the virtual camera.

[0078] <Example 2> In the first embodiment, a process for changing the playback speed of a virtual viewpoint video when the speed of a subject exceeds a threshold is described. In the second embodiment, a process for determining the playback speed in consideration of viewpoint information of a virtual camera is described.

[0079] The following description will be focused on a process for not changing the playback speed when the subject that the operator wishes to photograph with the virtual camera has already been photographed with the virtual camera.

[0080] For example, when generating a virtual viewpoint image of a volleyball, assume that the operator wishes to use the virtual camera to capture the ball spiked by a player or the estimated impact point of the ball based on the ball's trajectory. If the virtual camera is already capturing the ball or the impact point of the ball, there is little need to move the virtual camera. On the other hand, if the ball or the impact point of the ball is not captured, it is necessary to operate the virtual camera to capture that subject with the virtual camera. Therefore, in this embodiment, a process will be described in which the playback speed is not changed if the virtual camera has already captured the subject that the operator wishes to capture with the virtual camera, and the playback speed is changed if the subject is not captured.

[0081] The use of this embodiment is not limited to when the subject the operator wishes to photograph is not captured by the virtual camera. For example, the playback speed may be changed only when the subject the operator wishes to photograph is captured by the virtual camera, thereby preventing the playback speed from changing when the subject is not captured by the virtual camera.

[0082] In this embodiment, the same components as those in the first embodiment are denoted by the same reference numerals, and detailed description thereof will be omitted.

[0083] 7 is a system configuration diagram of a virtual viewpoint image generating device 700 according to Example 2. In the virtual viewpoint image generating device 700, a storage unit 702 has a configuration for transmitting position information of a subject to a virtual camera control unit 706 in addition to the functional configuration of the storage unit 102 of Example 1. Also, the virtual camera control unit 706 has a configuration for receiving position information of a subject from the virtual camera control unit 106.

[0084] By setting an object that is used as a reference for changing the playback speed in the movement speed calculation unit 105, the operator can set the object that he or she desires to photograph with the virtual camera in this system.

[0085] 8 is a diagram showing the virtual camera control unit 706 of the second embodiment. A subject position acquisition unit 805 and a determination unit 806 are added to the configuration of FIG.

[0086] In addition to the functions of the playback speed determination unit 201 in the first embodiment, when the playback speed determination unit 801 receives a determination result from the determination unit 806 that a subject is captured by the virtual camera, it can perform processing to prevent the playback speed from being changed even if the moving speed of the subject exceeds a threshold value.

[0087] In addition to the functions of the viewpoint information determination unit 203 in the first embodiment, the viewpoint information determination unit 803 transmits viewpoint information to the determination unit 806 .

[0088] The subject position acquisition unit 805 acquires the position information of the foreground model stored in the storage unit 702. If multiple foreground models are captured, the position information of all of the foreground models may be acquired. Also, models not included in the foreground model, such as structures, may be stored in advance in the storage unit 702, and their position information may be acquired.

[0089] The determination unit 806 acquires information on at least the virtual camera's position, angle of view, and shooting direction from the viewpoint information determination unit 803, and acquires position information of the foreground model from the subject position acquisition unit 805. Then, the determination unit 806 determines whether a subject that the operator wishes to capture with the virtual camera is captured, and transmits the determination result to the playback speed determination unit 801. At this time, if there are multiple subjects that the operator wishes to capture with the virtual camera, the determination to change the playback speed may be transmitted only if all of the subjects are not captured. Conversely, the determination to change the playback speed may be transmitted if one or more of the multiple subjects are not captured.

[0090] Note that even if the foreground model is within the imaging range of the virtual camera, if it is close to the boundary between the imaging range and outside the imaging range, the determination unit 806 may change the playback speed because there is a possibility that the foreground model will go outside the imaging range of the virtual camera. Also, even if the foreground model is captured by the virtual camera, calculations may be performed using the values ​​of the movement speed and movement direction of the foreground model for that frame, and it may be predicted that the foreground model will go out of the imaging range of the virtual camera within several frames, and the playback speed may be changed accordingly.

[0091] Fig. 9 is a flowchart for explaining the processing for determining the playback speed for virtual camera operation performed by the playback speed determination unit 801 in the second embodiment. The processing described in Fig. 9 is a repetitive processing that is executed for each frame. A branch of S905 is added to the configuration in Fig. 4. Also, in Fig. 9, S401 is changed to S901, S402 to S902, S403 to S903, and S404 to S904.

[0092] In step S901, the playback speed determination unit 801 acquires the moving speed of the subject calculated by the moving speed calculation unit 105.

[0093] In step S902, it is determined whether the moving speed of the subject exceeds a threshold value based on the playback speed designation method 501 and the value of the threshold value 502 designated by the playback speed setting unit 202. If the moving speed is equal to or less than the threshold value, the process of step S903 is executed, and if it exceeds the threshold value, the process of step S905 is executed.

[0094] In step S903, the playback speed is not changed and the playback speed determination process for the frame is terminated.

[0095] In step S904, the playback speed is changed based on the playback speed 503 setting in the playback speed designation method 501, and the playback speed determination process is terminated.

[0096] In step S905, based on the determination result of the determination unit 806, if the subject is captured by the virtual camera, the process of step S903 is executed, and if not, the process of step S904 is executed. In the above process, the playback speed can be changed only if the subject is not captured by the virtual camera.

[0097] When the subject is within the angle of view and the playback speed is changed, in step S905, if the subject is captured by the virtual camera, the processing of step S904 is executed, and if the subject is not captured, the processing of step S905 is executed.

[0098] When the delay from capturing an image of a subject to generating a virtual viewpoint image increases due to a slowdown in the playback speed, if the subject's moving speed returns to a threshold or below, control may be added to reduce the delay by speeding up the playback speed. Also, when the delay becomes too large, control may be added to return the playback speed to the normal playback speed so that the delay does not increase any further.

[0099] As described above, when a subject is captured by the virtual camera, processing can be performed without changing the playback speed.

[0100] A computer program that realizes all or part of the control in this embodiment and the functions of the above-described embodiment may be supplied to an image processing system or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the image processing system or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0101] The disclosure of the present embodiment includes the following configurations, methods, systems, and programs.

[0102] (Configuration 1) An information processing device that generates information indicating a trajectory of a virtual camera for generating a virtual viewpoint video, a specifying means for specifying a first speed which is a moving speed of a subject in a virtual viewpoint image being displayed; a control means for controlling the moving speed of the subject to a second speed slower than the first speed when the first speed is greater than a threshold value; An information processing device comprising:

[0103] (Configuration 2) 2. The information processing device according to configuration 1, wherein the moving speed of the subject is the first speed when the first speed is equal to or less than a threshold value.

[0104] (Configuration 3) The information processing device according to configuration 1, characterized in that the first speed is determined based on the position of the subject at a first time, which is the time of the virtual viewpoint image being displayed, and the position of the subject at a second time, which is a time before the first time.

[0105] (Configuration 4) 4. The information processing device according to configuration 3, wherein the position of the subject is a three-dimensional position in a virtual space.

[0106] (Configuration 5) the virtual viewpoint video is composed of a plurality of frames of virtual viewpoint images, 2. The information processing device according to configuration 1, wherein the information indicating the trajectory of the virtual camera is configured with information indicating the position and orientation of the virtual camera in each of the plurality of frames.

[0107] (Configuration 6) The information processing device according to configuration 1, wherein the control means performs control to display on the display means information indicating that control is performed to change the moving speed of the subject to the second speed when the first speed is greater than a threshold value.

[0108] (Configuration 7) the specifying means specifies a third speed which is a moving speed of a virtual camera corresponding to the virtual viewpoint image being displayed; 2. The information processing apparatus according to configuration 1, wherein the control means performs control to change the moving speed of the virtual camera to a fourth speed slower than the third speed when the first speed is greater than a threshold value.

[0109] (Configuration 8) 2. The information processing device according to configuration 1, wherein the virtual viewpoint video is generated based on a plurality of captured images.

[0110] (Configuration 9) 2. The information processing device according to configuration 1, wherein the threshold value is set based on a user operation.

[0111] (Configuration 10) 2. The information processing device according to configuration 1, wherein the second speed is set based on a user operation.

[0112] (method) An information processing method for generating information indicating a trajectory of a virtual camera for generating a virtual viewpoint video, comprising: a specifying step of specifying a first speed which is a moving speed of the subject within the virtual viewpoint image being displayed; a control step of controlling the moving speed of the subject to a second speed slower than the first speed when the first speed is greater than a threshold value; An information processing method comprising:

[0113] (program) 11. A program for causing a computer to function as the information processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0114] 100 Virtual viewpoint image generation device 105 Movement speed calculation section 106 Virtual camera control unit

Claims

1. An information processing device that generates information indicating a trajectory of a virtual camera for generating a virtual viewpoint video, a specifying means for specifying a first speed which is a moving speed of a subject in a virtual viewpoint image being displayed; a control means for controlling the moving speed of the subject to a second speed slower than the first speed when the first speed is greater than a threshold value; An information processing device comprising:

2. 2 . The information processing apparatus according to claim 1 , wherein the moving speed of the subject is the first speed when the first speed is equal to or less than a threshold value.

3. 2. The information processing device according to claim 1, wherein the first speed is determined based on a position of the subject at a first time, which is the time of the virtual viewpoint image being displayed, and a position of the subject at a second time, which is a time before the first time.

4. 4. The information processing apparatus according to claim 3, wherein the position of the subject is a three-dimensional position in a virtual space.

5. the virtual viewpoint video is composed of a plurality of frames of virtual viewpoint images, 2. The information processing apparatus according to claim 1, wherein the information indicating the trajectory of the virtual camera is composed of information indicating the position and orientation of the virtual camera in each of the plurality of frames.

6. 2. The information processing apparatus according to claim 1, wherein the control means controls the display means to display information indicating that control is being performed to change the moving speed of the subject to the second speed when the first speed is greater than the threshold value.

7. the specifying means specifies a third speed which is a moving speed of a virtual camera corresponding to the virtual viewpoint image being displayed; 2. The information processing apparatus according to claim 1, wherein the control means controls the moving speed of the virtual camera to a fourth speed slower than the third speed when the first speed is greater than a threshold value.

8. The information processing apparatus according to claim 1 , wherein the virtual viewpoint video is generated based on a plurality of captured images.

9. The information processing apparatus according to claim 1 , wherein the threshold value is set based on a user operation.

10. The information processing apparatus according to claim 1 , wherein the second speed is set based on a user operation.

11. An information processing method for generating information indicating a trajectory of a virtual camera for generating a virtual viewpoint video, comprising: a specifying step of specifying a first speed which is a moving speed of the subject within the virtual viewpoint image being displayed; a control step of controlling the moving speed of the subject to a second speed slower than the first speed when the first speed is greater than a threshold value; An information processing method comprising:

12. A program for causing a computer to function as the information processing device according to any one of claims 1 to 10.

Images