Image processing device, image processing method, and program

Abstract

To make virtual viewpoint images more attractive to a user.SOLUTION: An image processing device for generating virtual viewpoint images of an object is provided. The image processing device is configured to acquire a three-dimensional model of the object, acquire information indicative of the state of the object, and generate a virtual viewpoint image of the object according to the three-dimensional model of the object and the information indicative of the state of the object.SELECTED DRAWING: Figure 7

Description

[Technical field]

[0001] The present invention relates to an image processing device, an image processing method, and a program, and more particularly to displaying a virtual viewpoint image. [Background technology]

[0002] A technology that performs synchronized shooting using multiple image capturing devices arranged at different positions and generates a virtual viewpoint image from a specified virtual viewpoint using images from multiple viewpoints thus obtained is attracting attention. With this technology, for example, highlight scenes of a soccer or basketball game can be viewed from various angles. Therefore, compared to normal images, users can feel a high sense of realism.

[0003] There is also known a technique for estimating the posture of a person who is a subject by using images of the subject captured by multiple imaging devices. For example, in Non-Patent Document 1, a sport such as soccer is captured by multiple cameras. Also, two-dimensional posture information of a player in each captured image is calculated. Then, the multiple two-dimensional posture information is combined to estimate three-dimensional posture information of the player. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Lewis Bridgeman et al. "Multi-person 3d pose estimation and tracking in sports." Proceedings of the IEEE / CVF conference on computer vision and pattern recognition workshops. 2019. Summary of the Invention [Problem to be solved by the invention]

[0005] There is a demand for increasing the appeal of virtual viewpoint image content. For example, since a virtual viewpoint can be set, various virtual viewpoint images can be generated. In some virtual viewpoint images, it may not be easy to grasp the scene represented by the virtual viewpoint image. It is believed that the appeal of the virtual viewpoint image for users will increase by making it easier to grasp the scene represented by the virtual viewpoint image.

[0006] The present invention aims to make virtual viewpoint images more attractive to users. [Means for solving the problem]

[0007] An image processing apparatus according to an embodiment of the present invention includes the following components: An image processing device for generating a virtual viewpoint image of a subject, A model acquisition means for acquiring a three-dimensional model of the subject; An information acquisition means for acquiring information indicating a state of the subject; a generating means for generating a virtual viewpoint image of the subject according to a three-dimensional model of the subject and information indicating a situation of the subject; Equipped with. Effect of the Invention

[0008] This can make the virtual viewpoint image more attractive to the user. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of a virtual viewpoint video generation system according to an embodiment. [Diagram 2] FIG. 1 is a diagram showing an example of a camera arrangement. [Diagram 3] 4A and 4B are diagrams showing examples of a captured image and a foreground mask image. [Figure 4] FIG. [Diagram 5] FIG. 1 is a diagram illustrating a three-dimensional model. [Figure 6] 1A to 1C are diagrams illustrating a method of coloring a three-dimensional model. [Figure 7] FIG. 1 is a block diagram showing an example of the functional arrangement of an image processing apparatus according to an embodiment. [Figure 8] FIG. 4 is a diagram showing an example of posture information. [Figure 9] 4A to 4C are diagrams for explaining a method of estimating the posture of a subject. [Figure 10] FIG. 13 is a diagram showing an example of a virtual viewpoint image. [Figure 11] 11A and 11B are diagrams showing examples of virtual viewpoint images generated in accordance with subject information. [Figure 12] 11A and 11B are diagrams showing examples of virtual viewpoint images generated in accordance with subject information. [Figure 13] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing apparatus according to an embodiment. [Figure 14] 1 is a flowchart of an image processing method according to an embodiment. [Figure 15] 11A and 11B are diagrams showing examples of a user interface for selecting a display mode. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although the embodiments describe a plurality of features, not all of these features are necessarily essential, and the plurality of features may be combined in any manner. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (Overall composition) An embodiment of the present disclosure relates to a system for generating a virtual viewpoint image. Fig. 1 is a block diagram showing a configuration of an image generation system 1. The image generation system 1 includes a camera array 10, a foreground extraction device group 11, a control device 12, a model generation device 13, an image generation device 14, and a database 15.

[0012] The camera array 10 includes multiple cameras 10a-10r. The multiple cameras 10a-10r capture the subject from various angles. The cameras 10a-10r output images to corresponding foreground extraction device groups 11a-11r. The cameras 10a-10r can be arranged around a shooting area 20 as shown in FIG. 2. In the example of FIG. 2, the shooting area is shown as a rectangle. In this example, the shooting area is a sports field such as soccer. The camera array 10 captures the shooting area 20 from various angles at synchronized times. In this way, the subjects, people 30a-30j and a ball 30k, can be captured.

[0013] The foreground extraction device group 11 includes a plurality of foreground extraction devices 11a-11r. The foreground extraction devices 11a-11r receive captured images from the corresponding cameras 10a-10r. Then, the foreground extraction devices 11a-11r extract a foreground area in which the foreground exists from the captured images. The foreground refers to an object displayed in a virtual viewpoint image. In the following example, the foreground is a three-dimensional object such as people 30a-30j and a ball 30k that are subjects in a shooting area.

[0014] Specifically, the foreground extraction devices 11a-11r can extract a foreground mask image from a captured image. FIG. 3(A) shows an example of a captured image. FIG. 3(A) is a captured image obtained by the camera 10g. This captured image is a texture image representing the texture of a subject. This captured image represents the texture of the foreground, and is also referred to as a foreground texture image below. For example, this captured image may be a color image. That is, the color of each pixel of this captured image is represented by the three primary colors R (red), G (green), and B (blue).

[0015] FIG. 3(B) shows an example of a foreground mask image extracted from the captured image shown in FIG. 3(A) by the foreground extraction devices 11a-11r. The foreground mask image shows the foreground region in the captured image. In this example, the foreground mask image is a binary image. In the foreground mask image, the value of pixels included in the foreground region is 1 (white in FIG. 3(B)). Also, the value of pixels included in the background region is 0 (black in FIG. 3(B)). This foreground mask image shows the silhouette of the foreground.

[0016] The method of detecting the foreground region by the foreground extraction device 11a-11r is not particularly limited. For example, the foreground extraction device 11a-11r can detect the foreground region by comparing a background image stored in advance with a captured image. Specifically, the foreground extraction device 11a-11r can determine an area with a large luminance difference or color difference between the background image and the captured image as the foreground region.

[0017] The control device 12 outputs information indicating the positions and orientations of the cameras 10a-10r to the model generation device 13. The control device 12 can calculate camera parameters indicating the positions and orientations of each of the cameras 10a-10r from the captured images obtained by the camera array 10 at synchronized times.

[0018] The camera parameters may include extrinsic parameters and intrinsic parameters. The extrinsic parameters indicate the position and orientation of the camera. The extrinsic parameters may be composed of a rotation matrix and a translation matrix. The intrinsic parameters indicate the imaging conditions of the image captured by the camera. For example, the intrinsic parameters may include information indicating the settings of the optical system of the camera, such as the focal length or optical center of the camera. The intrinsic parameters may indicate the angle of view of the camera or the size of the imaging sensor.

[0019] The process of calculating the camera parameters is called calibration. For the calibration, a specific pattern such as a checkerboard can be captured using the cameras 10a-10r. The control device 12 can perform the calibration using a plurality of captured images thus obtained. For example, the control device 12 can perform the calibration using the correspondence between points in a three-dimensional space according to the world coordinate system and corresponding points on a two-dimensional captured image.

[0020] The model generating device 13 generates a three-dimensional model of the subject. The model generating device 13 can generate a three-dimensional model of each of the subjects constituting the foreground. The configuration of the three-dimensional model is not particularly limited. For example, the three-dimensional model may be a set of voxels. A voxel is, for example, a tiny cube. FIG. 5(A) shows an example of a single voxel. FIG. 5(B) shows a set of voxels showing the space for which the three-dimensional model is to be generated. FIG. 5(C) shows an example of the subject. FIG. 5(D) shows a three-dimensional model representing the subject. The three-dimensional model shown in FIG. 5(D) is obtained by approximating the shape of the subject shown in FIG. 5(C) by a set of voxels. The smaller the voxel, the more precisely the three-dimensional model can reproduce the subject.

[0021] The model generating device 13 acquires camera parameters from the control device 12. The model generating device 13 also acquires a foreground mask image and a foreground texture image from the foreground extraction device group 11. The model generating device 13 can generate a three-dimensional model based on the camera parameters and the foreground mask image. The method of generating the three-dimensional model is not particularly limited. For example, the model generating device 13 can generate a three-dimensional model using a volume intersection method. That is, the model generating device 13 can generate a three-dimensional model of a subject represented by a voxel set using a foreground mask image based on an image captured by the cameras 10a-10r using a volume intersection method. The model generating device 13 can output the generated three-dimensional model of the subject to the database 15. The model generating device 13 can also output the acquired foreground texture image and camera parameters to the database 15.

[0022] Here, the visual hull intersection method will be described with reference to Figs. 4(A) and (B). As shown in Fig. 4(A), a captured image corresponding to an imaging plane 3006 is obtained by capturing an image of an object 3001 with a camera 3002a. As described above, a foreground extraction device (not shown) corresponding to the camera 3002a generates a foreground mask image 3003 representing a two-dimensional silhouette of the object from the captured image. Here, a frustum extending in a three-dimensional space defined by lines extending from the imaging center of the camera 3002a toward each point on the contour of the foreground mask image 3003 arranged on the imaging plane 3006 is called a visual hull 3004 of the object 3001 by the camera 3002a. As shown in Fig. 4(B), a visual hull by each of the cameras 3002a, 3002b, and 3002c can be defined. A common area 3005 of these visual hulls corresponds to the three-dimensional shape of the object 3001. In this manner, a three-dimensional model of the object can be generated based on the common area 3005.

[0023] The database 15 stores data related to the object. For example, the database 15 may store a three-dimensional model of the object, a foreground mask image, a foreground texture image, and camera parameters.

[0024] The image generating device 14 generates a virtual viewpoint image of a subject. The image generating device 14 can acquire a three-dimensional model of the subject, a foreground texture image, and camera parameters from the database 15. The image generating device 14 can then generate a virtual viewpoint image of the subject using this information. For example, the image generating device 14 can generate a virtual viewpoint image by projecting the three-dimensional model of the subject and the foreground texture image onto a virtual viewpoint. The virtual viewpoint may be specified by a user. The virtual viewpoint may be any point in a three-dimensional space. The virtual viewpoint can be represented by the above-mentioned camera parameters.

[0025] The method of generating the virtual viewpoint image is not particularly limited. For example, the image generating device 14 can color the three-dimensional model of the subject. For example, as shown in FIG. 6, the image generating device 14 arranges a foreground texture image 604, which is an image captured by the camera 603, on the imaging plane of the camera 603 indicated by the camera parameters. This foreground texture image 604 is an image captured by the subject represented by the three-dimensional model 601. This foreground texture image 604 can be an image used by the model generating device 13 to generate the three-dimensional model 601. Furthermore, the image generating device 14 projects a voxel 602 constituting the three-dimensional model 601 of the subject onto the foreground texture image 604. Then, the image generating device 14 assigns the pixel value of a pixel 605 on the foreground texture image 604 corresponding to the position of the voxel 602 to the voxel 602. In this manner, the voxel is colored. The image generating device 14 can generate a colored three-dimensional model by coloring each voxel constituting the three-dimensional model as described above. However, the coloring of the three-dimensional model may be performed by the model generating device 13.

[0026] The image generating device 14 can generate a virtual viewpoint image by projecting a colored three-dimensional model onto a virtual viewpoint according to camera parameters. For example, the virtual viewpoint image can be generated according to a ray tracing method. Specifically, the pixel value of a pixel in the virtual viewpoint image can be determined according to the color of the three-dimensional model at the intersection of a line extending from the virtual viewpoint in a direction corresponding to the pixel and the three-dimensional model. The image generating device 14 may also add a background image to the virtual viewpoint image.

[0027] The image generating device 14 can output the virtual viewpoint image thus generated. The image generating device 14 may output the virtual viewpoint image to a display connected to the image generating device 14. The image generating device 14 may also output the virtual viewpoint image to an image display device such as a personal computer or a smartphone connected to the image generating device 14 via a network. For example, the image generating device 14 may be a server. In this case, the image generating device 14 can generate a virtual viewpoint image according to a request from a user terminal and deliver the virtual viewpoint image to the user terminal. The request from the user terminal may include information specifying the virtual viewpoint. The user can view the virtual viewpoint image on the user terminal.

[0028] In this way, the image generating device 14 generates a virtual viewpoint image of the subject according to a three-dimensional model of the subject. Meanwhile, the image generating device 14 generates such a virtual viewpoint image according to information indicating the situation of the subject. A specific method of generating the virtual viewpoint image will be described later.

[0029] The image generating system 1 can generate a virtual viewpoint video composed of a plurality of virtual viewpoint images. That is, the camera array 10 can capture images of a subject synchronously at a plurality of different times. The model generating device 13 can generate a three-dimensional model of the subject at each time based on a plurality of foreground mask images at each time obtained from a plurality of captured images at each time. Based on this information, a virtual viewpoint image at each time can be generated. The virtual viewpoint video includes the time-series virtual viewpoint images thus generated.

[0030] 1, the foreground extraction devices 11a-11r and the model generation device 13 are connected in a star topology. However, the connection format is not particularly limited. For example, the foreground extraction devices 11a-11r and the model generation device 13 may be connected in a ring topology using a daisy chain connection or in a bus topology.

[0031] (Image Generation Device) The configuration of the image generating device 14, which is an image processing device according to one embodiment, will be described below with reference to the block diagram of FIG.

[0032] The foreground data acquisition unit 101 acquires a three-dimensional model of the subject. The foreground data acquisition unit 101 can acquire the three-dimensional model of the subject from the database 15. The foreground data acquisition unit 101 can also acquire camera parameters of the cameras 10a-10r, a foreground texture image, and a foreground mask image from the database 15. Note that the foreground data acquisition unit 101 may generate a three-dimensional model of the subject like the model generation device 13.

[0033] The posture estimation unit 102 acquires posture information indicating the posture of the subject. For example, the posture estimation unit 102 can estimate the posture of the subject. The method of estimating the posture of the subject by the posture estimation unit 102 is not particularly limited. The posture estimation unit 102 can estimate the posture of the subject based on the captured images of the subject by each of the multiple cameras. In the following example, the posture estimation unit 102 estimates the posture of the subject, which is a person, based on the captured images (foreground texture images) by the cameras 10a-10r and the camera parameters acquired by the foreground data acquisition unit 101.

[0034] According to posture estimation, the posture of a person can be visualized. Data calculated by posture estimation is called posture information. The posture estimation unit 102 can estimate the posture of the subject based on the positions of the joints. The posture information is expressed, for example, by a collection of objects imitating skeletons called bones that connect the joints of the detected person. FIG. 8 shows bones 41 representing posture information about a person as an example of posture information. The bones 41 indicate skeletons corresponding to the waist, abdomen, chest, neck, head, shoulders, arms, hands, legs, and ankles of a person. The skeletons representing posture information are not limited to those shown above. For example, the skeletons representing posture information may be based on fingers or facial expressions.

[0035] The posture estimation unit 102 can detect two-dimensional or three-dimensional coordinates of features such as a person's joints, eyes, or nose. Then, the posture estimation unit 102 can generate posture information based on these coordinates. As a specific example, the posture estimation unit 102 can estimate a two-dimensional posture of the subject based on each captured image obtained by the cameras 10a-10r. Then, the posture estimation unit 102 can estimate a three-dimensional posture of the subject by combining the two-dimensional posture of the subject estimated from each captured image with the camera parameters of each camera 10a-10r.

[0036] 9(A) to 9(D) show an example of estimating a three-dimensional posture based on the estimation result of a two-dimensional posture based on a plurality of captured images. FIG. 9(A) is an overhead view showing the state of capturing an image of a subject 4002 using two cameras 4001a and 4001b. FIG. 9(B) shows the estimation result of the position of a joint 4003a, which is a characteristic point of the subject 4002, in the image captured by the camera 4001a. In the example of FIG. 9(B), the head, chin, neck, chest, abdomen, waist, shoulders, elbows, wrists, thighs, groin, knees, and ankles are detected as joints. FIG. 9(C) similarly shows the estimation result of the position of a joint 4003b, which is a characteristic point of the subject 4002, in the image captured by the camera 4001b. The method of estimating the position of the joint is not particularly limited. For example, markers can be attached to the joints of a person in advance. In this case, the position of the joint can be detected based on the position of the marker detected from the captured image. In this way, the posture of the subject in the two-dimensional captured image can be estimated.

[0037] FIG. 9(D) shows a method for calculating the three-dimensional positions of the subject's joints based on the images captured by the two cameras 4001a and 4001b. As shown in FIG. 9(D), the three-dimensional position of the subject's chest can be geometrically estimated using triangulation based on the positions of the cameras 4001 and 4001b in three-dimensional space and the estimated position of the subject's chest on the captured images. The calculation of the three-dimensional position by triangulation can be performed according to a normal method. By performing such processing for each feature point, the three-dimensional positions of the feature points of the subject can be estimated, and therefore the three-dimensional posture of the subject can be estimated.

[0038] However, the method of estimating the posture of the subject is not limited to the above method. For example, instead of using markers, the posture estimation unit 102 may estimate the positions of the joints based on the feature points of the person detected from the captured image. In addition, the posture estimation unit 102 may estimate the posture of the subject using a distance sensor or the like.

[0039] The information acquisition unit 103 acquires information indicating the state of the subject (hereinafter referred to as subject information). The subject information is information determined based on, for example, the position, shape, or appearance of the subject. The information acquisition unit 103 can generate the subject information based on a captured image of the subject. However, the information acquisition unit 103 may acquire subject information that has been generated in advance.

[0040] In this embodiment, the information acquisition unit 103 generates subject information based on the estimation result of the subject's posture. That is, the information acquisition unit 103 acquires the posture information calculated by the posture estimation unit 102, and generates subject information based on the posture information. As described above, the posture information is generated based on the captured images representing the position, shape, and appearance of the subject by each of the multiple cameras. In addition, the posture information itself also represents the position and shape of the subject. In this way, the information acquisition unit 103 can generate subject information based on the position, shape, or appearance of the subject using the posture information. Then, the information acquisition unit 103 outputs the acquired subject information to the image generation unit 104.

[0041] The subject information in this embodiment is information (hereinafter referred to as statistical information) calculated based on the position, posture, shape, or appearance of the subject over multiple times. Examples of the statistical information include tracking information, motion analysis information, and event information.

[0042] The tracking information is information that specifies the position of the subject in time series. For example, the tracking information may be an ID that can uniquely identify a player participating in a match. Such an ID may be attached to the posture information of the subject at each time calculated by the posture estimation unit 102. The information acquisition unit 103 can determine posture information at a past time that corresponds to the posture information at a new time. For example, the information acquisition unit 103 can associate the posture information at a new time with the posture information at a past time based on the proximity of the position of the subject represented by the posture information or based on the movement of the subject represented by the posture information. Then, the information acquisition unit 103 can assign the same ID to the posture information at the new time as the posture information at the corresponding past time. In this way, the information acquisition unit 103 can assign the same ID to the same subject by determining the subject represented by the posture information based on the posture information. The tracking information may also be information that indicates the movement trajectory of the subject. The information acquisition unit 103 can generate such a movement trajectory based on the above-mentioned association of posture information.

[0043] Event information is information indicating an event that has occurred with respect to a subject. Event information may be information indicating a movement, a change in posture, a change in shape, or a classification result of a situation of a subject. Event information may be, for example, information indicating that a specific movement or change in posture has occurred with respect to a subject. Furthermore, for example, event information about a sport may be information indicating that a goal has been scored, information indicating that a foul has been committed, or information indicating that a specific play has been performed. As a specific example, event information about soccer may be information indicating that a shot, a goal, or a penalty kick has occurred.

[0044] Such event information can be generated according to a time change in the movement or posture information of one or more subjects. For example, the information acquisition unit 103 can determine that a specific event has occurred by comparing a time change in the posture information of one or more subjects with a pattern of a posture information change that is predefined for an event. For example, the information acquisition unit 103 can determine that a shot has occurred by comparing a time change in the foot position indicated by the posture information of the player and a time change in the ball position indicated by the posture information of the ball with a pattern of a posture information change corresponding to a shot.

[0045] An example of event information is injury risk information. The injury risk information is information indicating that a risk of injury has been detected for a subject such as a person. The information acquisition unit 103 can detect the risk of injury, for example, when a specific movement occurs for the subject. The postures with a high risk of injury and a low risk of injury can be predetermined according to the structure of a person's joints. For example, the risk of injury to the foot can be determined based on the positions of the ankle and knee. The information acquisition unit 103 can detect the risk of injury by comparing the posture information of the person with a predetermined posture. The information acquisition unit 103 may take into account the speed of change in posture to detect the risk of injury. For example, the information acquisition unit 103 can determine the presence or absence of a risk of injury to the foot based on the angle and speed of the ankle and knee indicated by the posture information and a predetermined threshold value. The information acquisition unit 103 may also determine the magnitude of the risk of injury.

[0046] The motion analysis information is information indicating an analysis result of the motion of the subject. For example, the motion analysis information may be information indicating a statistical value regarding the motion of the subject. Specifically, the motion analysis information may be information indicating the moving speed of the subject. For example, the motion analysis information may indicate an average moving speed or a maximum moving speed within a predetermined period (for example, during a match). Furthermore, the motion analysis information may indicate a moving distance, a moving time, or a number of times or time of moving exceeding a threshold speed within a predetermined period (for example, during a match). The information acquisition unit 103 may calculate such motion analysis information based on the tracking information of the subject corresponding to a player or the time transition of posture information corresponding to a player. Furthermore, the motion analysis information may indicate the number of times the subject performs a specific motion (for example, a shot) within a predetermined period. The information acquisition unit 103 may calculate such motion analysis information based on event information. Furthermore, the motion analysis information may indicate the degree of fatigue of the subject. The degree of fatigue of the subject may be calculated based on, for example, a comparison between a moving speed within a past predetermined period and a moving speed within a recent predetermined period.

[0047] The image generating unit 104 generates a virtual viewpoint image of the subject according to a three-dimensional model of the subject. The virtual viewpoint image includes an object according to the three-dimensional model of the subject. In this specification, the object in the virtual viewpoint image may be simply referred to as the subject. In this embodiment, the image generating unit 104 generates a virtual viewpoint image of the subject according to the three-dimensional model of the subject and information indicating the situation of the subject. For example, the image generating unit 104 can generate a virtual viewpoint image having a display form according to the subject information about the subject shown in the virtual viewpoint image. As a specific example, the image generating unit 104 may generate the virtual viewpoint image while performing different processing according to the situation of the subject. As another specific example, the image generating unit 104 may generate the virtual viewpoint image so as to include information indicating the situation of the subject. Then, the image generating unit 104 can output the virtual viewpoint image thus obtained. Note that, when generating a virtual viewpoint image of the subject for a specific time, the image generating unit 104 can perform processing according to the situation of the subject before or after the specific time.

[0048] As already described, the image generating unit 104 can color the three-dimensional model. Also, as already described, the image generating unit 104 can generate a virtual viewpoint image according to the colored three-dimensional model and the virtual viewpoint. FIG. 10 shows an example of a virtual viewpoint image generated in this way. When generating the virtual viewpoint image shown in FIG. 10, no processing according to the subject information is performed. Note that the virtual viewpoint image may include an object according to a three-dimensional model of a subject (e.g., a player) that follows a real appearance and shape. For the sake of explanation, a subject with a simplified shape is shown in the virtual viewpoint image shown in FIG. 10. On the other hand, the virtual viewpoint image may include an object corresponding to a three-dimensional model of a subject that differs from the real appearance and shape of the subject.

[0049] In one embodiment, the image generating unit 104 changes the display of the subject according to the subject information. The change in the display of the subject may be a change in the display of the subject itself, or a change in the display around the subject. For example, the change in the display of the subject may be a change in the size of the subject, a change in the color of the subject, a change in the color around the subject, or an addition of an object to the vicinity of the subject. The image generating unit 104 may change the display of the subject when the subject information satisfies a specific condition. Examples of the specific condition include a specific movement such as a shot, a specific event such as a goal, or a movement speed or a total travel distance exceeding a threshold. The image generating unit 104 may also change the display of the subject according to a specific parameter indicated by the subject information (for example, according to motion analysis information such as a movement speed or a total travel distance). In this way, the image generating unit 104 can change the process used to generate a virtual viewpoint image according to the subject information.

[0050] The image generating unit 104 can change the display of a specific subject based on the subject information of the specific subject. The subject information of the specific subject can be, for example, motion analysis information on the motion of the specific subject, or event information on the movement performed by the specific subject. For example, when the specific subject performs a specific movement such as a shot, the image generating unit 104 can change the display of the specific subject. Also, the image generating unit 104 can change the display of a subject related to a specific event such as a goal (for example, a player who has scored a goal). For example, when a specific event has occurred in the past, is currently occurring, or will occur in the future, the image generating unit 104 can change the display of a subject related to the event. Note that a method of determining a subject related to a specific event may be defined in advance.

[0051] For example, the image generating unit 104 can change the appearance of the subject according to the subject information. Specifically, the image generating unit 104 can change the size of the three-dimensional model of the subject according to the subject information. Specifically, the image generating unit 104 can enlarge the three-dimensional model of the subject when the subject information satisfies a specific condition. Also, as shown in FIG. 11(A), the image generating unit 104 can enlarge the three-dimensional model of the subject according to the subject information (for example, according to the motion analysis information). The image generating unit 104 can change the size of the three-dimensional model of the subject by changing the size of the voxels of the three-dimensional model of the subject.

[0052] As another example, the image generating unit 104 can change the color of the subject according to the subject information. For example, the image generating unit 104 can change the color to be assigned to the voxel according to the subject information. Specifically, the image generating unit 104 may change the color of the subject according to the subject information (for example, according to the motion analysis information). Also, the image generating unit 104 may change the color of the subject when the subject information satisfies a specific condition. As another method, the image generating unit 104 may give a light-emitting effect to the subject according to the subject information (for example, when the subject information satisfies a specific condition). In this case, the image generating unit 104 can generate a virtual viewpoint image taking into account the light emitted from the subject. In this case, the subject and the surroundings of the subject become brighter in the virtual viewpoint image.

[0053] Furthermore, the image generating unit 104 can change the appearance of the surroundings of the subject according to the subject information. For example, the image generating unit 104 can change the color of the surroundings of the subject according to the subject information. Specifically, the image generating unit 104 can darken the surroundings of the subject according to the subject information (for example, when the subject information satisfies a specific condition). For example, the image generating unit 104 can darken the background around the subject so that the subject stands out.

[0054] As another example, the image generating unit 104 can add an object to the virtual viewpoint image according to the subject information. The object may be a three-dimensional object or a two-dimensional object. The image generating unit 104 can add an object related to a specific subject based on the subject information about the specific subject. Such an object can be placed in the vicinity of the specific subject.

[0055] For example, the image generating unit 104 can add a three-dimensional object or a two-dimensional object around the subject according to the subject information. Specifically, the image generating unit 104 can add a production voxel around the three-dimensional model of the subject when the subject information satisfies a specific condition (for example, when there is a specific movement such as a shot, or when the moving speed exceeds a threshold). For example, the image generating unit 104 can add a star-shaped three-dimensional object around the three-dimensional object of the subject according to the subject information (for example, according to the moving speed). FIG. 11(B) shows a virtual viewpoint video thus generated. The number of objects added may be increased according to the subject information (for example, according to the moving speed).

[0056] Furthermore, the image generating unit 104 can add an object showing information showing the situation of the subject. This object can be, for example, a two-dimensional information display area. In this way, the image generating unit 104 can superimpose information showing the situation of the subject on the virtual viewpoint image. For example, the statistical information as described above can be used to provide information to a user or to analyze a team's tactics. For this reason, the image generating unit 104 can add an object showing the statistical information to the virtual viewpoint image.

[0057] The image generating unit 104 can add an object indicating information about a specific subject to the vicinity of the specific subject. The object can indicate, for example, motion analysis information. In the example of FIG. 12(B), the speed, fatigue level, and running time of the subject are displayed in an information display area located near the subject. Also, in the example of FIG. 12(B), an object according to tracking information of the subject is added to the vicinity of the subject. The object indicates the movement trajectory of the ball. In this way, the image generating unit 104 can superimpose information indicating the movement trajectory of the subject on the virtual viewpoint image. Note that the object may indicate other information about the subject. For example, as shown in FIG. 12(B), the object may include information that identifies the subject, such as the name of a player. The information acquiring unit 103 may identify the subject by image recognition.

[0058] The image generating unit 104 may superimpose the object described above at a position where it does not overlap with the foreground. With this configuration, it becomes easier to visually grasp useful information without hiding the foreground.

[0059] As in the above example, a noteworthy subject can be presented to the user as a dramatic effect by enlarging the subject, giving the subject a conspicuous color, or adding an object near the subject. With such a configuration, the scene represented by the virtual viewpoint image can be more easily grasped, and the appeal of the virtual viewpoint image is improved. Furthermore, by generating a virtual viewpoint image according to the above statistical information, the user can easily visually recognize the statistical information. By referring to the statistical information, the scene can be more easily grasped, and the appeal of the virtual viewpoint image is further improved.

[0060] So far, the case where statistical information is generated based on posture information has been described. However, the method of generating statistical information or subject information is not particularly limited. For example, the statistical information or subject information may be generated based on a three-dimensional model of the subject. Here, the subject information may be information indicating a change in the shape of the subject. As a specific example, the information acquisition unit 103 may calculate the amount of deformation of the shape of the subject. An example of the amount of deformation is the amount of displacement of the shape of a ball used in a ball game. A larger amount of displacement from a sphere indicates a stronger force applied to the ball. As an example, the image generation unit 104 can change the display related to the ball in the same manner as the above example based on the amount of displacement of the shape of the ball.

[0061] So far, the case where the subject information is statistical information has been described. However, the subject information is not limited to statistical information. For example, the subject information or the above-mentioned event information may be information indicating that the position, posture, shape, or appearance of the subject satisfies a predetermined condition. Furthermore, the subject information may be posture information. That is, the image generating unit 104 may generate a virtual viewpoint image according to the posture information.

[0062] For example, the image generating unit 104 may add an object to the virtual viewpoint image according to the posture information. For example, the image generating unit 104 may superimpose an object according to posture information, which indicates the posture of a specific subject, on a specific subject. FIG. 12(A) shows an example of a virtual viewpoint image in which an object indicating a posture is superimposed on the virtual viewpoint image. In FIG. 12(A), a bone according to the posture information is superimposed on the virtual viewpoint image. With this configuration, the user can easily visually grasp the posture of a player in a goal scene or the like, thereby improving the appeal of the virtual viewpoint image.

[0063] 13 is a block diagram showing an example of a hardware configuration of the image generating device 14 according to an embodiment. The CPU 111 is a central processing unit. The CPU 111 controls the entire image generating device 14 by executing a program stored in the ROM 112 or the RAM 113. For example, the CPU 111 can load a predetermined program stored in the storage device 114 into the RAM 113 and execute the program loaded into the RAM 113.

[0064] The ROM 112 is a read-only non-volatile memory. The RAM 113 is a memory that can be read and written at any time. For example, a dynamic random access memory (DRAM) can be used as the RAM 113. The storage device 114 is a large-capacity storage device such as a hard disk.

[0065] The input / output unit 115 accepts various user inputs to the image generating device 14. The input / output unit 115 can also output various information. The input / output unit 115 can include, for example, one or more of a keyboard, a mouse, a display, and a touch panel. The communication unit 116 controls communication between the image generating device 14 and an external device. In this embodiment, the control device 12 and the database 15 are connected to the communication unit 116. The communication unit 116 can also be connected to an external device via a network such as a LAN.

[0066] In this way, the image processing device according to an embodiment can be realized by a computer including a processor and a memory. That is, a processor such as a CPU 111 executes a program stored in a memory such as a ROM 112 or a RAM 113, thereby realizing the functions of each unit shown in FIG. 7 and the like. On the other hand, each functional unit of the image generating device 14 may be realized by cooperation between software and hardware such as a dedicated IC. Also, some or all of the functional units of the image generating device 14 may be realized only by hardware. Also, the image generating device 14 according to an embodiment may be configured by a plurality of information processing devices connected via a network, for example. That is, the processing of each functional unit of the image generating device 14 may be distributed and executed in a plurality of information processing devices. In this way, the functions of the image generating device 14 may be provided as a cloud service.

[0067] Fig. 14 shows a processing flow of an image processing method performed by the image generating device 14 in one embodiment. According to the processing in Fig. 14, a virtual viewpoint image corresponding to each time is generated. The virtual viewpoint image generated in this manner corresponds to each frame of the virtual viewpoint video.

[0068] In S201, foreground data acquisition unit 101 acquires camera parameters for all cameras 10a-10r from database 15. In S202, foreground data acquisition unit 101 acquires a three-dimensional model of the subject, a foreground mask image, and a foreground texture image from database 15, as described above. In S201 and S202, foreground data acquisition unit 101 acquires these data for the new time. If data for the new time cannot be acquired in S203, the processing in FIG. 14 ends.

[0069] If data for a new time can be acquired in S203, the process proceeds to S204. In S204, the posture estimation unit 102 calculates posture information of the subject as described above. In S205, the information acquisition unit 103 acquires subject information as described above. The information acquisition unit 103 may refer to information of the subject at past or future times (e.g. posture information, a three-dimensional model, or other information) to calculate statistical information. In S206, the image generation unit 104 generates a virtual viewpoint image based on the subject information as described above. In S207, the image generation unit 104 outputs the image generated in S206. Thereafter, the process returns to S202.

[0070] As described above, by generating a virtual viewpoint image according to information indicating the situation of a subject, it is possible to increase the attractiveness of the virtual viewpoint image for a user. For example, according to some embodiments, it is possible to make it easier for a user to grasp the situation of a subject. Also, according to some embodiments, it is possible to generate an attractive virtual viewpoint image to which a dramatic effect is added.

[0071] (Select display mode) As described above, the image generating unit 104 can generate a virtual viewpoint image while using various processes according to the subject information. Moreover, the above processes can be used in combination. Furthermore, the process used to generate the virtual viewpoint image may be selectable by the user. For example, the user can select which process to use. Furthermore, the user can select what subject information should be referred to in order to control the process. The user may select one display mode from a plurality of display modes. In each of the plurality of display modes, a virtual viewpoint image is generated using the subject information by different processes. That is, in each of the plurality of display modes, a virtual viewpoint image in a different display form is displayed. The virtual viewpoint image displayed in each display mode may be, for example, the virtual viewpoint image shown in FIG. 11(A), FIG. 11(B), FIG. 12(A), or FIG. 12(B).

[0072] For example, in the user interface shown in Fig. 15, in addition to the virtual viewpoint image, a group of buttons 5001 for the user to select a display mode are arranged. By selecting button 5001a, a display mode in which the color of the subject is changed (or the subject is illuminated) according to the subject information is selected. By selecting button 5001b, a display mode in which the size of the subject is changed according to the subject information is selected. By selecting button 5001c, a display mode in which voxels are added around the subject for effect is selected. By selecting button 5001d, a display mode in which information about the subject, such as a moving speed, is displayed is selected.

[0073] The image generation unit 104 can output such a user interface to the user, for example, via the input / output unit 115 or the communication unit 116. Furthermore, the image generation unit 104 can receive a user input for selecting a display mode, for example, via the input / output unit 115 or the communication unit 116. Then, the image generation unit 104 can generate a virtual viewpoint image according to the display mode selected by the user in this manner.

[0074] The multiple display modes may include a display mode that outputs a virtual viewpoint image as shown in Fig. 10. In this case, the image generating unit 104 does not generate a virtual viewpoint image based on the subject information. In this case, the processing by the posture estimating unit 102 and the information acquiring unit 103 can be omitted.

[0075] (Granting rights corresponding to subject information or display mode) Only when the user has a right corresponding to the specific display mode, a virtual viewpoint image according to the specific display mode may be output. For example, the image generating unit 104 can search for a display mode for which the user has a right. Then, the image generating unit 104 can generate a virtual viewpoint image according to the display mode for which the user has a right. In this case, only buttons corresponding to the display modes for which the user has a right may be displayed in the button group 5001.

[0076] The image generating unit 104 can also receive a request for output of a virtual viewpoint image corresponding to a specific display mode from a user or a user terminal. At this time, the image generating unit 104 can determine whether or not the user has a right corresponding to the specific display mode. In response to the determination that the user has a right corresponding to the specific display mode, the image generating unit 104 can generate a virtual viewpoint image according to the subject information corresponding to this display mode. That is, in response to the determination that the user has a right corresponding to the specific display mode, the image generating unit 104 can generate a virtual viewpoint image according to the specific display mode. For example, the button group 5001 may include a button corresponding to a display mode to which the user has a right and a button corresponding to a display mode to which the user does not have a right. In this case, the image generating unit 104 can determine whether or not the user has a right corresponding to the selected display mode. In addition, if the user does not have a right corresponding to the selected display mode, the image generating device 14 may start a process for the user to acquire a right corresponding to this display mode. In one embodiment, if the user does not have a right corresponding to the display mode, a virtual viewpoint image as shown in FIG. 10 is output.

[0077] Also, the user may have a right to the subject information, for example, the statistical information or the posture information. For example, the user may have a right to a specific type of subject information, or a right to subject information about a specific subject. Then, only when the user has a right corresponding to the specific subject information, the virtual viewpoint image generated as described above according to the subject information may be output. In one embodiment, the image generating unit 104 can determine whether or not the user has a right corresponding to the specific subject information. Then, in response to the determination that the user has a right corresponding to the specific subject information, the image generating unit 104 can generate a virtual viewpoint image of the subject as described above according to the specific subject information. Note that the right corresponding to the specific display mode may be a right to the subject information used for generating a virtual viewpoint image in the specific display mode. That is, in response to the determination that the user has a right corresponding to the specific subject information, the image generating unit 104 can generate a virtual viewpoint image according to a display mode in which a virtual viewpoint image corresponding to the specific subject information is output.

[0078] A user may purchase a right to specific subject information. Similarly, a user may purchase a right corresponding to a specific display mode. For example, a user may purchase a right corresponding to a display mode in which a staged object is added around a subject, thereby enabling the user to view an attractive virtual viewpoint image such as that shown in FIG. 11(A)-(B) or FIG. 12(A)-(B).

[0079] Such rights to specific object information or rights corresponding to a specific display mode can be managed using an NFT. NFT stands for Non-Fungible-Token. An NFT is a token that expresses ownership of a unique item in digital data. Codes such as item information and ownership information can be assigned to the NFT. The NFT is stored on a blockchain. In this case, an NFT corresponding to object information or a display mode can be issued. Such an NFT may be issued for each of various types of object information. For example, an NFT may be issued for tracking information, motion analysis information, event information, injury risk information, movement speed, or information indicating that a shot has been taken. In addition, such an NFT may be issued for each of various objects (for example, each player or ball). A user can acquire rights corresponding to object information or a display mode corresponding to this NFT by purchasing an NFT. In this example, a virtual viewpoint image generated according to specific object information or a virtual viewpoint image according to a specific display mode is provided only to a user who has an NFT.

[0080] Blockchain is a technology that uses encryption technology to synchronize and record data on multiple computers that make up a distributed network. In blockchain, data generated within a certain period of time is organized into blocks. In addition, data is verified by each computer. In this way, correct records are linked like a chain. In such a blockchain, even if some computers falsify data, the correct data is selected by majority vote of the computers, so data falsification can be prevented.

[0081] In such a configuration, the image generating unit 104 can determine whether or not the user owns an NFT corresponding to specific subject information. Such verification can be performed according to a normal blockchain technology. In order to generate a virtual viewpoint image, the image generating unit 104 can generate a virtual viewpoint image according to the subject information in response to a determination that the user owns an NFT corresponding to specific subject information, and output the generated virtual viewpoint image to the user. Similarly, in response to a determination that the user owns an NFT corresponding to a specific display mode, the image generating unit 104 can generate a virtual viewpoint image according to the display mode, and output the generated virtual viewpoint image to the user. The image generating unit 104 can issue such an NFT. In addition, an NFT management unit (not shown) included in the image generating device 14 or the image generating system 1 can issue such an NFT.

[0082] According to this configuration, the right to view an attractive virtual viewpoint image can be treated as an asset.

[0083] (Other Examples) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0084] The disclosure of this specification includes the following image processing device, image processing method, and program. (Item 1) An image processing device for generating a virtual viewpoint image of a subject, A model acquisition means for acquiring a three-dimensional model of the subject; An information acquisition means for acquiring information indicating a state of the subject; a generating means for generating a virtual viewpoint image of the subject according to a three-dimensional model of the subject and information indicating a situation of the subject; An image processing device comprising: (Item 2) 2. The image processing device according to item 1, further comprising a posture estimation unit that estimates a posture of the subject based on images of the subject captured by each of a plurality of cameras. (Item 3) 3. The image processing device according to item 2, wherein the subject is a person, and the posture estimation means estimates the posture of the subject based on positions of the person's joints. (Item 4) 4. The image processing device according to any one of items 1 to 3, wherein the information acquisition means generates information indicating a state of the subject based on a captured image of the subject. (Item 5) 5. The image processing device according to any one of items 1 to 4, wherein the information acquisition means generates information indicating a state of the subject based on an estimation result of the posture of the subject. (Item 6) 6. The image processing device according to any one of items 1 to 5, characterized in that the information indicating the state of the subject is information calculated based on changes in the position, posture, shape, or appearance of the subject over multiple times. (Item 7) 7. The image processing device according to any one of items 1 to 6, wherein the information indicating the state of the subject is information indicating an event that has occurred regarding the subject. (Item 8) 8. The image processing device according to any one of items 1 to 7, wherein the information indicating the state of the subject is information indicating the movement, posture change, shape change, or state classification result of the subject. (Item 9) 9. The image processing device according to any one of items 1 to 8, wherein the information indicating the state of the subject is information indicating that a specific movement or posture change has occurred with respect to the subject. (Item 10) 7. The image processing device according to any one of items 1 to 6, wherein the information indicating the state of the subject is information indicating an analysis result of the movement of the subject. (Item 11) 7. The image processing device according to any one of items 1 to 6, wherein the information indicating the state of the subject is information indicating a moving speed or a deformation amount of the shape of the subject. (Item 12) 7. The image processing device according to any one of items 1 to 6, wherein the information indicating the state of the subject is information specifying a time-series position of the subject or information indicating a posture of the subject. (Item 13) 13. The image processing device according to any one of items 1 to 12, wherein the generating means changes a display relating to the subject in the virtual viewpoint image according to information indicating a situation of the subject. (Item 14) 14. The image processing device according to any one of items 1 to 13, wherein the generating means changes a size or a color of the subject in the virtual viewpoint image according to information indicating the situation of the subject. (Item 15) 14. The image processing device according to any one of items 1 to 13, wherein the generating means adds an object around the subject in accordance with information indicating a situation of the subject. (Item 16) 14. The image processing device according to any one of items 1 to 13, wherein the generating means superimposes information indicating a situation of the subject on the virtual viewpoint image. (Item 17) The image processing device described in any one of items 1 to 16, characterized in that the generation means determines whether or not a user has a right corresponding to a display mode, and based on a determination that the user has a right corresponding to the display mode, generates a virtual viewpoint image of the subject according to information indicating a status of the subject that corresponds to the display mode. (Item 18) The image processing device described in any one of items 1 to 17, characterized in that the generation means determines whether or not a user owns an NFT corresponding to the information indicating the situation of the subject, and generates a virtual viewpoint image of the subject according to the information indicating the situation of the subject based on a determination that the user owns an NFT corresponding to the information indicating the situation of the subject. (Item 19) 1. An image processing method for generating a virtual viewpoint image of a subject, comprising: obtaining a three-dimensional model of the object; acquiring information indicative of a state of the subject; generating a virtual viewpoint image of the subject according to a three-dimensional model of the subject and information indicating a situation of the subject; 13. An image processing method comprising: (Item 20) 19. A program for causing a computer to function as the image processing device according to any one of items 1 to 18.

[0085] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. Accordingly, the following claims are appended to disclose the scope of the present disclosure. [Explanation of symbols]

[0086] 14: Image generating device, 101: Foreground data acquisition unit, 102: Pose estimation unit, 103: Information acquisition unit, 104: Image generating unit

Claims

1. An image processing device for generating a virtual viewpoint image of a subject, A model acquisition means for acquiring a three-dimensional model of the subject; An information acquisition means for acquiring information indicating a state of the subject; a generating means for generating a virtual viewpoint image of the subject according to a three-dimensional model of the subject and information indicating a situation of the subject; An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, further comprising: a posture estimation unit for estimating a posture of the subject based on images of the subject captured by each of a plurality of cameras.

3. 3. The image processing apparatus according to claim 2, wherein the subject is a person, and the posture estimation means estimates the posture of the subject based on positions of the joints of the person.

4. The image processing apparatus according to claim 1 , wherein the information acquisition means generates information indicating a state of the subject based on a captured image of the subject.

5. The image processing apparatus according to claim 1 , wherein the information acquisition means generates information indicating a state of the subject based on a result of estimating the posture of the subject.

6. The image processing device according to claim 1 , wherein the information indicating the state of the subject is information calculated based on changes in the position, posture, shape, or appearance of the subject over a plurality of times.

7. The image processing device according to claim 1 , wherein the information indicating the state of the subject is information indicating an event that has occurred regarding the subject.

8. The image processing device according to claim 1 , wherein the information indicating the state of the subject is information indicating a movement, a change in posture, a change in shape, or a classification result of the state of the subject.

9. The image processing device according to claim 1 , wherein the information indicating the state of the subject is information indicating that a specific movement or change in posture has occurred with respect to the subject.

10. The image processing device according to claim 1 , wherein the information indicating the state of the subject is information indicating an analysis result of the movement of the subject.

11. The image processing device according to claim 1 , wherein the information indicating the state of the subject is information indicating a moving speed or a deformation amount of the shape of the subject.

12. The image processing device according to claim 1 , wherein the information indicating the state of the subject is information specifying a time-series position of the subject, or information indicating a posture of the subject.

13. The image processing apparatus according to claim 1 , wherein the generating means changes a display relating to the subject in the virtual viewpoint image in accordance with information indicating a state of the subject.

14. 2. The image processing apparatus according to claim 1, wherein said generating means changes a size or a color of said subject in said virtual viewpoint image in accordance with information indicating a state of said subject.

15. The image processing device according to claim 1 , wherein the generating means adds an object around the subject in accordance with information indicating a state of the subject.

16. The image processing apparatus according to claim 1 , wherein the generating means superimposes information indicating a state of the subject on the virtual viewpoint image.

17. 2. The image processing device according to claim 1, wherein the generation means determines whether or not a user has a right corresponding to a display mode, and based on a determination that the user has a right corresponding to the display mode, generates a virtual viewpoint image of the subject according to information indicating a status of the subject that corresponds to the display mode.

18. The image processing device described in claim 1, characterized in that the generation means determines whether or not a user owns an NFT corresponding to the information indicating the situation of the subject, and generates a virtual viewpoint image of the subject according to the information indicating the situation of the subject based on a determination that the user owns an NFT corresponding to the information indicating the situation of the subject.

19. 1. An image processing method for generating a virtual viewpoint image of a subject, comprising: obtaining a three-dimensional model of the subject; acquiring information indicative of a state of the subject; generating a virtual viewpoint image of the subject according to a three-dimensional model of the subject and information indicating a situation of the subject; 13. An image processing method comprising:

20. A program for causing a computer to function as the image processing device according to any one of claims 1 to 18.

Images