Image processing device, image processing method and program

Abstract

This realizes tracking that is robust even against occlusion. [Solution] Three-dimensional shape data of an object is generated from images captured by multiple image capture devices, three-dimensional areas in the image capture space of the multiple image capture devices that correspond to the three-dimensional shape data are extracted, an image capture device that captured an image to be used to identify the object is selected from the multiple image capture devices based on spatial information indicating the extracted three-dimensional areas, and the object corresponding to the three-dimensional shape data is identified based on the image captured by the selected image capture device.

Description

[Technical Field]

[0001] The present disclosure relates to a technique for tracking three-dimensional shape data of an object. [Background technology]

[0002] In recent years, volumetric video technology, which reconstructs data representing the three-dimensional shape of an object (subject) (commonly referred to as a "3D model") in three-dimensional space and visualizes it from any viewpoint using computer graphics, has become common in the field of video production. There is a demand for using this volumetric video technology to generate virtual viewpoints from the player's point of view during a game, for example, or to analyze the movements of each player. To achieve this, it is necessary to generate a 3D model of each player frame by frame based on video data from multiple cameras, assign a unique identifier to each 3D model, and track them. In this regard, Patent Document 1 discloses a technology for quickly and robustly recognizing and identifying objects from video data captured by multiple cameras. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2019-101526 Summary of the Invention [Problem to be solved by the invention]

[0004] In the technology disclosed in Patent Document 1, the orientation of an object is estimated from changes in position information between the current frame and previous frames, and tracking is performed by performing texture recognition using images from a camera selected based on the estimated orientation. For example, when using the technology disclosed in Patent Document 1 to track players in a baseball game, the orientation of a player (e.g., a pitcher) whose position information changes little between frames cannot be accurately estimated, resulting in inaccurate texture recognition. Furthermore, because orientation information alone cannot take into account so-called occlusion, texture recognition breaks down, making tracking errors more likely. [Means for solving the problem]

[0005] The image processing device according to the present disclosure includes a generation means for generating three-dimensional shape data of an object from images captured by a plurality of imaging devices, an extraction means for extracting a three-dimensional area in the imaging space of the plurality of imaging devices that corresponds to the three-dimensional shape data, a selection means for selecting an imaging device from the plurality of imaging devices that captured an image to be used to identify the object based on spatial information indicating the three-dimensional area extracted by the extraction means, and an identification means for identifying the object corresponding to the three-dimensional shape data based on the image captured by the imaging device selected by the selection means. [Effects of the Invention]

[0006] According to the present disclosure, tracking that is robust even against occlusion can be achieved. [Brief explanation of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of a virtual viewpoint video generation system. [Figure 2] FIG. 1 is a diagram showing the hardware configuration of an image processing device and control device. [Figure 3] 10A and 10B are diagrams illustrating a method for selecting a camera using depth of field information. [Figure 4]10A to 10D are diagrams illustrating a method for selecting a camera using a foreground extraction region mask. [Figure 5] 10(a) to 10(c) are diagrams illustrating a method for selecting a camera using a distance map. [Figure 6] 10(a) to 10(d) are diagrams illustrating a camera selection method using a weight map. [Figure 7] 10 is a flowchart showing the flow of a process for identifying which object a 3D model corresponds to. [Figure 8] 10A and 10B are diagrams showing an example of the results of texture recognition processing. DETAILED DESCRIPTION OF THE INVENTION

[0008] Hereinafter, the present invention will be described in detail based on preferred embodiments thereof with reference to the drawings. Note that the configurations shown in the following embodiments are merely examples and are not limited to the configurations shown in the drawings.

[0009] [Embodiment 1] In this embodiment, a camera that captures the texture of a 3D model of interest is selected, and texture recognition is performed using the image from the selected camera, thereby enabling tracking that is robust even against occlusion.

[0010] <System configuration> 1 is a block diagram showing an example of the configuration of a virtual viewpoint video generation system according to this embodiment. The virtual viewpoint video generation system in FIG. 1 includes a camera group 10, a control device 20, and an image processing device 30.

[0011] A camera group 10, consisting of multiple imaging devices (cameras), is installed to surround an imaging space where a subject (object) exists, and all cameras are synchronized to capture video images. Each camera also has the function of performing foreground extraction processing on the acquired video frame by frame to generate an image representing the silhouette of the foreground object (hereinafter referred to as a "foreground silhouette"). For the foreground extraction processing, a background subtraction method or a machine learning method may be used. In this case, the foreground extraction area information specifies the area in the captured image from which foreground extraction is performed. In this embodiment, the foreground extraction area information is a mask image (hereinafter referred to as a "foreground extraction area mask") that represents the area to be extracted using a binary value: a pixel value of "1 (white)" for the area to be extracted, and a pixel value of "0 (black)" for the area not to be extracted. The foreground extraction area mask is used to extract the foreground silhouette from only the area required for generating a 3D model when generating a foreground silhouette from the captured image. Furthermore, each camera also has the function of generating an image (hereinafter referred to as "foreground texture") by clipping out an area representing the texture of an object in the captured image, in accordance with the circumscribing rectangle of the silhouette represented by the foreground silhouette. Note that there is no particular limitation on the number of cameras constituting camera group 10; for example, if the capture space is a rugby or soccer stadium, several tens to several hundred cameras are installed around the stadium. Furthermore, although the present embodiment will be described taking the case of a moving image as an example, still images may also be used.

[0012] The control device 20 provides the image processing device 30 with data such as camera parameters for each camera in the camera group 10, foreground extraction area masks for each camera, a lookup table used for object recognition, and a textured background 3D model. The camera parameters consist of external parameters such as the camera's position and orientation, and internal parameters such as the focal length and the center coordinates of the image sensor. The lookup table used for object recognition is a table that describes feature information for each object. For example, when generating a virtual viewpoint image from footage of a sports game, the feature information describes facial feature points and uniform number information for each player. Information representing a person's facial feature points includes, but is not limited to, information on specific facial features such as the corners of the eyes and the corners of the mouth. For example, feature information obtained by common techniques such as SIFT (Scale-Invariant Feature Transform) or machine learning may also be used. Furthermore, the "uniform number" refers to a unique number assigned to each player and printed in large letters on the back of the player's uniform. This concept includes cases where the number is printed on the chest or near the shoulder of the uniform. The control device 20 provides this data before the image processing device 30 begins generating and tracking a 3D model. The control device 20 also generates and provides a so-called camera path required for generating a virtual viewpoint image.

[0013] The image processing device 30 performs a series of image processing such as generation and analysis of a 3D model, tracking, texture recognition, and generation of a virtual viewpoint image. The image processing device 30 of this embodiment includes a data storage unit 100, a 3D model generation unit 101, an analysis unit 102, an object region extraction unit 103, a tracking unit 104, a selection unit 105, a recognition unit 106, a setting unit 107, and a virtual viewpoint image generation unit 108. Each unit of the image processing device 30 will be described below.

[0014] <Details of the image processing device 30> The data storage unit 100 stores raw data for generating a virtual viewpoint image in addition to the above-mentioned data provided in advance by the control device 20. This raw data includes foreground silhouettes and foreground textures input from the camera group 10, 3D models that are the processing results of each unit in the image processing device 30, distance maps, weight maps, visibility information, bounding boxes, and the like.

[0015] The 3D model generation unit 101 generates a 3D model, which is three-dimensional shape data of an object, by a method such as volume intersection based on multiple foreground silhouettes input from the camera group 10. The data format of the 3D model is not particularly limited and may be polygon format, point cloud format, voxel format, etc. In this embodiment, a description will be given taking a voxel format 3D model as an example.

[0016] The analysis unit 102 uses the camera parameters of each camera to generate an image called a "distance map" or "distance image," which has the same width and height as the captured image and stores a distance value for each pixel, as distance information representing the distance from the camera to each element that constitutes the 3D model. The distance map is generated as follows. First, the coordinates X of a voxel P of interest in the voxels that constitute the 3D model are calculated. w , the extrinsic matrix T e By multiplying it, the camera coordinate X in the camera coordinate system c where the extrinsic matrix T e is a matrix consisting of the external parameters of the camera. If the camera position is the origin and the direction in which the camera lens is facing is the positive z-axis of the camera coordinate system, then the camera coordinate X c The z coordinate of voxel P is the distance value when viewed from the camera. Next, the camera coordinate X c Image coordinate X i Calculate the distance map coordinates to store the distance values. i is the camera coordinate X c is normalized by the z-coordinate to the normalized camera coordinates, and the intrinsic matrix T i It is calculated by multiplying the internal matrix T i is a matrix consisting of the camera's internal parameters. iIf the pixel in the image contains a distance value of another voxel that has already been calculated, the distance value and the image coordinate X i If the z coordinate is smaller, the z coordinate is compared with the image coordinate X iThe distance map for one camera is generated by performing these processes on all voxels P in the voxel group that constitutes the 3D model. Furthermore, distance maps for all cameras can be obtained by performing similar processes on each camera in the camera group 10. The method for generating the distance map is not limited to the above example. For example, each camera may be equipped with a distance sensor, and the distance map may be generated based on sensor values ​​indicating the distance to the object obtained by the distance sensor. Furthermore, the analysis unit 102 generates visibility information and weight information based on the generated distance map. First, visibility information will be described. Visibility information can be obtained by determining the visibility of all voxels that constitute the 3D model using the distance map and camera parameters of each camera. Specifically, the process is as follows: First, the coordinate Xw of a target voxel P in the voxels that constitute the 3D model is converted into the camera coordinate system and the image coordinate system, and the camera coordinate Xc and the image coordinate Xi are calculated. Then, the pixel value of the distance map at the image coordinate Xi is compared with the z coordinate of the camera coordinate Xc (the distance from the camera to voxel P). If the result of this comparison is that the difference d is equal to or less than a threshold, the voxel P is determined to be visible from that camera; if the difference d exceeds the threshold, the voxel P is determined to be invisible from that camera. By performing the above process for all cameras, it is possible to determine whether the voxel P of interest is visible from each camera. Then, by performing the same process for all voxels constituting the 3D model, visibility information is obtained. Next, weight information will be described. A virtual viewpoint image is an image that represents the appearance of an object from a virtual viewpoint and is obtained by, for example, applying a foreground texture to a 3D model. In this embodiment, as weight information, an image called a "weight map" or "weight image" is generated, which has the same width and height as the captured image and in which each pixel stores a weight value indicating the effectiveness of the foreground texture applied to the 3D model when generating the virtual viewpoint image. For example, a weight map can be obtained by setting a low weight for areas in the distance map where the distance value is not infinite and a high weight for areas where the distance value changes little.

[0017] The object region extraction unit 103 extracts a corresponding three-dimensional region in the imaging space for each 3D model generated by the 3D model generation unit 101, and generates spatial information indicating the extracted three-dimensional region. In this embodiment, spatial information indicating a circumscribed rectangular parallelepiped (hereinafter referred to as a "BBox") of each 3D model, generally called a "bounding box," is generated.

[0018] The tracking unit 104 tracks each 3D model generated by the 3D model generation unit 101 between frames based on the spatial information for each 3D model extracted by the object region extraction unit 103. Specifically, it calculates the distance between the center of gravity coordinates obtainable from the BBox of the 3D model of interest in the current frame and the center of gravity coordinates of the BBoxes of all 3D models generated in the previous frame. It then identifies the BBox with the smallest distance among the BBoxes of the 3D models generated in the previous frame and associates it with the BBox of the 3D model of interest in the current frame. This makes it possible to determine which 3D model in a different frame each 3D model generates corresponds to.

[0019] The selection unit 105 selects a camera suitable for texture recognition of each 3D model generated by the 3D model generation unit 101 based on the spatial information for each 3D model generated by the object region extraction unit 103 and selection data, the contents of which vary for each camera. The selection data here includes a foreground extraction region mask, a distance map, a weight map, and depth of field information, and camera selection is performed for each 3D model using at least one of these. A specific camera selection method will be described later. Information on the camera selected for each 3D model is output as a selected camera list.

[0020] The recognition unit 106 performs texture recognition processing based on the selected camera list output from the selection unit 105 and feature information for each object prepared in advance, and identifies which object the generated 3D model belongs to. The texture recognition processing will be described in detail later.

[0021] The setting unit 107 assigns object information to each 3D model based on the results of the texture recognition process received from the recognition unit 106. Here, the object information is information that includes at least an identifier (object ID) that can identify which object the target 3D model belongs to, and may also include the name and uniform number of the object. The 3D models to which the object information has been assigned are stored in the data storage unit 100 and provided to the virtual viewpoint image generation unit 108.

[0022] The virtual viewpoint image generation unit 108 generates a virtual viewpoint image that represents how an object appears from a virtual viewpoint by, for example, applying a foreground texture to a 3D model generated frame by frame based on the camera path provided by the control device 20.

[0023] It is not necessary for one device to have all of the above-mentioned functional units 101 to 108, and the image processing device 30 may be realized by distributing the functional units among a plurality of devices.

[0024] <Hardware configuration of image processing device 30> FIG. 2 is a block diagram showing the basic hardware configuration of the image processing device 30. The control device 20 and each camera constituting the camera group 10 also have a similar hardware configuration. The image processing device 30 includes a CPU 211, a ROM 212, a RAM 213, an auxiliary storage device 214, a display unit 215, an operation unit 216, a communication I / F 217, and a bus 218. The CPU 211 controls the entire device using computer programs and data stored in the ROM 212 and RAM 213, thereby realizing each function of the image processing device 30. Note that the image processing device 30 may include one or more dedicated hardware components separate from the CPU 211, and at least a portion of the processing performed by the CPU 211 may be executed by the dedicated hardware components. Examples of the dedicated hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor). The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214 and data supplied from an external device via the communication I / F 217. The auxiliary storage device 214 is configured, for example, by a hard disk drive or the like, and stores various data such as image data, audio data, etc. The display unit 215 is configured, for example, by a liquid crystal display, LED, etc., and displays a GUI (Graphical User Interface) and the like for the user to operate the image processing device 30.

[0025] <Details of selection unit 105> Next, the camera selection process in the selection unit 105 will be described in detail. As described above, the camera selection process uses at least one of depth of field information, a foreground extraction region mask, a distance map, and a weight map as selection data. Below, we will explain how to select a camera in each case.

[0026] <When using depth of field information> In the selection using depth of field information, a camera that can capture an image with an object in focus is selected from among the cameras constituting the camera group 10. Here, depth of field refers to the range of clearly visible areas before and after the focus point, and varies depending on conditions such as the lens focal length, aperture value (F-number), focus position, and shooting distance. Figure 3(a) shows an example of a table describing the depth of field information for each camera. It consists of five items: "camera ID," "x coordinate of gaze point," "y coordinate of gaze point," "z coordinate of gaze point," "forward focus distance," and "backward focus distance." The gaze point is the position in the imaging space where the camera focuses, and is represented by a three-dimensional coordinate system consisting of the x-, y-, and z-axes. In the table shown in Figure 3(a), the three-dimensional coordinates (x, y, z) are expressed as the distance from the origin, each expressed in mm. The origin (x, y, z) = (0, 0, 0) is an arbitrary position in the imaging space, such as the center of the court (the center of the center circle) in a basketball game. The forward focus distance represents the distance from the gaze point of the target camera to the position where the near side is in focus, and the backward focus distance represents the distance from the gaze point of the target camera to the position where the far side is in focus. The units of the forward focus distance and the backward focus distance are also (mm). FIG. 3(b) is an overhead view showing two cameras 301a and 301b constituting the camera group 10 capturing an image of an object 309 present in the imaging space. Cameras 301a and 301b are currently facing gaze points 302a and 302b, respectively, indicated by crosses, and sector-shaped areas 303a and 303b indicate the areas that cameras 301a and 301b can capture. Hatched trapezoidal areas 304a and 304b indicate the in-focus areas within the depth of field of cameras 301a and 301b. Furthermore, line segment 307a on focus area 304a indicates a position that is closer to camera 301a from point of gaze 302a by forward focus distance 305a, and line segment 308a on focus area 304a indicates a position that is farther from the camera from point of gaze 302a by backward focus distance 306a.Similarly, line segment 307b on focused area 304b indicates a forward focused position that is closer to camera 301b from point of gaze 302b by forward focused distance 305b, and line segment 308b on focused area 304b indicates a backward focused position that is farther away from camera 301b from point of gaze 302b by backward focused distance 306b. As shown in FIG. 3(b), object 309, indicated by a black circle, is not included in focused area 304a for camera 301a, and therefore is not in focus. This means that object 309 is not in focus in the image captured by camera 301a, and the foreground texture based on the captured image is not suitable for texture recognition by recognition unit 106. On the other hand, object 309 is in focus for camera 301b because it is included in focused area 304b. This means that the image captured by camera 301b is in focus on object 309, and the foreground texture based on this captured image is suitable for texture recognition by recognition unit 106. Therefore, in the example of FIG. 3(b), the foreground texture of camera 301a is determined to be unsuitable for texture recognition of the 3D model of object 309, and camera 301a is not selected. On the other hand, the foreground texture of camera 301b is determined to be suitable, and camera 301b is selected. Here, to determine whether a 3D model is included in the focal area of ​​a certain camera, for example, the following equation (1) is used. TIFF2026041153000002.tif16150 In the above formula (1), when flag=1, it indicates that the image is included in the in-focus area, and when flag=0, it indicates that the image is not included in the in-focus area. bb is the distance to the 3D model, d front is the distance from the camera to the front focus position, and d rear and indicate the distance from the camera to the rear focus position, and are calculated by the following equations (2) to (4), respectively. TIFF2026041153000003.tif21105 In the above formulas (2) to (4), x cam , y cam , z cam represents the camera coordinates, and xp , y p , z p represents the camera's gaze point coordinate, and x bb , y bb , z bb represents the position coordinates of the 3D model. In this embodiment, the coordinates of the center of gravity of the BBox are used as the position coordinates of the 3D model. front represents the forward focusing distance, and DOF rear represents the rear focus distance. The determination using the above formula (1) is performed for each 3D model, and a list of valid cameras (selected camera list) that include the target 3D model within their focus area is obtained.

[0027] <When using a foreground extraction area mask> In the selection using the foreground extraction region mask, a 3D model is projected onto the foreground extraction region mask, and a camera whose entire projection region is included in the target region for foreground extraction in the foreground extraction region mask is selected from among the cameras constituting the camera group 10. FIG. 4(a) shows an image 400 captured by a certain camera capturing a basketball game. FIG. 4(b) shows a foreground extraction region mask 420 for that camera, where white indicates regions for which the foreground region is to be extracted from the captured image and black indicates regions for which the foreground region is not to be extracted from the captured image. In the case of the foreground extraction region mask 420 in FIG. 4(b), a portion of the texture of the player is obscured by the goal, resulting in occlusion, and therefore the region is designated as a region not targeted for foreground extraction. FIG. 4(c) shows a foreground silhouette 410 obtained from the captured image 400 in FIG. 4(a) and the foreground extraction region mask 420 in FIG. 4(b). In the foreground silhouette 410, the white area 411 represents the silhouette of player 401 extracted from the captured image 400, and the white area 412 represents the silhouette of player 402 extracted from the captured image 400. The black area in the foreground silhouette 410 represents the background that was not extracted as the foreground. In the foreground silhouette 410, part of the silhouette of player 402 is missing. This is because part of player 402 falls within the non-target area (goal area) of foreground extraction in the foreground extraction area mask 420 and was not extracted as the foreground. FIG. 4(d) shows the results obtained by projecting 3D models corresponding to players 401 and 402 onto the foreground extraction area mask 420 in FIG. 4(b). In the foreground extraction area mask 420′ after projection, the area 421 indicated by the dashed line represents the projected area of ​​the 3D model corresponding to player 401, and the area 422 indicated by the dashed line represents the projected area of ​​the 3D model corresponding to player 402. Now, the entire projection area 421 is included in the target area for foreground extraction in the foreground extraction area mask 420. In this case, for the 3D model corresponding to player 401, the image captured by that camera is determined to be suitable for texture recognition, and that camera is selected. On the other hand, a portion of the projection area 422 is included in the non-target area for foreground extraction in the foreground extraction area mask 420.In this case, the image captured by the camera is determined to be unsuitable for texture recognition for the 3D model corresponding to player 402, and the camera is not selected. In this case, if the portion included in the non-target area of ​​foreground extraction exceeds a certain percentage (for example, several percent to several tens of percent) of the entire projection area, the image captured by the camera may be determined to be unsuitable for texture recognition.

[0028] <When using a distance map> In the selection using a distance map, a 3D model is projected onto the distance map, and based on the distance values ​​within the projection area on the distance map, a camera corresponding to a captured image in which the target object is presumably not obstructed by other players or the like is selected from among the cameras constituting the camera group 10. FIG. 5(a) shows an image 500 captured by a camera capturing a basketball game, and FIG. 5(b) shows a distance map 510 corresponding to the captured image 500. In the distance map 510, each pixel value is a normalized value of the distance from the camera to the 3D model within a range of 0 to 1, with the minimum pixel value "0.00" being black and the maximum pixel value "1.00" being white. Because player 501 is closer to the camera than player 502, the gray area 511 corresponding to player 501 in the distance map 510 is darker than the gray area 512 corresponding to player 502. Note that to ensure consistency between the normalized values ​​for each camera, the normalization is performed using minimum and maximum distance values ​​specified in advance by the user via a UI (not shown). For example, if a minimum distance value of 5000 mm and a maximum distance value of 50,000 mm are specified, the pixel value is normalized as "0.00" when the distance from the camera is 5,000 mm and "1.00" when the distance from the camera is 50,000 mm. As a result, pixel values ​​are clipped to "0.00" when the distance from the camera is less than 5,000 mm, and to "1.00" when the distance from the camera is greater than 50,000 mm. In such a distance map-based selection, the distance value within the projection area when the 3D model is projected onto the distance map is compared with the closest distance value from the camera to the 3D model. If the comparison result indicates that no occlusion occurs in the 3D model, the image captured by the camera corresponding to the distance map is determined to be suitable for texture recognition. Figure 5(c) shows the results of projecting 3D models corresponding to players 501 and 502 onto the distance map 510 in Figure 5(b). In the projected distance map 510' of Figure 5(c), the area 521 indicated by the dashed line indicates the projection area of ​​the 3D model corresponding to player 501, and the area 522 indicated by the dashed line indicates the projection area of ​​the 3D model corresponding to player 502.As shown in captured image 500 in FIG. 5(a), player 501 is positioned in front of player 502, obscuring his lower body from the waist down, as viewed from the camera. Projection area 521 and projection area 522 partially overlap in projected distance map 510' in FIG. 5(c). The pixel values ​​of this overlapping area on distance map 510 represent the distance from the camera to player 501. In other words, the minimum pixel value of projection area 521 and the minimum pixel value of projection area 522 are the same value, and in this example, both are assumed to be 0.2. Meanwhile, the minimum distance value from the camera to each vertex coordinate of the BBox of the 3D model corresponding to player 501 is normalized to 0.18, and the minimum distance value from the camera to each vertex coordinate of the BBox of the 3D model corresponding to player 502 is normalized to 0.29. Here, when focusing on the 3D model corresponding to player 501, the pixel value "0.18" indicating the distance from the camera is smaller than the minimum pixel value "0.20" of the projection area 521, so it can be inferred that no occlusion has occurred. Therefore, in this case, it is determined that the image captured by the camera corresponding to that distance map is suitable for texture recognition of the 3D model corresponding to player 501, and that camera is selected. On the other hand, when focusing on the 3D model corresponding to player 502, the pixel value "0.29" indicating the distance from the camera is larger than the minimum pixel value "0.20" of the projection area 522, so it can be inferred that occlusion has occurred. Therefore, in this case, it is determined that the image captured by the camera corresponding to that distance map is not suitable for texture recognition of the 3D model corresponding to player 502, and that camera is not selected.

[0029] <When using a weight map> In the selection using a weight map, a 3D model is projected onto the weight map, and based on the weight values ​​within the projection area of ​​the weight map, a camera corresponding to a captured image in which the player's face or uniform number is not obscured by the player's own arm or the like is selected from among the cameras constituting the camera group 10. FIG. 6(a) shows an image 600 captured by a camera capturing a basketball game. FIG. 6(b) shows a distance map 610 corresponding to the captured image 600, and FIG. 6(c) shows a weight map 620 corresponding to the distance map 610. As described above, in the distance map 610, each pixel value is a value obtained by normalizing the distance value from the camera to the 3D model within the range of 0 to 1, with the minimum pixel value "0.00" being represented as black and the maximum pixel value "1.00" being represented as white. Because the distances from the camera to players 601 and 602 are approximately the same, the gray area 611 corresponding to player 601 and the gray area 612 corresponding to player 602 in the distance map 610 have the same shade of gray. The weight map 620, based on the distance map 610, is composed of white regions corresponding to the silhouettes of the players and the ball and black regions corresponding to the rest of the background. The weights, which are pixel values ​​in the weight map 620, range from 0 to 1, with a weight value of "0" representing a black region and a weight value of "1" representing a white region. The black regions (low-weight regions) within the silhouettes of players 601 and 602 indicate a steep change in distance value in the distance map 620 compared to the adjacent white regions. In other words, there is a high possibility that occlusion has occurred due to a player's own arm or other obstruction. Therefore, a 3D model is projected onto the weight map, and it is determined whether the proportion of pixels with a weight of "0" in the projected region is below a preset threshold. In this embodiment, where the player's face and uniform number are used for texture recognition, the texture of the player's upper body is important. Therefore, only a 3D model corresponding to a portion of the BBox—specifically, the 3D model corresponding to the upper half of the BBox—is projected onto the weight map. FIG. 6(d) shows a weight map 620' after projecting the upper half of the 3D models corresponding to players 601 and 602 onto the weight map 620 of FIG. 6(c).In the weight map 620' in FIG. 6(d), an area 623 indicated by a dashed line indicates the projection area of ​​the upper half of the 3D model corresponding to player 601, and an area 624 indicated by a dashed line indicates the projection area of ​​the upper half of the 3D model corresponding to player 602. In this case, for example, assume that the proportion of pixels with a weight of "0" among the pixels included in projection area 623 is 10%, the proportion of pixels with a weight of "0" among the pixels included in projection area 624 is 30%, and the preset threshold is 20%. In this case, for the 3D model corresponding to player 601, the image captured by the camera corresponding to weight map 620 is determined to be suitable for texture recognition of the 3D model corresponding to player 601, and that camera is selected. On the other hand, for the 3D model corresponding to player 602, the image captured by the camera corresponding to weight map 620 is determined to be unsuitable for texture recognition of the 3D model corresponding to player 602, and that camera is not selected.

[0030] The above is a description of specific selection methods according to the content of selection data in this embodiment. The above four selection methods may all be executed sequentially or in parallel, or only some of the selection methods may be used. When two or more selection methods are used, the results of each selection method may be integrated by, for example, taking the logical sum or product of the results of each selection method. Alternatively, priorities may be assigned to each selection method in advance, and if different results are obtained, the result with the higher priority may be used. The above selection method allows the selection of a camera with an image (foreground texture) suitable for texture recognition even if there is little movement of the 3D model between the previous and next frames. This allows only the image captured by the selected camera to be used for texture recognition, while the image captured by the unselected camera is discarded, thereby improving the accuracy of texture recognition and reducing the processing time required for texture recognition.

[0031] <Operation flow of the image processing device 30> Next, the flow of processing performed by each unit of the image processing device 30 to identify which object a 3D model generated frame by frame corresponds to, which is a feature of this embodiment, will be described with reference to the flowchart of FIG. 7. The series of processes shown in the flowchart of FIG. 7 are executed frame by frame, for example, from the start frame to the end frame of the camera path. Note that prior to the flow of FIG. 7, various data that does not change between frames is assumed to be provided in advance by the control device 20 and stored in the data storage unit 100. Specifically, this includes camera parameters and feature information of each camera constituting the camera group 10, and the depth of field information and foreground extraction area mask among the above-mentioned selection data. In the following description, the symbol "S" denotes a step.

[0032] In S701, all BBoxes corresponding to all 3D models related to the frame of interest (current frame) to be processed are read from the data storage unit 100. All the read BBoxes of the current frame are output to the tracking unit 104 and the selection unit 105.

[0033] In S702, all BBoxes corresponding to all 3D models related to the previous frame (for example, the frame immediately before the current frame) are read from the data storage unit 100. All BBoxes of the read previous frame are output to the tracking unit 104.

[0034] In S703, a BBox to be focused on as a processing target (hereinafter referred to as a "focused BBox") is determined from all BBoxes of the current frame acquired in S701.

[0035] In S704, the tracking unit 104 performs tracking processing on the BBox of interest. That is, the distance between the coordinates of the center of gravity of the BBox of interest in the current frame and the coordinates of the centers of gravity of all BBoxes in the previous frame is calculated, and the BBox in the previous frame with the smallest distance is identified. The tracking unit 104 then outputs the object information assigned to the 3D model of the BBox in the previous frame (the object information assigned to the previous frame in S710, described below) to the setting unit 107 (or the recognition unit 106) as the tracking result. Note that if there is no previous frame immediately after the start of processing, this step is skipped.

[0036] In S705, the selection unit 105 reads and acquires from the data storage unit 100 the selection data to be used in the next step S706, i.e., at least one of the foreground extraction area mask, distance map, weight map, and depth of field information corresponding to each camera.

[0037] In S706, the selection unit 105 selects, from among the cameras constituting the camera group 10, cameras having foreground textures suitable for texture recognition of the 3D model corresponding to the BBox of interest, by applying one or more selection methods, based on the selection data acquired in S705. Specific selection methods have already been described in detail. Then, a selected camera list based on the selection results is output to the recognition unit 106.

[0038] In S707, the recognition unit 106 reads and acquires from the data storage unit 100 the foreground textures corresponding to the cameras indicated in the selected camera list for the 3D model corresponding to the target BBox.

[0039] In S708, the recognition unit 106 performs texture recognition processing using the foreground texture of each camera acquired in S707. Specifically, a 3D model corresponding to the BBox of interest is projected onto the foreground texture of each camera, and a matching process is performed to compare facial feature points and uniform numbers obtained from the texture data of each projection area with the facial feature points and uniform numbers of each player included in the feature information. This matching process uses, for example, a known machine learning technique. For example, in the case of facial feature point matching, feature points are extracted from the area corresponding to the face in the texture of the projection area, and the extracted feature points are sequentially compared with the facial feature points of each player registered in the feature information to calculate the similarity for each player. If the highest similarity calculated for each player is equal to or greater than a predetermined threshold, matching is successful; if it is less than the threshold, matching is unsuccessful. In the case of uniform number matching, optical character recognition (OCR) is performed on the texture of the projection area, and a probability indicating the likelihood that the numbers recognized by OCR match each uniform number registered in the feature information is calculated. If the calculated probability is equal to or greater than a predetermined threshold, matching is successful; if it is less than the threshold, matching is unsuccessful. Figure 8(a) shows an example of a table that summarizes the matching results for facial feature points and uniform numbers using the foreground textures of each camera selected for a 3D model corresponding to a certain BBox of interest. The table in Figure 8(a) consists of the following fields: "Camera ID," "Player ID," and "Uniform Number." "Camera ID" contains the identifier assigned to each camera. "Player ID," which serves as an object ID, contains the identifier assigned to each player. "Uniform Number" contains the uniform number of each player. In the table in Figure 8(a), eight cameras with camera IDs "Cam1," "Cam3," "Cam8," "Cam20," "Cam24," "Cam46," "Cam51," and "Cam56" are selected. Texture recognition processing was performed using the foreground textures of these eight cameras. Based on facial feature points, the remaining cameras, except for camera ID "Cam46," successfully matched the face of the player with object ID "0001." For the camera with the camera ID "Cam46", the status is "unknown", indicating that matching failed.Furthermore, when matching is based on uniform numbers, the camera with the camera ID "Cam1" successfully matches the uniform number "010," while the cameras with the camera IDs "Cam3," "Cam20," "Cam24," "Cam51," and "Cam56" successfully match the uniform number "003." Furthermore, the cameras with the camera IDs "Cam8" and "Cam46" fail the match, resulting in an "unknown" result. A uniform number match fails when OCR fails or when a uniform number matching the OCR-recognized number is not registered as feature information. Based on the texture recognition results obtained for each selected camera, the player ID and uniform number corresponding to the 3D model in the BBox of interest are identified. Identification is performed according to predetermined conditions. For example, the most common successful match for each player ID and uniform number, and the result that is found by more than half of the selected cameras, is used. According to these conditions, in the example table of FIG. 8(a) described above, the player ID is identified as "0001" and the uniform number is identified as "003." If the player ID and uniform number can be identified for the 3D model of the target BBox in this way, object information such as that shown in FIG. 8(b) is output to the setting unit 107 as the result of the texture recognition process. If identification according to the conditions is not possible, for example, if there are two or more most common successful matching results, or if there is no successful matching result that exceeds the majority of the selected cameras, the object is treated as unidentifiable. If identification is not possible, object information in which "unknown," indicating unidentifiable, is added to the "ID" and / or "Number" in FIG. 8(b) is output to the setting unit 107 as the result of the texture recognition process. Note that instead of outputting unidentifiable object information, object information as the tracking result input from the tracking unit 104 may be output to the setting unit 107 as the result of the texture recognition process. Furthermore, in this embodiment, the uniform number written in large letters on the back of the uniform is used as feature information, but the chest number written on the front of the uniform may also be used depending on, for example, the camera resolution.

[0040] In S709, the setting unit 107 sets object information for the 3D model of the BBox of interest based on the results of the texture recognition processing output from the recognition unit 106. Specifically, if the texture recognition processing succeeds in identifying the player ID and uniform number and identifies which object the 3D model belongs to, object information including the player ID and uniform number is assigned to the 3D model. On the other hand, if the processing fails to identify at least one of the player ID and uniform number and identifies which object the 3D model belongs to, object information for the previous frame input from the tracking unit 104 (or the recognition unit 106) is assigned to the 3D model. The 3D model of the BBox of interest with the object information assigned is then sent to and stored in the data storage unit 100 and provided to the virtual viewpoint image generation unit 108.

[0041] The above is the flow of processing to identify which object each 3D model corresponds to, which is a feature of this embodiment. This enables highly accurate texture recognition and also reduces the processing time required for texture recognition, which has a low chance of success.

[0042] <Modification> In the above-described embodiment, feature information for each object, such as a player's facial feature points and uniform number, is stored in the data storage unit 100 in advance. However, this is not necessarily limited to this. For example, feature information may be generated or updated based on the results of texture recognition processing obtained by the recognition unit 106. Specifically, if a player's facial feature points or uniform number are not registered in the feature information and matching in the texture recognition processing fails, the facial feature points extracted from the texture of the projection area and the recognized number corresponding to the uniform number obtained by OCR are linked and registered as new feature information. This allows feature information to be created or updated while capturing images and generating virtual viewpoint images, thereby achieving the highly accurate texture recognition processing described in the above-described embodiment without having to prepare feature information in advance. Furthermore, for example, object information for each 3D model may be presented to the user using a separate application, allowing new information, such as a player's name, to be added to each object information. This makes it possible to more efficiently generate virtual viewpoint images from the player's perspective and analyze player movements.

[0043] (Other Examples) The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0044] The present disclosure also includes the following configurations and methods.

[0045] [Configuration 1] a generating means for generating three-dimensional shape data of an object from images captured by a plurality of imaging devices; an extraction means for extracting a three-dimensional area in an imaging space of the plurality of imaging devices, the three-dimensional area corresponding to the three-dimensional shape data; a selection means for selecting, from the plurality of image capture devices, an image capture device that captures an image used to identify the object, based on spatial information indicating the three-dimensional area extracted by the extraction means; an identification means for identifying an object corresponding to the three-dimensional shape data based on an image captured by the imaging device selected by the selection means; An image processing device comprising:

[0046] [Configuration 2] 2. The image processing device according to configuration 1, wherein the selection means makes the selection based on selection data whose contents differ for each of the plurality of image capturing devices.

[0047] [Configuration 3] 3. The image processing device according to configuration 2, wherein the selection data is depth of field information of each of the plurality of image capturing devices.

[0048] [Configuration 4] 3. The image processing device according to configuration 2, wherein the selection data is foreground extraction area information that specifies an area to be subjected to foreground extraction from the images captured by each of the plurality of image capturing devices.

[0049] [Configuration 5] the foreground extraction region information is a mask image indicating a region to be subjected to foreground extraction; 5. The image processing device according to configuration 4, wherein the selection means makes the selection based on a result of projecting the shape indicated by the three-dimensional shape data onto the mask image.

[0050] [Configuration 6] 3. The image processing device according to configuration 2, wherein the selection data is distance information representing a distance between an element constituting the three-dimensional shape data generated by the generation means and each of the plurality of imaging devices.

[0051] [Configuration 7] the distance information is a distance image having the same width and height as the images captured by the plurality of image capturing devices, with a distance value stored in each pixel; 7. The image processing device according to configuration 6, wherein the selection means makes the selection based on a result of projecting the shape indicated by the three-dimensional shape data onto the distance image.

[0052] [Configuration 8] 3. The image processing device according to configuration 2, wherein the selection data is weight information indicating the effectiveness of the texture when the texture of the object in the images captured by the plurality of imaging devices is pasted onto the three-dimensional shape data generated by the generation means.

[0053] [Configuration 9] the weighting information is a weighting image having the same width and height as the images captured by the plurality of image capturing devices, and in which a weighting value is stored for each pixel; 9. The image processing device according to configuration 8, wherein the selection means performs the selection based on a result of projecting the shape indicated by the three-dimensional shape data onto the weighted image.

[0054] [Configuration 10] 10. The image processing device according to claim 9, wherein the selection means, when projecting the shape indicated by the three-dimensional shape data onto the weighted image, projects a portion of the shape indicated by the three-dimensional shape data in which features of the object exist.

[0055] [Configuration 11] 11. The image processing device according to configuration 10, wherein the features of the object are feature points of a human face.

[0056] [Configuration 12] The feature of the object is a number written on a uniform worn by a person. 11. The image processing device according to configuration 10.

[0057] [Configuration 13] the selection data is at least two or more pieces of information selected from the group consisting of depth of field information for each of the plurality of image capturing devices, foreground extraction area information for specifying an area to be subjected to foreground extraction in the images captured by each of the plurality of image capturing devices, distance information for indicating a distance between each of the plurality of image capturing devices and an element constituting the three-dimensional shape data generated by the generation means, and weight information for indicating a degree of effectiveness of the texture when the texture of the object in the images captured by the plurality of image capturing devices is pasted onto the three-dimensional shape data generated by the generation means; determining, from among the plurality of imaging devices, an imaging device that captured an image to be used for identifying the object, based on the selection result obtained using the two or more pieces of information; 3. The image processing device according to configuration 2.

[0058] [Configuration 14] The image processing device according to any one of configurations 1 to 13, wherein the identification means projects a shape indicated by the three-dimensional shape data onto a texture of the object in the image captured by the imaging device selected by the selection means, and compares features of the object obtained from the texture data of the projection area with feature information indicating features of each object, thereby identifying the object corresponding to the three-dimensional shape data.

[0059] [Configuration 15] 15. The image processing device according to configuration 14, further comprising an update means for updating the feature information based on features of the object obtained from texture data of the projection area.

[0060] [Configuration 16] 16. The image processing device according to any one of configurations 1 to 15, further comprising: an assigning unit that assigns information indicating the object identified by the identifying unit to the three-dimensional shape data generated by the generating unit.

[0061] [Configuration 17] the images captured by the plurality of imaging devices are moving images consisting of a plurality of frames, 17. The image processing device according to configuration 16, further comprising a tracking means for tracking the three-dimensional shape data generated by the generating means between frames based on the spatial information extracted by the extracting means.

[0062] [Configuration 18] 18. The image processing device according to configuration 17, wherein, when the identification means fails to identify a frame of interest among the plurality of frames in the moving image, the assignment means assigns, based on the tracking result by the tracking means, information indicating the object assigned to the three-dimensional shape data of the frame preceding the frame of interest to the three-dimensional shape data generated by the generation means for the frame of interest.

[0063] [Method 1] a generating means for generating three-dimensional shape data of an object from images captured by a plurality of imaging devices; an extraction step of extracting a three-dimensional area in an imaging space of the plurality of imaging devices corresponding to the three-dimensional shape data; a selection step of selecting, from the plurality of image capture devices, an image capture device that captures an image used to identify the object, based on spatial information indicating the three-dimensional region extracted in the extraction step; an identification step of identifying an object corresponding to the three-dimensional shape data based on an image captured by the imaging device selected in the selection step; An image processing method comprising:

[0064] [Configuration 19] 19. A program for causing a computer to function as the image processing device according to any one of configurations 1 to 18.

Claims

1. a generating means for generating three-dimensional shape data of an object from images captured by a plurality of imaging devices; an extraction means for extracting a three-dimensional area in an imaging space of the plurality of imaging devices, the three-dimensional area corresponding to the three-dimensional shape data; a selection means for selecting, from the plurality of image capture devices, an image capture device that captures an image used to identify the object, based on spatial information indicating the three-dimensional area extracted by the extraction means; an identification means for identifying an object corresponding to the three-dimensional shape data based on an image captured by the imaging device selected by the selection means; An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the selection means makes the selection based on selection data whose contents differ for each of the plurality of image capturing devices.

3. The image processing device according to claim 2 , wherein the selection data is depth of field information for each of the plurality of image capturing devices.

4. 3. The image processing device according to claim 2, wherein the selection data is foreground extraction area information that specifies an area to be subjected to foreground extraction from the images captured by each of the plurality of image capturing devices.

5. the foreground extraction region information is a mask image indicating a region to be subjected to foreground extraction; 5. The image processing apparatus according to claim 4, wherein the selection means makes the selection based on a result of projecting the shape indicated by the three-dimensional shape data onto the mask image.

6. 3. The image processing device according to claim 2, wherein the selection data is distance information representing distances between elements constituting the three-dimensional shape data generated by the generating means and each of the plurality of image capturing devices.

7. the distance information is a distance image having the same width and height as the images captured by the plurality of image capturing devices, with a distance value stored in each pixel; 7. The image processing apparatus according to claim 6, wherein said selection means makes said selection based on a result of projecting the shape indicated by said three-dimensional shape data onto said distance image.

8. 3. The image processing device according to claim 2, wherein the selection data is weight information indicating the effectiveness of the texture when the texture of the object in the images captured by the plurality of imaging devices is pasted onto the three-dimensional shape data generated by the generation means.

9. the weighting information is a weighting image having the same width and height as the images captured by the plurality of image capturing devices, and in which a weighting value is stored for each pixel; 9. The image processing apparatus according to claim 8, wherein said selection means makes said selection based on a result of projecting the shape indicated by said three-dimensional shape data onto said weighted image.

10. 10. The image processing device according to claim 9, wherein the selection means, when projecting the shape indicated by the three-dimensional shape data onto the weighted image, projects a portion of the shape indicated by the three-dimensional shape data in which a feature of the object exists.

11. The image processing device according to claim 10, wherein the features of the object are feature points of a human face.

12. The feature of the object is a number written on a uniform worn by a person.

11. The image processing device according to claim 10.

13. the selection data is at least two or more pieces of information selected from the group consisting of depth of field information for each of the plurality of image capturing devices, foreground extraction area information for specifying an area to be subjected to foreground extraction in the images captured by each of the plurality of image capturing devices, distance information for indicating a distance between each of the plurality of image capturing devices and an element constituting the three-dimensional shape data generated by the generation means, and weight information for indicating a degree of effectiveness of the texture when the texture of the object in the images captured by the plurality of image capturing devices is pasted onto the three-dimensional shape data generated by the generation means; determining, from the plurality of image capturing devices, an image capturing device that captured an image to be used for identifying the object, based on the selection result obtained using the two or more pieces of information; 3. The image processing device according to claim 2.

14. 2. The image processing device according to claim 1, wherein the identification means projects the shape indicated by the three-dimensional shape data onto the texture of the object in the image captured by the imaging device selected by the selection means, and compares features of the object obtained from the texture data of the projection area with feature information indicating features of each object to identify the object corresponding to the three-dimensional shape data.

15. 15. The image processing apparatus according to claim 14, further comprising an updating unit that updates the feature information based on features of the object obtained from texture data of the projection area.

16. 2. The image processing apparatus according to claim 1, further comprising: an assigning unit for assigning information indicating the object identified by said identifying unit to the three-dimensional shape data generated by said generating unit.

17. the images captured by the plurality of imaging devices are moving images consisting of a plurality of frames, 17. The image processing apparatus according to claim 16, further comprising a tracking unit that tracks the three-dimensional shape data generated by the generating unit between frames based on the spatial information extracted by the extracting unit.

18. 18. The image processing device according to claim 17, wherein, when the identification means fails to identify a frame of interest among the plurality of frames in the moving image, the assignment means assigns, based on the result of the tracking by the tracking means, information indicating the object assigned to the three-dimensional shape data of the frame preceding the frame of interest to the three-dimensional shape data generated by the generation means for the frame of interest.

19. a generating means for generating three-dimensional shape data of an object from images captured by a plurality of imaging devices; an extraction step of extracting a three-dimensional area in an imaging space of the plurality of imaging devices corresponding to the three-dimensional shape data; a selection step of selecting, from the plurality of image capture devices, an image capture device that captures an image used to identify the object, based on spatial information indicating the three-dimensional region extracted in the extraction step; an identification step of identifying an object corresponding to the three-dimensional shape data based on an image captured by the imaging device selected in the selection step; An image processing method comprising:

20. A program for causing a computer to execute the image processing method according to claim 19.

Images