Image processing device, image processing method, and program

Abstract

To improve the accuracy of estimating the position of the target object. [Solution] The image processing apparatus 102 according to the present disclosure acquires shape data showing the three-dimensional shape of each object present in the target space, estimates the number of three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data for each shape data, separates the three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data based on the estimated number and a height image showing the height of the three-dimensional shape obtained by projecting the three-dimensional shape shown by the shape data onto a horizontal plane from above, and identifies the position of the separated three-dimensional shapes corresponding to the target object.

Description

Technical Field

[0001] The present disclosure relates to a technique for estimating the position of an object.

Background Art

[0002] There is a technique for generating an image corresponding to a view from an arbitrary virtual viewpoint (hereinafter referred to as a "virtual viewpoint") specified by a user or the like, based on a plurality of captured images obtained by capturing with a plurality of imaging devices (hereinafter referred to as "multi-viewpoint images"). Patent Document 1 discloses the following technique as preprocessing for generating the image (hereinafter referred to as a "virtual viewpoint image"). Specifically, in each captured image constituting the multi-viewpoint images, a region with a large temporal change is extracted as a foreground region, and a region with a small change is extracted as a background region. Subsequently, based on the extracted foreground region, a three-dimensional shape corresponding to an object existing in a space to be imaged by the plurality of imaging devices is estimated. Subsequently, the position of the object is estimated based on the estimated three-dimensional shape.

Prior Art Literature

Patent Literature

[0003]

Patent Document 1

Summary of Invention

Problem to be Solved by the Invention

[0004] The technology disclosed in Patent Document 1 assumes imaging in a stable state, such as a studio floor, and extracts the three-dimensional shape near the floor. However, in sports such as baseball, the state of the field surface changes over time due to play. In such cases, the region of the image in the captured image where the field surface has changed may be detected as the foreground region, and a three-dimensional shape corresponding to a non-existent object (hereinafter referred to as "false shape") may be estimated. Therefore, when the state of the field surface of the object being imaged changes over time, the technology disclosed in Patent Document 1 has the problem that the accuracy of estimating the position of the target object decreases due to the estimated false shape. [Means for solving the problem]

[0005] The image processing apparatus according to this disclosure includes: shape acquisition means for acquiring shape data showing the three-dimensional shape of each object present in a target space; and identification means for estimating the number of three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data for each piece of shape data, and based on the estimated number and a height image showing the height of the three-dimensional shape obtained by projecting the three-dimensional shape shown by the shape data onto a horizontal plane from above, separating the three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data, and identifying the position of the separated three-dimensional shapes corresponding to the target object. [Brief explanation of the drawing]

[0006] [Figure 1] This is a block diagram showing an example of the configuration of an image processing system according to the first embodiment. [Figure 2] This is a block diagram showing an example of the logical configuration of an image processing device according to the first embodiment. [Figure 3] This is a block diagram showing an example of the hardware configuration of an image processing device according to the first embodiment. [Figure 4] This flowchart shows an example of the processing flow of the image processing apparatus according to the first embodiment. [Figure 5] This flowchart shows an example of the flow of specific processing in the specific unit according to the first embodiment. [Figure 6] This is a diagram illustrating an example of processing in a specific unit according to the first embodiment. [Figure 7] This flowchart shows an example of the flow of separation and identification processes in the identification section according to the first embodiment. [Figure 8] This figure shows an example of a selection screen for receiving instructions to select an identifier according to the first embodiment. [Figure 9] This figure shows an example of automated piloting from a virtual viewpoint according to the first embodiment. [Figure 10] This figure shows an example of a display image according to the first embodiment. [Modes for carrying out the invention]

[0007] [First Embodiment] (Configuration of the image processing system) Figure 1 is a block diagram showing an example of the configuration of an image processing system according to the first embodiment. The image processing system includes a plurality of imaging devices 100, a synchronization device 101, an image processing device 102, and a display device 103. The image processing system may have one image processing device 102 or a plurality of image processing devices 102. In the following description, the image processing system will be described as having one image processing device 102.

[0008] Multiple imaging devices 100 are composed of digital video cameras and the like, and they synchronizely image the space to be imaged (hereinafter referred to as the "imaging area") from different directions. Specifically, each of the multiple imaging devices 100 receives a synchronization signal from the synchronization device 101 and performs imaging based on the received synchronization signal. The data of multiple imaged images (multi-view images) obtained by the synchronized imaging by the multiple imaging devices 100 is output to the image processing device 102. The multiple imaging devices 100 are installed, for example, to surround the imaging area so that objects (hereinafter simply referred to as "objects") present within the imaging area can be imaged from multiple directions.

[0009] The image processing device 102 acquires data from multi-view images output from multiple imaging devices 100, generates a virtual viewpoint image corresponding to the view from an arbitrary virtual viewpoint, and outputs data of a display image including the generated virtual viewpoint image to the display device 103. The display device 103 is composed of a liquid crystal display or the like, acquires data of the display image output from the image processing device 102, and displays the said display image.

[0010] (Configuration of the image processing device) Figure 2 is a block diagram showing an example of the logical configuration of the image processing device 102 according to the first embodiment. The image processing device 102 has an image acquisition unit 201, a shape estimation unit 202, a viewpoint acquisition unit 203, an image generation unit 204, a specific unit 205, and an output control unit 206 as its logical configuration. The processing of each part of the logical configuration of the image processing device 102 is performed by processing hardware such as an ASIC (Application Specific Integrated Circuit) built into the image processing device 102. This processing may also be performed by software using a computing device such as a CPU (Central Processor Unit) or GPU (Graphics Processor Unit) and memory built into the image processing device 102. Details of the processing of each part of the logical configuration of the image processing device 102 will be described later.

[0011] Referring to Figure 3, the hardware configuration of the image processing device 102 will be described when the processing of each part of the logical configuration of the image processing device 102 is performed by software execution. Figure 3 is a block diagram showing an example of the hardware configuration of the image processing device 102 according to the first embodiment. The image processing device 102 is composed of a computer, and the computer has a hardware configuration of a CPU 301, a GPU 302, a ROM 303, a RAM 304, a VRAM 305, and an auxiliary storage device 306. The computer also has a hardware configuration of a display unit 307, an operation unit 308, a communication unit 309, and a bus 310.

[0012] The CPU 301 controls the computer using programs and data stored in the ROM 303 or auxiliary storage device 306, etc., thereby causing the image processing device 102 to function as each part of its logical configuration. The ROM 303 stores programs and various data that do not require modification. The auxiliary storage device 306 is configured, for example, as a hard disk drive, and stores programs and various data such as image data or audio data. The RAM 304 operates as the work area of ​​the CPU 301 and temporarily stores programs and data supplied from the ROM 303 or auxiliary storage device 306, or data supplied from the outside via the communication unit 309.

[0013] The GPU 302 controls the computer in cooperation with the CPU 301 by using programs or data stored in the ROM 303, the auxiliary storage device 306 or the like, so as to function as each unit included in the logical configuration of the image processing apparatus 102. The VRAM 305 is a memory for graphics processing, operates as a work area for the GPU 302, and temporarily stores programs and data supplied from the ROM 303, the auxiliary storage device 306 or the like. Note that the image processing apparatus 102 may include one or more dedicated processing hardware different from the CPU 301 and the GPU 302, and the processing hardware may execute at least part of the processing performed by the CPU 301 or the GPU 302. Examples of dedicated processing hardware include ASIC, FPGA (Field Programmable Gate Array), and DSP (Digital Signal Processor).

[0014] The display unit 307 is configured by a liquid crystal display, an LED (light-emitting diode) or the like. The display unit 307 displays a GUI (Graphical User Interface) for a user to operate the image processing apparatus 102, a GUI for a user to view the status of the image processing apparatus 102, or the like. The operation unit 308 is configured by a keyboard, a mouse, a touch panel or the like, receives an operation performed by a user, and inputs various instructions corresponding to the operation to the CPU 301. The CPU 301 also operates as a display control unit that controls the display unit 307 and an operation control unit that controls the operation unit 308.

[0015] The communication unit 309 is used for communication with devices external to the image processing apparatus 102. For example, when the image processing apparatus 102 is wired-connected to an external device, a communication cable is connected to the communication unit 309. When the image processing apparatus 102 has a function of performing wireless communication with an external device, the communication unit 309 includes an antenna. The bus 310 communicably connects the above-described hardware configurations of the image processing apparatus 102 and transmits information. Hereinafter, a description is given on the assumption that the display unit 307 and the operation unit 308 exist inside the image processing apparatus 102, but at least one of the display unit 307 and the operation unit 308 may exist as a separate device outside the image processing apparatus 102.

[0016] The image acquisition unit 201 acquires data of multi-view images output from a plurality of imaging apparatuses 100. The image acquisition unit 201 also acquires camera parameters when each of the plurality of imaging apparatuses 100 captures each captured image constituting the multi-view image. The camera parameters include external parameters related to the position and orientation of the imaging apparatus that captured the captured image, and internal parameters related to focal length, principal point, resolution, and the like. The camera parameters may be stored in advance in the auxiliary storage device 306 or the like, or may be output from each imaging apparatus 100. The multi-view image data acquired by the image acquisition unit 201 and the camera parameters corresponding to each captured image obtained by imaging by each imaging apparatus or the plurality of imaging apparatuses are output to and stored in the auxiliary storage device 306 or the like.

[0017] The shape estimation unit 202 estimates the three-dimensional shape of an object using the multi-view images acquired by the image acquisition unit 201 and the camera parameters corresponding to each captured image that makes up the multi-view images. Specifically, for example, the shape estimation unit 202 first extracts the region containing the image of the object in each captured image as a silhouette by performing foreground-background separation on each captured image. The foreground-background separation method is well known, so its explanation is omitted. Note that if foreground-background separation is performed by the background subtraction method or the like, the background image data corresponding to each captured image is assumed to be stored in advance in the RAM 304 or auxiliary storage device 306, etc. Next, the shape estimation unit 202 estimates the three-dimensional shape of the object using the camera parameters of each imaging device 100 acquired by the image acquisition unit 201, using a method such as the view volume cross-section method. The view volume cross-section method is well known, so its explanation is omitted. The data regarding the three-dimensional shape of the object estimated by the shape estimation unit 202 is output to and stored in the auxiliary storage device 306, etc. Here, an object refers to a natural person or an item handled by a natural person whose three-dimensional shape is to be estimated.

[0018] The identification unit 205 uses the three-dimensional shape estimated by the shape estimation unit 202 to identify the position of a natural person in the imaging area. Furthermore, based on the identified position of each natural person, the identification unit 205 performs tracking for each natural person, generating information about the position of each natural person that changes over time (hereinafter referred to as "tracking information"). The tracking information generated by the identification unit 205 is output to and stored in the auxiliary storage device 306, etc. If the natural people are separated from each other at the time of imaging of the multi-view images used for estimating the three-dimensional shape by the shape estimation unit 202, then individual three-dimensional shapes can be estimated for each natural person. However, if the natural people are in contact with or close to each other, there is a possibility that the three-dimensional shapes of multiple natural people may be estimated as a single three-dimensional shape. Therefore, the identification unit 205 considers the possibility that the three-dimensional shapes of multiple natural people may be estimated as a single three-dimensional shape and performs the following processing. Specifically, the identification unit 205 separates the three-dimensional shape corresponding to each natural person from a three-dimensional shape that may include three-dimensional shapes corresponding to multiple natural persons, and identifies the position of each three-dimensional shape corresponding to a natural person.

[0019] The viewpoint acquisition unit 203 acquires information related to a virtual viewpoint (hereinafter referred to as "virtual viewpoint information"). The virtual viewpoint information includes the position of the virtual viewpoint, which corresponds to the external parameters of the imaging device, information related to the direction of the line of sight at the virtual viewpoint, information related to the focal length and principal point, which correspond to the internal parameters, and information related to the time code of the virtual viewpoint image to be generated. Hereinafter, the direction of the line of sight at the virtual viewpoint will be referred to as the "virtual viewpoint orientation". Specifically, for example, the user inputs the position and orientation of the virtual viewpoint, the focal length and principal point at the virtual viewpoint, and the time code of the virtual viewpoint image to be generated by operating the operation unit 308. The viewpoint acquisition unit 203 acquires the virtual viewpoint information by receiving a signal corresponding to the input from the operation unit 308. Alternatively, for example, the viewpoint acquisition unit 203 may acquire information related to a virtual viewpoint by using tracking information generated by the identification unit 205 to determine the position and orientation of a virtual viewpoint that automatically tracks a predetermined object (natural person), for example.

[0020] The image generation unit 204 generates a virtual viewpoint image. Specifically, the image generation unit 204 first acquires the necessary materials for generating a virtual viewpoint image corresponding to a time code, based on the time code included in the virtual viewpoint information acquired by the viewpoint acquisition unit 203, by reading them from the auxiliary storage device 306, etc. The materials necessary for generating a virtual viewpoint image include, for example, data relating to the three-dimensional shape of an object, captured image data, camera parameters, and background model and background texture image data, all corresponding to the time code. Subsequently, the image generation unit 204 uses the acquired materials to generate a virtual viewpoint image corresponding to the view from the virtual viewpoint indicated by the virtual viewpoint information. The method for generating a virtual viewpoint image corresponding to an arbitrary virtual viewpoint using data relating to the three-dimensional shape of an object, captured image data, camera parameters, and background model and background texture image data is well known, so a detailed explanation is omitted. The virtual viewpoint image data generated by the image generation unit 204 is output to the output control unit 206.

[0021] The output control unit 206 generates a display image including the virtual viewpoint image generated by the image generation unit 204, outputs the data of the generated display image to the display device 103, and causes the display device 103 to display the display image.

[0022] (Operation of the image processing device) An example of the operation of the image processing device 102 will be described with reference to Figures 4 and 5. Figure 4 is a flowchart showing an example of the processing flow of the image processing device 102 according to the first embodiment. In the following description, the symbol "S" means step. First, in S401, the image acquisition unit 201 acquires a multi-view image and camera parameters corresponding to each captured image that makes up the multi-view image. The data of the multi-view image and camera parameters acquired in S401 are stored in the auxiliary storage device 306, etc. Next, in S402, the shape estimation unit 202 estimates the three-dimensional shape of the object using the multi-view image acquired in S401 and the camera parameters corresponding to each captured image. The data of the three-dimensional shape estimated in S402 is stored in the auxiliary storage device 306, etc. Next, in S403, the viewpoint acquisition unit 203 acquires virtual viewpoint information. In S404, the identification unit 205 executes the identification process described later. Details of the processing from S405 onwards will be described later.

[0023] (Specific processing in a specific section) Figure 5 is a flowchart illustrating an example of the identification process in the identification unit 205 according to the first embodiment, that is, the flow of the identification process in S404 shown in Figure 4. In this embodiment, as an example, the imaging target is a baseball game, and the identification unit 205 is described as performing the identification of the position of each natural person playing the baseball game (hereinafter referred to as "player") and the generation of tracking information related to each player. In particular, in this embodiment, as an example, a case will be described in which the three-dimensional shapes corresponding to each of the three players and the false shapes estimated in accordance with the changes in the state of the field surface over time are included in a single three-dimensional shape as a result of estimation by the shape estimation unit 202.

[0024] Figure 6 is a diagram illustrating an example of processing in the specific unit 205 according to the first embodiment. Specifically, Figure 6(a) shows the three-dimensional shapes corresponding to each of the three players obtained by estimation by the shape estimation unit 202, and the false shapes 610. Figures 6(b) to (d) will be described later.

[0025] In the processing of S404, first, in S501, the identification unit 205 selects an arbitrary time code from the period indicated by the time code included in the virtual viewpoint information acquired in S403 (hereinafter referred to as the "processing period"). The identification unit 205 sequentially selects one or more time codes included in the processing period in S501 by performing the repetitive processing described later. In the repetitive processing described later, the identification unit 205 will be described as selecting the time code closest to the start of the processing period from among the unselected time codes in all the time codes included in the processing period. Next, in S502, the identification unit 205 acquires all three-dimensional shapes corresponding to the time code selected in S501 (hereinafter referred to as the "selected time code"). Specifically, the identification unit 205 acquires all three-dimensional shape data corresponding to the selected time code by reading it from the auxiliary storage device 306, etc.

[0026] Next, in S503, the identification unit 205 selects an arbitrary three-dimensional shape from the three-dimensional shapes acquired in S502. Then, in S504 and S505, the identification unit 205 determines, based on the three-dimensional shape selected in S503 (hereinafter referred to as the "selected shape"), whether or not the selected shape corresponds to the three-dimensional shape of the target object, which is a natural person (player). Specifically, the identification unit 205 determines whether or not the size of the selected shape selected in S503 is less than or equal to a predetermined threshold.

[0027] Here, the size of the three-dimensional shape refers to at least one of the volume, surface area, and external dimensions of the three-dimensional shape. For example, if the data of the three-dimensional shape is represented by a dense point cloud, the identification unit 205 can obtain an approximate volume of the object based on the number of points included in the point cloud. Also, if the data of the three-dimensional shape is represented by a point cloud indicating the surface shape of the object, the identification unit 205 can obtain an approximate surface area of ​​the object based on the number of points included in the point cloud of the three-dimensional shape. Regarding the external dimensions, at least one of the width, depth, and height of the three-dimensional shape can be used.

[0028] The threshold mentioned above is based on the surface area of ​​a typical adult, which is 1.6 m². 2 Assuming it is approximately (square meters), and taking into account changes in surface area due to factors such as body size, clothing, and posture, for example, 1.6m 2 A threshold of 1.2 square meters, which is approximately 70% of the total surface area, can be used. Also, the volume of a typical adult is 60,000 cm³. 3 Assuming it is approximately (cubic centimeters), for example, taking that change into consideration, 60,000 cm 3 42,000 cm, which is about 70% of the total. 3 This can be used as the volume threshold. Furthermore, considering the average adult height and shoulder width, 40 cm (centimeters) can be used as the width, depth, or height threshold. With such thresholds, even when a natural person is crouching, their external dimensions will not normally fall within these thresholds.

[0029] Hereafter, the data for the three-dimensional shape will be described as being represented by a point cloud indicating the surface shape of the object. In this case, for example, the identification unit 205 determines in S504 whether the surface area of ​​the selected shape is less than or equal to a predetermined threshold, and in S505 whether the external dimensions of the selected shape are less than or equal to a predetermined threshold. If it is determined in S504 or S505 that the dimensions are less than or equal to the threshold, the identification unit 205 determines that the selected shape is not a three-dimensional shape corresponding to a natural person and executes the process in S510, which will be described later. If it is determined in S504 and S505 that the dimensions are not less than or equal to the threshold, the identification unit 205 determines that the selected shape is a three-dimensional shape that includes three-dimensional shapes corresponding to one or more natural people and executes the process in S506, which will be described later.

[0030] In S506, the specific unit 205 generates an image (hereinafter referred to as the "height image") that shows the depth of the selected shape selected in S503 when viewed from above, i.e., the height from the field surface. Figures 6(b) to (d) show examples of height images. Specifically, the height image shown in Figure 6(b) corresponds to the three-dimensional shape shown in Figure 6(a) when viewed from above. Figures 6(c) and (d) will be described later. In the height images shown in Figures 6(b) to (d), as an example, areas with lower heights, i.e., areas with a larger distance from above, are represented by lower brightness, and areas with higher heights, i.e., areas with a smaller distance from above, are represented by higher brightness.

[0031] Next, in S507, the specific unit 205 performs a shrinkage process on the height image generated in S506, thereby changing the pixel values ​​corresponding to pixels with low brightness in the height image to pixel values ​​corresponding to black. Through this process, pixel values ​​in areas corresponding to three-dimensional shapes (false shapes) near the field surface that were mistakenly detected as objects due to changes in the field floor surface over time can be removed from the height image. Similarly, pixel values ​​in areas corresponding to fine three-dimensional shapes such as noise generated by other processes such as foreground extraction and shape estimation can also be removed.

[0032] Next, in S508, the identification unit 205 identifies the number of three-dimensional shapes corresponding to each natural person included in the selected shape, based on the modified height image obtained by the modification process in S507 (hereinafter referred to as the "modified height image"). Specifically, for example, the identification unit 205 first performs a binarization process on the modified height image. Subsequently, the identification unit 205 identifies the number of three-dimensional shapes corresponding to natural people included as three-dimensional shape images in the modified height image obtained in S507, based on the area of ​​the region composed of pixels other than black in the binarized modified height image (hereinafter referred to as the "effective region"). Specifically, the projected area from directly above a natural person is 0.5m 2 Since it is approximately (square meters), the specific unit 205 sets the number of pixels included in the effective area to 0.5 m 2By dividing by the corresponding number of pixels, the number of three-dimensional shapes corresponding to natural people included as three-dimensional images in the height image is identified. For example, if the value of the quotient is approximately 3, the identification unit 205 identifies that the modified height image contains three three-dimensional images corresponding to three natural people.

[0033] Next, in S509, the identification unit 205 performs separation processing of three-dimensional shapes corresponding to natural persons and position identification processing of the separated three-dimensional shapes, based on the modified height image obtained in S507 and the number of three-dimensional shapes corresponding to natural persons identified in S508. Through the separation and identification processing in S509, even if the selected shape includes three-dimensional shapes corresponding to multiple natural persons, the position of the three-dimensional shape corresponding to each natural person can be identified. Details of the separation and identification processing in S509 will be described later.

[0034] Next, in S510, the identification unit 205 determines whether all three-dimensional shapes acquired in S502 have been selected as selected shapes in S503. If it is determined in S510 that all three-dimensional shapes have been selected, the identification unit 205 executes the process in S511, which will be described later. Otherwise, the identification unit 205 returns to the process in S503 and repeatedly executes the processes from S503 to S510 until it is determined in S510 that all three-dimensional shapes have been selected. In the process in S503 of the above-mentioned repetitive process, the identification unit 205 selects an arbitrary three-dimensional shape from among one or more three-dimensional shapes that have not been selected so far as the selected shape. Through the repetitive process from S503 to S510, the position of the three-dimensional shape corresponding to each of the natural people present in the imaging area at the selected time code is determined.

[0035] In S511, the identification unit 205 performs tracking processing for the three-dimensional shape corresponding to each natural person. Specifically, the identification unit 205 performs tracking processing by comparing the positions of the three-dimensional shapes corresponding to each natural person in the selected time code identified in S510 and in time codes different from the selected time code selected in the repeating process. Specifically, if only one time code has been selected in S510, the identification unit 205 performs the following processing. For example, in this case, the identification unit 205 associates the information indicating the position of the three-dimensional shape corresponding to each natural person in the selected time code selected in S501 with an identifier such as a serial number that can uniquely identify the selected time code and the three-dimensional shape. Subsequently, the identification unit 205 stores the associated information as tracking information in the auxiliary storage device 306 or the like.

[0036] In contrast, if multiple time codes are selected in S501 through repeated processing, the identification unit 205 performs the following processing. For example, in this case, the identification unit 205 compares the positions of the three-dimensional shapes corresponding to each natural person in the selected time code and the time code immediately preceding the selected time code. Next, the identification unit 205 identifies the three-dimensional shape corresponding to the natural person in the immediately preceding time code whose position is closest to the position of each three-dimensional shape corresponding to the natural person in the selected time code. Next, the identification unit 205 associates the selected time code with the same identifier that is associated with the identified position of the three-dimensional shape corresponding to the natural person in the immediately preceding time code. Next, the identification unit 205 stores the associated information as tracking information in the auxiliary storage device 306 or the like.

[0037] After S511, in S512, the identification unit 205 determines whether all time codes for the processing period were selected as selected time codes in S501. If it is determined in S512 that at least some of the time codes were not selected as selected time codes, the identification unit 205 returns to the process of S501. In this case, the identification unit 205 repeatedly executes the processes from S501 to S512 until it is determined in S512 that all time codes have been selected as selected time codes. If it is determined in S512 that all time codes have been selected as selected time codes, the identification unit 205 terminates the process shown in the flowchart in Figure 5, i.e., the process of S404.

[0038] In this embodiment, it is explained that captured images corresponding to all time codes for the processing period are pre-stored in the auxiliary storage device 306, etc. However, the scope of application of the technology of this disclosure is not limited to such cases. For example, the image processing device 102 may acquire the data of the captured image obtained by imaging by the imaging device 100 each time the imaging device 100 outputs the data, and sequentially execute the processing of the flowchart described above based on the data. In this case, it is desirable that the specific unit 205 executes the processing of the flowchart shown in Figure 5, that is, the processing of S404 shown in Figure 4, after waiting for the processing of S403 based on the newly acquired multi-view image corresponding to the latest time code to be completed.

[0039] (Separation and specific processing in specific parts) Figure 7 is a flowchart showing an example of the flow of the separation and identification processes in the identification unit 205 according to the first embodiment, that is, the separation and identification processes in S509 shown in Figure 5. In S509, first, in S701, the identification unit 205 performs a region segmentation process (also called "segmentation") on the effective region of the modified height image after binarization obtained by the binarization process in S508.

[0040] Figure 6(c) shows an example of segmentation results for a modified height image obtained by binarization processing using S508. In the example shown in Figure 6(c), region 601 contains only the three-dimensional shape image corresponding to one natural person, while region 302 contains the three-dimensional shape images corresponding to two natural people.

[0041] Following S701, in S702, the identification unit 205 selects an arbitrary independent region from among the one or more segments (also called "independent regions") obtained as a result of the segmentation in S701. Next, in S703, the identification unit 205 determines whether the area of ​​the independent region selected in S702 (hereinafter referred to as the "selected region") is less than or equal to a predetermined threshold. Here, the threshold is, for example, an area that can contain a three-dimensional image corresponding to one natural person, which is at least 0.3 m². 2 Let it be so.

[0042] If it is determined in S703 that the value is below the threshold, the identification unit 205 executes the process in S708, described later, assuming that the size of the selected region is such that it cannot contain an image of a three-dimensional shape corresponding to a natural person. Otherwise, in S704, the identification unit 205 determines whether the size of the selected region corresponds to the size of an image of a three-dimensional shape corresponding to one natural person. If it is determined in S704 that the size corresponds to the size of an image of a three-dimensional shape corresponding to one natural person, the identification unit 205 executes the process in S705, described later, assuming that the selected region is a region that contains an image of a three-dimensional shape corresponding to one natural person. Otherwise, the identification unit 205 executes the process in S706, described later, assuming that the selected region is a region that contains images of three-dimensional shapes corresponding to two or more natural people.

[0043] In S705, the identification unit 205 identifies the position of the three-dimensional shape corresponding to the selected region. Specifically, for example, the identification unit 205 identifies an arbitrary position, such as the centroid of the three-dimensional shape, as the position of the three-dimensional shape. The method for identifying the position of the three-dimensional shape corresponding to the selected region in the identification unit 205 is not limited to this, and for example, the position may be identified by the following method. For example, the identification unit 205 first identifies a predetermined position, such as the centroid of the selected region, on the plane of the modified height image as the position on the horizontal plane of the three-dimensional shape corresponding to the selected region. Subsequently, the identification unit 205 identifies the vertical position of the three-dimensional shape corresponding to the selected region by calculating the height of the three-dimensional shape based on the pixel values ​​of the selected region in the modified height image. After S705, the identification unit 205 executes the process in S708, which will be described later.

[0044] In S706, the identification unit 205 changes the pixel values ​​(pixel values) of pixels with low brightness in the modified height image to pixel values ​​corresponding to black. After S706, in S707, the identification unit 205 uses the image obtained from the modification process in S706 as the modified height image and recursively executes processes corresponding to the processes from S701 to S708 described later. Through such recursive processing, a region containing images of three-dimensional shapes corresponding to two or more natural persons is separated into multiple regions, each containing an image of a three-dimensional shape corresponding to one natural person, and the position of the three-dimensional shape corresponding to each natural person can be identified. After S707, the identification unit 205 executes the process in S708 described later.

[0045] Figure 6(d) shows an example of a modified height image obtained by recursive region segmentation processing. Specifically, the modified height image shown in Figure 6(d) shows three segments (independent regions 611 to 613) obtained by recursive region segmentation processing on region 302 shown in Figure 6(c). Even with recursive processing, region separation may not be properly performed until a region containing an image of a three-dimensional shape corresponding to a single natural person is obtained. For this reason, an upper limit on the number of recursive processing executions may be set in advance, and the identification unit 205 may stop the recursive processing when the number of executions reaches the upper limit, and identify the position of the three-dimensional shape corresponding to the independent region being processed at that time.

[0046] In S708, the identification unit 205 determines whether all independent regions obtained as a result of segmentation in S702 have been selected. If it is determined in S708 that at least some independent regions have not been selected, the identification unit 205 returns to the process in S702 and repeatedly executes the processes from S702 to S708 until it is determined in S708 that all independent regions have been selected. If it is determined in S708 that all independent regions have been selected, in S709, the identification unit 205 counts the number of three-dimensional shapes whose positions were determined in S705 and in the recursive process in S705.

[0047] After S709, in S710, the identification unit 205 determines whether the number of three-dimensional shapes aggregated in S709 matches the number of three-dimensional shapes corresponding to each natural person identified in S508. If it is determined in S710 that they match, the identification unit 205 terminates the process shown in the flowchart in Figure 7, i.e., the process shown in S509 in Figure 5. Otherwise, since proper separation processing may not have been performed, the identification unit 205 may change the parameters related to the separation processing and execute the processes from S701 to S710 again.

[0048] Specifically, for example in this case, first, in S711, the identification unit 205 determines whether the parameter modification process related to the separation process, which will be described later, has already been executed. If it is determined in S711 that the parameter modification process related to the separation process has not yet been executed, then in S712, the identification unit 205 modifies the parameters related to the separation process. Specifically, for example, the identification unit 205 modifies the parameter related to the separation process that relates to the interval for changing pixels with low brightness to the pixel value corresponding to black. After S712, the identification unit 205 discards the processing results from S701 to S710 before the parameter modification related to the separation process, returns to the process in S701, and executes the processes from S701 onwards.

[0049] If it is determined in S711 that the parameter modification process related to the separation process has already been executed, the identification unit 205 terminates the process shown in the flowchart in Figure 7, i.e., the process in S509 shown in Figure 5. In this case, the identification unit 205 executes the processes from S510 onwards shown in Figure 5, using, for example, the results of the separation process using the modified parameters obtained in S705 and the process in S705 in the recursive process. This is because there may be errors in the identification of the number of natural persons identified in S508, or separation may be difficult depending on the degree of overlap between natural persons. In this embodiment, a configuration in which the parameter modification related to the separation process is performed only once is described, but the modification may be performed multiple times. If the modification is performed multiple times, the processing time will increase, and the process may not be completed within a predetermined period. Therefore, it is desirable that the number of times the modification is performed be appropriately determined considering the processing capacity of the CPU 301, the period, etc.

[0050] Furthermore, by using the tracking information generated and stored in S511, it is possible to track a natural person corresponding to a three-dimensional shape corresponding to a specific identifier within the imaging area. In addition, depending on the sport, the field surface may change over time due to roughness, etc. However, in the process of separating the image of the three-dimensional shape corresponding to a natural person, the image processing device 102 according to this embodiment sequentially deletes the three-dimensional shape near the field surface. Therefore, according to the image processing device 102 according to this embodiment, even if false shapes are generated near the field surface due to changes in the field surface over time, it is possible to perform the tracking process while eliminating the influence of the false shapes.

[0051] Furthermore, in the tracking process, during the comparison process between the position of the three-dimensional shape of the timecode to be processed (selected timecode) and the position of the three-dimensional shape of the timecode corresponding to a past point in time, the direction and speed of movement of the three-dimensional shape can be determined from the difference between these positions. In this case, the identification unit 205 may include information regarding the direction and speed of movement for each identified three-dimensional shape in the tracking information and store it in the auxiliary storage device 306 or the like.

[0052] The processing from S405 onwards will now be explained. After S404, in S405, the image generation unit 204 generates a virtual viewpoint image based on the multi-view images and camera parameters acquired in S401, the three-dimensional shape data estimated in S402, and the virtual viewpoint information acquired in S403. Next, in S406, the output control unit 206 generates a display image including the virtual viewpoint image generated in S405, outputs the data of the generated display image to the display device 103, and displays the display image on the display device 103. After S406, the image processing device 102 completes the processing shown in the flowchart in Figure 4.

[0053] In this embodiment, information such as the position and orientation of the virtual viewpoint used in the virtual viewpoint image generation process by the image generation unit 204 is pre-included in the virtual viewpoint information, and the virtual viewpoint information is described as being set by the user operating the operation unit 308. However, the position and orientation of the virtual viewpoint are not limited to those set by the user operating the operation unit 308.

[0054] For example, the viewpoint acquisition unit 203 may acquire information on the position and orientation of the virtual viewpoint by determining the position and orientation of the virtual viewpoint based on the tracking information generated in S511. Specifically, for example, the viewpoint acquisition unit 203 first acquires the tracking information corresponding to a predetermined identifier from the tracking information corresponding to the processing period indicated by the time code etc. included in the virtual viewpoint information by reading it from the auxiliary storage device 306 etc. The identifier of the tracking information acquired by the viewpoint acquisition unit 203 is determined by, for example, the following method. First, the image generation unit 204 generates a virtual viewpoint image for receiving instructions to select an identifier. The generated virtual viewpoint image is displayed on the display device 103 via processing by the output control unit 206.

[0055] Figure 8 shows an example of a selection screen for receiving instructions to select an identifier according to the first embodiment. In the selection screen shown in Figure 8, for example, images 801,801 of identifiers associated with the three-dimensional shape corresponding to each player are superimposed and displayed near the player's image in the virtual viewpoint image. The user confirms the contents of the identifier images 801,801 and selects an identifier by operating the operation unit 308. In this way, the image processing device 102 receives instructions from the user to select an identifier.

[0056] When the user provides an instruction to select an identifier, the viewpoint acquisition unit 203 acquires tracking information corresponding to the selected identifier. Subsequently, based on the acquired tracking information, the viewpoint acquisition unit 203 determines the position and orientation of the virtual viewpoint so that it fits within the field of view of the virtual viewpoint image, following the movement of the three-dimensional shape associated with the selected identifier. Hereafter, the virtual viewpoint image will be described as a moving image, and at least one of the position and orientation of the virtual viewpoint will be automatically changed based on the tracking information in accordance with the movement of the three-dimensional shape associated with the selected identifier. Hereafter, the automatic change of at least one of the position and orientation of the virtual viewpoint based on the tracking information will be referred to as "automatic control of the virtual viewpoint."

[0057] Figure 9 shows an example of automatic control of a virtual viewpoint according to the first embodiment. The viewpoint acquisition unit 203 may, for example, as shown in Figure 9(a), rotate the virtual viewpoint with the position 900 of the three-dimensional shape corresponding to the natural person indicated by the tracking information as the rotation center of the virtual viewpoint, and determine the position and orientation of the virtual viewpoint so that the direction of the line of sight of the virtual viewpoint is directed toward position 900. Alternatively, the viewpoint acquisition unit 203 may, for example, as shown in Figure 9(b), fix the position of the virtual viewpoint and determine the position and orientation of the virtual viewpoint so that the direction of the line of sight of the virtual viewpoint is directed toward position 900. The method of automatic control of a virtual viewpoint shown in Figure 9 is merely an example and is not limited thereto.

[0058] Furthermore, when the viewpoint acquisition unit 203 performs automatic control of a virtual viewpoint based on tracking information, it is desirable to perform smoothing processing on the time-series position of the three-dimensional shape corresponding to the natural person obtained from the tracking information, thereby smoothing the position and movement of the virtual viewpoint. The reason for performing smoothing processing is that the position of the three-dimensional shape points identified by the identification unit 205 may contain errors due to the influence of the natural person's posture or the estimation accuracy in the shape estimation process. Therefore, if smoothing processing is not performed, the position and posture of the virtual viewpoint may contain minor fluctuations.

[0059] In the example described above, identifier images 801 and 802 were superimposed on a virtual viewpoint image, and the user selected one of these identifiers. However, the method is not necessarily limited to this. For example, the image processing device 102 may be configured to acquire an image that allows the entire field used in the competition to be seen (hereinafter referred to as an "overhead viewpoint image"), and to superimpose the identifier images onto the overhead viewpoint image. In this case, for example, the image processing device 102 may be configured so that the user can change the target object that is followed by the automatic control of the virtual viewpoint by operating the operation unit 308 to select the identifier images superimposed on the overhead viewpoint image.

[0060] When automatic control of the virtual viewpoint based on tracking information is in operation, the position indicated by the tracking information may become discontinuous if a natural person corresponding to the three-dimensional shape associated with a selected identifier leaves the imaging area or enters the imaging area. If automatic control based on tracking information is continued in such cases, the position or orientation of the virtual viewpoint may change rapidly, resulting in a virtual viewpoint image that is difficult to view. Therefore, for example, the viewpoint acquisition unit 203 may stop automatic control of the virtual viewpoint if the position of the three-dimensional shape corresponding to the natural person to be processed in the time code indicated by the tracking information is more than a predetermined distance away from the position in the previous time code. Also, when the image processing device 102 is operating by automatic control, if the playback speed of the virtual viewpoint image is increased to double speed playback, etc., the viewpoint acquisition unit 203 may perform the following processing. Specifically, in this case, for example, the viewpoint acquisition unit 203 increases the aforementioned distance threshold for determining whether a rapid change in the position or orientation of the virtual viewpoint is possible in proportion to the playback speed.

[0061] Furthermore, the image processing device 102 may indicate to the user whether or not the image processing device 102 is operating by automatic control of the virtual viewpoint. Figure 10 is a diagram showing an example of a display image shown in the display device 103 according to the first embodiment. As shown as an example in Figure 10, by highlighting or otherwise indicating that automatic control of the virtual viewpoint is being performed, the user can easily grasp the state of the virtual viewpoint in the image processing device 102.

[0062] The image processing device 102 configured as described above can suppress the decrease in the accuracy of estimating the position of the target object due to false shapes that occur as a result of changes in the field surface over time.

[0063] [Modified version of the first embodiment] In the first embodiment, the identification unit 205 was described in which the number of three-dimensional shapes corresponding to natural persons included in the three-dimensional shape was determined by the area obtained when the three-dimensional shape obtained by the estimation process by the shape estimation unit 202 is projected onto the field surface. However, the method for determining the number of three-dimensional shapes corresponding to natural persons is not limited to this. For example, the number of three-dimensional shapes corresponding to natural persons may be determined using the size of the three-dimensional shape, such as the volume, surface area, or external dimensions of the three-dimensional shape obtained by the estimation process by the shape estimation unit 202. In this case, the index of the size of the three-dimensional shape corresponding to one natural person may be, for example, the volume, surface area, or external dimensions of a typical adult as described above in the first embodiment.

[0064] Furthermore, in the first embodiment, the three-dimensional shape data obtained by the estimation process by the shape estimation unit 202 is stored in an auxiliary storage device 306 or the like, and the identification unit 205 acquires and processes the three-dimensional shape data by reading it from the auxiliary storage device 306 or the like. However, the method of acquiring the three-dimensional shape data by the identification unit 205 is not limited to this. For example, the shape estimation unit 202 may store the estimated three-dimensional shape data in the auxiliary storage device 306 or the like and transmit it to the identification unit 205. In this case, the identification unit 205 may be configured to receive the three-dimensional shape data transmitted from the shape estimation unit 202 and sequentially perform processing on the received three-dimensional shape data.

[0065] Furthermore, in the first embodiment, a method was described in which the specific unit 205 removes the false shape 610 as shown in Figure 6(a) by performing a shrinkage process on the height image to change the pixel values ​​of the region corresponding to the false shape 610 in the height image to pixel values ​​corresponding to black. However, the method of removing false shapes is not limited to this. For example, the specific unit 205 may remove false shapes by performing the following process. Specifically, for example, first, the specific unit 205 performs region division processing (segmentation) on the effective region in the height image. Subsequently, the specific unit 205 considers the divided segments (independent regions) to be false shapes if their area is less than or equal to a predetermined size, such as an area corresponding to 1000 pixels, and removes them from subsequent processing.

[0066] Furthermore, although the first embodiment was described using baseball as an example, the scope of application of the technology disclosed herein is not limited to baseball. For example, the technology disclosed herein may be applied to imaging other sports such as soccer or bicycle racing, or to imaging performances in studios, stages, etc.

[0067] Furthermore, in the first embodiment, the identification unit 205 changed the parameters related to the separation process in the S712 process and then re-executed the processes from S701 onwards using the changed parameters related to the separation process. However, the processing when the number of three-dimensional shapes aggregated in S709 does not match the number of three-dimensional shapes corresponding to each natural person identified in S508 is not limited to the process of changing the parameters related to the separation process. For example, in this case, the identification unit 205 may be configured to change the separation process method itself. Specifically, for example, in this case, the identification unit 205 changes the separation process method in the identification unit 205 as follows. For example, first, in S706, the identification unit 205 performs a process to change the value of pixels with low brightness in the modified height image (pixel value) to the pixel value corresponding to black. Subsequently, the identification unit 205 performs a process to remove the fine areas created by the change by image contraction and expansion processing.

[0068] [Other embodiments] The technology of this disclosure can also be realized by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be realized by a processing circuit such as an ASIC (Application-Specific Integrated Circuit) that implements one or more functions.

[0069] Furthermore, within the scope of this disclosure, the technologies described herein allow for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0070] [Technical Features of This Disclosure] This disclosure includes the following configurations, methods, and programs.

[0071] (Composition 1) A shape acquisition means for acquiring shape data that shows the three-dimensional shape of each object present in the target space, For each of the shape data, the number of three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data is estimated, and based on the estimated number and a height image showing the height of the three-dimensional shape obtained by projecting the three-dimensional shape shown by the shape data onto a horizontal plane from above, the three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data are separated, and the position of the separated three-dimensional shapes corresponding to the target object is identified. An image processing apparatus characterized by having

[0072] (Configuration 2) The identification means estimates the number of three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data, based on at least one of the volume, surface area, external dimensions, and area when the three-dimensional shape shown by the shape data is projected onto a horizontal plane. An image processing apparatus according to configuration 1, characterized by the above.

[0073] (Composition 3) The identifying means, if the number of three-dimensional shape images corresponding to the target object that may be included in the independent region of the height image is less than the estimated number, divides the independent region into a number of independent regions corresponding to the estimated number by deleting low-luminance regions in the height image. An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0074] (Composition 4) The identifying means determines that, for those independent regions in the height image whose size is less than or equal to a predetermined size, the image does not contain a three-dimensional shape corresponding to the target object. An image processing apparatus according to any one of configurations 1 to 3 characterized by the above.

[0075] (Composition 5) A viewpoint acquisition means that acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight in the said virtual viewpoint, Image generation means for generating a virtual viewpoint image corresponding to the view from the virtual viewpoint based on the shape data and the virtual viewpoint information, Having further, An image processing apparatus according to any one of configurations 1 to 4 characterized by the above.

[0076] (Composition 6) The identification means identifies a three-dimensional shape corresponding to the same target object based on the difference between the positions of the three-dimensional shapes corresponding to the target object included in the three-dimensional shape indicated by the shape data corresponding to each identified time point, and generates tracking information that associates the same identifier with the identified three-dimensional shapes corresponding to the same target object. An image processing apparatus according to any one of configurations 1 to 5 characterized by the above.

[0077] (Composition 7) A viewpoint acquisition means that acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight in the said virtual viewpoint, Image generation means for generating a virtual viewpoint image corresponding to the view from the virtual viewpoint based on the shape data and the virtual viewpoint information, It further possesses, The viewpoint acquisition means acquires the virtual viewpoint information by determining the position of the virtual viewpoint and the direction of the line of sight at the virtual viewpoint, based on the tracking information, such that a three-dimensional image of the target object corresponding to a specific identifier is included in the virtual viewpoint image. The image processing apparatus according to configuration 6, characterized by the above.

[0078] (method) A shape acquisition process that acquires shape data showing the three-dimensional shape of each object present in the target space, For each of the shape data, the number of three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data is estimated, and based on the estimated number and a height image showing the height of the three-dimensional shape obtained by projecting the three-dimensional shape shown by the shape data onto a horizontal plane from above, the three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data are separated, and the position of the separated three-dimensional shapes corresponding to the target object is identified. Image processing methods including [specific details omitted].

[0079] (program) A program for causing a computer to function as an image processing device described in any one of configurations 1 through 7. [Explanation of Symbols]

[0080] 102 Image Processing Device 201 Image Acquisition Unit 202 Shape estimation part 203 Viewpoint acquisition unit 204 Image Generation Unit 205 Specific section

Claims

1. A shape acquisition means for acquiring shape data that shows the three-dimensional shape of each object present in the target space, For each of the shape data, the number of three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data is estimated, and based on the estimated number and a height image showing the height of the three-dimensional shape obtained by projecting the three-dimensional shape shown by the shape data onto a horizontal plane from above, the three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data are separated, and the position of the separated three-dimensional shapes corresponding to the target object is identified. An image processing apparatus characterized by having

2. The identification means estimates the number of three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data, based on at least one of the volume, surface area, external dimensions, and area when the three-dimensional shape shown by the shape data is projected onto a horizontal plane. The image processing apparatus according to claim 1, characterized in that

3. The identifying means, if the number of three-dimensional shape images corresponding to the target object that may be included in the independent region of the height image is less than the estimated number, divides the independent region into a number of independent regions corresponding to the estimated number by deleting low-luminance regions in the height image. The image processing apparatus according to claim 1, characterized in that

4. The aforementioned identification means determines that, for those independent regions in the height image whose size is less than or equal to a predetermined size, the image does not contain a three-dimensional shape corresponding to the target object. The image processing apparatus according to claim 1, characterized in that

5. A viewpoint acquisition means that acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight in the said virtual viewpoint, Image generation means for generating a virtual viewpoint image corresponding to the view from the virtual viewpoint based on the shape data and the virtual viewpoint information, Having further, The image processing apparatus according to claim 1, characterized in that

6. The identification means identifies a three-dimensional shape corresponding to the same target object based on the difference between the positions of the three-dimensional shapes corresponding to the target object included in the three-dimensional shape indicated by the shape data corresponding to each identified time point, and generates tracking information that associates the same identifier with the identified three-dimensional shapes corresponding to the same target object. The image processing apparatus according to claim 1, characterized in that

7. A viewpoint acquisition means that acquires virtual viewpoint information indicating the position of a virtual viewpoint and the direction of the line of sight in the said virtual viewpoint, Image generation means for generating a virtual viewpoint image corresponding to the view from the virtual viewpoint based on the shape data and the virtual viewpoint information, It further possesses, The viewpoint acquisition means acquires the virtual viewpoint information by determining the position of the virtual viewpoint and the direction of the line of sight at the virtual viewpoint, based on the tracking information, such that a three-dimensional image of the target object corresponding to a specific identifier is included in the virtual viewpoint image. The image processing apparatus according to claim 6, characterized by the following:

8. A shape acquisition process that acquires shape data showing the three-dimensional shape of each object present in the target space, For each of the shape data, the number of three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data is estimated, and based on the estimated number and a height image showing the height of the three-dimensional shape obtained by projecting the three-dimensional shape shown by the shape data onto a horizontal plane from above, the three-dimensional shapes corresponding to the target object included in the three-dimensional shape shown by the shape data are separated, and the position of the separated three-dimensional shapes corresponding to the target object is identified. Image processing methods including [specific details omitted].

9. A program for causing a computer to function as an image processing device according to any one of claims 1 to 7.

Images