Image processing system, control method, and storage medium

Abstract

To provide a technique for confirming which subject is included in a predetermined range among subjects included in captured images acquired by a plurality of imaging devices.SOLUTION: An image processing system 100 includes an acquisition unit that acquires an image based on imaging by an imaging unit 201 and depth information indicating a range of a predetermined distance from the imaging unit 201, and a display control unit 206 that displays a subject present in the range of the predetermined distance and a subject not present in the range of the predetermined distance among subjects in the image in an identifiable manner.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image processing system, a control method, and a program. [Background technology]

[0002] Known methods for generating a three-dimensional model of a subject from images acquired by multiple imaging devices include a method of generating a three-dimensional model using a volume intersection method using a silhouette image obtained by separating a portion corresponding to the subject from an image containing the subject of interest.

[0003] In order to generate a three-dimensional model with high accuracy based on the shape of the subject, it is necessary to accurately separate the subject area contained in each captured image. Patent Document 1 discloses a method for checking the results of separating the subject area of ​​each imaging device. In this technology, images of the object (subject area) for which a three-dimensional model is to be generated are obtained from each captured image, and these images are displayed on a monitoring screen as images representing the captured state of the object.

[0004] Depending on the installation position of the imaging device relative to the target space for generating a 3D model, a portion of the target space for generating a 3D model may be outside the focus range of the imaging device. For example, when a large space such as a soccer stadium is used as the target space for generating a 3D model, if the imaging device cannot be installed at a sufficiently high position relative to the target space, the depression angle of the imaging device becomes small, causing a portion of the target space included in the captured image to be outside the focus range. If a subject exists in an area outside the focus range of a certain imaging device, the subject may be captured blurred, making it difficult to accurately separate the subject. In response to this, Patent Document 2 discloses generating a 3D model using images captured by an imaging device determined to include an object for which a 3D model is to be generated within the focus range. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Publication No. 2018-129736 [Patent Document 2] Japanese Patent Publication No. 2022-110751 Summary of the Invention [Problem to be solved by the invention]

[0006] Using the technology of Patent Document 1, it is possible to confirm on a display device the results of separating the subject from each captured image. However, when a three-dimensional model is generated using only images of objects contained within the in-focus range as in Patent Document 2, there is a problem in that it is not possible to confirm which part of the subject contained in the captured image is an object contained within a predetermined range such as the in-focus range.

[0007] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a technology for confirming which subjects contained in images acquired by multiple imaging devices are included within a specified range. [Means for solving the problem]

[0008] An image processing system according to one aspect of the present disclosure that achieves the above object includes: an acquisition means for acquiring an image based on imaging by an imaging means and depth information indicating a range of a predetermined distance from the imaging means; a display control means for displaying, among the subjects in the image, subjects that exist within the predetermined distance range and subjects that do not exist within the predetermined distance range in a distinguishable manner; The present invention is characterized by having the following. [Effects of the Invention]

[0009] According to the present disclosure, it is possible to check which of the subjects included in the captured images acquired by a plurality of imaging devices are included within a predetermined range. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to an embodiment. [Figure 2] FIG. 1 is a diagram showing an example of the functional configuration of an image processing system according to an embodiment. [Figure 3] FIG. 10 is an explanatory diagram of an effective depth range according to an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of the hardware configuration of an image processing system according to an embodiment. [Figure 5] 1 is a flowchart showing the operation of an image processing system according to an embodiment. [Figure 6] 10 is a flowchart showing the operation of a partial region specifying unit according to an embodiment. [Figure 7] FIG. 2A is a diagram showing an example of an image captured by a camera 102A according to an embodiment, and FIG. 2B is a diagram showing an example of a silhouette image according to an embodiment. [Figure 8] (a) A diagram showing an example of the effective depth range of camera 102A according to one embodiment, (b) a diagram showing an example of the generation result of a three-dimensional model according to one embodiment, and (c) a diagram showing an example of a partial region image according to one embodiment. [Figure 9] FIG. 10 is a diagram showing an example of images captured by all cameras according to an embodiment. [Figure 10] FIG. 10 is a diagram showing an example of silhouette images of all cameras according to an embodiment. [Figure 11] FIG. 10 is a diagram showing images of partial areas (areas used to generate a 3D model) from all cameras according to an embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present disclosure according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] For example, camera 102A and camera 102B shown in Figure 1(a) represent different instances that have the same function. Note that having the same function means having at least a specific function (such as an image capture function), and for example, camera 102A and camera 102B may have different functions and performance.

[0013] (Embodiment 1) In this embodiment, an example will be described in which partial regions, which are subject regions used to generate a three-dimensional model in images captured by multiple cameras installed to capture an image of a target space for generating a three-dimensional model, are superimposed on a silhouette image of the subject included in the captured images. In generating a three-dimensional model, a depth range (effective depth range) used to generate the three-dimensional model within the range captured by each camera is set for each camera, and the subject region used to generate the three-dimensional model is identified as a partial region.

[0014] <System configuration> FIG. 1(a) is a diagram showing an example of the configuration of an image processing system according to this embodiment. FIG. 1(b) is a cross-sectional view showing the positional relationship between an arbitrary camera 102 and a space 101 for which a 3D model is to be generated. Note that, although a soccer stadium is the subject of imaging in this embodiment, this is not a limitation. This embodiment can also be applied to other types of stadiums, such as those for baseball, basketball, and volleyball, or concert venues.

[0015] A playing field 104 and a subject 105 for which a three-dimensional model is to be generated are present in a three-dimensional model generation target space 101. In the case of soccer, the subject 105 includes subject 105A, a subject 105B, which are players, and subject 105C, which is a soccer ball.

[0016] Cameras 102, including cameras 102A to 102P, are arranged around a space 101 for which a three-dimensional model is to be generated. Each of the cameras 102 captures a portion of the space 101 for which a three-dimensional model is to be generated. In FIG. 1(a), the cameras 102 are arranged to surround a playing field 104 on a plan view of the space 101 for which a three-dimensional model is to be generated. The cameras 102 are installed at a certain height. An imaging range 106 indicates the range captured by the camera 102. In this example, as shown in FIG. 1(b), subjects 105A, 105B, and 105C are included within the imaging range 106 of the camera 102, and these subjects will be included in the captured image.

[0017] The image processing server 103 is an image processing device that receives images captured by each camera 102 and performs processing to generate a 3D model of the subject. The image processing server 103 displays a partial area, which is the subject area in each captured image used to generate the 3D model, superimposed on a silhouette image of the subject included in the captured image. This makes it possible to monitor the imaging status of the subject by each camera 102 and to adjust the partial area of ​​the subject used to generate the 3D model.

[0018] <Functional configuration> 2 is a diagram showing an example of the functional configuration of an image processing system 100 according to this embodiment. The image processing system 100 includes an imaging unit 201 and an image processing unit 210. The imaging unit 201 captures an image of a subject for which a three-dimensional model is to be generated. The image processing unit 210 then generates a three-dimensional model of the subject, and identifies and displays a subject region (partial region) that indicates a portion of the subject included in each captured image captured by the imaging unit 201 that was used to generate the three-dimensional model.

[0019] The imaging unit 201 is a group of imaging devices arranged and installed so as to capture images of the target space 101 for which a three-dimensional model is to be generated from all directions. Each imaging device corresponds to the camera 102 in FIGS. 1(a) and 1(b), and captures an image of a part of the target space 101 for which a three-dimensional model is to be generated. The imaging unit 201 transmits the captured images acquired by each camera 102 to the image processing unit 210.

[0020] The image processing unit 210 includes a foreground separation unit 202, a depth information storage unit 203, a shape estimation unit 204, a partial region identification unit 205, and a display control unit 206. Each process of the image processing unit 210 is executed by the image processing server 103 shown in FIG.

[0021] The foreground separation unit 202 performs a process of separating and excluding a subject region from each captured image. The foreground separation unit 202 outputs a silhouette image of the subject obtained as a result of the separation to the shape estimation unit 204 and the partial region identification unit 205. For example, a background subtraction method or a machine learning method can be applied to the foreground separation process.

[0022] The background subtraction method generates a silhouette image of a subject by calculating the difference between a background image corresponding to an image in which the subject is not present in the captured image and an image in which the subject is present. One method for obtaining a background image is to use an image captured at a moment when the subject is not present as the background image. Another method that can be applied is to obtain captured images over a certain period of time, observe changes in pixel values ​​on a pixel or small region basis, and, if the changes in pixel values ​​are within a certain amount, use the average value over the certain period or the latest pixel value as the pixel value that constitutes the background image.

[0023] The machine learning method uses a model that has been trained in advance to extract only the subject portion of a captured image. This training model can be realized, for example, by a convolutional neural network (CNN) model that executes a semantic segmentation task to identify classes for each pixel of an input image.

[0024] The method for generating a silhouette image of a subject used by the foreground separation unit 202 is not limited to the background subtraction method or the machine learning method.

[0025] The depth information storage unit 203 stores information indicating which portion of the imaging range of each camera 102 is to be used for generating a three-dimensional model. The depth information indicates a range of a predetermined distance from the imaging unit 201, and is information indicating a range in the depth direction within the imaging range of a certain camera 102 that is to be used for generating a three-dimensional model, such as the effective depth range 301 shown in FIG. 3. The camera 102 in FIG. 3 captures an area included in the imaging range 106 of the target space 101 for three-dimensional model generation, but the area used for generating a three-dimensional model is only the area shown in the effective depth range 301. The effective depth range is determined as a range that allows for accurate separation of the subject area included in the captured image.

[0026] One way to determine the effective depth range is to use the in-focus range determined from the camera's sensor information and lens settings (focal length, subject distance, aperture). This determination method is based on the idea that subject regions can be separated accurately as long as the range is within which the subject is captured in focus.

[0027] However, the effective depth range does not necessarily have to be set to coincide with the focus range, and the user may determine a range that allows accurate separation of the subject area. For example, if the size of the subject captured is small even within the focus range, accurate separation may not be possible depending on the subject area separation method. In this case, the effective depth range may be set so that the back side of the focus range is outside the effective area.

[0028] Furthermore, the range within which the subject region can be accurately separated must be adjusted by checking the silhouette image of the subject obtained by actual separation. For example, the pixel values ​​(brightness and hue) of the subject to be 3D modeled or the competition field in the captured image change depending on the weather or field conditions. This changes the relative pixel difference value between the subject portion to be separated and the other portions, and the accuracy may not be confirmed without actually separating the subject region using the captured image. Therefore, it is necessary to check the separation result and adjust the effective depth range. In this case, the subject silhouette included in the partial region (the region used to generate the 3D model) displayed by the display control unit 206 (described later) is checked and adjusted. In other words, the depth information may indicate the effective depth range adjusted by the user or may be set by the user. Furthermore, the depth information may be determined by the settings of the imaging unit 201 or may be information based on the focal length of the imaging unit 201.

[0029] The depth information storage unit 203 outputs the effective depth range of each camera 102 within the 3D model generation target space 101 to the shape estimation unit 204 and the partial region identification unit 205 .

[0030] The shape estimation unit 204 performs a shape estimation process to generate a three-dimensional model using a silhouette image of the subject obtained by separating the subject region from each captured image and the effective depth range of each camera 102. For example, the shape of silhouette method can be used as the shape estimation process. In the shape of silhouette method, first, rectangular parallelepipeds with unit volume called voxels are tiled in the target space for generating the three-dimensional model. Of these voxels, those included in the imaging range and effective depth range of each camera 102 are projected onto the captured image of the camera 102, and voxels not included in the subject silhouette are excluded. This process is repeated for all cameras 102 to generate a three-dimensional model.

[0031] In this embodiment, the three-dimensional model is described as a point cloud, which is a collection of voxels, but the three-dimensional model is not limited to a point cloud. For example, the three-dimensional model may be a three-dimensional mesh obtained by converting a point cloud generated by a volume intersection method. Alternatively, the three-dimensional model may be a three-dimensional mesh generated by another three-dimensional model generation method. The shape estimation unit 204 outputs the generated three-dimensional model to the partial region identification unit 205.

[0032] The partial area identification unit 205 identifies a subject present within a predetermined distance range based on a three-dimensional model generated based on the captured image and included within a range in virtual space corresponding to a predetermined distance range from the imaging unit 201. The subject present within the predetermined distance range is a subject corresponding to a three-dimensional model generated based on the captured image and included within a range in virtual space corresponding to a predetermined distance range from the imaging unit 201. More specifically, the partial area identification unit 205 identifies a partial area used (contributed) to the generation of the three-dimensional model from a silhouette image of the subject included in each captured image. To identify the partial area, the partial area identification unit 205 uses a silhouette image obtained by separating and excluding the subject area in the foreground separation unit 202, depth information (e.g., effective depth range) received from the depth information storage unit 203, and a three-dimensional model generated by the shape estimation unit 204. A method for identifying a partial area (a region used to generate the three-dimensional model) using these information will be described later in the description of the operation using a flowchart. The partial area identification unit 205 outputs information about the identified partial area to the display control unit 206.

[0033] The display control unit 206 displays, among the subjects in the captured image, subjects that are present within a predetermined distance range from the imaging unit 201 and subjects that are not present within the predetermined distance range in a distinguishable manner. More specifically, the display control unit 206 displays, for each captured image, a partial region used to generate a three-dimensional model superimposed on a silhouette image of the subject included in the captured image on the display unit 405. An example of a display method in this embodiment will be described later by explaining the operation using a flowchart.

[0034] <Hardware configuration> Next, the hardware configuration of the image processing unit 210 in the image processing system 100 according to this embodiment will be described with reference to Fig. 4. The image processing unit 210 has a CPU 401, a ROM 402, a RAM 403, an auxiliary storage device 404, a display unit 405, an operation unit 406, a communication I / F 407, and a bus 408.

[0035] The CPU 401 controls the entire image processing unit 210 using computer programs and data stored in the ROM 402 and RAM 403, and realizes each function of the image processing unit 210. Note that the image processing unit 210 may have one or more pieces of dedicated hardware different from the CPU 401, and at least a part of the processing by the CPU 401 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0036] The ROM 402 stores programs that do not require modification. The RAM 403 temporarily stores programs and data supplied from the auxiliary storage device 404, and data supplied from the outside via the communication I / F 407. The auxiliary storage device 404 is configured, for example, with a hard disk drive, and stores various data. The RAM 403 and the auxiliary storage device 404 hold captured images input from the imaging unit 201, intermediate data during processing by the image processing unit 210, and output data for display.

[0037] The display unit 405 is configured with, for example, a liquid crystal display or an LED (light-emitting diode), and displays a GUI (Graphical User Interface) for the user to control the entire image processing system 100. In addition, in order to monitor the imaging status of the subject by each camera, the display unit 405 displays a partial region of the subject used to generate a three-dimensional model from images captured by the multiple cameras, superimposed on a silhouette image of the subject included in the captured images.

[0038] The operation unit 406 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and receives operations from a user to input various instructions to the CPU 401. The CPU 401 operates as a display control unit that controls the display unit 405 and an operation control unit that controls the operation unit 406.

[0039] The communication I / F 407 is used for communication with external devices. It is used to output control instructions for the entire image processing system 100 including the imaging unit 201, and to input captured images from the imaging unit 201. This I / F is realized by a wired network I / F such as Ethernet, a wireless network I / F such as wireless LAN, or an SDI (Serial Digital Interface) I / F that transmits and receives video signals. The bus 408 connects the various units that make up the image processing unit 210 to transmit information.

[0040] In this embodiment, the display unit 405 and the operation unit 406 are assumed to exist inside the image processing unit 210, but at least one of the display unit 405 and the operation unit 406 may exist as a separate device outside the image processing unit 210.

[0041] <Processing> Next, the operation of the image processing system according to this embodiment will be described with reference to Figures 5 to 11. Figure 5 is a flowchart showing the operation of the image processing system 100 according to this embodiment.

[0042] In S501, the image processing system 100 starts a processing loop for all cameras. In the example shown in Fig. 1, the loop is from camera 102A to camera 102P.

[0043] In S502, the imaging unit 201 acquires a captured image. When each camera 102 constituting the imaging unit 201 acquires a captured image, it transmits the acquired image to the image processing unit 210. An example of a captured image is shown in captured image 701 in FIG. 7(a). Captured image 701 indicates a captured image obtained by camera 102A capturing an image of the space 101 to which a three-dimensional model is to be generated in the image processing system 100 of this embodiment. Captured image 701 includes the competition field 104 and subjects 105A to 105C present in the space 101 to which a three-dimensional model is to be generated. Camera 102A included in the imaging unit 201 outputs captured image 701 to the foreground separation unit 202.

[0044] In S503, the foreground separation unit 202 performs a process of separating the object region included in the captured image. The separation process is performed using the background subtraction method or machine learning method described above. An example of the result of the object region separation process is shown in silhouette image 702 in FIG. 7(b). In the silhouette image 702, the portion where the object exists and has been separated as the object region is shown as a white region, and the other portion is shown as a filled-in gray region. The silhouette image 702 includes object silhouettes 710A to 710C corresponding to the objects 105A to 105C included in the captured image 701. The foreground separation unit 202 outputs the silhouette image 702, which is the result of the object region separation process, to the shape estimation unit 204 and the partial region identification unit 205.

[0045] In S504, the depth information storage unit 203 acquires depth information. The depth information is information set for each camera 102. As shown in the effective depth range 301 in FIG. 3, an area used for shape estimation processing to generate a three-dimensional model is set in the depth direction of the imaging range of the camera 102. An example of the effective depth range of the camera 102A is shown as the effective depth range 301A in FIG. 8(a). The user of the image processing system 100 sets such effective depth ranges 301 for all cameras 102 in advance. The depth information storage unit 203 outputs the effective depth range 301 of each camera 102 to the shape estimation unit 204 and the partial region identification unit 205.

[0046] In S505, if the image processing system 100 has completed the loop for all cameras, it proceeds to S506. If there are still cameras remaining, it returns to S501 and repeats the process. In the example shown in Fig. 1, this loop ends when the processes from S502 to S504 are completed for each of cameras 102A to 102P.

[0047] In S506, the shape estimation unit 204 performs a shape estimation process for the object and generates a three-dimensional model of the object. The shape estimation unit 204 generates the three-dimensional model using the silhouette images 702 of all cameras 102 and the effective depth range 301. For example, a three-dimensional model of the object 105 included in the three-dimensional model generation target space 101 is generated using the volume intersection method described above. FIG. 8B shows an example of the generated three-dimensional model. The shape estimation unit 204 can identify the voxel range that each camera 102 should exclude from the effective depth range 301 of that camera 102. For example, the silhouette image of camera 102A is used only to generate three-dimensional model 720B, which is included in the effective depth range 301A in FIG. 8B. Even though three-dimensional models 720A and 720C are captured within the angle of view, they are outside the effective depth range 301A and are therefore not used to generate these models. The three-dimensional models 720 of the subject are generated as three-dimensional models 720A to 720C at positions corresponding to the actual subjects 105A to 105C in the three-dimensional model generation target space 101. The shape estimation unit 204 outputs the generated three-dimensional models 720 of the subject to the partial region identification unit 205.

[0048] In S507, the partial area identification unit 205 identifies, within the camera angle of view, the subject area (partial area) used to generate the three-dimensional model from among the subjects captured by each camera 102. The processing content of S507 will be described with reference to the flowchart shown in FIG.

[0049] In S601, the partial region identification unit 205 acquires a three-dimensional model. For example, the partial region identification unit 205 acquires the three-dimensional model by receiving the three-dimensional model 720 from the shape estimation unit 204.

[0050] In S602, the partial region identification unit 205 starts a loop for all cameras 102. In the example shown in Fig. 1, the loop is from camera 102A to camera 102P.

[0051] In S603, the partial region identification unit 205 acquires depth information. In this step, the partial region identification unit 205 acquires the effective depth range 301 of each camera 102 as the depth information.

[0052] In S604, partial region identification unit 205 backprojects the three-dimensional model within the effective depth range. Here, with reference to FIG. 8(b), a method for backprojecting the three-dimensional model included in the effective depth range 301A of camera 102A out of the three-dimensional model 720 will be described. Since the three-dimensional model 720 included in the effective depth range 301A of camera 102A is three-dimensional model 720B corresponding to subject 105B, this three-dimensional model 720B is backprojected onto silhouette image 702 of camera 102A. This makes it possible to identify the partial region within silhouette image 702 that was used in the generation process of the three-dimensional model.

[0053] An example of the partial region identification result is shown in partial region image 703 in FIG. 8(c). In partial region image 703, subject silhouette 710B corresponding to subject 105B in silhouette image 702 is shown as used region 730B, which is the partial region used to generate the three-dimensional model. On the other hand, subject silhouettes 710A and 710C corresponding to subjects 105A and 105C, which are not included in effective depth range 301, are shown as unused regions 740A and 740C, indicated by diagonal lines. Partial region identification unit 205 outputs a partial region image including these used and unused regions to display control unit 206 as partial region information. In partial region image 703, subject silhouette 710B may be displayed in any manner as long as it is distinguishable from subject silhouettes 710A and 710C.

[0054] In S605, the partial area identification unit 205 ends the process if the loop has been completed for all cameras 102. If there are still cameras 102 remaining, the process returns to S602 and is repeated. In the example shown in Fig. 1, this loop ends when the processes of S603 and S604 are completed for each camera from camera 102A to camera 102P.

[0055] Returning to the description of the flowchart in Fig. 5, in S508, the display control unit 206 displays the partial regions (regions used to generate the three-dimensional model). The display control unit 206 displays partial region images including the partial regions, which are subject regions used to generate the three-dimensional model in each camera 102, identified in S507. The display control unit 206 can also display captured images and silhouette images of each camera 102. Display examples of the captured images, silhouette images, and partial region images are shown in Figs. 9, 10, and 11, respectively.

[0056] Fig. 9 shows captured images 801A to 801P corresponding to cameras 102A to 102P. Each image captures a subject included in the imaging range of each camera 102. Fig. 10 shows silhouette images 802A to 802P corresponding to cameras 102A to 102P. Silhouette image 802 includes subject silhouettes corresponding to subjects included in the images captured by each camera 102. Fig. 11 shows partial region images 803A to 803P corresponding to cameras 102A to 102P. Of the subject silhouettes corresponding to subjects included in the images captured by each camera 102, the used regions used in generating the three-dimensional model are indicated by white areas, and unused regions are indicated by hatched areas in the partial region images.

[0057] In this way, by displaying the used area and the unused area based on the silhouette image of each camera 102, it is possible to easily check on the image which of the subjects captured by each camera 102 contributed to the generation of the three-dimensional model. Note that in the present embodiment, an example has been shown in which the partial area and the other area are distinguished by white and hatched areas, respectively, but a display method in which each is filled with a different color is also possible. In other words, any display mode is acceptable as long as the used area and the unused area can be distinguished by being displayed in different display modes. Note that it is not necessary to fill the entire area with a specific color; for example, the color of the edge of the area may be different between the used area and the unused area. In other words, the display color of the edge of the used area may be different from the display color of the edge of the unused area.

[0058] This makes it possible, for example, to adjust the separation parameters of subjects with low separation accuracy included in the silhouette image of the subject used to generate the three-dimensional model, or to adjust the effective depth range for each camera, while checking the partial area image.

[0059] As described above, in this embodiment, a partial area indicating a subject area that contributed to the generation of the three-dimensional model is identified based on the three-dimensional model and the depth information corresponding to each image capture device. This makes it possible to identify which subjects, among the subjects included in the captured images acquired by the multiple image capture devices, contributed to the generation of the three-dimensional model.

[0060] Furthermore, in this embodiment, among the subjects in the captured images, subjects that exist within a predetermined distance range from the imaging unit 201 and subjects that do not exist within the predetermined distance range are displayed in a distinguishable manner. More specifically, a partial region that is the subject region used to generate the three-dimensional model is displayed superimposed on the silhouette image of the subject. This makes it easy to confirm which part (which subject) of the silhouette images of the subject included in the captured images acquired by the multiple imaging devices was used to generate the three-dimensional model.

[0061] This makes it easy to adjust the parameters for separating the object regions. Also, by avoiding unnecessary parameter adjustment for objects that are not used in generating a 3D model, it is possible to suppress a decrease in the accuracy of separating objects that are captured within the focus range.

[0062] [Variations] In the above embodiment, the silhouette image of the subject and the partial area used to generate the three-dimensional model are displayed superimposed on each other, and the display colors of the partial area and other areas are changed, but the display method is not limited to this.

[0063] For example, by calculating the distance between the camera and the three-dimensional model in Fig. 8(b), it is possible to superimpose a depth map, in which the hue or shade of color varies depending on the depth, on the silhouette image and partial region of the subject when displaying the image. The depth map is map information indicating the distance from the imaging unit 201 to the subject. When the depth map is superimposed, for example, whether or not it is a partial region may be indicated by indicating the boundary color between the portion used in generating the three-dimensional model and the portion not used.

[0064] This allows the user to check the relative position of a subject in the effective depth range, making it easy to adjust the effective depth range. Furthermore, the depth map used for this superimposed display can be obtained not only from the relative positional relationship between the camera and the three-dimensional model, but also from a depth sensor mounted on the camera.

[0065] The disclosure of this specification includes the following image processing system, control method, and program.

[0066] (Item 1) an acquisition means for acquiring an image based on imaging by an imaging means and depth information indicating a range of a predetermined distance from the imaging means; a display control means for displaying, among the subjects in the image, subjects that exist within the predetermined distance range and subjects that do not exist within the predetermined distance range in a distinguishable manner; An image processing system comprising:

[0067] (Item 2) 2. The image processing system according to item 1, wherein the depth information is set by a user.

[0068] (Item 3) 2. The image processing system according to item 1, wherein the depth information is determined by settings of the imaging means.

[0069] (Item 4) 4. The image processing system according to any one of items 1 to 3, wherein the depth information is information based on a focal length of the imaging means.

[0070] (Item 5) 5. The image processing system according to any one of items 1 to 4, wherein the display control means displays subjects that exist within the predetermined distance range and subjects that do not exist within the predetermined distance range in different display modes.

[0071] (Item 6) 6. The image processing system according to any one of items 1 to 5, wherein the display control means displays subjects that exist within the predetermined distance range and subjects that do not exist within the predetermined distance range in different colors.

[0072] (Item 7) 7. The image processing system according to any one of items 1 to 6, wherein the display control means further displays a depth map indicating a distance from the imaging means to a subject.

[0073] (Item 8) 8. The image processing system according to any one of items 1 to 7, further comprising an identifying unit that identifies a subject that exists within the range of the predetermined distance.

[0074] (Item 9) Item 9. The image processing system according to item 8, wherein the identification means identifies a subject present within the range of the predetermined distance based on a three-dimensional model generated based on the image and included in a range corresponding to the range of the predetermined distance in virtual space.

[0075] (Item 10) 10. The image processing system according to any one of items 1 to 9, wherein the subject present within the range of the predetermined distance is a subject corresponding to a three-dimensional model generated based on the image and included in a range in a virtual space corresponding to the range of the predetermined distance.

[0076] (Item 11) an acquisition step of acquiring an image based on imaging by an imaging means and depth information indicating a range of a predetermined distance from the imaging means; a display control step of displaying, among the subjects in the image, subjects that exist within the predetermined distance range and subjects that do not exist within the predetermined distance range in a distinguishable manner; A control method comprising:

[0077] (Item 12) Item 12. A program for causing a computer to execute the control method according to Item 11.

[0078] (Other embodiments) The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0079] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0080] 100: Image processing system, 103: Image processing server, 210: Image processing unit, 201: Imaging unit, 206: Display control unit

Claims

1. an acquisition means for acquiring an image based on imaging by an imaging means and depth information indicating a range of a predetermined distance from the imaging means; a display control means for displaying, among the subjects in the image, subjects that exist within the predetermined distance range and subjects that do not exist within the predetermined distance range in a distinguishable manner; An image processing system comprising:

2. 2. The image processing system according to claim 1, wherein the depth information is set by a user.

3. 2. The image processing system according to claim 1, wherein the depth information is determined by settings of the imaging means.

4. 2. The image processing system according to claim 1, wherein the depth information is information based on a focal length of the imaging means.

5. 2. The image processing system according to claim 1, wherein the display control means displays an object that exists within the predetermined distance range and an object that does not exist within the predetermined distance range in different display modes.

6. 2. The image processing system according to claim 1, wherein the display control means displays an object that exists within the predetermined distance range and an object that does not exist within the predetermined distance range in different colors.

7. 2. The image processing system according to claim 1, wherein the display control means further displays a depth map indicating a distance from the imaging means to a subject.

8. 2. The image processing system according to claim 1, further comprising: a specifying unit for specifying a subject that exists within the predetermined distance range.

9. The image processing system according to claim 8, characterized in that the identification means identifies a subject present within the predetermined distance range based on a three-dimensional model generated based on the image and included in a range in virtual space corresponding to the predetermined distance range.

10. The image processing system according to claim 1, characterized in that the subject present within the range of the predetermined distance is a subject corresponding to a three-dimensional model generated based on the image and included in a range in a virtual space corresponding to the range of the predetermined distance.

11. an acquisition step of acquiring an image based on imaging by an imaging means and depth information indicating a range of a predetermined distance from the imaging means; a display control step of displaying, among the subjects in the image, subjects that exist within the predetermined distance range and subjects that do not exist within the predetermined distance range in a distinguishable manner; A control method comprising:

12. A program for causing a computer to execute the control method according to claim 11.

Images