Image processing system, image processing method, and program

Abstract

To provide an image processing system, a method, and a program for easily setting and adjusting various parameters in foreground / background separation.SOLUTION: An image processing method in an image processing system includes: learning means for training a learning model for estimating a parameter for extracting a silhouette of an object from a captured image using learning data in which a correct silhouette image is associated with each of a plurality of captured images obtained by a plurality of imaging devices; estimation means for estimating the parameter for extracting a silhouette of an object from a captured image obtained by a specific imaging device whose imaging condition matches with that of any imaging device of the plurality of imaging devices using a learned model obtained by the learning means; and generation means for generating a silhouette image by extracting a silhouette of an object from a captured image obtained by the specific imaging device using the parameter estimated by the estimation means.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating a silhouette image from a captured image. [Background technology]

[0002] A technology for obtaining an image (silhouette image) showing the silhouette of a subject (object) in the foreground from an image captured by an imaging device is used in various fields. For example, in a virtual viewpoint video technology that shows the view from a virtual viewpoint, a three-dimensional shape of a foreground object is estimated using a plurality of silhouette images obtained from a plurality of captured images from different viewpoints. There are various methods for obtaining a silhouette image from a captured image, such as a background difference method based on the difference with a background image, an inter-frame difference method based on the difference between frames, and a method using machine learning. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-182320 Summary of the Invention [Problem to be solved by the invention]

[0004] To obtain a highly accurate silhouette image by a method other than machine learning, such as background subtraction or interframe subtraction, it is necessary to set and adjust parameters for separating the silhouette of the foreground object from the background. However, in an actual imaging environment, for example, the imaging conditions between multiple imaging devices may differ, and the background within the angle of view of each imaging device may differ. Therefore, it is difficult to apply the same parameters to multiple images captured by multiple imaging devices with different viewpoints to obtain a highly accurate silhouette image corresponding to each captured image. Furthermore, for example, in an event held in a stadium, about 100 imaging devices may be installed, and it is very time-consuming to individually set and adjust optimal parameters for each image captured by a large number of imaging devices. In this regard, Patent Document 1 discloses a technology for estimating a threshold value, which is one of the parameters in the background subtraction method, based on a silhouette image obtained using a model that has learned the characteristics of the silhouette. According to the technology of Patent Document 1, it is possible to efficiently set an appropriate threshold value. However, Patent Document 1 does not take into consideration parameters other than the threshold value in the background subtraction method. This disclosure has been made in consideration of the above points. [Means for solving the problem]

[0005] The image processing system according to the present disclosure is characterized by having a learning means for training a learning model that estimates parameters for extracting a silhouette of an object from a captured image using learning data in which a correct silhouette image is associated with each of a plurality of captured images obtained by a plurality of imaging devices, an estimation means for estimating parameters for extracting a silhouette of an object from a captured image obtained by a specific imaging device having at least one imaging condition matching one of the plurality of imaging devices, using the trained model obtained by the learning means, and a generation means for extracting a silhouette of an object from an captured image obtained by the specific imaging device using the parameters estimated by the estimation means to generate a silhouette image. Effect of the Invention

[0006] According to the present disclosure, various parameters for foreground / background separation can be easily set and adjusted. [Brief description of the drawings]

[0007] [Figure 1] FIG. 1 is a block diagram showing an example of a hardware configuration of an image processing apparatus. [Diagram 2] FIG. 2 is a block diagram showing an example of a functional configuration of an image processing apparatus. [Diagram 3] Diagram of DNN configuration. [Figure 4] FIG. 4 is an explanatory diagram showing an outline of operations in a learning phase. [Diagram 5] 11 is a flowchart showing the process flow of a learning phase. [Figure 6] (a) is an example of a captured image, (b) is an example of its correct silhouette image, and (c) is an example of a region of interest mask. [Figure 7] FIG. 1 is an explanatory diagram illustrating an outline of an operation in an inference phase. [Figure 8] 11 is a flowchart showing the flow of processing in an inference phase. [Figure 9] 11 is a flowchart showing the flow of a silhouette extraction process. [Figure 10] 13A to 13D are diagrams illustrating modified examples. [Figure 11] 11 is a flowchart showing the flow of a process for determining whether or not adjustment of an extraction parameter is required. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0008] Hereinafter, with reference to the drawings, an embodiment for carrying out the present invention will be described. Note that the following embodiment does not limit the present invention, and not all of the combinations of features described in the present embodiment are necessarily essential to the solution of the present invention. Note that the same configurations are described with the same reference numerals, and duplicated descriptions are omitted. In addition, terms with different alphabets added after the reference numerals indicate different instances of devices having the same functions.

[0009] [Embodiment 1] In this embodiment, an image processing system is described as an example in which extraction parameters in a background subtraction method, which is one method of foreground-background separation, are estimated using a machine learning model, and a silhouette image is generated and output based on the obtained extraction parameters. Note that the background subtraction method is a method of comparing a background image in which no object exists (i.e., only the background is shown) with a captured image in which an object exists (i.e., the foreground and background are shown), and extracting the silhouette of the object based on the difference between the two. In this embodiment, a specific example will be described in which a silhouette image is generated from each of a plurality of captured images obtained by a plurality of imaging devices (cameras) arranged to surround a basketball court.

[0010] <Hardware configuration of image processing device> 1 is a block diagram showing an example of a hardware configuration of an image processing device 100 according to this embodiment. The image processing device 100 is an information processing device such as a PC, and includes a CPU 111, a ROM 112, a RAM 113, an auxiliary storage device 114, a display unit 115, an operation unit 116, a communication I / F 117, and a bus 118.

[0011] The CPU 111 controls the entire image processing device 100 using computer programs and data stored in the ROM 112 and the RAM 113. The image processing device 100 may have one or more dedicated hardware components different from the CPU 111, and at least a part of the processing by the CPU 111 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include a GPU (graphics processing unit), an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0012] ROM 112 stores programs and the like that do not require modification. RAM 113 temporarily stores programs and data supplied from auxiliary storage device 114, as well as data and the like supplied from the outside via communication I / F 117. Auxiliary storage device 114 is, for example, a hard disk drive, and stores various data such as image data and audio data.

[0013] The display unit 115 is, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) or the like for the user to operate the image processing device 100. The operation unit 116 is, for example, composed of a keyboard, a mouse, a joystick, a touch panel, and the like, and receives operations by the user to input various instructions to the CPU 111. The CPU 111 operates as a display control unit that controls the display unit 115 and an operation control unit that controls the operation unit 116.

[0014] The communication I / F 117 is an interface that communicates with an external device. For example, when the image processing device 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 117. When the image processing device 100 has a function of wirelessly communicating with an external device, the communication I / F 117 includes an antenna. The bus 118 connects each unit of the image processing device 100 to transmit information.

[0015] In this embodiment, the display unit 115 and the operation unit 116 are assumed to exist inside the image processing device 100, but at least one of the display unit 115 and the operation unit 116 may exist outside the image processing device 100 as a separate device.

[0016] <Functional configuration of image processing device> 2 is a block diagram showing an example of the functional configuration of the image processing device 100 according to this embodiment. The image processing device 100 has an image input unit 201, a parameter estimation unit 202, a database (DB) 203, a region of interest setting unit 204, a silhouette extraction unit 205, and an image output unit 206. Each unit will be described below.

[0017] The image input unit 201 receives input of captured images from each of a plurality of cameras (not shown). In the present embodiment, captured images obtained by a plurality of cameras installed so as to surround a court on which basketball is played are input. The input captured images corresponding to each camera are sent to an attention area setting unit 204 and / or a silhouette extraction unit 205 depending on the purpose.

[0018] The parameter estimation unit 202 estimates parameters used by the silhouette extraction unit 205 described later when generating a silhouette image from a captured image, using a trained model. The trained model is obtained by training a machine learning model with appropriate learning data. The machine learning model is a learning model based on a machine learning algorithm. Specific machine learning algorithms include the nearest neighbor method, the Naive Bayes method, a decision tree, and a support vector machine. In this embodiment, a case where learning is performed using a DNN, which is a type of deep learning that uses a neural network to generate features and connection weighting coefficients for learning, will be described as an example. FIG. 3 is a configuration diagram of a DNN. In the learning phase, learning data for multiple cameras, background images corresponding to each captured image included in the learning data, and attention area masks are input to the input layer 301. In addition, in the inference phase, captured images, background images, and attention area masks for each of the multiple cameras are input. The intermediate layers A302 to D305 branch according to the contents of the extraction parameters to be output, and feature extraction is performed in each intermediate layer. Finally, the extracted parameters of the target are estimated at each output layer 306. The extracted parameters to be estimated include the following:

[0019] - Threshold for comparing with the calculated difference in background subtraction Color difference weights (e.g. R:0.3, G:0.4, B:0.3) Weight when converting to brightness value and extracting - The type and size of the filter used in the filtering process after silhouette extraction (median filter, expansion / contraction filter, etc.) Color space when captured images are input in RAW image format The output values ​​from each output layer are sent as extraction parameters to the silhouette extraction unit 205. Note that the configuration diagram shown in Fig. 3 is just an example, and the numbers of input layers, intermediate layers and output layers, the connections between layers, etc. are not limited to this.

[0020] DB203 is a database that stores data necessary for this image processing system, such as background image data used in the background subtraction method and learning data used in machine learning.

[0021] The attention area setting unit 204 sets an attention area in the captured image that emphasizes accuracy when extracting the silhouette of an object from the captured image. In this embodiment, a mask image (hereinafter referred to as "attention area mask") indicating an area in which an object of interest is likely to exist in the captured image of each camera is provided to the parameter estimation unit 205 as attention area information. The attention area mask is statically or dynamically generated for each camera. If the object of interest is, for example, a player, an attention area mask is generated common to each frame, with the area of ​​interest being the area excluding the area in the field of view where the player cannot exist, such as the spectator seats (static generation). Alternatively, if the object of interest is, for example, a ball, the position of the ball may be identified for each frame by object tracking or the like, and an attention area mask for each frame may be generated with the limited area in the field of view being the area of ​​interest (dynamic generation). As a generation method, for example, a bounding box according to the position and size of the object of interest is arranged in the three-dimensional space where the image is captured, and the bounding box is back-projected onto the captured image of each camera.

[0022] The silhouette extraction unit 205 generates a silhouette image by performing silhouette extraction (foreground / background separation) of an object appearing in the input captured image using the extraction parameters estimated by the parameter estimation unit 202. In this embodiment, a background difference method is used for silhouette extraction, but it goes without saying that the present invention is not limited to this.

[0023] The image output unit 206 outputs the silhouette image generated by the silhouette extraction unit 205 via, for example, the communication I / F 117 to, for example, an external device (not shown) that generates a three-dimensional model from the silhouette image.

[0024] Note that each of the functional units shown in Fig. 2 may be realized by a plurality of devices. For example, it may be configured as an image processing system including a device that performs processes up to estimating the extraction parameters and a device that generates and outputs a silhouette image from a captured image using the estimated extraction parameters.

[0025] <Details of the parameter estimation part> Next, the details of the operation of the parameter estimation unit 202 will be described, divided into a learning phase and an inference phase. The learning phase is executed in advance using learning data for multiple cameras created based on image data captured during the previous match held in the same environment, for example. The inference phase is executed on a camera-by-camera basis during preparation for the current match, for example, using a trained model obtained by executing the prior learning phase. In this embodiment, both the learning phase and the inference phase are performed within one device, but they may be performed by multiple devices in a shared manner.

[0026] Learning Phase First, the learning phase will be described in detail with reference to the explanatory diagram showing an outline of the operation in FIG. 4 and the flowchart in FIG.

[0027] In S501, previously prepared learning data, a background image, and an attention area mask are acquired. Here, the learning data is data consisting of an image pair in which a captured image and a correct silhouette image in the captured image are associated with each other. FIG. 6(a) is an example of a captured image at the moment when a player shoots a ball is captured, and FIG. 6(b) is an example of a correct silhouette image corresponding to the captured image. FIG. 6(c) is an example of an attention area mask in which a ball as an object of interest is represented by a binary value of white pixels (pixel value = 1) and the other parts are represented by a binary value of black pixels (pixel value = 0). In the case of the attention area mask of FIG. 6(c), it means that the extraction accuracy of a silhouette belonging to an area with a high pixel value is given more importance than that of a silhouette belonging to an area with a low pixel value. Note that the attention area mask may be a multi-value image in which each pixel has a pixel value exceeding two values. It is desirable to prepare as much learning data as possible for each object that may be in the foreground for each of the multiple cameras (for each captured image with different viewpoints) installed in the captured space. The background image is an image corresponding to each captured image included in the learning data, in which no foreground objects (players or ball) are captured. In this embodiment, the learning data and the background image are obtained from the DB 203, and the learning area mask is obtained from the attention area setting unit 204.

[0028] In S502, a process of applying an attention area mask to the captured image in the image pair of interest among the image pairs included in the acquired learning data is performed. This results in a captured image in which each pixel value of the captured image is weighted based on each pixel value of the attention area mask. Note that, in this example, the attention area mask is applied to the captured image before inputting it to the input layer 301, but the attention area mask and the captured image may be input to the input layer 301 separately.

[0029] In S503, the captured image to which the attention area mask has been applied and a background image corresponding to the captured image are input to a machine learning model (here, a DNN).

[0030] In S504, features are extracted in the intermediate layers of the DNN. In the following S505, extracted parameters are estimated in each output layer.

[0031] In S506, an estimated silhouette image showing the silhouette of an object in the captured image is generated by a background subtraction method using the captured image in the image pair of interest and a corresponding background image, using the estimated extraction parameters.

[0032] In S507, the weights of the DNN are updated using, for example, backpropagation so as to minimize a loss function based on the generated estimated silhouette image and the ground truth silhouette image in the image pair of interest. For example, the loss function uses the sum of squares error or the cross entropy error. In addition, known learning methods such as so-called stacked autoencoder, dropout, noise addition, and sparse regularization may be used in combination to improve the processing accuracy.

[0033] In S508, it is determined whether learning has been completed for all image pairs included in the learning data acquired in S501. If unprocessed image pairs remain, the process returns to S502, and the next image pair of interest is determined from among the unprocessed image pairs, and the above-mentioned process is repeated. On the other hand, if learning has been completed for all image pairs included in the learning data, this process ends.

[0034] The above is the content of the processing in the learning phase. The next inference phase is executed using the trained model obtained in this way. Note that additional learning may be performed on the trained model at any time.

[0035] Inference Phase Next, the inference phase will be described in detail with reference to the explanatory diagram showing the operation overview in FIG. 7 and the flowchart in FIG. 8. As described above, the inference phase is executed for each of the multiple cameras installed on the day of the game to be imaged, for example. Note that the "multiple cameras" in the learning phase and the "multiple cameras" in the inference phase do not need to be physically the same. For example, when the image capture for learning data and the image capture for estimating the extraction parameters are performed on different days, the same number of cameras may be installed at the same venue under approximately the same image capture conditions. Here, for example, the image capture conditions are the camera settings such as the camera position, the camera angle of view, the exposure conditions, and the shutter speed. Also, for example, the image capture conditions may be the degree of agreement of the features of the foreground or background in the image captured by the camera, or the degree of agreement of the light of the lighting reflected in the image. Even in this case, the inference phase of this embodiment can be executed without any problems. In this case, the number of cameras and the image capture conditions of each camera do not need to be completely identical, and it is sufficient that at least one of the image capture conditions is identical. Also, the agreement of the image capture conditions here is not limited to a complete agreement, and it is sufficient that the camera having the closest condition from the multiple image capture conditions in the learning phase and the inference phase is associated.

[0036] In S801, a captured image obtained by a target camera, a background image corresponding to the captured image, and a region of interest mask are obtained.

[0037] In S802, a process of applying a region of interest mask to the acquired captured image is performed.

[0038] In S803, the captured image to which the attention area mask has been applied and a background image corresponding to the captured image are input to a trained model (here, a DNN).

[0039] In S804, features are extracted in the intermediate layers of the DNN. In the following S805, extracted parameters are estimated in each output layer.

[0040] In S805, all the extraction parameters estimated in each output layer are provided to the silhouette extraction unit 205. Here, the estimated extraction parameters may be subjected to a predetermined post-processing and provided to the silhouette extraction unit 205. The post-processing includes, for example, a clipping process that is performed so that the difference between the estimated extraction parameter value and a reference value falls within a predetermined value, as well as processes such as normalization and integer conversion. As the reference value in the clipping process, a value applied to a previous frame when performing continuous silhouette extraction, or a predetermined value may be used. Note that the estimated (or post-processed) extraction parameters may be temporarily stored in the RAM 113 or the auxiliary storage device 114, and the silhouette extraction unit 205 may later read and acquire them.

[0041] The above is the content of the processing in the inference phase. The silhouette extraction processing described below is executed using the extraction parameters obtained in this way.

[0042] <Silhouette extraction processing> Next, a description will be given of the silhouette extraction process performed by the silhouette extraction unit 205. Fig. 9 is a flowchart showing the flow of the silhouette extraction process. As in the inference phase, this silhouette extraction process is also executed for each of the multiple cameras installed on the day of the match.

[0043] In S901, a captured image obtained by a target camera, a background image corresponding to the captured image, and extraction parameters estimated for the target camera are acquired.

[0044] In S902, the acquired extraction parameters are applied to a frame of interest among the frames constituting the acquired captured image, and a silhouette image showing the silhouette of an object appearing in that frame is generated.

[0045] In S904, the generated silhouette image is sent to the image output unit 206. Then, in S905, it is determined whether or not the generation of silhouette images has been completed for all frames constituting the acquired captured image. If there are any unprocessed frames remaining, the process returns to S902, where the next frame of interest is determined and the process continues. On the other hand, if the generation of silhouette images has been completed for all frames constituting the acquired captured image, this process ends.

[0046] The above is the content of the silhouette extraction process. The silhouette image thus generated is output from the image output unit 206 to, for example, an external device (not shown) that generates a virtual viewpoint video.

[0047] <Modification> For example, when the event to be imaged is a baseball or soccer game, the game may be imaged by more than 100 cameras installed around the field. In such a case, it takes a lot of time to estimate the extraction parameters for all the installed cameras using the method of the first embodiment. Therefore, default extraction parameters are set for each camera at first, and then, based on the silhouette image actually obtained, it is determined whether the extraction parameters need to be adjusted. Then, only for the cameras that are determined to need adjustment, the extraction parameters are estimated using the method of the first embodiment, and updated to the estimated extraction parameters. Here, the default extraction parameters may be extraction parameters estimated in the past using a trained model under a similar imaging environment, or parameters set by an operator based on empirical rules.

[0048] In this case, the necessity of adjustment is determined based on whether the shape accuracy of the silhouette image generated using default extraction parameters is equal to or lower than a certain level. Specifically, for example, data (three-dimensional model) representing the three-dimensional shape of the object generated using a volume intersection method based on a plurality of silhouette images is used. (a) to (d) of FIG. 10 are diagrams for explaining this modified example. FIG. 10(a) shows how a scene in which a player 10a and a ball 10b exist as objects is captured using three cameras 11a to 11c. FIG. 10(b) shows the captured images by the three cameras 11a to 11c, respectively, and FIG. 10(c) shows a silhouette image generated from the captured images. FIG. 11 is a flowchart showing the flow of a process for determining the necessity of adjustment of the extraction parameters using a three-dimensional model. The following will be described with reference to the flowchart of FIG. 11.

[0049] In S1101, the three-dimensional shape of the object represented by the three-dimensional model is back-projected onto each captured image. The image obtained by this (back-projection image) is, for example, a binary image in which the pixel value of the area corresponding to the three-dimensional model is represented as "1" and the pixel value of the other area is represented as "0".

[0050] In S1102, the difference between each backprojection image obtained in S1101 and each corresponding silhouette image is calculated. Fig. 10(d) shows the calculated difference visualized on each silhouette image shown in Fig. 10(c). In each image shown in Fig. 10(d), the white and black areas represent areas without difference, and the gray areas represent areas with difference.

[0051] In S1103, the extraction parameters that need to be adjusted are determined based on the difference calculated in S1102. For example, the extraction parameters that need to be adjusted are determined to be the extraction parameters that correspond to the camera that captured the captured image with the largest difference region among all captured images. Alternatively, when the ratio of the difference region to the entire captured image exceeds a predetermined threshold, the extraction parameters that correspond to the camera that captured the captured image may be determined to be the extraction parameters that need to be adjusted.

[0052] The above is the content of the process for determining whether the extraction parameters need to be adjusted. For the cameras corresponding to the extraction parameters determined to need adjustment in this way, inference is performed using the trained model described above, and the extracted parameters are updated to the obtained parameters.

[0053] In the above example, the necessity of adjustment is determined based on the difference between the backprojection image and the silhouette image, but the determination method is not limited to this. For example, it may be determined based on whether the three-dimensional shape represented by the above-mentioned three-dimensional model is missing or corrupted. If the three-dimensional shape is missing or corrupted, it can be considered that the accuracy of one of the silhouette images has decreased. Whether the three-dimensional shape is missing or corrupted can be determined, for example, by whether the size of the three-dimensional model (specified by the number of voxels / number of points or the size of the bounding box surrounding the three-dimensional shape) is smaller than the lower limit size (lower limit threshold). If it is determined that the three-dimensional shape is missing or corrupted, it is sufficient to determine that the extraction parameters corresponding to the camera that captured the image with the smallest difference area among all the captured images are the extraction parameters that need to be adjusted. For example, in three-dimensional shape estimation using the volume intersection method, voxels, etc. are excessively removed by a silhouette image in which the silhouette shape is smaller than the actual shape (largely missing), and the resulting remaining voxel group / point group represents a three-dimensional shape that is smaller and less accurate than the actual shape. In the backprojection image onto each captured image based on the three-dimensional model created by excessively removing voxels, etc., the shape part that should exist does not exist. Therefore, the difference is large when compared with the silhouette image obtained by normal silhouette extraction. Therefore, it can be inferred that a silhouette image with a small difference area has a defect that causes the voxels, etc. to be excessively removed during three-dimensional shape estimation. Note that, when the ratio of the difference area to the entire captured image falls below a predetermined threshold, the extraction parameter corresponding to the camera that captured the captured image may be determined to be an extraction parameter that needs to be adjusted.

[0054] When it is determined that adjustment is necessary as described above, the extraction parameters corresponding to the camera that captured the target captured image can be estimated by the parameter estimation unit 202. In this way, only the extraction parameters that need to be adjusted are estimated using a trained model and updated, thereby further reducing the amount of calculation and the processing time.

[0055] As described above, according to this embodiment, various extraction parameters for generating a silhouette image of an object from a captured image can be easily obtained.

[0056] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0057] The present disclosure also includes the following configurations and methods.

[0058] [Configuration 1] a learning means for training a learning model that estimates parameters for extracting a silhouette of an object from a captured image, using learning data in which a correct silhouette image is associated with each of a plurality of captured images obtained by a plurality of imaging devices; an estimation means for estimating, by using the trained model obtained by the learning means, parameters for extracting a silhouette of an object from a captured image obtained by a specific imaging device having imaging conditions matching those of any one of the imaging devices among the plurality of imaging devices; a generating means for extracting a silhouette of an object from an image captured by the specific image capturing device using the parameters estimated by the estimating means to generate a silhouette image; 1. An image processing system comprising:

[0059] [Configuration 2] The image processing system according to configuration 1, characterized in that the learning means trains the learning model based on information indicating an area of ​​interest in each of a plurality of captured images included in the learning data, so as to emphasize silhouette accuracy in each area of ​​interest.

[0060] [Configuration 3] The image processing system according to configuration 1 or 2, wherein the learning means trains the learning model so as to minimize a loss function based on a silhouette image generated using parameters output from the learning model and a corresponding ground truth silhouette image included in the learning data.

[0061] [Configuration 4] 4. The image processing system according to any one of configurations 1 to 3, wherein the learning model is a neural network having an input layer, an intermediate layer, and an output layer according to the type of the parameter.

[0062] [Configuration 5] The image processing system according to configuration 4, characterized in that the types of parameters include at least one of a weight of color difference in the captured image, a weight when the color in the captured image is converted to a brightness value and input, a type and / or size of a filter used in a filter process after silhouette extraction, and a color space when the captured image is input in a RAW image format.

[0063] [Configuration 6] 6. The image processing system according to any one of configurations 1 to 5, wherein the generating means extracts a silhouette of an object from an image captured by the specific imaging device by a background subtraction method.

[0064] [Configuration 7] 7. The image processing system according to configuration 6, wherein the types of parameters further include a threshold value in the background subtraction method.

[0065] [Configuration 8] The image processing system according to configuration 1, wherein the estimation means performs the estimation for an image captured by the specific imaging device based on information indicating a region of interest in the captured image, such that emphasis is placed on accuracy of the silhouette in the region of interest.

[0066] [Configuration 9] 9. The image processing system according to configuration 2 or 8, wherein the information indicating the region of interest is a mask image in which the region of interest and other regions in the captured image are expressed in binary form.

[0067] [Configuration 10] The image processing system according to configuration 1, further comprising a determination means for determining the specific imaging device based on a silhouette image generated by the generation means using default parameters different from the parameters estimated by the estimation means.

[0068] [Configuration 11] The image processing system according to configuration 10, characterized in that, when the shape accuracy of the silhouette image generated by the generation means is equal to or lower than a certain level, the determination means determines the imaging device corresponding to the captured image that was the source of the silhouette image as the specific imaging device.

[0069] [Configuration 12] 12. The image processing system according to claim 10, wherein the different parameters for the particular imaging device determined by the determination means are updated to parameters estimated by the estimation means.

[0070] [Configuration 13] The image processing system according to configuration 10, characterized in that the default parameters are parameters estimated using a previously trained model under a similar imaging environment or parameters set based on empirical rules.

[0071] [Method 1] a learning step of training a learning model that estimates parameters for extracting a silhouette of an object from a captured image, using learning data in which a correct silhouette image is associated with each of a plurality of captured images obtained by a plurality of imaging devices; an estimation step of estimating, using the trained model obtained in the learning step, parameters for extracting a silhouette of an object from an image captured by a specific imaging device having imaging conditions matching those of any one of the imaging devices among the plurality of imaging devices; a generating step of extracting a silhouette of an object from an image captured by the specific image capturing device using the parameters estimated in the estimating step to generate a silhouette image; 13. An image processing method comprising:

[0072] [Configuration 14] A program for causing a computer to function as a means of the image processing system according to any one of configurations 1 to 13.

Claims

1. a learning means for training a learning model that estimates parameters for extracting a silhouette of an object from a captured image, using learning data in which a correct silhouette image is associated with each of a plurality of captured images obtained by a plurality of imaging devices; an estimation means for estimating, by using the trained model obtained by the learning means, parameters for extracting a silhouette of an object from a captured image obtained by a specific imaging device having at least one imaging condition that matches that of any one of the imaging devices; a generating means for extracting a silhouette of an object from an image captured by the specific image capturing device using the parameters estimated by the estimating means to generate a silhouette image; 1. An image processing system comprising:

2. The image processing system according to claim 1, characterized in that the learning means trains the learning model based on information indicating an area of ​​interest in each of the multiple captured images included in the learning data, so as to emphasize silhouette accuracy in each area of ​​interest.

3. 3. The image processing system according to claim 1, wherein the learning means trains the learning model so as to minimize a loss function based on a silhouette image generated using parameters output from the learning model and a corresponding correct silhouette image included in the learning data.

4. 2. The image processing system according to claim 1, wherein the learning model is a neural network having an input layer, an intermediate layer, and an output layer according to the type of the parameter.

5. The image processing system according to claim 4, characterized in that the types of parameters include at least one of a weight of color difference in the captured image, a weight when the color in the captured image is converted to a brightness value and input, a type and / or size of a filter used in filtering after silhouette extraction, and a color space when the captured image is input in a RAW image format.

6. 2. The image processing system according to claim 1, wherein the generating means extracts a silhouette of an object from the image captured by the specific image capturing device by a background subtraction method.

7. 7. The image processing system according to claim 6, wherein the types of parameters further include a threshold value in the background subtraction method.

8. The image processing system according to claim 1, characterized in that the estimation means performs the estimation for an image captured by the specific imaging device based on information indicating a region of interest in the captured image, with emphasis on accuracy of the silhouette in the region of interest.

9. 9. The image processing system according to claim 2, wherein the information indicating the region of interest is a mask image in which the region of interest and other regions in the captured image are expressed by binary values.

10. 2. The image processing system according to claim 1, further comprising a determination means for determining the specific imaging device based on a silhouette image generated by the generation means using default parameters different from the parameters estimated by the estimation means.

11. 11. The image processing system according to claim 10, wherein when a shape accuracy of the silhouette image generated by the generation means is equal to or lower than a certain level, the determination means determines an imaging device corresponding to a captured image that is a source of the silhouette image as the specific imaging device.

12. 12. The image processing system according to claim 10, wherein the different parameters for the specific imaging device determined by the determining means are updated to parameters estimated by the estimating means.

13. The image processing system according to claim 10 , wherein the default parameters are parameters estimated using a previously trained model under a similar imaging environment or parameters set based on empirical rules.

14. a learning step of training a learning model that estimates parameters for extracting a silhouette of an object from a captured image, using learning data in which a correct silhouette image is associated with each of a plurality of captured images obtained by a plurality of imaging devices; an estimation step of estimating, using the trained model obtained in the learning step, parameters for extracting a silhouette of an object from an image captured by a specific imaging device having imaging conditions matching those of any one of the imaging devices among the plurality of imaging devices; a generating step of extracting a silhouette of an object from an image captured by the specific image capturing device using the parameters estimated in the estimating step to generate a silhouette image; 13. An image processing method comprising:

15. A program for causing a computer to execute the image processing method according to claim 14.

Images