Image processing apparatus, image processing method, and program

Abstract

To appropriately update a background image even when a target object does not move.SOLUTION: An image processing apparatus 102 according to the present disclosure acquires data of an input image, data of a background image used for extracting a foreground area from the input image, and data of a three-dimensional model generated on the basis of the foreground area extracted from the input image, and updates the background image on the basis of the input image and the three-dimensional model.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to techniques for extracting foreground regions from images. [Background technology]

[0002] There is a technology (hereinafter referred to as "foreground extraction technology") that extracts a foreground region in an image by analyzing the image. For example, a background difference method is known as a foreground extraction technology that extracts a region in a target image where a difference from a background image is detected as the foreground region. In order to accurately extract a foreground region in the background difference method, a highly accurate background image is required. Patent Document 1 discloses a technology that updates a background image based on the status of an object corresponding to a foreground region in an input image within a predetermined period of time. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2021-163303 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the background image updating method disclosed in Patent Document 1, if the predetermined period is set to a short time, the background image may be updated so that the image area corresponding to the object is included in the background image if the object does not move within the predetermined period. As a result, accurate extraction of the foreground area may not be possible. Furthermore, if the predetermined period is set to a long time, the background image may not be updated at the appropriate time, making it impossible to accurately extract the foreground area.

[0005] Therefore, an object of the present disclosure is to provide a technology that can appropriately update a background image even when a target object does not move. [Means for solving the problem]

[0006] The image processing device according to the present disclosure includes an image acquisition means for acquiring data of an input image, a background acquisition means for acquiring data of a background image used to extract a foreground region from the input image, a model acquisition means for acquiring data of a three-dimensional model generated based on the foreground region extracted from the input image, and an update means for updating the background image based on the input image and the three-dimensional model. [Effects of the Invention]

[0007] According to the present disclosure, the background image can be updated appropriately even if the target object does not move. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of the configuration of an image processing system according to a first embodiment. [Figure 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Figure 3] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Figure 4] 4 is a flowchart showing an example of a processing flow in the image processing device according to the first embodiment. [Figure 5] 5 is a diagram for explaining a projection process in an update unit according to the first embodiment. FIG. [Figure 6] 10 is a flowchart showing an example of a processing flow in an image processing device according to a second embodiment. [Figure 7] 10 is a diagram for explaining a background image update process in an update unit according to the first embodiment. FIG. [Figure 8] FIG. 10 is a block diagram showing an example of the functional configuration of an image processing device according to a third embodiment. [Figure 9] 11 is a flowchart showing an example of a processing flow in an image processing device according to a third embodiment. [Figure 10]FIG. 11 is a diagram for explaining a projection process in an update unit according to the third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Preferred embodiments of the present disclosure will be described in detail below with reference to the drawings. Note that the following embodiments do not necessarily limit the means for solving the problems according to the present disclosure. Furthermore, not all of the combinations of features described in the following embodiments are necessarily essential to the means for solving the problems according to the present disclosure.

[0010] [Embodiment 1] A first embodiment will be described with reference to Figures 1 to 5. In the first embodiment, an acquired three-dimensional model indicating the three-dimensional shape of an object is used to identify whether or not a three-dimensional model exists, and thereby, for each imaging device, it is determined whether or not to update the background image corresponding to the imaging device.

[0011] <Image processing system configuration> 1 is a block diagram showing an example of the configuration of an image processing system 100 according to embodiment 1. The image processing system 100 includes an imaging device 101, an image processing device 102, an information processing device 103, and a database 104. The image processing system 100 is a system that generates a three-dimensional model corresponding to an object based on an image obtained by imaging using the imaging device 101 (hereinafter referred to as a "captured image").

[0012] The imaging device 101 has a plurality of imaging devices, each of which is constituted by a digital still camera, a digital video camera, or the like, and each imaging device of the imaging device 101 captures an image of an object present in an imaging area. Data of captured images obtained by imaging with the imaging device 101, i.e., data of a plurality of captured images obtained by imaging with the plurality of imaging devices of the imaging device 101, is transmitted to the image processing device 102. In this embodiment, the description will be given assuming that each imaging device of the imaging device 101 captures a moving image and sequentially transmits data of each frame constituting the moving image to the image processing device 102 as captured image data.

[0013] The image processing device 102 uses the background image to perform foreground / background separation processing to separate each captured image (frame) received from each imaging device of the imaging device 101 into a foreground region including an image of an object and a background region not including an image of an object. A plurality of foreground images obtained by this separation, which indicate the foreground region in each captured image (frame), are transmitted to the information processing device 103 and the database 104.

[0014] The information processing device 103 generates a three-dimensional model corresponding to the object using data of the multiple foreground images received from the image processing device 102. Specifically, for example, the information processing device 103 generates the three-dimensional model corresponding to the object by executing a well-known shape estimation process for estimating the three-dimensional shape of the object, such as a volume intersection method. Data of the three-dimensional model obtained by this generation is transmitted to the image processing device 102 and the database 104.

[0015] The image processing device 102 updates the background image used in the foreground / background separation process using the three-dimensional model received from the information processing device 103. The database 104 receives the three-dimensional model data transmitted from the information processing device 103 and the data of multiple foreground images transmitted from the image processing device 102, and stores these in association with each other. By using this data stored in the database 104, it becomes possible to reproduce the appearance from any virtual viewpoint position, for example.

[0016] <Hardware configuration of image processing device> FIG. 2 is a block diagram showing an example of the hardware configuration of the image processing device 102 according to the first embodiment. The image processing device 102 includes a central processing unit (CPU) 201, a random access memory (RAM) 202, a read-only memory (ROM) 203, and a network interface (IF) 204. The various components of the hardware configuration of the image processing device 102 are communicably connected to one another via a bus 205. The CPU 201 executes predetermined arithmetic processing or various programs using a control program stored in the ROM 203, thereby realizing various functions of the image processing device 102. The image processing device 102 may include one or more dedicated hardware components different from the CPU 201. In this case, the image processing device 102 may execute at least a portion of the processing performed by the CPU 201 using the dedicated hardware components. Examples of the dedicated hardware include an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), and a digital signal processor (DSP).

[0017] The RAM 202 operates as the main memory of the CPU 201 and is used as a temporary storage area such as a work area for the CPU 201. The ROM 203 stores a boot program, a control program, and various data such as parameters used when the image processing device 102 operates. The network IF 204 is an interface used for communication between the image processing device 102 and an external device. The network IF 204 is connected to a wired network such as a LAN (Local Area Network). The image processing device 102 transmits and receives various information to and from the external device via the network IF 204.

[0018] <Functional configuration of image processing device> 3 is a block diagram showing an example of the functional configuration of the image processing device 102 according to the first embodiment. The image processing device 102 includes an image acquisition unit 301, an extraction unit 302, a background acquisition unit 303, an update unit 304, and a model acquisition unit 305. Each unit included in the functional configuration of the image processing device 102 is realized by the CPU 201 executing a program stored in the ROM 203, using the RAM 202 as a work memory. Note that not all of the processes described below necessarily need to be executed by the CPU 201, and the image processing device 102 may be configured such that part or all of the processes are executed by one or more processing circuits other than the CPU 201.

[0019] The image acquisition unit 301 acquires, from the imaging device 101, data of a plurality of captured images (frames) obtained by imaging using a plurality of imaging devices included in the imaging device 101, as data of input images. The input image data acquired by the image acquisition unit 301 is transmitted to the extraction unit 302, the background acquisition unit 303, and the update unit 304. Note that, in this embodiment, the image acquisition unit 301 is described as acquiring captured image data directly from the imaging device 101, but the source from which the image acquisition unit 301 acquires captured image data is not limited to the imaging device 101. For example, the image acquisition unit 301 may acquire captured image data from an external device other than the imaging device 101, such as a storage device not shown in FIG. 1 .

[0020] The background acquisition unit 303 receives input image data transmitted from the image acquisition unit 301 and generates a background image based on the received input image. Specifically, for example, the background acquisition unit 303 acquires a background image by generating a background image based on two or more input images captured at different times, assuming that pixels with no or little fluctuation in pixel value over a predetermined period correspond to pixels corresponding to the background. The background acquisition unit 303 may acquire, as background image data, data of a captured image obtained by previously capturing an image in which no object is present in the imaging area of ​​the imaging device 101. In this case, for example, the background image data is stored in advance in the ROM 203 or the like, and the background acquisition unit 303 acquires the background image data by reading the background image data stored in the ROM 203 or the like. Furthermore, if the update unit 304 has already completed updating the background image, the background acquisition unit 303 acquires, as background image data, data of the background image updated by the update unit 304 (hereinafter referred to as the "updated background image"). The background image data acquired by the background acquisition unit 303 is sent to the extraction unit 302 and the update unit 304 .

[0021] The extraction unit 302 receives input image data transmitted from the image acquisition unit 301 and background image data transmitted from the background acquisition unit 303, and compares the received input image with the background image to extract a foreground region in the input image. There are various methods for comparing the input image with the background image. For example, the extraction unit 302 may identify pixels in which the difference in RGB values ​​between corresponding pixels in the input image and the background image is equal to or greater than a threshold as pixels included in the foreground region of the input image. The above-described comparison method is merely an example, and the extraction unit 302 may compare the input image with the background image using any method that can compare images. Foreground image data indicating the foreground region corresponding to each input image extracted by the extraction unit 302 is transmitted to the information processing device 103 and the database 104.

[0022] The model acquisition unit 305 acquires three-dimensional model data corresponding to an object present in the imaging area of ​​the imaging device 101. Specifically, the model acquisition unit 305 acquires the three-dimensional model data by receiving the three-dimensional model data generated and transmitted by the information processing device 103. The three-dimensional model data acquired by the model acquisition unit 305 is transmitted to the update unit 304.

[0023] The update unit 304 receives input image data transmitted from the image acquisition unit 301, background image data transmitted from the background acquisition unit 303, and three-dimensional model data transmitted from the model acquisition unit 305. The update unit 304 uses the received input image and three-dimensional model to update the background image transmitted from the background acquisition unit 303. The updated background image data obtained by the update process by the update unit 304 is transmitted to the background acquisition unit 303 and the extraction unit 302.

[0024] When the background image is updated by update unit 304, extraction unit 302 extracts the foreground region in the input image using the data of the updated background image. Specifically, in this case, extraction unit 302 extracts the foreground region in the input image by comparing the data of the input image received from image acquisition unit 301 with the data of the updated background image received from background acquisition unit 303. Foreground image data indicating the foreground region corresponding to each input image extracted by extraction unit 302 is transmitted to information processing device 103 and database 104.

[0025] <Operation of image processing device> 4 is a flowchart showing an example of the flow of processing in the image processing device 102 according to the first embodiment. The processing of this flowchart is repeatedly executed, for example, each time data of a plurality of frames obtained by capturing images synchronized with each other is input from a plurality of image capturing devices included in the image capturing apparatus 101. Note that "capturing images synchronized with each other" does not mean capturing images at the same time, but rather capturing images after performing synchronization processing. In other words, the capturing images synchronized with each other do not need to be performed at exactly the same time, and also include modes in which the capturing images are performed at approximately the same time. Note that, hereinafter, each processing step (process) will be denoted by adding "S" to the beginning of the reference numeral.

[0026] First, in S401, the background acquisition unit 303 acquires background image data. Here, if the update unit 304 has already completed updating the background image, the background acquisition unit 303 acquires data of the updated background image as background image data. That is, the background acquisition unit 303 acquires the latest background image data. The background image data acquired by the background acquisition unit 303 is transmitted to the extraction unit 302 and the update unit 304. Next, in S402, the image acquisition unit 301 acquires captured image data as input image data from each imaging device included in the imaging device 101. The input image data acquired in S401 is transmitted to the extraction unit 302, the background acquisition unit 303, and the update unit 304.

[0027] Next, in S403, the extraction unit 302 extracts a foreground region in each input image by comparing the corresponding input image and background image based on the input image data acquired in S402 and the background image data acquired in S401. The foreground image data indicating the foreground region corresponding to each input image extracted in S403 is transmitted to the information processing device 103. The information processing device 103 receives the foreground image data corresponding to each input image transmitted in S403 and generates a 3D model corresponding to the object. Note that, in this embodiment, information such as camera parameters required for generating a 3D model is assumed to be stored in advance in the information processing device 103, but the information processing device 103 may receive this information from the image processing device 102 or the imaging device 101 as appropriate. The 3D model data generated by the information processing device 103 is transmitted to the image processing device 102.

[0028] Next, in S404, the model acquisition unit acquires data of the three-dimensional model generated by the information processing device 103. The data of the three-dimensional model acquired in S404 is transmitted to the update unit 304. Next, in S405, the update unit 304 projects the three-dimensional model acquired in S404 onto each input image, and identifies the projection status of the three-dimensional model onto each input image. The processing of S405 will be described with reference to FIG. 5.

[0029] FIG. 5 is a diagram illustrating the projection processing in the update unit 304 according to the first embodiment. FIG. 5 illustrates three image capturing devices 511, 512, and 513 as an example of the multiple image capturing devices included in the image capturing device 101. In FIG. 5, input images 521, 522, and 523 indicate captured images obtained by image capturing using the image capturing devices 511, 512, and 513, respectively. The update unit 304 projects the three-dimensional model 501 onto each of the input images 521 to 523 using the camera parameters of the image capturing devices 511 to 513. As shown as an example in FIG. 5, the three-dimensional model 501 is projected onto the input images 521 and 522, but is not projected onto the input image 523. It is assumed that the camera parameters of the image capturing devices 511 to 513 are stored in advance in the update unit 304, but the update unit 304 may receive the camera parameters from the image capturing device 101 as appropriate.

[0030] After S405, in S406, the update unit 304 determines for each input image whether a three-dimensional model is projected onto the input image. For a background image corresponding to an imaging device that captured an input image for which it was determined in S406 that a three-dimensional model is not to be projected, in S407 the update unit 304 updates the background image. In the example of FIG. 5, since the three-dimensional model 501 is not projected onto the input image 523, in S407 the update unit 304 updates the background image corresponding to the imaging device 513 that captured the input image 523. Specifically, in S407, the update unit 304 updates the background image by replacing the background image with the input image. In the example of FIG. 5, the update unit 304 updates the background image corresponding to the imaging device 513 by replacing the background image corresponding to the imaging device 513 with the input image 523. Note that the update unit 304 does not execute the process of S407 for a background image corresponding to an imaging device that captured an input image for which it was determined in S406 that a three-dimensional model is to be projected. After S407, the image processing device 102 ends the processing of the flowchart shown in FIG. 4, and when a plurality of captured images (frames) obtained by the next synchronized imaging are input, executes the processing of the flowchart again.

[0031] As described above, in the first embodiment, the image processing device 102 is configured to use a three-dimensional model to determine whether or not a target object is reflected in each input image, and to update the background image if no object is reflected. With the image processing device 102 configured in this way, the background image can be updated immediately when a three-dimensional model is not projected onto the input image, and the background image can be updated appropriately even if the target object does not move for a predetermined period of time. As a result, the image processing device 102 enables highly accurate foreground / background separation processing.

[0032] [Embodiment 2] <Image processing system configuration> A second embodiment will be described with reference to Figures 6 and 7. In the second embodiment, an acquired three-dimensional model is used to identify an area where no three-dimensional model exists, and information about the area is used to update a background image. Note that the configuration of the image processing system 100 according to the second embodiment, and the hardware configuration and functional configuration of the image processing device 102 are the same as those of the first embodiment, and therefore, in this embodiment, only the differences from the first embodiment will be described.

[0033] <Operation of image processing device> FIG. 6 is a flowchart showing an example of the processing flow in the image processing device 102 according to the second embodiment. The processing of this flowchart is repeatedly executed, for example, each time data of multiple frames obtained by synchronized imaging is input from multiple imaging devices included in the imaging device 101. Note that "synchronized imaging" does not mean simultaneous imaging, but rather imaging after synchronization processing. In other words, synchronized imaging does not need to be performed at exactly the same time, and also includes modes in which imaging is performed approximately at the same time. Note that processing steps shown in FIG. 6 that are similar to processing steps shown in FIG. 4 are assigned the same reference numerals as in FIG. 4 and will not be described again. First, the image processing device 102 executes processing from S401 to S405. After S405, in S607, the update unit 304 updates the background image. Specifically, the update unit 304 uses an area in each input image where a 3D model is not projected to update the area of ​​the background image corresponding to that area.

[0034] Fig. 7 is a diagram illustrating the background image update process in the update unit 304 according to the first embodiment. Note that in Fig. 7, the same components as those in Fig. 5 are denoted by the same reference numerals, and their description will be omitted. In the input image 521 shown in Fig. 7, an area 701 is an area onto which the three-dimensional model 501 is projected, and an area 702 indicated by a hatched texture is an area onto which the three-dimensional model 501 is not projected. Similarly, in the input image 522 shown in Fig. 7, an area 703 is an area onto which the three-dimensional model 501 is projected, and an area 704 indicated by a hatched texture is an area onto which the three-dimensional model 501 is not projected.

[0035] The updating unit 304 updates the background image corresponding to the imaging device 511 by replacing pixel values ​​of a region 702 in the background image corresponding to the imaging device 511, where the three-dimensional model is not projected, with the pixel values ​​of the region 702 in the input image 521. Similarly, the updating unit 304 updates the background image corresponding to the imaging device 512 by replacing pixel values ​​of a region 704 in the background image corresponding to the imaging device 512, where the three-dimensional model is not projected, with the pixel values ​​of the region 704 in the input image 522. Note that in the input image 523, the three-dimensional model 501 is not projected over the entire image area, so the updating unit 304 updates the background image corresponding to the imaging device 513 by replacing it with the input image 523. After S607, the image processing device 102 ends the processing of the flowchart shown in FIG. 6, and when a plurality of captured images (frames) obtained by the next synchronized imaging are input, executes the processing of the flowchart again.

[0036] As described above, in the second embodiment, the image processing device 102 is configured to identify an area where a target object exists for each imaging device using a three-dimensional model, and to perform an update process for areas in the background image where no object exists. The image processing device 102 configured in this way can immediately update the background image for areas where a three-dimensional model is not projected onto the input image, and can appropriately update the background image even if the target object is not moving. As a result, the image processing device 102 enables highly accurate foreground / background separation processing.

[0037] [Embodiment 3] <Image processing system configuration> A third embodiment will be described with reference to Figures 8 to 10. In the third embodiment, an acquired three-dimensional model is used to identify an area where a three-dimensional model corresponding to a predetermined object does not exist, and information about the area is used to update a background image. Note that the configuration of the image processing system 100 according to the second embodiment and the hardware configuration of the image processing device 102 are the same as those of the first and second embodiments, and therefore, in this embodiment, only the differences from the first and second embodiments will be described.

[0038] <Functional configuration of image processing device> 8 is a block diagram showing an example of the functional configuration of an image processing device 102 according to the third embodiment. The image processing device 102 includes an image acquisition unit 301, an extraction unit 302, a background acquisition unit 303, an update unit 304, a model acquisition unit 305, an identification unit 801, and an information acquisition unit 802. Each unit included in the functional configuration of the image processing device 102 is realized by the CPU 201 executing a program stored in the ROM 203, using the RAM 202 as a work memory. In FIG. 8, components that perform the same processes as those in the functional configuration shown in FIG. 3 are denoted by the same reference numerals, and their description will be omitted. In this embodiment, three-dimensional model data acquired by the model acquisition unit 305 is transmitted to the identification unit 801.

[0039] The information acquisition unit 802 acquires information (hereinafter referred to as "object information") that can identify the object to be imaged, such as the feature amounts of the object to be imaged. In this embodiment, it is assumed that the object information corresponding to the object to be imaged is set in advance by the user and stored in the ROM 203 or the like. That is, the information acquisition unit 802 acquires the object information from the ROM 203 or the like by reading out the object information stored in the ROM 203 or the like. The object information acquired by the information acquisition unit 802 is transmitted to the identification unit 801.

[0040] The identifying unit 801 identifies a three-dimensional model received from the model acquiring unit 305 that corresponds to the object to be imaged, using the object information received from the information acquiring unit 802. The identifying unit 801 transmits data of the three-dimensional model that has been identified as the three-dimensional model that corresponds to the object to be imaged, from among the three-dimensional models received from the model acquiring unit 305, to the updating unit 304.

[0041] <Operation of image processing device> 9 is a flowchart showing an example of the flow of processing in the image processing device 102 according to the third embodiment. The processing of this flowchart is repeatedly executed, for example, each time data of a plurality of frames obtained by capturing images synchronized with each other is input from a plurality of image capturing devices included in the image capturing device 101. Note that "capturing images synchronized with each other" does not mean capturing images simultaneously, but rather refers to capturing images after performing synchronization processing. In other words, the capturing images synchronized with each other do not need to be performed at exactly the same time, and also includes modes in which the capturing images are performed at approximately the same time. Note that, in the processing steps shown in FIG. 9, the same steps as those shown in FIG. 4 or 6 are denoted by the same reference numerals as those in FIG. 4 or 6, and description thereof will be omitted.

[0042] First, the image processing device 102 executes the processes from S401 to S404. The three-dimensional model data acquired in S404 is transmitted to the identification unit 801. After S404, in S901, the information acquisition unit 802 acquires object information. The object information acquired in S901 is transmitted to the identification unit 801. Next, in S902, the identification unit 801 uses the object information acquired in S901 to identify, for each of the one or more three-dimensional models acquired in S404, whether or not the three-dimensional model corresponds to the object to be imaged. Data of the three-dimensional model identified in S902 as the three-dimensional model corresponding to the object to be imaged is transmitted to the update unit 304 and used in the processes of S905 and S907. Next, in S905, the update unit 304 projects the three-dimensional model identified in S902 as the three-dimensional model corresponding to the object to be imaged onto each input image, and identifies the projection status of the three-dimensional model onto each input image. The process of S905 will be described with reference to FIG. 10 .

[0043] FIG. 10 is a diagram illustrating the projection processing in the update unit 304 according to the third embodiment. Note that in FIG. 7, the same components as those in FIG. 5 or 7 are denoted by the same reference numerals, and descriptions thereof will be omitted. In FIG. 10, a three-dimensional model 501 is a three-dimensional model corresponding to an object to be imaged, and three-dimensional models 1002 to 1004 are three-dimensional models corresponding to objects other than the object to be imaged. Three-dimensional models corresponding to objects other than the object to be imaged, such as the three-dimensional model 1002, are unintended three-dimensional models that have been extracted as a foreground region due to, for example, unintended dirt or unevenness on the ground, and have been generated by the information processing device 103. In S905, the update unit 304 projects only the three-dimensional model 501 corresponding to the object to be imaged in S902 onto the input image, and does not project the other three-dimensional models 1002 to 1004 onto the input image.

[0044] In the example shown in FIG. 10 , the object information includes, for example, feature quantities of a natural person as the object to be imaged. By using such object information, the identification unit 801 can distinguish and identify the three-dimensional model 501 corresponding to the natural person from the three-dimensional models 1002 to 1004 corresponding to objects other than natural people. The object information is not limited to the information described above. For example, the object information may include information indicating the height range of the object. For example, if the object to be imaged is a natural person and the object to be excluded from the imaged subject is dirt or unevenness on the ground, information indicating a predetermined height, such as a height of one meter or more, can be set as the object information.

[0045] After S905, in S907, the update unit 304 updates the background image. Specifically, the update unit 304 uses an area in each input image onto which a three-dimensional model corresponding to the object to be imaged, identified in S905, is not projected to update the area of ​​the background image corresponding to that area. The background image update process in S907 is similar to the background image update process in S607, and therefore a detailed description thereof will be omitted. After S907, the image processing device 102 ends the process of the flowchart shown in FIG. 9, and when a plurality of captured images (frames) obtained by the next synchronized imaging are input, executes the process of the flowchart again.

[0046] As described above, in the third embodiment, the image processing device 102 is configured to identify in advance a three-dimensional model corresponding to an object to be captured, and to identify, for each imaging device, an area in which the object to be captured is present using the identified three-dimensional model. Also, as described above, in the third embodiment, the image processing device 102 is configured to perform an update process for areas in a background image in which no object is present. The image processing device 102 configured in this manner can immediately perform background update for areas in an input image in which a three-dimensional model corresponding to the object to be captured is not projected. As a result, the image processing device 102 can appropriately update the background image even when the target object is not moving, enabling highly accurate foreground / background separation processing.

[0047] [Other embodiments] In the above-described embodiment, the image processing device 102 has been described as including, as an example, the extraction unit 302, which extracts a foreground region from an input image. However, since the image processing device 102 only needs to acquire foreground image data, the foreground region extraction process may be performed by an external device. Also, in the above-described embodiment, the image processing system 100 has been described as including, as an example, the information processing device 103, which generates a three-dimensional model corresponding to an object. However, the three-dimensional model may be generated by the image processing device 102. In this case, for example, the model acquisition unit 305 receives foreground image data transmitted from the extraction unit 302 and generates a three-dimensional model based on the received foreground image data, thereby acquiring the three-dimensional model data. The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0048] It should be noted that within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0049] [Configuration of the present disclosure] The present disclosure includes the following configurations, methods, and programs.

[0050] <Configuration 1> image acquisition means for acquiring input image data; background acquisition means for acquiring background image data used to extract a foreground region from the input image; a model acquisition means for acquiring data of a three-dimensional model generated based on the foreground region extracted from the input image; an update means for updating the background image based on the input image and the three-dimensional model; 1. An image processing device comprising:

[0051] <Configuration 2> the updating means updates the background image by replacing the background image with the input image on which the three-dimensional model is not projected when the three-dimensional model is projected onto the input image; 2. The image processing device according to configuration 1,

[0052] <Configuration 3> the updating means updates the background image by replacing an area in the input image onto which the three-dimensional model is not projected with an area in the background image corresponding to the area, when the three-dimensional model is projected onto the input image; 3. The image processing device according to configuration 1 or 2, characterized in that:

[0053] <Configuration 4> information acquisition means for acquiring object information relating to a target object; an identification means for identifying a target 3D model corresponding to the target object from among the 3D models based on the object information; and the updating means updates the background image based on the input image and the target three-dimensional model; 4. The image processing device according to any one of configurations 1 to 3, characterized in that:

[0054] <Configuration 5> the updating means updates the background image by replacing the background image with the input image on which the target three-dimensional model is not projected when the target three-dimensional model is projected onto the input image; 5. The image processing device according to configuration 4,

[0055] <Configuration 6> the updating means updates the background image by replacing an area in the input image where the target 3D model is not projected with an area in the input image where the target 3D model is not projected, the area corresponding to the area being projected; 6. The image processing device according to configuration 4 or 5,

[0056] <Configuration 7> the image acquisition means acquires, as the input image data, image data obtained by imaging using an imaging device; 7. The image processing device according to any one of configurations 1 to 6,

[0057] <Configuration 8> the updating means performs a projection process of the three-dimensional model onto the input image based on camera parameters of the imaging device; 8. The image processing device according to claim 7,

[0058] <Configuration 9> extraction means for extracting the foreground region from the input image using the background image; further comprising: 9. The image processing device according to any one of configurations 1 to 8,

[0059] <Configuration 10> the model acquisition means acquires data of the three-dimensional model by generating the three-dimensional model based on the foreground region; 10. The image processing device according to any one of configurations 1 to 9, characterized in that:

[0060] <Method> an image acquisition step of acquiring input image data; a background acquisition step of acquiring background image data used to extract a foreground region from the input image; a model acquisition step of acquiring data of a three-dimensional model generated based on the foreground region extracted from the input image; an updating step of updating the background image based on the input image and the three-dimensional model; An image processing method comprising:

[0061] <Program> 11. A program for causing a computer to function as the image processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0062] 102 Image processing device 301 Image Acquisition Unit 303 Background acquisition section 304 Update Department 305 Model Acquisition Department

Claims

1. image acquisition means for acquiring input image data; background acquisition means for acquiring background image data used to extract a foreground region from the input image; a model acquisition means for acquiring data of a three-dimensional model generated based on the foreground region extracted from the input image; an update means for updating the background image based on the input image and the three-dimensional model; 1. An image processing device comprising:

2. the updating means updates the background image by replacing the background image with the input image on which the three-dimensional model is not projected when the three-dimensional model is projected onto the input image; 2. The image processing device according to claim 1, wherein:

3. the updating means updates the background image by replacing an area in the input image onto which the three-dimensional model is not projected with an area in the background image corresponding to the area, when the three-dimensional model is projected onto the input image; 2. The image processing device according to claim 1, wherein:

4. information acquisition means for acquiring object information relating to a target object; an identification means for identifying a target 3D model corresponding to the target object from among the 3D models based on the object information; and the updating means updates the background image based on the input image and the target three-dimensional model; 2. The image processing device according to claim 1, wherein:

5. the updating means updates the background image by replacing the background image with the input image on which the target three-dimensional model is not projected when the target three-dimensional model is projected onto the input image; 5. The image processing device according to claim 4, wherein:

6. the updating means updates the background image by replacing an area in the input image where the target 3D model is not projected with an area in the input image where the target 3D model is not projected, the area corresponding to the area being projected; 5. The image processing device according to claim 4, wherein:

7. the image acquisition means acquires, as the input image data, image data obtained by imaging using an imaging device; 2. The image processing device according to claim 1, wherein:

8. the updating means performs a projection process of the three-dimensional model onto the input image based on camera parameters of the imaging device; 8. The image processing device according to claim 7,

9. extraction means for extracting the foreground region from the input image using the background image; further comprising:

2. The image processing device according to claim 1, wherein:

10. the model acquisition means acquires data of the three-dimensional model by generating the three-dimensional model based on the foreground region; 2. The image processing device according to claim 1, wherein:

11. an image acquisition step of acquiring input image data; a background acquisition step of acquiring background image data used to extract a foreground region from the input image; a model acquisition step of acquiring data of a three-dimensional model generated based on the foreground region extracted from the input image; an updating step of updating the background image based on the input image and the three-dimensional model; An image processing method comprising:

12. A program for causing a computer to function as the image processing device according to any one of claims 1 to 10.

Images