Image processing device, image processing method, and program

Abstract

To generate information indicating an appropriate foreground area.SOLUTION: An image processing apparatus comprises acquisition means, setting means, first generation means, and second generation means. The acquisition means acquires a picked-up image acquired through imaging performed by an imaging apparatus. The setting means sets, for the picked-up image, a first area and a second area different from the first area. The first generation means generates information indicating a first foreground area from the first area based on the picked-up image and an image including at least a background area corresponding to the picked-up image. The second generation means generates information indicating a second foreground area different from the first foreground area from the second area based on the picked-up image and another picked-up image acquired through imaging performed by the imaging apparatus at the same position and in the same direction as and at a different timing from the imaging performed by the imaging apparatus.SELECTED DRAWING: Figure 4

Description

Technical Field

[0001] The present disclosure relates to a technique for generating information indicating a foreground region from a captured image.

Background Art

[0002] There is a technique for generating foreground information from a captured image captured by an imaging device. Patent Document 1 discloses a method (so-called background difference) of generating a background image from an input image and extracting a foreground by the difference between the input image and the background image. Patent Document 2 discloses a method (so-called frame difference) of extracting a foreground by the difference between images with different foreground positions.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Patent Document 2

Summary of the Invention

Problems to be Solved by the Invention

[0004] However, in Patent Document 1 and Patent Document 2, there is a possibility that an object other than the object to be extracted as the foreground is also extracted from the captured image as the foreground.

[0005] The present disclosure provides a technique for appropriately generating information indicating a foreground region.

Means for Solving the Problems

[0006] An image processing apparatus according to an aspect of the present disclosure includes an acquisition unit that acquires a captured image acquired by imaging of an imaging device, Setting means for setting a first region and a second region in the captured image, among the captured images The aforementioned extracts a difference between the captured image and a background image generated in advance in a first region, and among the captured images The aforementionedThe system is characterized by having an extraction means that extracts the difference between the captured image and a captured image acquired by the acquisition means prior to the captured image in the second region. [Effects of the Invention]

[0007] According to this disclosure, it is possible to generate information that appropriately indicates the foreground region. [Brief explanation of the drawing]

[0008] [Figure 1] This figure shows an example configuration of an image processing system that generates virtual viewpoint images. [Figure 2] This diagram shows an example of the internal configuration of a camera adapter. [Figure 3] This figure shows an example of the hardware configuration of an image processing device. [Figure 4] This figure shows an example of the functional configuration of an image processing device. [Figure 5] This is a diagram illustrating the generation of foreground information. [Figure 6] This is a flowchart showing the processing flow performed by the image processing device. [Figure 7] This flowchart shows the process flow for generating the first foreground information and the second foreground information. [Figure 8] This figure shows a modified example of the functional configuration of an image processing device. [Figure 9] This figure shows an example of the functional configuration of an image processing device. [Figure 10] This is a diagram illustrating the generation of foreground information. [Modes for carrying out the invention]

[0009] Embodiments of this disclosure will be described below with reference to the drawings. Note that the following embodiments are not limiting to this disclosure, and not all combinations of features described in these embodiments are essential to the solutions of the present invention. Identical components are denoted by the same reference numerals.

[0010] <Embodiment 1> This embodiment describes a method for generating foreground information (first foreground information, second foreground information) used to generate a virtual viewpoint image. Specifically, this embodiment describes a method in which first foreground information generated using the background subtraction method and second foreground information generated using the frame subtraction method are selected (generated) for each predetermined first and second region in the captured image acquired by imaging by the imaging device. In this embodiment, an example is described in which foreground information is used to generate a virtual viewpoint image, but it is not necessarily required to be used to generate a virtual viewpoint image. Therefore, it is not bound by the relative geometric installation conditions between multiple imaging devices. Furthermore, it is not necessary to use multiple imaging devices. For example, it may be used to generate the foreground of an image captured by a single imaging device, such as a surveillance imaging device installed on premises, in a remote location, or outdoors.

[0011] First, to facilitate understanding of this embodiment, a brief overview of virtual viewpoint image generation will be provided. There is a technique that generates virtual viewpoint images from imaging devices that do not actually exist by setting up multiple imaging devices at different locations and performing synchronized imaging, and using the images from multiple viewpoints obtained from this imaging. For example, using virtual viewpoint images allows users to view highlight scenes of soccer or basketball from various angles, thus providing users with a greater sense of presence compared to normal images.

[0012] When generating virtual viewpoint images, images captured synchronously by multiple imaging devices are aggregated in an image processing unit such as a server. This unit then performs processes such as separating the foreground (the main subject / object) from the background, modeling it, and rendering it. Finally, the processing results are transmitted from the server to the user's terminal for display, enabling user viewing.

[0013] When modeling (generating) the foreground, information about the foreground mask, which corresponds to the silhouette of the foreground as seen from multiple imaging devices, and information about the texture of the foreground (for example, the R, G, and B color information of each pixel in the foreground) are required.

[0014] The process of separating the foreground from the background part is called foreground-background separation processing. Foreground-background separation processing is a process of estimating the foreground region and is generally performed by the background subtraction method or the frame difference method. The background subtraction method is to obtain the difference between the background image and the input image including the foreground, and regard the region where the set of pixels whose difference value is determined to be greater than or equal to a predetermined threshold as the foreground region. Frame difference means that from the difference image of temporally consecutive images (frames), the region where the set of pixels whose difference value is determined to be greater than or equal to a predetermined threshold is regarded as the foreground region.

[0015] (System Configuration) Here, an image processing system for generating a virtual viewpoint image will be described using a figure. FIG. 1 is a diagram showing a configuration example of an image processing system for generating a virtual viewpoint image. The image processing system 100 includes imaging modules 110a to 110z, a database (DB) 250, a server 270, a control device 300, a switching hub 180, and an end-user terminal 190. That is, the image processing system 100 has three functional domains: an image collection domain, a data storage domain, and an image generation domain. The image collection domain includes imaging modules 110a to 110z, the data storage domain includes the DB 250 and the server 270, and the image generation domain includes the control device 300 and the end-user terminal 190.

[0016] The control device 300 performs management of the operating state and parameter setting control, etc. on each block constituting the image processing system 100 through a network. Here, the network may be GbE (Gigabit Ethernet) compliant with the IEEE standard of Ethernet (registered trademark) or 10GbE, or may be configured by combining an interconnect Infiniband, an industrial local area network, etc. Also, it is not limited to these, and other types of networks may be used.

[0017] First, we will explain the operation of transmitting images from 26 sets of imaging modules 110a to 110z from imaging module 110z to server 270. Each imaging module 110a to 110z has one camera each, from 112a to 112z. In the following, the 26 sets of systems from imaging modules 110a to 110z may be referred to simply as "imaging module 110" without distinction. Similarly, the devices within each imaging module 110 may be referred to as "camera 112" and "camera adapter 120." Note that although the number of imaging modules 110 is given as 26 sets, this is merely an example and is not limited to this.

[0018] The imaging modules 110a to 110z are connected in a daisy-chain configuration. This connection configuration has the effect of reducing the number of connection cables and simplifying wiring work when increasing the resolution of captured images to 4K or 8K and increasing the frame rate, which leads to larger image data volumes. The connection configuration is arbitrary; for example, an imaging module 110a to 110z may be connected to a switching hub 180, and data transmission and reception between imaging modules 110 may be performed via the switching hub 180 in a star-type network configuration.

[0019] In this embodiment, each imaging module 110 consists of a camera (imaging device) 112 and a camera adapter 120, but is not limited to this. For example, it may have a microphone, a pan / tilt head, or an external sensor. Also, in this embodiment, the camera 112 and the camera adapter 120 are configured separately, but they may be integrated in the same housing. The image captured by the camera 112a in imaging module 110a is subjected to image processing described later in the camera adapter 120a and then transmitted to the camera adapter 120b of imaging module 110b. Similarly, imaging module 110b transmits the image captured by camera 112b, together with the image captured from imaging module 110a, to imaging module 110c. By continuing this operation, 26 sets of captured images are transmitted from imaging module 110z to the switching hub 180, and then to the server 270.

[0020] In this embodiment, the evaluation of whether an image is in the foreground is performed within each individual camera adapter 120. However, the embodiment is not limited to this configuration, and a server 270 that receives 26 sets of captured images may generate silhouette images corresponding to each captured image.

[0021] (Camera adapter configuration) Next, the camera adapter 120 will be described in detail. Figure 2 is a functional block diagram showing an example of the internal configuration of the camera adapter 120. The camera adapter 120 includes a network adapter 121, a transmission unit 122, an image processing device 123, and a camera control unit 124.

[0022] The network adapter 121 communicates data with other camera adapters 120, server 270, and control device 300. It also complies with, for example, the IEEE 1588 standard Ordinary Clock, saving timestamps of data transmitted and received with server 270, and synchronizing time with server 270. Time synchronization with a time server may also be achieved using other EtherAVB standards or proprietary protocols. In this embodiment, a NIC (Network Interface Card) is used as the network adapter 121, but it is not limited to this.

[0023] The transmission unit 122 controls the transmission of data to the switching hub 180, etc., via the network adapter 121. The transmission unit 122 has the function of compressing the data to be transmitted and received by applying a predetermined compression method, compression ratio, and frame rate, and the function of decompressing the compressed data. It also has the function of determining the routing destination of the received data and the data processed by the image processing device 123, and the function of transmitting the data to the determined routing destination. It also has the function of creating a message for transferring image data to another camera adapter 120 or server 270. The message contains metadata of the image data. This metadata includes the time code or sequence number at the time of sampling of image acquisition, the data type, and the identifier of the camera 112. The image data to be transmitted may be compressed. It also receives a message from another camera adapter 120 and restores the data information, which has been fragmented into a packet size specified by the transmission protocol, into image data according to the data type contained in the message.

[0024] The image processing device 123 performs foreground separation and generates foreground information, including images of objects, based on image data captured by the camera 112 under the control of the camera control unit 124, and initialization information. It also performs processes such as dynamic calibration. By having each of the multiple camera adapters 120 generate the foreground, the load on the image processing system 100 can be distributed. Dynamic calibration is calibration performed during imaging and includes color correction processing to suppress color variations between cameras, and blur correction processing (electronic vibration damping processing) to stabilize the image position against blur caused by camera vibration. The foreground information generated by the image processing device 123 includes a foreground silhouette image showing the foreground region and an image extracted from the input image showing the foreground region. These are transmitted to an image processing unit such as a server to generate a virtual viewpoint image. The foreground information only needs to include either a foreground silhouette image showing the foreground region or an image extracted from the input image showing the foreground region (foreground texture image).

[0025] The camera control unit 124 connects to the camera 112 and performs functions such as controlling the camera 112, acquiring captured images, providing synchronization signals, and setting the time. Control of the camera 112 includes, for example, setting and referencing imaging parameters (number of pixels, color depth, frame rate, white balance settings, etc.), acquiring status information of the camera 112 (imaging, stopped, synchronizing, and error, etc.), starting and stopping imaging, and adjusting the focus.

[0026] (Hardware configuration of the image processing device) Next, an example of the hardware configuration of the image processing device will be described using a diagram. Figure 1 is a block diagram showing the schematic hardware configuration of the image processing device according to this embodiment. The image processing device 123 includes a CPU 311, ROM 312, RAM 313, auxiliary storage device 314, display unit 315, operation unit 316, and communication interface (I / F) 317. Each component is connected to each other via a bus 318 so as to be able to communicate with each other.

[0027] The CPU (Central Processing Unit) 311 executes various programs stored in the ROM 312 and auxiliary storage device 314, etc., and comprehensively controls the image processing device 123 to realize the various functions of the image processing device 123 shown in Figure 4. The image processing device 123 may have one or more dedicated hardware components separate from the CPU 311, and at least a portion of the processing performed by the CPU 311 may be executed by the dedicated hardware. Examples of dedicated hardware include ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0028] The ROM (Read Only Memory) 312 stores programs and other data that do not require modification. The RAM (Random Access Memory) 313 temporarily stores programs and data supplied from the ROM 312 and auxiliary storage device 314, as well as data supplied from external sources via the communication interface 317. The auxiliary storage device 314 includes, for example, a hard disk drive and stores various types of data such as image data and audio data.

[0029] The display unit 315 has, for example, a liquid crystal display or LEDs, and displays a GUI (Graphical User Interface) for the user to operate the image processing device 123. The operation unit 316 has, for example, a keyboard, mouse, joystick, touch panel, etc., and receives various instructions from the user and inputs them to the CPU 311. The CPU 311 operates as a display control unit that controls the display unit 315, and as an operation control unit that controls the operation unit 316.

[0030] The communication interface 317 is used for communication between the image processing device 123 and external devices. For example, if the image processing device 123 is connected to an external device by a wired connection, a communication cable is connected to the communication interface 317. If the image processing device 123 has the function of wirelessly communicating with an external device, the communication interface 317 is equipped with an antenna. The bus 318 connects the various parts of the image processing device 123 and transmits information.

[0031] In this embodiment, the display unit 315 and the operation unit 316 are located inside the image processing device 123, but at least one of the display unit 315 and the operation unit 316 may exist outside the image processing device 123 as a separate device.

[0032] [Functional Configuration of Image Processing Devices] Next, the functional configuration of the image processing device 123 will be explained using a diagram. Figure 4 is a functional block diagram of the image processing device 123. The functional configuration shown in Figure 4 is realized by supplying the program to the image processing device 123 shown in Figure 3, and then having the image processing device 123 execute it.

[0033] The image processing device 123 of this embodiment includes a storage unit 401, a background generation unit 402, a first information generation unit 403, a second information generation unit 404, and a selection unit 405. The storage unit 401, background generation unit 402, first information generation unit 403, and second information generation unit 404 of the image processing device receive image data of captured images (processed images) obtained by the camera (imaging device) 112. The image data of captured images input to the image processing device 123 is also called the current input image (processed image). Captured images are images such as frames of still images or moving images that are captured continuously over time.

[0034] The memory unit 401 stores a predetermined number of captured images (input images). The predetermined number of frames can be any number necessary for the second information generation unit 403 to generate the second foreground information, which will be described in detail later. For example, it can be the four frames from four frames before the current frame to the frame before that.

[0035] The background generation unit 402 generates a background image from the input image. The background generation unit 402 determines whether there is motion in the input image by comparing the current input image with past input images (past images) stored in the storage unit 401 at regular intervals, and generates the region determined to have no motion relatively as the background image. Past input images only need to be input images from one or more frames prior to the input image of the current frame. Known techniques are used to determine whether there is motion. For example, the presence or absence of motion is determined by deriving a difference between the input image of the current frame and input images from multiple past frames, and checking whether the number of frames in which the derived difference is less than a predetermined threshold reaches a predetermined number of times. That is, if the number of frames in which the derived difference is less than a predetermined threshold reaches a predetermined number of times, it is determined that there is no motion relatively. If the number of frames in which the derived difference is less than a predetermined threshold does not reach a predetermined number of times, it is determined that there is motion relatively. If no past input images are stored in the storage unit 401, the background generation unit 402 generates the input image as the background image.

[0036] The first information generation unit 403 generates first foreground information (first information) for generating a first foreground included in the captured image (input image) using the background subtraction method. The first information can also be said to include a first foreground region. Specifically, the first information generation unit 403 binarizes the difference (background difference) between the background image generated by the background generation unit 402 and the input image at a predetermined threshold, thereby designating the relatively moving region as the foreground region and generating a foreground silhouette image that indicates the foreground region. The first information generation unit 403 can also be said to generate first foreground information indicating the first foreground region from the difference between images. The background image generated by the background generation unit 402 can also be said to be an image that includes at least the background region corresponding to the input image. An image that includes at least the background region corresponding to the input image can also be said to be another captured image obtained by imaging by the imaging device at the same position and direction as the imaging device, but at a different timing. Here, the generation of first foreground information by the first information generation unit 403 will be explained using a figure.

[0037] Figure 5 is a diagram illustrating the generation of foreground information. In Figure 5, the input image shows a scene from a soccer match, including three players running on the field and spectators in the stands watching the game, with a ball kicked high into the air against the backdrop of the stands. Based on this input image, foreground information is generated. The three players and the ball are considered foreground objects. The field area is designated as the first area, and objects within the field area are considered the first foreground. The spectator stand area is designated as the second area, and objects within the spectator stand area are considered the second foreground. Figures 5(a) to 5(d) correspond to the same captured image. In Figures 5(a) to 5(d), the area A100 to the left of the dotted line corresponds to the soccer field (field area), and the area A200 to the right of the dotted line corresponds to the spectator stands surrounding the field (spectator stand area). Figure 5(a) is a diagram showing an example of the first foreground information (foreground silhouette image) generated by the first information generation unit 403. In the field area A100, the area corresponding to the three players running on the field is separated as the foreground M100. In other words, the first information generation unit 403 generates a foreground silhouette image (foreground mask) corresponding to the objects (first foreground) included in the field area A100. In the spectator seating area A200, the area corresponding to spectators making relatively small movements in the seating area is separated as the foreground M101. Also, in the spectator seating area A200, a ball kicked high by a player is captured with the spectators in the background, and the foreground M101 separated from the spectator seating area A200 also includes the area corresponding to the ball. In other words, the foreground M101 includes the area corresponding to the spectators making relatively small movements and the area corresponding to the ball. The area corresponding to the spectators making relatively small movements is an area that has no meaning in the foreground M101 (it is actually an unnecessary area). The example illustrated here is that background subtraction can achieve the desired separation of objects on a static background, such as a field, but it also separates unwanted areas into the foreground when the background is not static, such as spectator seats.In this embodiment, a background image sequentially updated by the background generation unit 402 is used, but for example, an image captured in advance when the players are not on the field may be used as the background image. However, in this case, all spectators in the spectator seating area will be separated as foreground. The first information generation unit 403 generates a foreground silhouette image that shows the foreground area by separating relatively moving objects from the input image as foreground, for example using the background subtraction method.

[0038] The second information generation unit 404 uses the frame difference method to generate second foreground information (second information) for generating a second foreground included in the captured image (input image), using the captured image and past input images read from the storage unit 401. It can also be said that the second information generation unit 404 generates second foreground information indicating a second foreground region from the difference between images. Past input images can also be said to be other captured images acquired by imaging by the imaging device at the same position and direction as the imaging device's imaging but at a different timing. The second information can also be said to include a second foreground region. Specifically, the second information generation unit 404 generates a difference image (first frame difference) between the current input image and, for example, a past input image two frames prior to the current input image. Furthermore, the second information generation unit 404 generates a difference image (second frame difference) between the past input image two frames prior and the past input image two frames prior to the past input image two frames prior. Next, the second information generation unit 404 subtracts the second frame difference image from the first frame difference image to separate only the moving regions in the current input image as frame difference images. Then, the second information generation unit 404 binarizes the frame difference images at a predetermined threshold, designating the relatively moving regions as foreground regions, and generates a foreground silhouette image representing the foreground regions. In other words, the second information generation unit 404 also generates foreground silhouette images (foreground masks) corresponding to objects (second foreground) included in the spectator seating area A200. Note that the past input images used to generate the second foreground information are not limited to input images from 2 frames and 4 frames prior to the current input image, but may also be input images from n frames and (n+2) frames prior to the current input image, where n is a natural number.

[0039] Figure 5(b) shows an example of second foreground information (foreground silhouette image) generated by the second information generation unit 404. As shown in Figure 5(b), in the field region A100, three players are separated as foreground M200. However, because the players move only slightly in the time difference between two consecutive frames, and do not move completely to a position that does not overlap with the position of the player in the previous frame, the players' outlines are separated as foreground M200. In other words, the region where the outlines of the players separated as foreground M200 are located can also be said to be a region in the processed image where the foreground cannot be detected in the difference between two consecutive frames. On the other hand, in the spectator seating region A200, the spectators are almost stationary in the time difference between two frames and are not separated as foreground, but the ball kicked high is moving at a high speed and is therefore separated as foreground M201. What is illustrated here is that in field difference, relatively slow-moving objects are separated as outlines, but relatively fast-moving objects can be separated as desired. In other words, the region containing the ball that is separated as foreground M201 is also the region in the processed image where the foreground can be detected by the difference between two consecutive frames.

[0040] The first information generation unit 403 outputs foreground information (first foreground information) that includes a foreground silhouette image (corresponding to Figure 5(a)) showing the region separated as the foreground, and a foreground image (not shown) obtained by cutting out the foreground region from the input image. The second information generation unit 404 outputs foreground information (second foreground information) that includes a foreground silhouette image (corresponding to Figure 5(b)) showing the region separated as the foreground, and a foreground image (not shown) obtained by cutting out the foreground region from the input image, similar to the first information generation unit 403.

[0041] The selection unit 405 selects and outputs either the first foreground information generated by the first information generation unit 403 or the second foreground information generated by the second information generation unit 404, based on the selection information. The selection information is information specified in advance by the user and pre-set for each region in the captured image (processed image), and is held (set) in the selection unit 405. Figure 5(c) shows an example of the selection information set in the selection unit 405. As shown in Figure 5(c), the selection information is 1-bit image information of the same size as the input image, for example, with the region corresponding to the field region A100 set to "1" and the region corresponding to the spectator seating region A200 set to "0". Furthermore, once the imaging angle of view of the imaging device is determined, the selection information can be set for the boundary between the field region and the spectator seating region using structures such as stands within the angle of view as a guide.

[0042] Figure 5(d) shows an example of the output of the selection unit 405. When the processing pixel is a pixel corresponding to the field area A100 with a selection of "1" in the selection information, the selection unit 405 selects and outputs the foreground information M100 generated by the first information generation unit 403. When the processing pixel is a pixel corresponding to the spectator seating area with a selection of "0" in the selection information, the selection unit 405 selects and outputs the foreground information M201 generated by the second information generation unit 404. As shown in Figure 5(d), the selection unit 405 will select and output either the first foreground information shown in Figure 5(a) or the second foreground information shown in Figure 5(b) based on the selection information shown in Figure 5(c).

[0043] [Processing flow executed by the image processing unit] The processes performed by the image processing device described above will be explained using the diagram. Figure 6 is a flowchart showing the flow of processes performed by the image processing device in this embodiment. The processes shown in this flowchart are realized when the CPU 311 of the image processing device 123 reads a control program stored in the ROM 312 or auxiliary storage device 314, loads it into the RAM 313, and executes it. The processes in this embodiment are described as being raster scanned in synchronization with the frames in which the input image is input, but are not limited to this. In the description of the flowchart, the symbol "S" represents a step (process). The same applies to the following description of the flowchart.

[0044] First, the image processing device executes the processes in S100 and S200.

[0045] In S100, the first information generation unit 403 binarizes the difference (background difference) between the current input image and the background image generated by the background generation unit 402 corresponding to the current input image using a predetermined threshold, separates the relatively moving region as the foreground region, and generates first foreground information. After generation, the first information generation unit 403 moves the processing to S300.

[0046] In S200, the second information generation unit 404 generates a first frame difference image by binarizing the difference between the current input image and a past input image read from the storage unit 401, and generates a second frame difference image by binarizing the difference of the past input image. Then, the second information generation unit 404 subtracts the second frame difference image from the first frame difference image to separate only the relatively moving region in the current input image as the foreground region and generates second foreground information. After generation, the second information generation unit 404 moves the processing to S300. Note that S100 and S200 may be processed in parallel.

[0047] In S300, the selection unit 405 of the image processing device determines, based on selection information set in advance for each region, whether the currently processed image (pixel of interest) in the input image is in the first region for selecting the first foreground information generated by the first information generation unit 403. If the selection unit 405 determines that the pixel of interest is in the first region (YES in S300), it proceeds to S400. If the selection unit 405 determines that the pixel of interest is in the second region for selecting the second foreground information generated by the second information generation unit 404, and is not in the first region (NO in S300), it proceeds to S500.

[0048] In S400, the selection unit 405 selects the first foreground information generated by the first information generation unit 403 as the output. After selection, the selection unit 405 proceeds to S600.

[0049] In S500, the selection unit 405 selects the second foreground information generated by the second information generation unit 404 as the output. After selection, the selection unit 405 proceeds to S600.

[0050] In S600, the image processing device determines whether processing has been completed for all pixels in the input image. If the determination result is that processing has not been completed for all pixels in the input image (NO in S600), the image processing device returns to processing in S300. The image processing device then executes processing in S100 to S500 for all unprocessed pixels in the input image. On the other hand, if the determination result is that processing has been completed for all pixels in the input image (YES in S600), the image processing device terminates the flow shown in Figure 6.

[0051] Figure 7(a) is a flowchart showing the detailed flow of the first foreground information generation process in S100 of Figure 6. The process shown in this flowchart is realized when the CPU 311 of the image processing device 123 reads the control program stored in the ROM 312 or auxiliary storage device 314, loads it into the RAM 313, and executes it.

[0052] In S1001, the background generation unit 402 determines whether there is motion in the input image by comparing the input image with past input images stored in the storage unit 401 that correspond to the input image, for example, at regular intervals, and generates a background image of the region where there is relatively no motion. For example, if the change in the pixel value of the same pixel is less than a threshold, it is determined that there is relatively no motion. The background generation unit 402 then generates a background image of the region containing the pixels that have been determined to have relatively no motion. If the change in the pixel value of the same pixel is greater than or equal to a threshold, it is determined that there is relatively motion, and the background generation unit 402 does not generate a background image of the region containing the pixels that have been determined to have relatively motion.

[0053] In S1002, the first information generation unit 403 generates a background difference image (background difference) between the input image and the background image corresponding to the input image, which was generated in S1001 by the background generation unit 402. For generating the background difference image, for example, the difference values ​​of the R, G, and B gradation values ​​in the processing pixels can be used.

[0054] In S1003, the first information generation unit 403 binarizes the background difference image generated in S1002 at a predetermined threshold and separates the relatively moving region as the foreground region. In other words, the first information generation unit 403 generates a foreground silhouette image from the image obtained by binarizing the background difference image of the input image and the background image at a predetermined threshold, in the region where a stable background image can be generated, showing the region separated as the foreground region.

[0055] In S1004, the first information generation unit 403 generates first foreground information which includes the foreground silhouette image generated in S1003 and a foreground image obtained by cutting out the foreground region from the corresponding input image and the foreground silhouette image.

[0056] Figure 7(b) is a flowchart detailing the process of generating the second foreground information in S200 of Figure 6. The process shown in this flowchart is achieved when the CPU 311 of the image processing device 123 reads the control program stored in the ROM 312 or auxiliary storage device 314, loads it into the RAM 313, and executes it.

[0057] In S2001, the second information generation unit 404 generates a difference image (first frame difference) between the current input image and, for example, an input image from two frames prior (past image) read from the storage unit 401. The second information generation unit 404 further generates a difference image (second frame difference) between the aforementioned input image from two frames prior (past image) and an input image from four frames prior (past image) corresponding to the input image and read from the storage unit 401. Then, the second information generation unit 404 subtracts the second frame difference image from the first frame difference image to separate (generate) only the relatively moving region in the current input image as the foreground region (frame difference image). For generating the frame difference image, for example, the difference values ​​of the R, G, and B gradation values ​​in the processing pixels can be used.

[0058] In S2002, the second information generation unit 404 binarizes the frame difference image generated in S2001 at a predetermined threshold and separates the relatively moving region as the foreground region. In other words, the second information generation unit 404 generates a foreground silhouette image from the image obtained by binarizing the frame difference image of the first frame image and the second frame image at a predetermined threshold, showing the region separated as the foreground region in areas where a stable background image cannot be generated.

[0059] In S2003, the second information generation unit 404 generates second foreground information which includes the foreground silhouette image generated in S2002 and a foreground image obtained by cutting out the foreground region from the corresponding input image and the foreground silhouette image.

[0060] [effect] As described above, according to this embodiment, by applying a foreground-background separation method suitable for each region of the captured image, the generation of unnecessary foreground information can be suppressed, and necessary foreground information can be separated (generated). In other words, appropriate foreground information can be generated.

[0061] Furthermore, as a result, in a system that transmits and aggregates foreground information to an image processing unit such as a server to generate a virtual viewpoint image, it becomes possible to generate a virtual viewpoint image that does not omit necessary foreground information while suppressing transmission costs.

[0062] [Example 1] In the above-described embodiment 1, the selection unit 405 selects and outputs either first foreground information or second foreground information generated by processing all pixels of the input image, based on the selection information.

[0063] However, the first foreground information and the second foreground information are not limited to information generated by processing all pixels of the input image. Therefore, for example, the first foreground information generated for the first region of the input image and the second foreground information generated for the second region of the input image may be combined and output. Next, the functional configuration of the image processing apparatus, which is a modified example 1 of the above-described embodiment 1, will be explained using a figure. Figure 8 is a functional block diagram of the modified example 1 of the image processing apparatus. The functional configuration shown in Figure 8 is realized by supplying a program to realize that function to the image processing apparatus 123 shown in Figure 3, and then having the image processing apparatus 123 execute it.

[0064] The image processing device includes a storage unit 401, a background generation unit 402, a first information generation unit 403, a second information generation unit 404, a first region selection unit 406, a second region selection unit 407, and a synthesis unit 408. In this modified example, all components except the first region selection unit 406, the second region selection unit 407, and the synthesis unit 408 shown in Figure 8 are the same as in Embodiment 1, so their explanation is omitted here.

[0065] The first region selection unit 406 outputs the input image and background image to the first information generation unit 403 when the processing pixels are in the first region, based on the first selection information. The first selection information is information that indicates the region corresponding to the first foreground, which is set in advance and is held in the first region selection unit 406. In the selection information shown in Figure 5(c), the first information corresponds to the first region A100. The first information generation unit 403 generates first foreground information, including a foreground silhouette image and a foreground image, from the input image and background image that correspond to the first selection information input from the first region selection unit 406.

[0066] The second region selection unit 407 outputs the current input image and the past input image to the second information generation unit 404 based on the second selection information when the processing pixel is in the second region. The second selection information is information that indicates the region corresponding to the second foreground, which is set in advance and is held in the second region selection unit 407. In the selection information shown in Figure 5(c), the second information corresponds to the second region A200. The second information generation unit 404 generates second foreground information, including a foreground silhouette image and a foreground image, from the current input image and the past input image that correspond to the second selection information input from the second region selection unit 407.

[0067] The synthesis unit 408 synthesizes the first foreground information generated by the first information generation unit 403 and the second foreground information generated by the second information generation unit 404, and outputs the information corresponding to the processing pixels as foreground information.

[0068] As explained above, according to this modified example, even if the first selection information corresponding to the first region and the second selection information corresponding to the second region are set individually, the generation of unnecessary foreground information can be suppressed and the necessary foreground information can be generated, similar to Embodiment 1. In other words, appropriate foreground information can be generated.

[0069] [Differentiation 2] In the above-described embodiment 1, an example was shown in which the first information generation unit 403 generates first foreground information by background difference, and the second information generation unit 404 generates second foreground information by frame difference.

[0070] However, the method for generating the first foreground information by the first information generation unit 403 is not limited to background difference. Therefore, for example, foreground information may be generated by frame difference in either the first information generation unit 403 or the second information generation unit 404. For example, the first information generation unit 403 generates the first foreground information from the difference between the current frame and a past frame that is relatively far behind the imaging time of the current frame. The first foreground information corresponds to a region containing a relatively slow-moving foreground. The second information generation unit 404 generates the second foreground information from the difference between the current frame and a past frame that is relatively far behind the imaging time of the current frame. The second foreground information corresponds to a region containing a relatively fast-moving foreground. Therefore, it is possible to generate both the first foreground information corresponding to a region containing a relatively slow-moving foreground and the foreground information corresponding to a region containing a relatively fast-moving foreground. The regions that are relatively fast-moving and regions that are relatively slow-moving can be changed depending on the processing target. For example, if soccer is the processing target, it may be set as follows. Players may be assigned to areas of relatively slow movement, and running players may be set to be separable from the processed image as foreground frames using frame differences. Alternatively, the soccer ball may be assigned to areas of relatively fast movement, and the soccer ball during passes, dribbles, and shots may be set to be separable from the processed image as foreground frames using frame differences. Past frames with a relatively long time interval from the current frame's acquisition time may be, for example, three or four past frames prior to the current frame when the frame rate is 60fps (frames per second). Past frames with a relatively short time interval from the current frame's acquisition time may be, for example, two past frames prior to the current frame when the frame rate is 60fps.

[0071] As explained above, this modified version can generate first foreground information for generating a first foreground corresponding to relatively slow-moving objects, and second foreground information for generating a second foreground corresponding to relatively fast-moving objects. Therefore, similar to Embodiment 1, it is possible to suppress the generation of unnecessary foreground information and separate (generate) the necessary foreground information. In other words, it is possible to generate appropriate foreground information.

[0072] <Embodiment 2> In the above-described embodiment 1, the selection unit 405 selects and outputs either the first foreground information generated by the first information generation unit 403 or the second foreground information generated by the second foreground information generation unit 404, based on pre-set selection information.

[0073] However, the selection information used by the selection unit 405 to select the first or second region is not limited to pre-set information. Therefore, for example, the selection unit 405 may select and output the first foreground information or the second foreground information based on selection information generated from the difference between the current captured image (frame) and a past captured image (frame). In this embodiment, an image processing device that generates selection information for each region, selecting either the first foreground information generated using the background difference method or the second foreground information generated using the frame difference method, based on the analysis results of the input image will be described. In this embodiment, the same reference numerals are used for components identical to those in Embodiment 1 described above, and their descriptions are omitted.

[0074] [Functional Configuration of Image Processing Devices] The functional configuration of the image processing apparatus of this embodiment will be explained using the figures. Figure 9 is a functional block diagram of the image processing apparatus 123. The functional configuration shown in Figure 9 is realized by supplying a program to the image processing apparatus 123 shown in Figure 3, and then having the image processing apparatus 123 execute it.

[0075] The image processing apparatus 123 of this embodiment includes a storage unit 401, a background generation unit 402, a first information generation unit 403, a second information generation unit 404, a selection unit 405, and a selected information generation unit 409.

[0076] The selection information generation unit 409, for example, compares the input image with past input images (past images) stored in the storage unit 401 that correspond to the input image at regular intervals to determine whether or not there is motion in the input image, and generates selection information to be used by the selection unit 405 based on the determination result. The selection information generated by the selection information generation unit 409 is output to the selection unit 405.

[0077] Generally, the presence or absence of motion in an input image is determined by using multiple consecutive frames of the input image, deriving the difference between the multiple frames as test frames and past frames, and checking whether the number of test frames in which the derived difference value is less than a predetermined threshold reaches a predetermined number of times. That is, if the number of test frames in which the derived difference value is less than a predetermined threshold reaches a predetermined number of times, it is determined that there is relatively no motion. If the number of test frames in which the derived difference value is less than a predetermined threshold does not reach a predetermined number of times, it is determined that there is relatively motion. In this embodiment, the field area is determined to be an area with almost no motion, and the spectator seating area is determined to be an area with relatively motion. Therefore, an area in which the difference from past frames is less than a predetermined threshold in all test frames is not determined to be an area with relatively no motion, and it is permitted for the difference from past frames to be greater than or equal to the threshold up to a predetermined number of test frames.

[0078] The selection unit 405 selects and outputs either the first foreground information generated by the first information generation unit 403 or the second foreground information generated by the second information generation unit 404 for each region, based on the selection information generated by the selection information generation unit 409.

[0079] Figure 10 is a diagram illustrating the generation of foreground information in this embodiment. Figures 10(a) to 10(d) are images corresponding to the same captured image. Figure 10(a) is a diagram showing an example of first foreground information (foreground silhouette image) generated by the first information generation unit 403, and is the same as Figure 5(a) described in Embodiment 1. Figure 10(b) is a diagram showing an example of second foreground information (foreground silhouette image) generated by the second information generation unit 404, and is the same as Figure 5(b) described in Embodiment 1. Figure 10(c) is a diagram showing an example of selection information generated by the selection information generation unit 409. The selection information is generated as 1-bit image information of the same size as the input image, for example, with the area corresponding to the field area A100 being "1" and the area corresponding to the spectator seating area A200 being "0". The selection information generated by the selection information generation unit 409 is almost the same as the selection information shown in Figure 5(c). Figure 10(d) is a diagram showing an example of the output of the selection unit 405. Furthermore, when the processing pixel is in the "1" region of the selection information, the selection unit 405 selects and outputs the first foreground information M100 generated by the first information generation unit 403. Also, when the processing pixel is in the "0" region of the selection information, the selection unit 405 selects and outputs the second foreground information M201 generated by the second information generation unit 404. As shown in Figure 10(d), the selection unit 405 selects and outputs either the first foreground information shown in Figure 10(a) or the second foreground information shown in Figure 10(b) based on the selection information shown in Figure 10(c). Therefore, the output by the selection unit 405 in this embodiment is almost the same as the output example in Figure 5(d).

[0080] [effect] As described above, according to this embodiment, without pre-setting selection information, it is possible to select and output either first foreground information or second foreground information based on selection information generated from the input image. Therefore, it is possible to dynamically respond to cases where it is not possible to set a selection area for structures within the field of view of the imaging device, such as when a special seating area is provided on the field. As a result, by applying an appropriate foreground-background separation method to each region of the captured image, it is possible to suppress the generation of unnecessary foreground information and separate the necessary foreground information. Consequently, in a system that transmits foreground information to an image processing unit such as a server and aggregates it to generate a virtual viewpoint image, it becomes possible to generate a virtual viewpoint image that does not omit the necessary foreground while suppressing transmission costs.

[0081] [Other embodiments] Furthermore, while the above-described embodiment illustrates the imaging of a soccer match, the objects to be imaged are not necessarily limited to this. For example, this embodiment can also be applied to the imaging of other sports matches such as rugby, tennis, ice skating, and basketball, as well as live performances and concerts.

[0082] This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by a process in which one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions. [Explanation of Symbols]

[0083] 403 First Information Generation Department 404 Second information generation section

Claims

1. An acquisition means for acquiring an image captured by imaging by an imaging device, Setting means for setting a first region and a second region in the captured image, Extraction means for extracting the difference between the captured image and a pre-generated background image in the first region of the captured image, and for extracting the difference between the captured image and a captured image acquired by the acquisition means prior to the captured image in the second region of the captured image, An image processing apparatus characterized by having

2. The image processing apparatus according to claim 1, further comprising a generation means for generating a foreground image based on the difference extracted by the extraction means.

3. The extraction means generates first foreground information from the first region and second foreground information from the second region. The image processing apparatus according to claim 2, characterized in that the generation means generates the foreground image based on the first foreground information and the second foreground information.

4. The image processing apparatus according to any one of claims 1 to 3, characterized in that the first region and the second region are regions that do not overlap with each other.

5. The image processing apparatus according to any one of claims 1 to 4, wherein the setting means sets the first region and the second region based on a selected image for setting the first region and the second region.

6. The image processing apparatus according to claim 5, characterized in that the selected image is specified in advance by the user.

7. The image processing apparatus according to claim 5 or 6, characterized in that the selected image is an image of the same size as the captured image, and represents information to be selected for each region corresponding to the first region and the region corresponding to the second region using 1 bit.

8. The aforementioned first region is a region that represents the field in which the object, a person, competes. The image processing apparatus according to any one of claims 1 to 7, characterized in that the second region is a region indicating the seating area where spectators are watching the competition.

9. A program for causing a computer to function as one of the means of an image processing apparatus according to any one of claims 1 to 8.

10. The acquisition process involves acquiring an image captured by imaging using an imaging device, A setting step of setting a first region and a second region in the captured image, An extraction step which extracts the difference between the captured image and a pre-generated background image in the first region of the captured image, and extracts the difference between the captured image and a captured image acquired in the acquisition step prior to the captured image in the second region of the captured image, An image processing method characterized by having the following features.

Images