Image processing apparatus, image processing method, and program

Abstract

To generate information indicating an appropriate foreground area.SOLUTION: An image processing apparatus comprises acquisition means, setting means, first generation means, and second generation means. The acquisition means acquires a picked-up image acquired through imaging performed by an imaging apparatus. The setting means sets, for the picked-up image, a first area and a second area different from the first area. The first generation means generates information indicating a first foreground area from the first area based on the picked-up image and an image including at least a background area corresponding to the picked-up image. The second generation means generates information indicating a second foreground area different from the first foreground area from the second area based on the picked-up image and another picked-up image acquired through imaging performed by the imaging apparatus at the same position and in the same direction as and at a different timing from the imaging performed by the imaging apparatus.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating information indicating a foreground region from a captured image. [Background technology]

[0002] There are technologies for generating foreground information from an image captured by an imaging device. Patent Document 1 discloses a method for generating a background image from an input image and extracting the foreground by the difference between the input image and the background image (so-called background difference). Patent Document 2 discloses a method for extracting the foreground by the difference between images with different foreground positions (so-called frame difference). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2010-219665 A [Patent Document 2] JP 2011-139288 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in Patent Documents 1 and 2, there is a possibility that an object other than an object desired to be extracted as the foreground may be extracted as the foreground from the captured image.

[0005] The present disclosure provides a technique for generating information that appropriately indicates a foreground region. [Means for solving the problem]

[0006] An image processing device according to one aspect of the present disclosure is characterized in that it comprises an acquisition means for acquiring an image acquired by imaging with an imaging device, a setting means for setting a first region and a second region different from the first region for the image, a first generation means for generating information indicating a first foreground region from the first region based on the image and an image including at least a background region corresponding to the image, and a second generation means for generating information indicating a second foreground region different from the first foreground region from the second region based on the image and another image acquired by imaging with the imaging device at the same position and in the same direction as the imaging with the imaging device but at a different timing. Effect of the Invention

[0007] According to the present disclosure, information that appropriately indicates a foreground region can be generated. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an image processing system that generates a virtual viewpoint image. [Diagram 2] 2 is a diagram illustrating an example of the internal configuration of a camera adapter. [Diagram 3] FIG. 2 is a diagram illustrating an example of a hardware configuration of an image processing apparatus. [Figure 4] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing device. [Diagram 5] FIG. 11 is a diagram for explaining generation of foreground information. [Figure 6] 4 is a flowchart showing the flow of processing executed by the image processing device. [Figure 7] 10 is a flowchart showing a flow of a process for generating first foreground information and second foreground information. [Figure 8] FIG. 13 is a diagram illustrating a modified example of the functional configuration of the image processing device. [Figure 9] FIG. 2 is a diagram illustrating an example of a functional configuration of an image processing device. [Figure 10] FIG. 11 is a diagram for explaining generation of foreground information. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The same configurations will be described with the same reference numerals.

[0010] <Embodiment 1> In this embodiment, a form of generating foreground information (first foreground information, second foreground information) used to generate a virtual viewpoint image will be described. Specifically, in this embodiment, a form of selecting (generating) the first foreground information generated using a background difference method and the second foreground information generated using a frame difference method for each of the first and second regions previously determined in the captured image acquired by imaging with the imaging device will be described. Note that, in this embodiment, an example will be described in which the foreground information is used to generate a virtual viewpoint image, but it does not necessarily have to be used to generate a virtual viewpoint image. For this reason, it is not limited to the relative geometric installation conditions between the multiple imaging devices. Furthermore, it is not necessary to use a multiple imaging device. For example, it may be used to generate the foreground of an image captured by a single imaging device such as a monitoring imaging device installed in a premises, a remote location, or outdoors.

[0011] First, to facilitate understanding of the present embodiment, an overview of the generation of a virtual viewpoint image will be briefly described. There is a technology in which a plurality of imaging devices are installed at different positions, and images of the plurality of viewpoints obtained by the imaging are synchronously captured, and a virtual viewpoint image from an imaging device that does not actually exist is generated using the images. For example, the use of a virtual viewpoint image allows the user to view and watch highlight scenes of soccer or basketball from various angles, which gives the user a high sense of realism compared to normal images.

[0012] When generating a virtual viewpoint image, images captured synchronously by multiple imaging devices are collected in an image processing unit such as a server, and the image processing unit performs processing such as separating the foreground, which is the main subject (object), from the background and modeling it, and rendering it. Finally, the processing result is transmitted from the server to a user terminal and displayed, enabling the user to view and browse it.

[0013] When modeling (generating) the foreground, information on the foreground mask, which corresponds to the silhouette of the foreground as seen from multiple imaging devices, and information on the foreground texture (e.g., R, G, B color information for each pixel in the foreground) are required.

[0014] The process of separating the foreground from the background is called foreground background separation. Foreground background separation is a process of estimating the foreground region, and is generally performed by a background subtraction method or a frame subtraction method. The background subtraction method calculates the difference between a background image and an input image including the foreground, and determines the foreground region as a region that is a collection of pixels whose difference value is determined to be equal to or greater than a predetermined threshold value. The frame subtraction method determines the foreground region as a region that is a collection of pixels whose difference value is determined to be equal to or greater than a predetermined threshold value from the difference image of temporally consecutive images (frames).

[0015] (System Configuration) Here, an image processing system that generates a virtual viewpoint image will be described with reference to the drawings. FIG. 1 is a diagram showing a configuration example of an image processing system that generates a virtual viewpoint image. The image processing system 100 has imaging modules 110a to 110z, a database (DB) 250, a server 270, a control device 300, a switching hub 180, and an end user terminal 190. That is, the image processing system 100 has three functional domains, namely, an image collection domain, a data storage domain, and an image generation domain. The image collection domain includes the imaging modules 110a to 110z, the data storage domain includes the DB 250 and the server 270, and the image generation domain includes the control device 300 and the end user terminal 190.

[0016] The control device 300 manages the operating states and controls parameter settings through a network for each block constituting the image processing system 100. Here, the network may be GbE (Gigabit Ethernet) or 10 GbE conforming to the IEEE standard, which is Ethernet (registered trademark), or may be configured by combining an interconnect Infiniband, an industrial local area network, etc. Furthermore, the network is not limited to these, and may be other types of networks.

[0017] First, an operation of transmitting 26 sets of captured images of the imaging modules 110a to 110z from the imaging module 110z to the server 270 will be described. The imaging modules 110a to 110z each have one camera 112a to 112z. In the following, the 26 sets of the imaging modules 110a to 110z may not be distinguished from each other and may simply be referred to as "imaging module 110". Similarly, the devices in each imaging module 110 may be referred to as "camera 112" and "camera adapter 120". Although the number of imaging modules 110 is 26, this is merely an example and is not limiting.

[0018] The imaging modules 110a to 110z are connected by a daisy chain. This connection configuration has the effect of reducing the number of connection cables and saving the labor of wiring work when the volume of image data increases with the increase in resolution of captured images to 4K, 8K, etc. and the increase in frame rate. Note that the connection configuration is arbitrary, and for example, a star-type network configuration may be used in which the imaging modules 110a to 110z are each connected to a switching hub 180 and data is transmitted and received between the imaging modules 110 via the switching hub 180.

[0019] In this embodiment, each imaging module 110 is composed of a camera (imaging device) 112 and a camera adaptor 120, but is not limited thereto. For example, a microphone, a camera platform, and an external sensor may be included. In this embodiment, the camera 112 and the camera adaptor 120 are configured to be separate, but may be integrated in the same housing. The captured image obtained by the camera 112a in the imaging module 110a is subjected to image processing described later in the camera adaptor 120a, and then transmitted to the camera adaptor 120b of the imaging module 110b. Similarly, the imaging module 110b transmits the captured image obtained by the camera 112b together with the captured image obtained from the imaging module 110a to the imaging module 110c. By continuing such an operation, 26 sets of captured images are transmitted from the imaging module 110z to the switching hub 180, and then transmitted to the server 270.

[0020] In this embodiment, the description will be given assuming that the evaluation as to whether or not an image is in the foreground is performed within each camera adapter 120. However, the present invention is not limited to this configuration, and the server 270 may be configured to receive 26 sets of captured images and generate silhouette images corresponding to each captured image.

[0021] (Camera adapter configuration) Next, a detailed description will be given of the camera adapter 120. Fig. 2 is a functional block diagram showing an example of the internal configuration of the camera adapter 120. The camera adapter 120 has a network adapter 121, a transmission unit 122, an image processing device 123, and a camera control unit .

[0022] The network adapter 121 performs data communication with other camera adapters 120, the server 270, and the control device 300. In addition, the network adapter 121 also stores a time stamp of data transmitted to and received from the server 270 and performs time synchronization with the server 270, conforming to, for example, the Ordinay Clock of the IEEE1588 standard. Note that time synchronization with a time server may be realized by other EtherAVB standards or a unique protocol. In this embodiment, a NIC (Network Interface Card) is used as the network adapter 121, but is not limited to this.

[0023] The transmission unit 122 controls data transmission to the switching hub 180 and the like via the network adapter 121. The transmission unit 122 has a function of compressing data to be transmitted and received by applying a predetermined compression method, compression ratio, and frame rate, and a function of decompressing the compressed data. The transmission unit 122 also has a function of determining a routing destination for the received data and the data processed by the image processing device 123, and a function of transmitting the data to the determined routing destination. The transmission unit 122 also has a function of creating a message for transferring image data to another camera adapter 120 or the server 270. The message includes meta information of the image data. This meta information includes a time code or sequence number at the time of sampling of the image capture, a data type, and an identifier of the camera 112. Note that the image data to be transmitted may be compressed. The transmission unit 122 also receives a message from another camera adapter 120, and restores the data information fragmented into a packet size specified by the transmission protocol to image data according to the data type included in the message.

[0024] The image processing device 123 performs a process of separating the foreground from the background based on the image data captured by the camera 112 under the control of the camera control unit 124 and the initialization information, and generates foreground information including an image of an object, which is the foreground. In addition, the image processing device 123 also performs processes such as dynamic calibration. By each of the multiple camera adapters 120 generating the foreground, the load in the image processing system 100 can be distributed. The dynamic calibration is a calibration performed during imaging, and includes a color correction process for suppressing color variations for each camera, and a blur correction process (electronic image stabilization process) for stabilizing the position of the image against blur caused by camera vibration. The foreground information generated by the image processing device 123 includes a foreground silhouette image showing the foreground area and an image obtained by cutting out the foreground area from the input image, and is transmitted to an image processing unit such as a server to generate a virtual viewpoint image. The foreground information may include either a foreground silhouette image showing the foreground area or an image obtained by cutting out the foreground area from the input image (foreground texture image).

[0025] The camera control unit 124 is connected to the camera 112, and controls the camera 112, acquires captured images, provides synchronization signals, sets time, etc. The control of the camera 112 includes, for example, setting and referring to imaging parameters (such as the number of pixels, color depth, frame rate, and white balance settings), acquiring status information of the camera 112 (imaging, stopped, synchronized, error, etc.), starting and stopping imaging, and adjusting focus.

[0026] (Hardware configuration of image processing device) Next, an example of the hardware configuration of an image processing device will be described with reference to the drawings. Fig. 1 is a block diagram showing a schematic hardware configuration of an image processing device according to this embodiment. The image processing device 123 has a CPU 311, a ROM 312, a RAM 313, an auxiliary storage device 314, a display unit 315, an operation unit 316, and a communication interface (I / F) 317. Each component is connected to each other via a bus 318 so as to be able to communicate with each other.

[0027] A CPU (Central Processing Unit) 311 executes various programs stored in a ROM 312, an auxiliary storage device 314, etc., and comprehensively controls the image processing device 123, thereby realizing each function of the image processing device 123 shown in Fig. 4. Note that the image processing device 123 may have one or more dedicated hardware devices different from the CPU 311, and at least a part of the processing by the CPU 311 may be executed by the dedicated hardware devices. Examples of the dedicated hardware devices include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor).

[0028] A ROM (Read Only Memory) 312 stores programs and the like that do not require modification. A RAM (Random Access Memory) 313 temporarily stores programs and data supplied from the ROM 312 and auxiliary storage device 314, as well as data and the like supplied from the outside via a communication I / F 317. The auxiliary storage device 314 has, for example, a hard disk drive, and stores various data such as image data and audio data.

[0029] The display unit 315 has, for example, a liquid crystal display or an LED, and displays a GUI (Graphical User Interface) or the like for the user to operate the image processing device 123. The operation unit 316 has, for example, a keyboard, a mouse, a joystick, a touch panel, and the like, and receives operations by the user to input various instructions to the CPU 311. The CPU 311 operates as a display control unit that controls the display unit 315 and an operation control unit that controls the operation unit 316.

[0030] The communication I / F 317 is used for communication with an external device of the image processing device 123. For example, when the image processing device 123 is connected to an external device by wire, a communication cable is connected to the communication I / F 317. When the image processing device 123 has a function of wirelessly communicating with an external device, the communication I / F 317 is equipped with an antenna. The bus 318 connects each unit of the image processing device 123 to transmit information.

[0031] In this embodiment, the display unit 315 and the operation unit 316 are assumed to exist inside the image processing device 123, but at least one of the display unit 315 and the operation unit 316 may exist outside the image processing device 123 as a separate device.

[0032] [Functional configuration of image processing device] Next, the functional configuration of the image processing device 123 will be described with reference to the drawing. Fig. 4 is a functional block diagram of the image processing device 123. Note that the functional configuration shown in Fig. 4 is realized by supplying a program for realizing the function to the image processing device 123 shown in Fig. 3 and further executing the program by the image processing device 123.

[0033] The image processing device 123 of this embodiment has a storage unit 401, a background generation unit 402, a first information generation unit 403, a second information generation unit 404, and a selection unit 405. Note that image data of a captured image (processed image) captured by the camera (imaging device) 112 is input to the storage unit 401, the background generation unit 402, the first information generation unit 403, and the second information generation unit 404 of the image processing device. The image data of the captured image input to the image processing device 123 is also referred to as a current input image (processed image). The captured image is an image such as a still image or a frame of a moving image captured continuously in time.

[0034] The storage unit 401 stores a predetermined number of frames of captured images (input images). The predetermined number of frames may be, for example, four frames from four frames before the current frame to one frame before the current frame, as long as it is a number required when the second information generating unit 403, which will be described in detail later, generates second foreground information.

[0035] The background generating unit 402 generates a background image from an input image. For example, the background generating unit 402 compares the current input image with past input images (past images) stored in the storage unit 401 at regular intervals to determine whether there is motion in the input image, and generates an area determined to have relatively no motion as a background image. The past input image may be an input image of a frame one or more frames before the input image of the current frame. A known technique is used to determine whether there is motion. For example, the presence or absence of motion is determined by deriving a difference between the input image of the current frame and input images of multiple past frames, and by determining whether the number of frames in which the derived difference is less than a predetermined threshold reaches a predetermined number. That is, if the number of frames in which the derived difference value is less than a predetermined threshold reaches a predetermined number, it is determined that there is relatively no motion. If the number of frames in which the derived difference value is less than a predetermined threshold does not reach a predetermined number, it is determined that there is relatively motion. Note that if no past input images are stored in the storage unit 401, the background generating unit 402 generates the input image as a background image.

[0036] The first information generating unit 403 generates first foreground information (first information) for generating a first foreground included in the captured image (input image) by using a background difference method. The first information can be said to include a first foreground region. Specifically, the first information generating unit 403 binarizes the difference (background difference) between the background image generated by the background generating unit 402 and the input image with a predetermined threshold value, thereby treating a relatively moving region as a foreground region and generating a foreground silhouette image indicating the foreground region. The first information generating unit 403 can be said to generate first foreground information indicating the first foreground region from the difference between the images. The background image generated by the background generating unit 402 can be said to be an image including at least a background region corresponding to the input image. The image including at least a background region corresponding to the input image can be said to be another captured image obtained by the imaging device at the same position and in the same direction as the imaging device at a different timing. Here, the generation of the first foreground information by the first information generating unit 403 will be described with reference to the drawings.

[0037] FIG. 5 is a diagram for explaining the generation of foreground information. In FIG. 5, it is assumed that foreground information is generated based on an input image of a scene of a soccer game, including three players running on the field and playing a game, and spectators in the spectator seats beside the field watching the game, with a ball kicked up high against the background of the spectator seats. The three players and the ball are foreground objects. The field area is the first area, and objects in the field area are the first foreground. The spectator seat area is the second area, and objects in the spectator seat area are the second foreground. In FIG. 5(a) to (d), images correspond to the same captured image. In FIG. 5(a) to (d), the area A100 to the left of the dotted line is the area corresponding to the soccer field (field area), and the area A200 to the right of the dotted line is the area corresponding to the spectator seats arranged around the field (spectator seat area). FIG. 5(a) is a diagram showing an example of first foreground information (foreground silhouette image) generated by the first information generating unit 403. In the field area A100, areas corresponding to three players running on the field are separated as the foreground M100. That is, it can be said that the first information generating unit 403 generates a foreground silhouette image (foreground mask) corresponding to an object (first foreground) included in the field area A100. In the spectator seat area A200, an area corresponding to spectators making relatively small movements in the spectator seat is separated as the foreground M101. Also, in the spectator seat area A200, a ball kicked up high by a player is photographed with the spectators in the background, and the foreground M101 separated from the spectator seat area A200 also includes an area corresponding to the ball. That is, the foreground M101 includes an area corresponding to spectators making relatively small movements and an area corresponding to the ball. The area corresponding to spectators making relatively small movements is an area that has no meaning in the foreground M101 (an area that is actually unnecessary). The example given here is that while background subtraction can achieve the desired separation of objects on a background that is generally stationary, such as a field, it can also separate unnecessary areas into the foreground when the background is not generally stationary, such as an auditorium.In this embodiment, the background image successively updated by the background generating unit 402 is used, but for example, an image captured in advance when the players are not on the field may be used as the background image. In this case, however, all spectators in the spectator seating area will be separated as the foreground. The first information generating unit 403 generates a foreground silhouette image showing the foreground area by separating relatively moving objects as the foreground from the input image, for example, using a background subtraction method.

[0038] The second information generating unit 404 uses a frame difference method to generate second foreground information (second information) for generating a second foreground included in the captured image (input image) by using a captured image and a past input image read from the storage unit 401. It can also be said that the second information generating unit 404 generates second foreground information indicating a second foreground area from the difference between images. The past input image can also be said to be another captured image acquired by capturing an image of the imaging device at a different timing from the image capturing of the imaging device at the same position and in the same direction as the image capturing of the imaging device. It can also be said that the second information includes a second foreground area. Specifically, the second information generating unit 404 generates a difference (first frame difference) image between the current input image and a past input image, for example, two frames before the current input image. In addition, the second information generating unit 404 generates a difference (second frame difference) image between the past input image of two frames before and the past input image of two frames before the past input image of two frames before. Next, the second information generating unit 404 separates only the moving area in the current input image as a frame difference image by subtracting the second frame difference image from the first frame difference image. Then, the second information generating unit 404 binarizes the frame difference image with a predetermined threshold value, sets the relatively moving area as the foreground area, and generates a foreground silhouette image indicating the foreground area. That is, it can be said that the second information generating unit 404 generates a foreground silhouette image (foreground mask) corresponding to the object (second foreground) included in the spectator seat area A200. Note that the past input images used to generate the second foreground information are not limited to the input images two frames before and four frames before the current input image, and input images n frames before and (n+2) frames before the current input image may be used. Here, n is a natural number.

[0039] FIG. 5(b) is a diagram showing an example of the second foreground information (foreground silhouette image) generated by the second information generating unit 404. As shown in FIG. 5(b), in the field area A100, three players are separated as the foreground M200. However, the players only move slightly in the time difference between two consecutive frames, and the players do not move completely to a position where they do not overlap with the positions of the players in the previous frame, so the outlines of the players are separated as the foreground M200. In other words, the area where the outlines of the players are separated as the foreground M200 can be said to be an area in which the foreground cannot be detected in the difference between two consecutive frames in the processed image. On the other hand, in the spectator seat area A200, the spectators are almost still in the time difference between two frames and are not separated as the foreground, but the ball kicked up high moves quickly and is separated as the foreground M201. This is exemplified in that, in the field difference, objects that move relatively slowly are separated as outlines, but objects that move relatively quickly can be separated as desired. In other words, the area containing the ball separated as the foreground M201 can be said to be an area in which the foreground can be detected based on the difference between two consecutive frames in the processed image.

[0040] The above-mentioned first information generating unit 403 outputs foreground information (first foreground information) including a foreground silhouette image (corresponding to FIG. 5(a)) indicating the area separated as the foreground, and a foreground image (not shown) obtained by cutting out the foreground area from the input image. The above-mentioned second information generating unit 404, like the first information generating unit 403, outputs foreground information (second foreground information) including a foreground silhouette image (corresponding to FIG. 5(b)) indicating the area separated as the foreground, and a foreground image (not shown) obtained by cutting out the foreground area from the input image.

[0041] The selection unit 405 selects and outputs either the first foreground information generated by the first information generation unit 403 or the second foreground information generated by the second information generation unit 404 based on the selection information. The selection information is information designated in advance by the user and set in advance for each area in the captured image (processed image), and is information that is held (set) in the selection unit 405. FIG. 5(c) is a diagram showing an example of the selection information set in the selection unit 405. As shown in FIG. 5(c), the selection information is, for example, 1-bit image information of the same size as the input image, in which an area corresponding to the field area A100 is set to "1" and an area corresponding to the spectator seat area A200 is set to "0". In addition, once the imaging angle of view of the imaging device is determined, the selection information can be set for the boundary between the field area and the spectator seat area using, for example, a structure such as a stand within the angle of view as a clue.

[0042] FIG. 5(d) is a diagram showing an example of output from the selection unit 405. When the pixel to be processed corresponds to the field area A100 with the selection information set to "1", the selection unit 405 selects and outputs the foreground information M100 generated by the first information generation unit 403. When the pixel to be processed corresponds to the spectator area with the selection information set to "0", the selection unit 405 selects and outputs the foreground information M201 generated by the second information generation unit 404. As shown in FIG. 5(d), the selection unit 405 selects and outputs the first foreground information shown in FIG. 5(a) or the second foreground information shown in FIG. 5(b) based on the selection information shown in FIG. 5(c).

[0043] [Processing flow executed by image processing device] The processing executed by the image processing device described above will be described with reference to the drawings. FIG. 6 is a flowchart showing the flow of processing executed by the image processing device in this embodiment. The processing shown in this flowchart is realized by the CPU 311 of the image processing device 123 reading out a control program stored in the ROM 312 or the auxiliary storage device 314, expanding it in the RAM 313, and executing it. The processing in this embodiment will be described as raster scanning processing in synchronization with the frame in which the input image is input, but is not limited to this. Note that the symbol "S" in the description of the flowchart represents a step (process). This also applies to the following description of the flowchart.

[0044] First, the image processing apparatus executes the process of S100 and the process of S200.

[0045] In S100, the first information generating unit 403 binarizes the difference (background difference) between the current input image and the background image corresponding to the current input image and generated by the background generating unit 402 using a predetermined threshold, separates a relatively moving area as a foreground area, and generates first foreground information. Then, after generation, the first information generating unit 403 shifts the process to S300.

[0046] In S200, the second information generating unit 404 binarizes the difference between the current input image and the past input image read from the storage unit 401 to generate a first frame difference image, and binarizes the difference between the past input images to generate a second frame difference image. The second information generating unit 404 then subtracts the second frame difference image from the first frame difference image to separate only the area in the current input image with relative movement as the foreground area, and generates second foreground information. After generation, the second information generating unit 404 then transitions to processing at S300. Note that S100 and S200 may be processed in parallel.

[0047] In S300, the selection unit 405 of the image processing device determines, based on selection information preset for each region, whether or not the pixel currently being processed in the input image (pixel of interest) is in a first region that selects first foreground information generated by the first information generation unit 403. If the determination result indicates that the pixel of interest is in the first region (YES in S300), the selection unit 405 transitions the process to S400. If the determination result indicates that the pixel of interest is in a second region that selects second foreground information generated by the second information generation unit 404 but is not in the first region (NO in S300), the selection unit 405 transitions the process to S500.

[0048] In S400, the selection unit 405 selects, as output, the first foreground information generated by the first information generation unit 403. After the selection, the selection unit 405 transitions to the process of S600.

[0049] In S500, the selection unit 405 selects, as output, the second foreground information generated by the second information generation unit 404. After the selection, the selection unit 405 transitions to the process of S600.

[0050] In S600, the image processing device determines whether or not processing has been completed for all pixels in the input image. If the image processing device obtains a determination result that processing has not been completed for all pixels in the input image (NO in S600), the image processing device returns the processing to S300. Then, the image processing device executes the processing from S100 to S500 for all unprocessed pixels in the input image. On the other hand, if the image processing device obtains a determination result that processing has been completed for all pixels in the input image (YES in S600), the image processing device ends the flow shown in FIG. 6.

[0051] Fig. 7(a) is a flowchart showing a detailed flow of the generation process of the first foreground information in S100 in Fig. 6. The process shown in this flowchart is realized by the CPU 311 of the image processing device 123 reading out a control program stored in the ROM 312 or the auxiliary storage device 314, expanding the program in the RAM 313, and executing the program.

[0052] In S1001, the background generation unit 402, for example, at regular intervals, compares the input image with a past input image corresponding to the input image and stored in the storage unit 401 to determine whether there is any motion in the input image, and generates an area with relatively no motion as a background image. For example, if the change in pixel value of the same pixel is less than a threshold, it is determined that there is relatively no motion. Then, the background generation unit 402 generates an area having pixels determined to have relatively no motion as a background image. Note that if the change in pixel value of the same pixel is equal to or greater than a threshold, it is determined that there is relatively motion, and the background generation unit 402 does not generate an area having pixels determined to have relatively motion as a background image.

[0053] In S1002, the first information generating unit 403 generates a difference (background difference) image between the input image and the background image corresponding to the input image and generated by the background generating unit 402 in S1001. The background difference image can be generated using, for example, difference values ​​of R, G, and B gradation values ​​at the processing pixel.

[0054] In S1003, the first information generating unit 403 binarizes the background difference image generated in S1002 with a predetermined threshold value, and separates a relatively moving area as a foreground area. In other words, it can be said that the first information generating unit 403 generates a foreground silhouette image indicating an area separated as a foreground area in an area where a stable background image can be generated from an image obtained by binarizing the background difference image between the input image and the background image with a predetermined threshold value.

[0055] In S1004, first information generating unit 403 generates first foreground information including the foreground silhouette image generated in S1003, the corresponding input image, and a foreground image obtained by cutting out a foreground region from the foreground silhouette image.

[0056] Fig. 7(b) is a flowchart showing a detailed flow of the generation process of the second foreground information in S200 in Fig. 6. The process shown in this flowchart is realized by the CPU 311 of the image processing device 123 reading out a control program stored in the ROM 312 or the auxiliary storage device 314, expanding the program in the RAM 313, and executing the program.

[0057] In S2001, the second information generating unit 404 generates a difference (first frame difference) image between the current input image and, for example, an input image (past image) two frames before the current input image that is read from the storage unit 401. The second information generating unit 404 further generates a difference (second frame difference) image between the above-mentioned input image (past image) two frames before and an input image (past image) four frames before that corresponds to the input image and is read from the storage unit 401. Then, the second information generating unit 404 separates (generates) only an area with relative movement in the current input image as a foreground area (frame difference image) by subtracting the second frame difference image from the first frame difference image. To generate the frame difference image, for example, a difference value of the R, G, and B gradation values ​​at the processing pixel can be used.

[0058] In S2002, the second information generating unit 404 binarizes the frame difference image generated in S2001 using a predetermined threshold value, and separates a region with relative movement as a foreground region. In other words, the second information generating unit 404 can be said to generate a foreground silhouette image indicating the region separated as the foreground region in the region where a stable background image cannot be generated, from an image obtained by binarizing the frame difference image between the first frame image and the second frame image using a predetermined threshold value.

[0059] In S2003, second information generating unit 404 generates second foreground information including the foreground silhouette image generated in S2002, the corresponding input image, and a foreground image obtained by cutting out a foreground region from the foreground silhouette image.

[0060] [effect] As described above, according to the present embodiment, by applying a foreground / background separation method suitable for each region of a captured image, it is possible to suppress the generation of unnecessary foreground information and separate (generate) necessary foreground information. In other words, it is possible to generate appropriate foreground information.

[0061] As a result, in a system that transmits foreground information to an image processing unit such as a server and aggregates it to generate a virtual viewpoint image, it is possible to generate a virtual viewpoint image that does not miss any necessary foreground while reducing transmission costs.

[0062] [Variation 1] In the above-described first embodiment, an example has been shown in which the selection unit 405 selects and outputs the first foreground information or the second foreground information generated by processing all pixels of the input image based on the selection information.

[0063] However, the first foreground information and the second foreground information are not limited to information generated by processing all pixels of the input image. Therefore, for example, the first foreground information generated for the first region of the input image and the second foreground information generated for the second region of the input image may be synthesized and output. Here, the functional configuration of the image processing device which is the first modification of the first embodiment described above will be described with reference to the drawings. FIG. 8 is a functional block diagram of the first modification of the image processing device. The functional configuration shown in FIG. 8 is realized by supplying a program for realizing the function to the image processing device 123 shown in FIG. 3 and further executing the program by the image processing device 123.

[0064] The image processing device has a storage unit 401, a background generation unit 402, a first information generation unit 403, a second information generation unit 404, a first area selection unit 406, a second area selection unit 407, and a synthesis unit 408. Note that in this modification, the components other than the first area selection unit 406, the second area selection unit 407, and the synthesis unit 408 shown in Fig. 8 are the same as those in the first embodiment, and therefore descriptions thereof will be omitted here.

[0065] When the processing pixel is in the first region, the first region selection unit 406 outputs the input image and the background image to the first information generation unit 403 based on the first selection information. The first selection information is information indicating a region corresponding to a first foreground that is set in advance, and is information held in the first region selection unit 406. In the selection information shown in FIG. 5(c), the first information is information corresponding to the first region A100. Note that the first information generation unit 403 generates first foreground information including a foreground silhouette image and a foreground image from the input image and background image corresponding to the first selection information input from the first region selection unit 406.

[0066] When the processing pixel is in the second region, the second region selection unit 407 outputs the current input image and the past input image to the second information generation unit 404 based on the second selection information. The second selection information is information indicating a region corresponding to the second foreground, which is set in advance, and is information held in the second region selection unit 407. In the selection information shown in FIG. 5(c), the second information is information corresponding to the second region A200. Note that the second information generation unit 404 generates second foreground information including a foreground silhouette image and a foreground image from the current input image and the past input image corresponding to the second selection information input from the second region selection unit 407.

[0067] A synthesis unit 408 synthesizes the first foreground information generated by the first information generation unit 403 and the second foreground information generated by the second information generation unit 404, and outputs information corresponding to the processed pixel as foreground information.

[0068] As described above, according to this modification, even if the first selection information corresponding to the first region and the second selection information corresponding to the second region are set separately, it is possible to suppress the generation of unnecessary foreground information and generate necessary foreground information, as in the first embodiment. In other words, it is possible to generate appropriate foreground information.

[0069] [Variation 2] In the above-described first embodiment, an example has been shown in which the first information generating unit 403 generates first foreground information using background difference, and the second information generating unit 404 generates second foreground information using frame difference.

[0070] However, the method of generating the first foreground information by the first information generating unit 403 is not limited to the background difference. Therefore, for example, in either the first information generating unit 403 or the second information generating unit 404, the foreground information may be generated by the frame difference. For example, the first information generating unit 403 generates the first foreground information from the difference between the current frame and a previous frame that is relatively long in time from the imaging time of the current frame. The first foreground information is information corresponding to an area including a foreground with relatively slow movement. The second information generating unit 404 generates the second foreground information from the difference between the current frame and a previous frame that is relatively short in time from the imaging time of the current frame. The second foreground information is information corresponding to an area including a foreground with relatively fast movement. Therefore, it is possible to simultaneously generate the first foreground information corresponding to an area including a foreground with relatively slow movement and the foreground information corresponding to an area including a foreground with relatively fast movement. The relatively fast-moving area and the relatively slow-moving area can be changed depending on the processing target. For example, when soccer is the processing target, the following setting may be performed. The players may be set to correspond to areas with relatively slow movement, and the running players may be set to be separable from the processed image as the foreground by frame difference. The soccer ball may be set to correspond to areas with relatively fast movement, and the soccer ball at the time of passing, dribbling, or shooting may be set to be separable from the processed image as the foreground by frame difference. The past frame having a relatively long time interval from the imaging time of the current frame may be, for example, the past frame three or four frames before the current frame when 60 fps (frames per second) is used. The past frame having a relatively short time interval from the imaging time of the current frame may be, for example, the past frame two frames before the current frame when 60 fps is used.

[0071] As described above, according to this modification, it is possible to generate first foreground information for generating a first foreground corresponding to an object that moves relatively slowly, and second foreground information for generating a second foreground corresponding to an object that moves relatively quickly. Therefore, similar to the first embodiment, it is possible to suppress the generation of unnecessary foreground information and separate (generate) necessary foreground information. In other words, it is possible to generate appropriate foreground information.

[0072] <Embodiment 2> In the above-described embodiment 1, an example has been shown in which the selection unit 405 selects and outputs the first foreground information generated by the first information generation unit 403 or the second foreground information generated by the second foreground information generation unit 404 based on preset selection information.

[0073] However, the selection information used by the selection unit 405 to select the first region or the second region is not limited to preset information. Therefore, for example, the selection unit 405 may select and output the first foreground information or the second foreground information based on the selection information generated from the difference between an input image (frame) which is a currently captured image and an input image (frame) which is a previously captured image. In this embodiment, an image processing device that generates selection information for selecting, for each region, either the first foreground information generated using a background difference method or the second foreground information generated using a frame difference method based on an analysis result of an input image will be described. In this embodiment, the same components as those in the above-mentioned embodiment 1 are denoted by the same reference numerals, and the description thereof will be omitted.

[0074] [Functional configuration of image processing device] The functional configuration of the image processing device of this embodiment will be described with reference to the drawing. Fig. 9 is a functional block diagram of the image processing device 123. The functional configuration shown in Fig. 9 is realized by supplying a program for realizing the function to the image processing device 123 shown in Fig. 3 and further executing it by the image processing device 123.

[0075] The image processing device 123 of this embodiment includes a storage unit 401 , a background generation unit 402 , a first information generation unit 403 , a second information generation unit 404 , a selection unit 405 , and a selection information generation unit 409 .

[0076] The selection information generating unit 409 compares the input image with a previous input image (past image) corresponding to the input image and stored in the storage unit 401 at regular intervals, for example, to determine the presence or absence of motion in the input image, and generates selection information to be used by the selecting unit 405 based on the determination result. The selection information generated by the selection information generating unit 409 is output to the selecting unit 405.

[0077] In general, the presence or absence of motion in an input image is determined by using a plurality of consecutive frames of the input image, deriving the difference from a previous frame using the plurality of frames as inspection frames, and determining whether the number of inspection frames in which the derived difference value is less than a predetermined threshold reaches a predetermined number. That is, it is determined that there is relatively no motion when the number of inspection frames in which the derived difference value is less than a predetermined threshold reaches a predetermined number. It is determined that there is relatively motion when the number of inspection frames in which the derived difference value is less than a predetermined threshold does not reach a predetermined number. In this embodiment, the field area is determined to be an area with relatively almost no motion, and the spectator seat area is determined to be an area with relatively motion. Therefore, an area in which the difference from a previous frame is less than a predetermined threshold in all inspection frames is not determined to be an area with relatively no motion, and the difference from a previous frame is allowed to be equal to or greater than the threshold up to a predetermined number of inspection frames.

[0078] Based on the selection information generated by the selection information generation unit 409, the selection unit 405 selects and outputs, for each region, either the first foreground information generated by the first information generation unit 403 or the second foreground information generated by the second information generation unit 404.

[0079] FIG. 10 is a diagram for explaining the generation of foreground information in this embodiment. Note that FIGS. 10(a) to 10(d) are images corresponding to the same captured image. FIG. 10(a) is a diagram showing an example of first foreground information (foreground silhouette image) generated by the first information generating unit 403, and is the same as FIG. 5(a) described in the first embodiment. FIG. 10(b) is a diagram showing an example of second foreground information (foreground silhouette image) generated by the second information generating unit 404, and is the same as FIG. 5(b) described in the first embodiment. FIG. 10(c) is a diagram showing an example of selection information generated by the selection information generating unit 409. The selection information is generated as 1-bit image information of the same size as the input image, in which the area corresponding to the field area A100 is "1" and the area corresponding to the spectator seat area A200 is "0". The selection information generated by the selection information generating unit 409 is almost the same as the selection information shown in FIG. 5(c). FIG. 10(d) is a diagram showing an example of the output of the selection unit 405. When the pixel to be processed is in the area of ​​"1" in the selection information, the selection unit 405 selects and outputs the first foreground information M100 generated by the first information generation unit 403. When the pixel to be processed is in the area of ​​"0" in the selection information, the selection unit 405 selects and outputs the second foreground information M201 generated by the second information generation unit 404. As shown in FIG. 10(d), the selection unit 405 selects and outputs the first foreground information shown in FIG. 10(a) or the second foreground information shown in FIG. 10(b) based on the selection information shown in FIG. 10(c). Therefore, the output by the selection unit 405 in this embodiment is almost the same as the output example in FIG. 5(d).

[0080] [effect] As described above, according to this embodiment, the first foreground information or the second foreground information can be selected and output based on the selection information generated from the input image without setting the selection information in advance. Therefore, it is possible to dynamically respond to a case where a structure selection area within the field of view of the imaging device cannot be set, such as when a special spectator seat area is provided on the field. This makes it possible to suppress the generation of unnecessary foreground information and separate necessary foreground information by applying an appropriate foreground background separation method for each area of ​​the captured image. As a result, in a system that transmits and aggregates foreground information to an image processing unit such as a server to generate a virtual viewpoint video, it is possible to generate a virtual viewpoint video that does not lack the necessary foreground while suppressing transmission costs.

[0081] [Other embodiments] In the above embodiment, the case of capturing images of a soccer match has been exemplified, but the subject of the image capturing is not necessarily limited to this. For example, the present embodiment can also be applied to capturing images of other sports matches such as rugby, tennis, ice skating, and basketball, as well as live performances and concerts.

[0082] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions. [Explanation of symbols]

[0083] 403 First Information Generation Department 404 Second information generation section

Claims

1. An acquisition means for acquiring a captured image acquired by imaging with an imaging device; A setting unit that sets a first area and a second area different from the first area in the captured image; a first generating means for generating information indicating a first foreground region from the first region based on the captured image and an image including at least a background region corresponding to the captured image; a second generation means for generating, from the second region, information indicating a second foreground region different from the first foreground region, based on the captured image and another captured image acquired by capturing an image by the imaging device at a different timing from the image captured by the imaging device at the same position and in the same direction as the image captured by the imaging device; 13. An image processing device comprising:

2. The setting means sets the first region to a region in which a foreground cannot be detected based on a difference between two consecutive frames in the captured image, and sets the second region to a region in which a foreground can be detected based on a difference between two consecutive frames in the captured image.

2. The image processing device according to claim 1,

3. a third generation means for generating selection information for setting the first region in a region determined to have relatively no motion and setting the second region in a region determined to have relatively motion by using a plurality of consecutive frames of the captured image and a frame difference; The setting means selects the first area or the second area based on the selection information generated by the third generation means.

3. The image processing device according to claim 1, wherein the first and second inputs are input to the image processing device.

4. The setting means sets the first area and the second area based on selection information designated in advance by a user.

2. The image processing device according to claim 1,

5. The selection information is generated as an image having the same size as the captured image, the image representing information to be selected for each region by one bit.

5. The image processing device according to claim 3, wherein the first and second inputs are input to the image processing apparatus.

6. the information indicating the first foreground region includes a first foreground silhouette image corresponding to the foreground of the first region, and an image of the first region cut out from the corresponding captured image and the first foreground silhouette image; The information indicating the second foreground region includes a second foreground silhouette image corresponding to the foreground of the second region, and an image of the second region cut out from the corresponding captured image and the second foreground silhouette image.

6. The image processing device according to claim 1, wherein the image processing device further comprises:

7. the information indicating the first foreground region is information indicating a region that is included in the captured image and corresponds to an object that moves relatively slowly, The information indicating the second foreground region is information indicating a region included in the captured image and corresponding to an object that moves relatively quickly.

7. The image processing device according to claim 1, wherein the image processing device further comprises: a first input unit;

8. 8. The image processing device according to claim 1, further comprising a synthesis unit that synthesizes information indicating the first foreground area and information indicating the second foreground area to generate foreground information corresponding to an object included in the captured image.

9. the image including at least a background region corresponding to the captured image is another captured image acquired by capturing an image of the imaging device at a different timing from the image of the imaging device in the same position and direction as the image of the imaging device, the other captured image used by the first generating means to generate information indicating the first foreground region is a past captured image that is captured a relatively long time interval from the capture time of the captured image, the other captured image used by the second generation means to generate information indicating the second foreground region is a previous captured image captured a relatively short time interval from a capture time of the captured image, The first generation means and the second generation means generate information indicating the first foreground region and information indicating the second foreground region, respectively, from a difference between images.

2. The image processing device according to claim 1,

10. the first area is an area showing a field on which a foreground object will play; 10. The image processing device according to claim 1, wherein the second area is an area showing a stand where spectators who watch the event are located.

11. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 10.

12. an acquisition step of acquiring an image acquired by imaging with an imaging device; a setting step of setting a first area and a second area different from the first area in the captured image; a first generation step of generating information indicating a first foreground region from the first region based on the captured image and an image including at least a background region corresponding to the captured image; a second generation step of generating, from the second region, information indicating a second foreground region different from the first foreground region, based on the captured image and another captured image acquired by capturing an image by the imaging device at a different timing from the captured image in the same position and direction as the image captured by the imaging device; 13. An image processing method comprising:

Images