Control device, image processing system, control method of image processing system, and program

Abstract

To solve the problem that a background image cannot be updated appropriately.SOLUTION: An image processing device includes: acquisition means for acquiring a plurality of captured images obtained by capturing of a plurality of imaging means; extraction means for extracting foreground regions from the plurality of captured images by a background difference method; and generation means for generating background images used in the background difference method while updating the background images on the basis of the plurality of captured images. The generation means updates the background images according to an update method of the set background images for each of groups to which the plurality of imaging means are classified on the basis of targets of interest of the respective plurality of imaging means in imaging using the captured images obtained by one or more imaging means included in the group.SELECTED DRAWING: Figure 8

Description

[Technical field]

[0001] The present disclosure relates to techniques for extracting foreground regions from images. [Background technology]

[0002] Methods for extracting an image region (hereinafter referred to as a "foreground region") corresponding to a foreground object (hereinafter referred to as a "foreground object") from an image captured by an imaging device (hereinafter referred to as a "captured image") are used for various purposes and there are various methods for extracting such an image region. For example, there is a so-called background difference method, which compares a captured image with an image of a background (hereinafter referred to as a "background image") obtained by some method, and extracts pixels whose pixel values ​​of corresponding pixels are equal to or greater than a predetermined threshold value as the foreground region. In the extraction of the foreground region by the background difference method, if the background changes dynamically or there is a change in environmental light such as sunlight or lighting, and the difference between the captured image and the background image becomes large, it becomes impossible to properly extract the foreground region.

[0003] In this regard, Patent Document 1 discloses a technique for updating to a background image generated at a set timing. By using the technique of Patent Document 1, even if the background changes dynamically or there is a change in the environmental light such as sunlight or lighting, the background is written as a background image and is not extracted as a foreground region. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2021-163303 A Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in the above Patent Document 1 had a problem that it was difficult to update the background image when a person was present for a long time in a part of the entire imaging area of ​​the multiple imaging devices. For example, in the case of baseball, the image is likely to change during a game as the white lines drawn on the field near the home base disappear, but the catcher and umpire are present near the home base for a long time, making it difficult to update the background image. [Means for solving the problem]

[0006] An image processing device according to one aspect of the present disclosure includes an acquisition means for acquiring a plurality of captured images obtained by imaging using a plurality of imaging means, an extraction means for extracting a foreground region from the plurality of captured images by a background difference method, and a generation means for generating a background image to be used in the background difference method while updating it based on the plurality of captured images, wherein the generation means updates the background image using captured images obtained by one or more imaging means included in each group in accordance with a background image update method set for each group into which the plurality of imaging means are classified based on an object of interest in imaging by each of the plurality of imaging means. Effect of the Invention

[0007] According to the present disclosure, the background image can be updated. [Brief description of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of a system configuration. [Diagram 2] FIG. 2 illustrates an example of a hardware configuration of the apparatus. [Diagram 3] FIG. 2 is a diagram illustrating an example of a software configuration of the image processing apparatus. [Figure 4] FIG. 13 is a diagram for explaining a background partial update function. [Diagram 5] FIG. 2 is a diagram illustrating an example of a software configuration of a control device. [Figure 6] FIG. 11 is a diagram showing an example of camera classification information. [Figure 7]FIG. 1 is a diagram for explaining camera classifications. [Figure 8] 10A is a flowchart showing the flow of setting control processing for background generation, and FIG. 10B is a flowchart showing the flow of setting processing for background generation. [Figure 9] FIG. 11 is a diagram illustrating an example of setting parameters for each background update mode. [Figure 10] FIG. 13 is a diagram illustrating an example of a UI for background update settings. [Figure 11] FIG. 13 is a diagram illustrating an example of a UI for background update settings. [Figure 12] 13 is a flowchart showing the flow of a background update process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, the embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments are not limited to the invention according to the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Note that in each drawing, the same elements are given the same reference numbers, and duplicated descriptions are omitted or simplified. Note that each process (step) in the flow chart is represented with an "S" at the beginning.

[0010] <Embodiment 1> A virtual viewpoint image generating system according to the present embodiment will be described. In the present embodiment, a plurality of imaging devices are grouped according to imaging targets, and the timing for updating the background is set for each grouped imaging device, and the background is updated at a moment when there is no person as a foreground object in the imaging targets.

[0011] (System Configuration) 1 is a diagram showing an example of the configuration of a virtual viewpoint image generation system according to this embodiment. The virtual viewpoint image generation system includes a plurality of cameras (imaging devices) 11-19, image processing devices 21-29 that process images captured by the cameras, a server 30, and a control device 40 that controls the cameras 11-19 and the image processing devices 21-29. The image processing devices 21-29 are connected to the server 30 and the control device 40 via a hub 50.

[0012] The cameras 11-19 are disposed at different positions and synchronously capture a plurality of images from a plurality of viewpoints. The cameras 11-19 capture images of foreground objects that are the targets of interest. That is, the cameras 11-19 are each set with targets to be captured. The image processing devices 21-29 process the captured images obtained by the corresponding cameras 11-19. The control device 40 controls the processing executed by the server 30.

[0013] The server 30 generates and displays a virtual viewpoint image seen from a virtual viewpoint (the viewpoint of a virtual camera). Here, the virtual camera is a virtual camera that can move freely in the imaging space. That is, the virtual camera can capture an image from a viewpoint different from any camera installed (placed) in the imaging space. However, a certain restriction may be imposed on the position and attitude of the virtual camera. The server 30 acquires the captured image and foreground information from the image processing devices 21 to 29. The server 30 also acquires in advance external parameters, which are coordinate information of the cameras 11 to 19, internal parameters of the lens, and distortion parameters. The server 30 generates a three-dimensional model of the foreground based on the foreground information and each parameter of the camera. In addition, a background three-dimensional model is registered in the server 30 in advance. The server 30 uses the captured image to map texture data to the foreground three-dimensional model based on the virtual viewpoint input by the user to generate a foreground image, and generates a background image from the background three-dimensional model.

[0014] The server 30 then combines the foreground image and the background image to continuously generate virtual viewpoint images, and displays the generated virtual viewpoint images. However, the method of generating the virtual viewpoint images is not limited to this. As a method of generating the virtual viewpoint images, various methods may be used, such as a method of generating the virtual viewpoint images by projective transformation of the captured images without using a three-dimensional model.

[0015] Additionally, as a supplementary note, in the following description, unless otherwise specified, the term "image" includes the concepts of both moving images and still images. In other words, the virtual viewpoint image generation system can process both still images and moving images.

[0016] (Hardware configuration) The hardware configuration of an information processing device such as the server 30 and the control device 40 will be described with reference to Fig. 2. Note that the image processing devices 21 to 29 basically have the same hardware configuration. The information processing device has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a communication I / F 205, an operation unit 206, a display unit 207, and a bus 208.

[0017] CPU 201 is a central processing unit, and controls the entire information processing device using programs and data stored in ROM 202 and RAM 203. Note that one or more dedicated hardware components different from CPU 201 may be provided, and at least a part of the processing by CPU 201 may be executed by the dedicated hardware components. Examples of the dedicated hardware components include ASICs (application specific integrated circuits), FPGAs (field programmable gate arrays), and DSPs (digital signal processors).

[0018] The ROM 202 stores setting data for the information processing device, programs that do not require modification, such as a boot program, etc. The RAM 203 has an area for temporarily storing programs and data loaded from the ROM 202, data acquired from the outside via the communication I / F 205, etc. Furthermore, the RAM 203 has a work area used when the CPU 201 executes various processes. That is, the RAM 203 can be allocated as a frame memory, for example, or can provide various other areas as appropriate.

[0019] The auxiliary storage device 204 is composed of, for example, an HDD or SSD. The auxiliary storage device 204 stores input data such as image data and audio data. The auxiliary storage device 204 also stores various data and programs such as tables referenced in various processes described below, an OS (operating system), and various application programs for causing the CPU 201 to realize the functions of each unit described below.

[0020] The communication I / F 205 can be connected to networks such as a LAN or the Internet, and other devices such as a projection device or a display device, and this information processing device can acquire and send various information via this communication I / F 205.

[0021] The operation unit 206 has a display device such as a liquid crystal display or LED, and an input device such as a keyboard or mouse, and receives operations by a user via a user interface or the like to input various instructions to the CPU 201. Note that a touch panel that combines the functions of a display device and an input device may also be used. The display unit 207 is formed of a display device such as a liquid crystal display or LED, and displays the results of processing by the CPU 201.

[0022] The computer programs and data stored in the auxiliary storage device 204 are loaded into the RAM 203 as appropriate under the control of the CPU 201, and are processed by the CPU 201. A bus 208 connects the above-mentioned components.

[0023] This embodiment can also be realized by supplying a storage medium on which a computer program code that realizes the above-mentioned functions is recorded to a system, and the system reads and executes the computer program code. In this case, the computer program code itself read from the storage medium realizes the functions of the above-mentioned embodiment, and the storage medium storing the computer program code constitutes the technology of the present disclosure. In addition, this also includes a case where an OS running on a computer uses the above-mentioned hardware to perform part or all of the actual processing based on the instructions of the program code, and the above-mentioned functions are realized by that processing.

[0024] Furthermore, the present invention may be realized in the following form: That is, the computer program code read from the storage medium is written into a memory provided in a function expansion card inserted into a computer or a function expansion unit connected to the computer. Then, based on the instructions of the computer program code, a CPU provided in the function expansion card or function expansion unit performs part or all of the actual processing to realize the above-mentioned functions.

[0025] When this embodiment is applied to the storage medium, the storage medium stores computer program code corresponding to the processes described above.

[0026] (Software configuration of image processing device) 3 is a diagram showing the software configuration of the image processing device 21. Note that the image processing devices 22 to 29 basically have the same software configuration. The image processing device 21 has an image receiving unit 301, a setting receiving unit 302, a foreground extraction unit 303, a background generation unit 304, and a data transmission unit 305.

[0027] The image receiving unit 301 receives a captured image (input image) captured by the camera 11. The received captured image is output to each of the foreground extraction unit 303 and the background generation unit 304. The setting receiving unit 302 receives various settings of the image processing device 21 that performs foreground / background separation processing from the control device 40 via the hub (HUB) 50, and executes settings for the foreground extraction unit 303, the background generation unit 304, the data transmission unit 305, etc. according to the various received settings.

[0028] The foreground extraction unit 303 compares a background image generated based on an input image (captured image) with the input image, and extracts a foreground region from the input image based on the comparison result. There are various methods for comparing a background image with an input image. For example, pixels whose RGB values ​​differ between the background image and the input image by a threshold value or more (foreground extraction threshold value) may be extracted as the foreground region from the input image. Any other image comparison method may be used. The information on the foreground region is a 1-bit image with the same image size as the captured image obtained by capturing an image with an imaging device, with the foreground region expressed in white and the background expressed in black. Any other information that can indicate the foreground region may be used, and the method is not particularly limited.

[0029] The background generating unit 304 generates an image consisting of still pixels as a background image by determining that still pixels are background based on input images (captured images) of multiple frames. The number of frames may be, for example, two or more, as long as the pixel values ​​of the same pixel can be compared. For example, the pixel values ​​of the same pixel in input images of multiple frames may be the same or within a threshold (threshold for background generation) for still pixels. Note that the threshold for background generation is smaller than the threshold for foreground extraction. The background generating unit 304 also receives settings related to background generation from the setting receiving unit 302, and generates a background image based on the received settings. The above settings include a setting called an update cycle for updating the background, and three settings related to the process of determining that a pixel is background, that is, a determination time, a determination cycle, and a determination threshold.

[0030] The background update period is a period for updating an existing background using a generated background. For example, if 4 seconds is set as the update period, the background is updated once every 4 seconds. The judgment time is the total time for judging whether it is a background, and the judgment period is a period for judging once whether it is a background. For example, if 64 seconds is set as the judgment time and 4 seconds is set as the judgment period, the judgment once every 4 seconds is repeated 16 times, and then a final judgment is made as to whether it is a background.

[0031] For background determination, each pixel is determined once per determination period as to whether it is background or not, and during the determination time, pixels that are always determined to be background are incorporated into the existing background image to generate a new background image.

[0032] The determination of whether it is background or not is based on whether it is still, that is, by comparing the captured image obtained by the imaging device one determination cycle ago with the captured image obtained by the current imaging device, and determining that pixels with color differences within the determination threshold are still background images. In this embodiment, colors are expressed by RGB with values ​​ranging from 0 to 1023, and the determination thresholds are also set to 0 to 1023 for each RGB.

[0033] Here, the method of updating the background will be described. If the judgment time is 64 seconds and the judgment cycle is 4 seconds, once every 4 seconds, the image captured by the imaging device 4 seconds ago is compared with the image captured by the current imaging device, and the image that is always judged to be the background after 16 comparisons is captured as the new background image. The pixels that are actually captured here are the pixels of the image captured by the imaging device at the time of the 16th judgment.

[0034] (background part update function) Furthermore, for background generation, a function for setting the validity / invalidity of background update for each region of an image may be provided. This function is a function for setting the background update target for each region by setting a partial update region setting image having the same image size as the input image, in which the valid region is white as the background update target region and the invalid region is black as the background update target region. This function is called the background partial update function. This background partial update function can be set to ON or OFF. When the background partial update function is set to ON, the background is partially updated based on the partial update region setting image set in advance. In other words, the background of only the target region of the entire input image is updated. When the background partial update function is set to OFF, the entire region of the entire input image is subject to background update.

[0035] Fig. 4 is a diagram for explaining the background portion update function, which shows an example of application of the background portion update function to a camera in which the image capture target is set to the catcher.

[0036] When the background partial update function is set to OFF for camera 401 in which the image capture target is set to the catcher, the entire area 411 of the captured image corresponding to the entire angle of view of camera 401 is treated as the background update target. When the background partial update function is set to ON for camera 401 in which the image capture target is set to the catcher, only a preset area 412 of the entire angle of view of camera 401 is treated as the background update target.

[0037] For example, if the subject of the image capture is baseball, the background portion update function is set to ON only for the area around home base in the image captured by the image capture device. By setting it in this way, even if there is a person within the camera's angle of view, if there is no person in the area around home base, which is the area set as the subject of the background portion update, the background can be updated.

[0038] The data transmission unit 305 transmits the captured image and the extracted foreground information.

[0039] (Control device software configuration) 5 is a diagram showing an example of the software configuration of the control device 40. The control device 40 includes a camera information storage unit 501, a camera classification information storage unit 502, a status acquisition unit 503, a UI generation unit 504, a display unit 505, a user input unit 506, a control unit 507, and a communication unit 508.

[0040] The camera information storage unit 501 stores information about the camera. The information includes camera identification information, connection group information to which the camera belongs, the connection destination of the camera, and installation coordinate information of the camera. For example, assume that multiple cameras are connected in a daisy chain, and there are two connection groups. In this case, the connection group information is information indicating which of the two connection groups the camera belongs to. The connection destination of the camera indicates the device connected by a transmission cable (network cable). The installation coordinate information of the camera is XYZ coordinate information when a predetermined position (for example, the center of the baseball field 710) is set as the origin.

[0041] The camera classification information holding unit 502 holds the camera classification information of each of the cameras 11 to 19 and the connected image processing devices 21 to 29. The camera classification information includes information on the subject imaged by the camera, the elevation angle of the camera, and the camera group.

[0042] (Camera classification information) 6 is a diagram showing an example of camera classification information 600. The camera classification information 600 includes information relating to a camera ID 610, a camera group 620, an imaging target 630, and an elevation angle 640 of the camera.

[0043] The camera classification information 600 is information that is manually set in advance by the user. In this embodiment, the classification information is set manually, but is not limited to this. For example, the camera's posture and direction may be automatically detected, and then the camera classification information may be automatically created by identifying the imaging target using the parameters. Also, in this embodiment, for simplicity, an example of managing information about nine cameras will be described, but is not limited to this. The camera classification information may manage information about fewer than nine cameras or more than nine cameras, for example, information about about 100 cameras may be managed.

[0044] The camera's imaging target 630 is an object that the camera captures, and in this embodiment, baseball is taken as an example, and each baseball position and baseball field are the targets. The method of expressing the target is not limited to this, and for example, the entire imaging area of ​​multiple cameras may be divided and each area may be numbered and expressed. For example, in soccer, it is possible to divide it into areas such as penalty area, right side, left side, etc. Also, depending on the camera, multiple positions may be the imaging target, and multiple imaging target information may be stored.

[0045] (Camera Classification) Fig. 7 is a diagram for explaining the classification of cameras. Fig. 7 shows a case where the system is applied to baseball. The system 700 has a plurality of cameras 721 set to capture an image of a pitcher (pitcher's mound) 711 in a baseball field 710, and a plurality of cameras 722 set to capture an image of a catcher (home base). That is, in the example of Fig. 7, the plurality of cameras are classified into cameras 721, 722, and 723 set to capture an image of the pitcher, catcher, and players other than the pitcher and catcher, respectively.

[0046] The camera's elevation angle 640 is the angle of the camera's imaging direction with respect to the ground when the ground is used as the reference. In the case of this system, the camera is installed at a position higher than the ground and takes images facing downward. Therefore, a camera with an elevation angle of 30 degrees or less is considered a low angle, and one with an elevation angle of more than 30 degrees is considered a high angle.

[0047] The camera group 620 is a group of multiple cameras grouped together according to the object to be controlled. For example, by classifying the cameras that capture the catcher into the same group, it is possible to issue an instruction such as background update to the cameras in the group at a moment when there is no person near the home base. When deciding which of the multiple groups each of the multiple cameras should be classified into, it may be further limited by the angle (camera elevation angle). That is, it may be decided which of the multiple groups each of the multiple cameras should be classified into according to the elevation angle at the time of capturing the image of the camera. In the case of a low-angle camera, there is an increased risk that players standing in front of or behind the home base will get in the way, in addition to those near the home base. Therefore, if only high-angle cameras are targeted, it is possible to further reduce the risk of being obstructed by players other than the object of capture. Here, a method of using camera information to decide which of the multiple groups each of the multiple cameras should be classified into is described, but the present invention is not limited to this. For example, a user may arbitrarily decide which of the multiple groups each of the multiple cameras should be classified into.

[0048] Returning to the explanation of Fig. 5, the status acquisition unit 503 acquires status information of the cameras 11-19 and status information of the image processing devices 21-29, and outputs the acquired information to the UI generation unit 504.

[0049] The UI generation unit 504 generates a UI image showing the status of the system and the camera based on the information acquired from the camera information storage unit 501, the camera classification information storage unit 502, and the status acquisition unit 503.

[0050] The display unit 505 has a frame buffer and a display panel. The display unit 505 stores (overwrites) the display image output from the UI generation unit 504 in the frame buffer. The display unit 505 then reads out the display image stored in the frame buffer at a predetermined refresh rate and displays it on the display panel. It can also be said that the display unit 505 performs display control so as to display a UI screen, which will be described in detail later, on the display panel. The display panel is, for example, a liquid crystal panel or an organic EL panel.

[0051] A user input unit 506 receives operation information input from a controller (not shown), and outputs the received information to a control unit 507. The controller is, for example, a keyboard, a mouse, a touch panel, a joystick, or the like.

[0052] The control unit 507 outputs control instructions to each unit of the control device 40 based on operation information from the user input unit 506. In this embodiment, the user instructs the settings for background generation, and the control unit 507 executes control in response to the user instruction input via the user input unit 506. At that time, the control unit 507 determines (designates) the image processing device to be controlled based on information acquired from the camera classification information storage unit 502. As a method for designation, any of the following five methods can be used.

[0053] (Specification method 1) Designation method 1 is a method for designating an image capture target. In this designation method, for example, a camera is designated that captures an image of a catcher. In this case, based on the camera classification information 600, the cameras with camera IDs 11, 15, and 17 are designated.

[0054] (Specification method 2) Designation method 2 is a method of designating by the elevation angle of the camera. In this designation method, for example, a camera with a high elevation angle is designated. In this case, based on the camera classification information 600, cameras with camera IDs 11, 13, 14, 15, 16, 18, and 19 are designated.

[0055] (Specification method 3) Designation method 3 is a method of designating a combination of the image capture target and the elevation angle of the camera. In this designation method, for example, a camera is designated when the image capture target is a catcher and the elevation angle of the camera is High. In this case, based on the camera classification information 600, the cameras with camera IDs 11 and 15 are designated.

[0056] (Specification method 4) Designation method 4 is a method for designating multiple imaging targets. In this designation method, for example, a camera is designated to image a catcher or pitcher. In this case, based on the camera classification information 600, the cameras with camera IDs 11, 12, 15, 17, and 18 are designated.

[0057] (Specification method 5) Designation method 5 is a method of designating by camera group. In this designation method, for example, a camera in camera group 3 is designated. In this case, based on the camera classification information 600, the cameras with camera IDs 13, 14, 16, and 19 are designated.

[0058] Furthermore, the control unit 507 instructs the image processing devices 21 to 29 to start and stop foreground / background separation and to start and stop data transmission.

[0059] The communication unit 508 outputs the control information from the control unit 507 to the outside. In addition, the communication unit 508 receives a notification indicating a result of the control, and outputs the received notification to the control unit 507.

[0060] (Background generation setting process) 8(a) is a flowchart showing the flow of the background generation setting control process. Hereinafter, the background generation setting control process will be described according to the flowchart.

[0061] In S801, the CPU 201 (user input unit 506) of the control device 40 checks (determines) whether there is a control instruction from the user. If the CPU 201 of the control device 40 determines that there is no control instruction from the user (NO in S801), it continues to wait for the control instruction. If the CPU 201 of the control device 40 determines that there is a control instruction from the user (YES in S801), the process of S802 is executed.

[0062] In S802, the CPU 201 (control unit 507) of the control device 40 checks the contents of the control instruction input to the user input unit 506. The contents of the control instruction are, for example, settings for background generation. The control contents include a setting target and setting contents. The setting target is either the identification information of the camera in the information about the camera, or the imaging target or the camera group in the camera classification information. In the case of the camera identification information, it is camera 11 indicating the camera ID. In the case of designation by imaging target in the camera classification information, it is information indicating the imaging target such as catcher or pitcher. In the case of camera group in the camera classification information, it is camera group 1. In addition, the setting contents are settings related to background generation as described above. Here, it is assumed that the setting target is a camera whose imaging target is a catcher, and the setting contents are contents to set the determination cycle of background generation to 32 seconds. Note that, here, a case where the setting contents is the determination cycle of background generation will be described, but is not limited thereto. The set contents may be the judgment time or the judgment threshold, any two of the judgment time, the judgment cycle, and the judgment threshold, or all of the judgment time, the judgment cycle, and the judgment threshold.

[0063] In S803, the CPU 201 (control unit 507) of the control device 40 acquires the camera classification information from the camera classification information storage unit 502.

[0064] In S804, the CPU 201 (control unit 507) of the control device 40 extracts image processing devices to be controlled. Here, the cameras 11, 15, and 17 capture images of the catcher, and the image processing devices 21, 25, and 27 to which the captured images are input are extracted as control devices.

[0065] In S805, the CPU 201 (control unit 507) of the control device 40 outputs background generation settings (determination cycle set to 32 seconds) to the image processing devices 21, 25, and 27 via the communication unit 508. Note that the image processing devices receive information regarding the background generation settings in S811, which will be described later.

[0066] In S806, the CPU 201 (communication unit 508) of the control device 40 receives the setting completion notification and outputs it to the control unit 507. The CPU 201 (control unit 507) of the control device 40 determines that the background generation setting is completed, and ends the background generation setting control process shown in FIG. 8(a).

[0067] (Background generation setting process) 8B is a flowchart showing the flow of the setting process for background generation. Hereinafter, the setting process for background generation will be described according to the flowchart. Note that since the processes of the image processing devices 21, 25, and 27 are equivalent, the process of the image processing device 21 will be described as an example.

[0068] In S811, the CPU 201 (setting receiving unit 302) of the image processing device 21 waits for background generation settings to be transmitted from the control device 40, and receives the background generation settings transmitted from the control device 40. The received background generation settings are output to the background generation unit 304.

[0069] In S812, the CPU 201 (background generating unit 304) of the image processing device 21 confirms the setting contents of the received background generation setting, and executes the confirmed setting contents to set the background generation setting, thereby reflecting the background generation setting. For example, the determination cycle for whether or not it is a background is set to 32 seconds.

[0070] In S813, the CPU 201 (background generating unit 304) of the image processing device 21 notifies the setting receiving unit 302 of the completion of the setting, and the CPU 201 (setting receiving unit 302) of the image processing device 21 transmits a setting completion notification to the control device 40.

[0071] (Background generation settings) Here, a case will be described in which a UI screen including a button associated with a background update mode that combines several settings prepared in advance is used to enable quicker work on background generation settings. That is, a case will be described in which a UI screen showing an update method in a user-specifiable manner is displayed.

[0072] First, there are five modes that change the speed at which the background updates: stop, slow, medium, fast, and instant. Each of these five modes can be associated with settings that correspond to that mode.

[0073] The "Stop" mode is a mode for setting a state in which the background is not updated at all. The "Slow" mode is a mode for setting a state in which the background is updated slowly, at a speed slower than the normal speed. The "Medium" mode is a mode for setting a state in which the background is updated at a normal speed. The "Fast" mode is a mode for setting a state in which the background is updated quickly, at a speed faster than the normal speed. The "Instant" mode is a mode for setting a state in which the background is updated instantly, at which the image captured by the camera is reflected directly in the background.

[0074] (Setting parameters for each background update mode) 9 is a diagram showing an example of setting parameters for realizing each background update mode. Setting parameters 900 have information on each of the items of background update mode 910, background update cycle (seconds) 920, judgment time (seconds) 930, judgment cycle (seconds) 940, and threshold 950. That is, information on each of the items of background update cycle (seconds) 920, judgment time (seconds) 930, judgment cycle (seconds) 940, and threshold 950 for determining background is linked to each of the background update modes 910, "stop," "low speed," "medium speed," "high speed," and "instantaneous."

[0075] For example, when the "stop" mode is selected for the background update mode 910, the "background update cycle (seconds)" 920 is set to ∞ (infinity) and the background update is stopped. Since ∞ (infinity) cannot actually be set as a numerical value, for example, a specific value such as 65535 is set to control the background update to be stopped.

[0076] When the "low speed" mode is selected for the background update mode 910, the "judgment time (seconds)" 930 is set to 64 seconds and the background is updated at a low speed. That is, when the "low speed" mode is selected, a judgment is made 16 times once every 4 seconds using a threshold (RGB pixel value), and then a final judgment is made as to whether or not it is the background. This makes it possible to prevent players and referees who do not move much during a match from being judged as being the background. When the "medium speed" mode is selected for the background update mode 910, the "judgment time (seconds)" 930 is set to 32 seconds and the background is updated at a medium speed. That is, when the "medium speed" mode is selected, a judgment is made 8 times once every 4 seconds using a threshold (RGB pixel value), and then a final judgment is made as to whether or not it is the background. When the "high speed" mode is selected for the background update mode 910, the "judgment time (seconds)" 930 is set to 16 seconds and the background is updated at a high speed. That is, when the "high speed" mode is selected, a threshold value (RGB pixel value) is used to repeat the determination once every four seconds four times, and then a final determination is made as to whether or not the object is a background.

[0077] When the "instant" mode is selected for the background update mode 910, the "judgment time (seconds)" 930 is set to 1 second, the "judgment cycle (seconds)" 940 is set to 1 second, and the "threshold" 950 for judgment is set to 1023, and the background is updated immediately. In other words, all pixels in the captured image are judged to be background and are updated.

[0078] A case where the above background update mode settings are applied to a baseball game will be described. For example, while a player is playing at a specific position on the ground, the background is captured slowly by setting the background update mode 910 to the "slow" mode. When a player leaves the specific position on the ground due to a change or the like, the background is quickly updated by setting the background update mode 910 to the "fast" mode or the "instant" mode. Note that, although the above description is given of the case of five modes, this is not limiting. As long as the update speeds are different from each other, for example, there may be fewer than five modes or more than five modes. In addition, the values ​​set in each of the items of the background update cycle (seconds) 920, the judgment time (seconds) 930, the judgment cycle (seconds) 940, and the threshold value 950 of the above five modes are merely examples, and other values ​​may be set.

[0079] (UI screen for background update settings) Fig. 10 is a diagram showing an example of a UI screen including a setting button that combines a setting target and a background update mode. In addition, in S801 shown in Fig. 8(a), it may be determined whether or not there is a control instruction from the user via the UI screen 1000.

[0080] The UI screen 1000 includes various buttons for setting a target camera and a background update mode. That is, the UI screen 1000 selectively displays a speed designation button for designating a speed at which the background image is updated and a partial update designation button for designating a predetermined area as a background image update target. By clicking buttons 1001 to 1011 on the UI screen 1000 with a mouse, the user can execute settings related to background generation associated with the clicked button. That is, the update method is set based on a user operation via the UI screen 1000.

[0081] Buttons 1001 to 1005 are buttons for setting the background generation mode for all cameras. That is, the "All Cameras Instant" button 1001 is a button for setting the background update mode 910 to "instant" mode for all cameras. The "All Cameras Fast" button 1002 is a button for setting the background update mode 910 to "fast" mode for all cameras. The "All Cameras Medium Speed" button 1003 is a button for setting the background update mode 910 to "medium speed" mode for all cameras.

[0082] An "all cameras slow" button 1004 is a button for setting the "slow" mode for the background update mode 910 for all cameras. An "all cameras stop" button 1005 is a button for setting the "stop" mode for the background update mode 910 for all cameras.

[0083] The UI screen 1000 includes buttons for setting each mode for the background update mode 910 for all cameras, so that a desired background update mode can be easily set for all cameras by operating one button. This makes it possible to control the background generation speed for all cameras.

[0084] Buttons 1006 to 1009 are buttons for setting a background generation mode for a specific camera. That is, the "catcher instant" button 1006 is a button for setting the background update mode 910 to the "instant" mode for a camera set to capture an image of the catcher. The "catcher slow" button 1007 is a button for setting the background update mode 910 to the "slow" mode for a camera set to capture an image of the catcher. When a player disappears from near home base during a change of pitches, for example, by quickly changing the background generation mode with these buttons, background updating can be performed even in a sport with few scenes without players.

[0085] A "Pitcher Instant" button 1008 is the same as the "Catcher Instant" button 1006, except that it is for a camera that is set to capture the pitcher. A "Pitcher Slow" button 1009 is the same as the "Catcher Slow" button 1007, except that it is for a camera that is set to capture the pitcher.

[0086] Buttons 1010 and 1011 are buttons for setting the background partial update function for all cameras. That is, the "ALL CAMERA BACKGROUND PARTIAL UPDATE ON" button 1010 is a button for enabling the background partial update function for all cameras. The "ALL CAMERA BACKGROUND PARTIAL UPDATE OFF" button 1011 is a button for disabling the background partial update function for all cameras.

[0087] Any one of the buttons 1001 to 1005, any one of the buttons 1006 to 1007, any one of the buttons 1008 to 1009, and any one of the buttons 1010 to 1011 may be set simultaneously. For example, "All Cameras Instant" 1001, "Catcher Slow" 1007, "Pitcher Slow" 1009, and "All Cameras Background Partial Update ON" 1010 may be set simultaneously. In this case, the background update mode "instant" is set for all cameras other than the cameras whose imaging targets are the catcher and pitcher, and the background update mode "slow" is set for the cameras whose imaging targets are the catcher. In addition, the background partial update function is set for all cameras. That is, it can be said that the update method is set based on the user's operation via the UI screen 1000 to use a partial area of ​​the captured image obtained by the camera to update the background image.

[0088] It should be noted that the buttons 1001 to 1011 shown on the UI screen 1000 shown in FIG. 10 are merely an example, and buttons corresponding to imaging targets and update speeds other than the pitcher and catcher may be present on the UI screen 1000.

[0089] (UI screen for background update settings) 11 is a diagram showing an example of a UI screen including a setting button that combines the progress of the game, the setting target, and the background update mode. In addition, in S801 shown in FIG. 8(a), it may be determined whether or not there is a control instruction from the user via the UI screen 1100.

[0090] The UI screen 1100 includes various buttons for setting a target camera and a background update mode for each progression of a baseball game. That is, the UI screen 1100 displays selectable speed designation buttons for designating the speed at which the background image is updated for each progression of the game captured by the camera. The user can execute settings related to background generation associated with the clicked button by clicking buttons 1101 to 1112 on the UI screen 1100 with a mouse, as in the case of the UI screen 1000. That is, the update method is set based on a user operation via the UI screen 1100.

[0091] In this embodiment, it is assumed that ground preparation is performed in the bottom of the fifth inning. Therefore, the UI screen 1100 is provided with buttons 1103 to 1107 so that the backgrounds near the home base and the pitcher's mound can be updated immediately at the bottom of the fifth inning. Also, it is assumed that there is a cheering time in the bottom of the seventh inning, during which the lighting fluctuates drastically. Therefore, the UI screen 1100 is provided with button 1110 so that the background update can be stopped for all cameras at the bottom of the seventh inning.

[0092] That is, on the UI screen 1100, an "all cameras fast" button 1101 and an "all cameras slow" button 1102 are prepared for the top of the first inning to the top of the fifth inning. The "all cameras fast" button 1101 and the "all cameras slow" button 1102 are buttons for setting the background update mode for all cameras from the top of the first inning to the top of the fifth inning. The "all cameras fast" button 1101 is a button for setting the background update mode to "fast" mode, and the "all cameras slow" button 1102 is a button for setting the background update mode to "slow" mode.

[0093] In the UI screen 1100, for the bottom of the fifth inning, a "catcher immediate" button 1103 and a "pitcher immediate" button 1104 are provided. In addition, a "all camera background partial update ON" button 1105, a "all camera slow" button 1106 and a "all camera background partial update OFF" button 1107 are provided. The "catcher immediate" button 1103 is a button for setting the background update mode for the camera whose imaging target is set to the catcher in the bottom of the fifth inning to the "immediate" mode. The "pitcher immediate" button 1104 is a button for setting the background update mode for the camera whose imaging target is set to the pitcher in the bottom of the fifth inning to the "immediate" mode. The "all camera background partial update ON" button 1105 is a button for setting the background partial update to be valid for all cameras in the bottom of the fifth inning. The "all camera slow" button 1106 is a button for setting the background update mode for all cameras in the bottom of the fifth inning to the "slow" mode. An "All Cameras Background Part Update OFF" button 1107 is a button for disabling background part updates for all cameras during the bottom of the fifth inning.

[0094] On the UI screen 1100, an "all cameras fast" button 1108 and an "all cameras slow" button 1109 are provided for the top of the sixth to seventh innings. The "all cameras fast" button 1108 and the "all cameras slow" button 1109 are buttons for setting the background update mode for all cameras from the top of the sixth to the top of the seventh innings. The "all cameras fast" button 1108 is a button for setting the background update mode to "fast" mode, and the "all cameras slow" button 1109 is a button for setting the background update mode to "slow" mode.

[0095] On the UI screen 1100, a "Stop all cameras" button 1110 is provided for the bottom of the seventh inning. The "Stop all cameras" button 1110 is a button for setting the background update mode for all cameras to "Stop" mode during the bottom of the seventh inning.

[0096] Furthermore, the UI screen 1100 provides an "all cameras fast" button 1111 and an "all cameras slow" button 1112 for the top of the 8th inning to the bottom of the 9th inning. The "all cameras fast" button 1111 and the "all cameras slow" button 1112 are buttons for setting the background update mode for all cameras from the top of the 8th inning to the bottom of the 9th inning. The "all cameras fast" button 1111 is a button for setting the background update mode to "fast," and the "all cameras slow" button 1112 is a button for setting the background update mode to "slow."

[0097] As described above, it is possible to arrange a plurality of buttons to which the same setting content is assigned on the UI screen 1100. Also, the buttons 1101 to 1112 shown on the UI screen 1100 are merely examples, and buttons corresponding to imaging targets and update speeds other than the pitcher and catcher may be provided on the UI screen 1100.

[0098] (Background update process) 12 is a flowchart showing the flow of a background update process. For example, assume that when no object exists in the imaging area of ​​a camera (imaging device), a captured image captured by the camera (imaging device) is acquired as a background image.

[0099] First, in S1201, timing is started. That is, the elapsed time from the start of the background image update process is measured.

[0100] Next, in S1202, the CPU 201 of the image processing apparatus determines whether the determination time has elapsed. The determination in S1202 is repeated until the determination time corresponding to the set update speed (low speed, medium speed, high speed, immediate) has elapsed.

[0101] In S1203, the CPU 201 of the image processing device determines whether the background image can be updated according to the set background update method. For example, when the "high speed" mode is set, a determination is made four times once every four seconds using a threshold value, and then a final determination is made as to whether the image is the background. If it is determined to be the background, it is determined that the background image can be updated. If the CPU 201 of the image processing device determines that the background image cannot be updated (NO in S1203), it ends the background update process shown in FIG. 12. If the CPU 201 of the image processing device determines that the background image can be updated (YES in S1203), the process proceeds to S1204.

[0102] In S1204, the CPU 201 of the image processing apparatus updates the background image. When the updating of the background image is completed, the flow of the background updating process shown in FIG.

[0103] In this embodiment, baseball is used as an example, but the present invention can be applied to other sports and other imaging subjects, and is not limited to any particular imaging subject. For example, the present invention can be applied to sports such as American football, rugby, basketball, soccer, cricket, racing (horse, boat, bicycle), badminton, and tennis.

[0104] As described above, in this embodiment, the cameras are classified into a plurality of groups based on the target of interest in the imaging of each camera, and a background generation method is set for each classified group. For each classified group, the plurality of background images are updated using the captured images obtained by one or more cameras included in the group according to the set background image update method. This makes it possible to update the background at the moment when there is no foreground object in the imaging area of ​​one or more cameras included in the group, and the background can be updated even if the object is present in the imaging area for a long time.

[0105] (Other embodiments) The present disclosure can also be realized by executing the following process. That is, the present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or various storage media, and having one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0106] The disclosure of this embodiment includes the following configuration examples.

[0107] (Configuration 1) An acquisition means for acquiring a plurality of captured images obtained by imaging using a plurality of imaging means; an extraction means for extracting a foreground region from the plurality of captured images by a background subtraction method; a generating means for generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; having The generating means includes: For each group into which the plurality of imaging means are classified based on a target object in imaging by each of the plurality of imaging means, the background image is updated using an image captured by one or more imaging means included in the group according to a set background image updating method. 13. An image processing device comprising:

[0108] (Configuration 2) The update method is set based on a user operation via a UI screen. 2. The image processing device according to claim 1,

[0109] (Configuration 3) As the updating method, a speed at which the background image is updated is set based on the user operation via the UI screen. 3. The image processing device according to configuration 2.

[0110] (Configuration 4) The speed at which the background image is updated is set for each of the classified groups. 4. The image processing device according to configuration 3.

[0111] (Configuration 5) The speed at which the background image is updated is set in units of the progress of the game captured by the multiple image capture means. 5. The image processing device according to configuration 3 or 4.

[0112] (Configuration 6) The speed at which the background image is updated is set to any one of a stop mode in which updating of the background image is stopped, an immediate mode in which the background image is immediately updated at a first speed, a high-speed mode in which the background image is updated at a second speed slower than the first speed, a medium-speed mode in which the background image is updated at a medium speed at a third speed slower than the second speed, and a low-speed mode in which the background image is updated at a low speed at a fourth speed slower than the third speed. 6. The image processing device according to any one of configurations 3 to 5.

[0113] (Configuration 7) When the immediate mode is set, The generating means reflects the captured image obtained by the imaging means as it is in the background image. 7. The image processing device according to configuration 6,

[0114] (Configuration 8) As the updating method, a partial area of ​​the captured image obtained by the imaging means is used as an update target of the background image, which is set based on the user operation via the UI screen. 8. The image processing device according to any one of configurations 2 to 7.

[0115] (Configuration 9) The plurality of imaging means are classified into the groups corresponding to the elevation angles at the time of imaging of the plurality of imaging means. 9. The image processing device according to any one of configurations 1 to 8.

[0116] (Configuration 10) As the update method, any one of an update period for updating the background image, a determination time for determining that the image is a background image, a determination period for determining that the image is a background image, and a determination threshold for the background image is set. 10. The image processing device according to any one of configurations 1 to 9.

[0117] (Configuration 11) A classification means for classifying the plurality of imaging means into a plurality of groups based on an object of interest in imaging of each of the plurality of imaging means; a setting means for setting, for each of the classified groups, a method of updating a background image used for extracting a foreground region by a background subtraction method from an image captured by one or more of the imaging means included in the group; A control device comprising:

[0118] (Configuration 12) A system including an image processing device and a control device that controls the image processing device, The image processing device includes: An acquisition means for acquiring a plurality of captured images obtained by imaging using a plurality of imaging means; an extraction means for extracting a foreground region from the plurality of captured images by a background subtraction method; a generating means for generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; having The control device includes: A setting means for setting a method for updating the background image is provided, The setting means is classifying the plurality of imaging means into a plurality of groups based on an object of interest in imaging each of the plurality of imaging means; A method of updating the background image is set for each of the classified groups; The generating means includes: For each of the classified groups, the background image is updated using the captured images obtained by one or more imaging means included in the group according to the set update method. A system characterized by:

[0119] (Configuration 13) An acquisition step of acquiring a plurality of captured images obtained by imaging using a plurality of imaging means; an extraction step of extracting a foreground region from the plurality of captured images by a background subtraction method; a generating step of generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; Including, In the generating step, For each group into which the plurality of imaging means are classified based on a target object in imaging by each of the plurality of imaging means, the background image is updated using an image captured by one or more imaging means included in the group according to a set background image updating method. 23. A method for controlling an image processing apparatus comprising the steps of:

[0120] (Configuration 14) A program for causing a computer to function as the image processing device according to any one of configurations 1 to 10.

Claims

1. An acquisition means for acquiring a plurality of captured images obtained by imaging using a plurality of imaging means; an extraction means for extracting a foreground region from the plurality of captured images by a background subtraction method; a generating means for generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; having The generating means includes: For each group into which the plurality of imaging means are classified based on a target object in imaging by each of the plurality of imaging means, the background image is updated using an image captured by one or more imaging means included in the group in accordance with a set background image updating method.

13. An image processing device comprising:

2. The update method is set based on a user operation via a UI screen.

2. The image processing device according to claim 1,

3. As the updating method, a speed at which the background image is updated is set based on the user operation via the UI screen.

3. The image processing device according to claim 2.

4. The speed at which the background image is updated is set for each of the classified groups.

4. The image processing device according to claim 3.

5. The speed at which the background image is updated is set in units of the progress of the game captured by the multiple image capture means.

4. The image processing device according to claim 3.

6. The speed at which the background image is updated is set to any one of a stop mode in which updating of the background image is stopped, an immediate mode in which the background image is immediately updated at a first speed, a high-speed mode in which the background image is updated at a second speed slower than the first speed, a medium-speed mode in which the background image is updated at a medium speed at a third speed slower than the second speed, and a low-speed mode in which the background image is updated at a low speed at a fourth speed slower than the third speed.

4. The image processing device according to claim 3.

7. When the immediate mode is set, The generating means reflects the captured image obtained by the imaging means as it is in the background image.

7. The image processing device according to claim 6,

8. The updating method is set based on the user's operation via the UI screen to use a partial area of ​​the captured image obtained by the imaging means for updating the background image.

3. The image processing device according to claim 2.

9. The plurality of imaging means are classified into the groups corresponding to the elevation angles at the time of imaging of the plurality of imaging means.

2. The image processing device according to claim 1,

10. As the update method, any one of an update period for updating the background image, a determination time for determining that the image is a background image, a determination period for determining that the image is a background image, and a determination threshold for the background image is set.

2. The image processing device according to claim 1,

11. A classification means for classifying the plurality of imaging means into a plurality of groups based on an object of interest in imaging of each of the plurality of imaging means; a setting means for setting, for each of the classified groups, a method of updating a background image used for extracting a foreground region by a background subtraction method from an image captured by one or more of the imaging means included in the group; The control device further comprises:

12. A system including an image processing device and a control device that controls the image processing device, The image processing device includes: An acquisition means for acquiring a plurality of captured images obtained by imaging using a plurality of imaging means; an extraction means for extracting a foreground region from the plurality of captured images by a background subtraction method; a generating means for generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; having The control device includes: A setting means for setting a method for updating the background image, The setting means is classifying the plurality of imaging means into a plurality of groups based on an object of interest in imaging each of the plurality of imaging means; A method of updating the background image is set for each of the classified groups; The generating means includes: For each of the classified groups, the background image is updated using the captured images obtained by one or more imaging means included in the group in accordance with the set update method. A system characterized by:

13. An acquisition step of acquiring a plurality of captured images obtained by imaging using a plurality of imaging means; an extraction step of extracting a foreground region from the plurality of captured images by a background subtraction method; a generating step of generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; Including, In the generating step, For each group into which the plurality of imaging means are classified based on a target object in imaging by each of the plurality of imaging means, the background image is updated using an image captured by one or more imaging means included in the group in accordance with a set background image updating method.

23. A method for controlling an image processing apparatus comprising the steps of:

14. A program for causing a computer to function as the image processing device according to claim 1.

Images