Image processing system, control method and program for image processing system

Abstract

To solve the problem that a background image cannot be updated appropriately.SOLUTION: An image processing device includes: acquisition means for acquiring a plurality of captured images obtained by capturing of a plurality of imaging means; extraction means for extracting foreground regions from the plurality of captured images by a background difference method; and generation means for generating background images used in the background difference method while updating the background images on the basis of the plurality of captured images. The generation means updates the background images according to an update method of the set background images for each of groups to which the plurality of imaging means are classified on the basis of targets of interest of the respective plurality of imaging means in imaging using the captured images obtained by one or more imaging means included in the group.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present disclosure relates to techniques for extracting foreground regions from images. [Background technology]

[0002] Techniques for extracting an image region (hereinafter referred to as a "foreground region") corresponding to a foreground object (hereinafter referred to as a "foreground object") from an image captured by an imaging device (hereinafter referred to as a "captured image") are used for various purposes, and there are many different techniques for doing so. One such technique is a so-called background subtraction method, which compares the captured image with an image of the background (hereinafter referred to as a "background image") obtained by some technique, and extracts pixels whose pixel value difference between corresponding pixels is equal to or exceeds a predetermined threshold value as the foreground region. Extracting the foreground region using the background subtraction method fails to properly extract the foreground region when the difference between the captured image and the background image increases due to dynamic changes in the background or changes in environmental light such as sunlight or lighting.

[0003] In this regard, Patent Document 1 discloses a technique for updating to a background image generated at a set timing. By using the technique of Patent Document 1, even if the background changes dynamically or there is a change in the ambient light such as sunlight or lighting, the background image is written as a background image and is not extracted as a foreground area. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-163303 Summary of the Invention [Problem to be solved by the invention]

[0005] The technology disclosed in Patent Document 1 above has a problem in that it is difficult to update the background image when a person remains for a long time in a part of the entire imaging area of ​​a plurality of imaging devices. For example, in the case of baseball, the image is likely to change during a game as the white lines drawn on the field near home base disappear, but the catcher and umpire remain near home base for a long time, making it difficult to update the background image. [Means for solving the problem]

[0006] An image processing system according to one aspect of the present disclosure includes a plurality of imaging means for capturing captured images by imaging, an extraction means for extracting a foreground region based on a difference between the captured images and a background image, and an update means for updating the background image based on the captured images, wherein the update means: When an instruction to update a background image for an update target in an imaging area is received, a background image corresponding to the update target is updated based on a captured image including the update target. the background image Subregion of The feature is that the above is updated. [Effects of the Invention]

[0007] According to the present disclosure, the background image can be updated. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 illustrates an example of a system configuration. [Figure 2] FIG. 2 illustrates an example of a hardware configuration of the apparatus. [Figure 3] FIG. 2 is a diagram illustrating an example of a software configuration of the image processing apparatus. [Figure 4] FIG. 10 is a diagram for explaining a partial background update function. [Figure 5] FIG. 2 is a diagram illustrating an example of a software configuration of a control device. [Figure 6] FIG. 10 is a diagram illustrating an example of camera classification information. [Figure 7] FIG. 1 is a diagram for explaining camera classifications. [Figure 8] 10A is a flowchart showing the flow of a setting control process for background generation, and FIG. 10B is a flowchart showing the flow of a setting process for background generation. [Figure 9] FIG. 10 is a diagram illustrating an example of setting parameters for each background update mode. [Figure 10] FIG. 10 is a diagram illustrating an example of a UI for background update settings. [Figure 11] FIG. 10 is a diagram illustrating an example of a UI for background update settings. [Figure 12] 10 is a flowchart showing the flow of a background update process. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments are not limited to the inventions according to the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the invention. Note that in each drawing, the same elements are given the same reference numerals, and duplicate explanations are omitted or simplified. Note that each process (step) in the flowchart is denoted by an "S" in front of it.

[0010] <Embodiment 1> A virtual viewpoint image generation system according to this embodiment will be described. In this embodiment, a plurality of image capture devices are grouped by image capture target, the timing for background update is set for each grouped image capture device, and the background update is performed at a moment when there is no person as a foreground object in the image capture target.

[0011] (System Configuration) 1 is a diagram showing an example of the configuration of a virtual viewpoint image generation system according to this embodiment. The virtual viewpoint image generation system includes a plurality of cameras (imaging devices) 11-19, image processing devices 21-29 that process images captured by the cameras, a server 30, and a control device 40 that controls the cameras 11-19 and the image processing devices 21-29. The image processing devices 21-29 are connected to the server 30 and the control device 40 via a hub 50.

[0012] Cameras 11-19 are placed at different positions and synchronously capture multiple images from multiple viewpoints. Each of cameras 11-19 captures an image of a foreground object that is the target of attention. That is, each of cameras 11-19 has a specific target to capture. Image processing devices 21-29 process the captured images obtained by the corresponding cameras 11-19. Control device 40 controls the processing executed by server 30.

[0013] The server 30 generates and displays a virtual viewpoint image viewed from a virtual viewpoint (the viewpoint of a virtual camera). The virtual camera is a virtual camera that can move freely within the imaging space. In other words, the virtual camera can capture images from a viewpoint different from that of any installed (disposed) camera within the imaging space. However, certain restrictions may be imposed on the position and orientation of the virtual camera. The server 30 acquires captured images and foreground information from the image processing devices 21-29. The server 30 also acquires in advance external parameters, which are coordinate information of the cameras 11-19, internal lens parameters, and distortion parameters. The server 30 generates a three-dimensional model of the foreground based on the foreground information and the camera parameters. A three-dimensional background model is also registered in the server 30 in advance. Based on the virtual viewpoint input by the user, the server 30 uses the captured image to map texture data onto the foreground three-dimensional model to generate a foreground image, and generates a background image from the background three-dimensional model.

[0014] The server 30 then combines the foreground image and the background image to continuously generate virtual viewpoint images and displays the generated virtual viewpoint images. However, the method for generating the virtual viewpoint images is not limited to this. Various methods may be used to generate the virtual viewpoint images, such as a method for generating the virtual viewpoint images by projective transformation of the captured images without using a three-dimensional model.

[0015] Additionally, in the following description, unless otherwise specified, the term "image" includes both moving and still images. In other words, the virtual viewpoint image generation system can process both still and moving images.

[0016] (Hardware configuration) The hardware configuration of an information processing device such as the server 30 and the control device 40 will be described with reference to Fig. 2. Note that the image processing devices 21 to 29 also have basically the same hardware configuration. The information processing device has a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a communication I / F 205, an operation unit 206, a display unit 207, and a bus 208.

[0017] The CPU 201 is a central processing unit that controls the entire information processing device using programs and data stored in the ROM 202 and RAM 203. Note that one or more pieces of dedicated hardware different from the CPU 201 may be provided, and at least part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (application-specific integrated circuit), an FPGA (field-programmable gate array), and a DSP (digital signal processor).

[0018] The ROM 202 stores setting data for the information processing device, programs that do not require modification, such as a boot program, etc. The RAM 203 has an area for temporarily storing programs and data loaded from the ROM 202, data acquired from the outside via the communication I / F 205, etc. The RAM 203 also has a work area used by the CPU 201 when executing various processes. That is, the RAM 203 can be allocated as a frame memory, for example, or can provide various other areas as needed.

[0019] The auxiliary storage device 204 is configured with, for example, an HDD or SSD. The auxiliary storage device 204 stores input data such as image data and audio data. The auxiliary storage device 204 also stores various data and programs, such as tables referenced in various processes described below, an OS (operating system), and various application programs that cause the CPU 201 to implement the functions of each unit described below.

[0020] The communication I / F 205 can be connected to networks such as a LAN or the Internet, and other devices such as a projection device or display device, and this information processing device can acquire and send various information via this communication I / F 205.

[0021] The operation unit 206 has a display device such as a liquid crystal display or LED, and an input device such as a keyboard or mouse, and receives operations from a user via a user interface or the like to input various instructions to the CPU 201. Note that a touch panel that combines the functions of a display device and an input device may also be used. The display unit 207 is formed from a display device such as a liquid crystal display or LED, and displays the results of processing by the CPU 201.

[0022] The computer programs and data stored in the auxiliary storage device 204 are loaded into the RAM 203 as appropriate under the control of the CPU 201, and are processed by the CPU 201. A bus 208 connects the above-mentioned components.

[0023] This embodiment can also be realized by providing a storage medium containing computer program code for implementing the above-described functions to a system, and then reading and executing the computer program code. In this case, the computer program code itself read from the storage medium implements the functions of the above-described embodiments, and the storage medium storing the computer program code constitutes the technology of the present disclosure. This also includes cases where an operating system running on a computer uses the above-described hardware to perform some or all of the actual processing based on instructions in the program code, thereby implementing the above-described functions.

[0024] Furthermore, the present invention may be realized in the following form: That is, computer program code read from a storage medium is written to memory provided on a function expansion card inserted into a computer or a function expansion unit connected to the computer, and the CPU provided on the function expansion card or function expansion unit then performs some or all of the actual processing based on the instructions of the computer program code, thereby realizing the above-mentioned functions.

[0025] When this embodiment is applied to the storage medium, the storage medium stores computer program code corresponding to the processing described above.

[0026] (Software configuration of image processing device) 3 is a diagram showing the software configuration of the image processing device 21. Note that the image processing devices 22 to 29 also have basically the same software configuration. The image processing device 21 has an image receiving unit 301, a setting receiving unit 302, a foreground extraction unit 303, a background generation unit 304, and a data transmission unit 305.

[0027] The image receiving unit 301 receives a captured image (input image) captured by the camera 11. The received captured image is output to the foreground extraction unit 303 and the background generation unit 304. The setting receiving unit 302 receives various settings of the image processing device 21 that performs foreground / background separation processing from the control device 40 via the hub 50, and executes settings for the foreground extraction unit 303, the background generation unit 304, the data transmission unit 305, etc. in accordance with the received various settings.

[0028] The foreground extraction unit 303 compares a background image generated based on an input image (captured image) with the input image, and extracts a foreground region from the input image based on the comparison result. There are various methods for comparing a background image with an input image. For example, pixels whose RGB values ​​differ by more than a threshold (foreground extraction threshold) between the background image and the input image may be extracted as the foreground region from the input image. Any other image comparison method may be used. The foreground region information is a 1-bit image with the same image size as the captured image obtained by capturing with an imaging device, in which the foreground region is expressed in white and the background in black. Any other information that can indicate the foreground region may be used, and the method is not particularly limited.

[0029] The background generation unit 304 determines that still pixels are background based on multiple frames of input images (captured images), and generates an image composed of still pixels as a background image. The multiple frames may be compared as long as the pixel values ​​of the same pixel can be compared, for example, two or more frames. A still pixel may be determined as, for example, a pixel whose pixel value is the same in multiple frames of input images, or within a threshold (background generation threshold). The background generation threshold is smaller than the foreground extraction threshold. The background generation unit 304 also receives settings related to background generation from the setting reception unit 302, and generates a background image based on the received settings. The settings include an update cycle setting for updating the background, and three settings related to the process of determining pixels as background: a determination time, a determination cycle, and a determination threshold.

[0030] The background update period is the period during which the existing background is updated using the generated background. For example, if 4 seconds is set as the update period, the background is updated once every 4 seconds. The judgment time is the total time for judging whether it is a background, and the judgment period is the period during which a judgment is made once whether it is a background. For example, if 64 seconds is set as the judgment time and 4 seconds is set as the judgment period, the judgment every 4 seconds is repeated 16 times before a final judgment is made whether it is a background.

[0031] For background determination, each pixel is determined once per determination period to be background or not, and during the determination time, pixels determined to be background are always incorporated into the existing background image to generate a new background image.

[0032] The determination of whether an image is background is made by comparing the image captured by the imaging device one determination cycle ago with the image captured by the current imaging device, and determining that pixels with color differences within the determination threshold are still background images. In this embodiment, colors are expressed by RGB values ​​ranging from 0 to 1023, and the determination thresholds are also set to 0 to 1023 for each RGB.

[0033] Here, we will explain how to update the background. If the judgment time is 64 seconds and the judgment cycle is 4 seconds, the image captured by the imaging device 4 seconds ago is compared with the image captured by the current imaging device once every 4 seconds, and the image that is consistently judged to be background after 16 comparisons is captured as the new background image. The pixels that are actually captured here are the pixels of the image captured by the imaging device at the 16th judgment.

[0034] (background part update function) Furthermore, for background generation, a function may be provided to set whether background updating is enabled or disabled for each region of the image. This function sets the background update target for each region by setting a partial update region setting image that has the same image size as the input image and in which valid regions are white and invalid regions are black as the background update target region. This function is called the background partial update function. This background partial update function can be set to ON or OFF. When the background partial update function is set to ON, the background is partially updated based on the preset partial update region setting image. In other words, the background of only the target region of the entire input image is updated. When the background partial update function is set to OFF, the entire region of the entire input image is subject to background updating.

[0035] Fig. 4 is a diagram for explaining the background partial update function, which shows an example of application of the background partial update function to a camera in which the image capture target is set to be a catcher.

[0036] When the partial background update function is set to OFF for camera 401 whose imaging target is set to catcher, the entire area 411 of the captured image corresponding to the entire angle of view of camera 401 is treated as the background update target. When the partial background update function is set to ON for camera 401 whose imaging target is set to catcher, only a predetermined area 412 of the entire angle of view of camera 401 is treated as the background update target.

[0037] For example, if the subject of the image capture is baseball, the background partial update function is set to ON only for the area around home base in the image captured by the imaging device. By setting it in this way, even if there is a person within the camera's angle of view, if there is no person in the area around home base, which is the area set as the background partial update target, the background can be updated.

[0038] The data transmission unit 305 transmits the captured image and the extracted foreground information.

[0039] (Control device software configuration) 5 is a diagram showing an example of the software configuration of the control device 40. The control device 40 includes a camera information storage unit 501, a camera classification information storage unit 502, a status acquisition unit 503, a UI generation unit 504, a display unit 505, a user input unit 506, a control unit 507, and a communication unit 508.

[0040] The camera information storage unit 501 stores information about the camera. This information includes camera identification information, connection group information to which the camera belongs, the camera's connection destination, and camera installation coordinate information. For example, assume that multiple cameras are connected in a daisy chain, with two connection groups. In this case, the connection group information is information indicating which of the two connection groups the camera belongs to. The camera's connection destination indicates the device connected via a transmission cable (network cable). The camera installation coordinate information is XYZ coordinate information with a predetermined position (for example, the center of the baseball field 710) as the origin.

[0041] The camera classification information storage unit 502 stores camera classification information for each of the cameras 11 to 19 and the connected image processing devices 21 to 29. The camera classification information includes information about the camera's imaging target, the camera's elevation angle, and the camera group.

[0042] (Camera classification information) 6 is a diagram showing an example of camera classification information 600. The camera classification information 600 includes information on a camera ID 610, a camera group 620, an imaging target 630, and an elevation angle 640 of the camera.

[0043] The camera classification information 600 is information that is manually set in advance by the user. In this embodiment, the classification information is set manually, but this is not limited to this. For example, the camera's posture and orientation may be automatically detected, and the camera classification information may be automatically created by identifying the imaging target using the parameters. Also, for simplicity, this embodiment will describe an example in which information about nine cameras is managed, but this is not limiting. The camera classification information may manage information about fewer than nine cameras or more than nine cameras, for example, information about approximately 100 cameras may be managed.

[0044] The camera's imaging target 630 is the object that the camera captures, and in this embodiment, baseball is used as an example, and the target is each baseball position or baseball field. The method of representing the target is not limited to this, and for example, the entire imaging area of ​​multiple cameras can be divided and each area can be numbered and represented accordingly. For example, in soccer, it is possible to divide it into areas such as penalty area, right side, left side, etc. Also, depending on the camera, multiple positions can be imaged, and multiple imaging target information can be stored.

[0045] (Camera classification) Fig. 7 is a diagram for explaining camera classification. Fig. 7 shows an example applied to baseball. System 700 has a plurality of cameras 721 set to capture an image of a pitcher (pitcher's mound) 711 in a baseball field 710, and a plurality of cameras 722 set to capture an image of a catcher (home base). That is, in the example of Fig. 7, the plurality of cameras are classified into cameras 721, 722, and 723 set to capture an image of the pitcher, catcher, and players other than the pitcher and catcher, respectively.

[0046] The camera's elevation angle 640 is the angle of the camera's imaging direction relative to the ground when the ground is used as the reference. In this system, the camera is installed higher than the ground and takes images facing downward. Therefore, a camera with an elevation angle of 30 degrees or less is considered a low angle, and one greater than that is considered a high angle.

[0047] The camera group 620 is a grouping of multiple cameras based on the target of control. For example, by classifying cameras capturing images of the catcher into the same group, it is possible to issue commands such as background update to the cameras in that group at a moment when no one is near home base. When determining which group each of the multiple cameras belongs to, the camera's angle (elevation angle) may also be used to further limit the group. That is, the camera's elevation angle at the time of capturing the image may determine which of the multiple groups each of the multiple cameras belongs to. With low-angle cameras, there is a higher risk of players standing in front of or behind home base, in addition to those near home base. Therefore, targeting only high-angle cameras can further reduce the risk of players other than the target being captured interfering with the capture. While the method described here uses camera information to determine which of the multiple groups each of the multiple cameras belongs to, this is not limiting. For example, the user may arbitrarily determine which of the multiple groups each of the multiple cameras belongs to.

[0048] Returning to the explanation of Fig. 5, the status acquisition unit 503 acquires status information of the cameras 11 to 19 and status information of the image processing devices 21 to 29, and outputs the acquired information to the UI generation unit 504.

[0049] The UI generation unit 504 generates UI images that indicate the status of the system and the camera based on the information acquired from the camera information storage unit 501, the camera classification information storage unit 502, and the status acquisition unit 503.

[0050] The display unit 505 has a frame buffer and a display panel. The display unit 505 stores (overwrites) the display image output from the UI generation unit 504 in the frame buffer. The display unit 505 then reads out the display image stored in the frame buffer at a predetermined refresh rate and displays it on the display panel. It can also be said that the display unit 505 performs display control so that a UI screen, which will be described in detail later, is displayed on the display panel. The display panel is, for example, a liquid crystal panel or an organic EL panel.

[0051] The user input unit 506 receives operation information input from a controller (not shown) and outputs the received information to the control unit 507. The controller is, for example, a keyboard, a mouse, a touch panel, a joystick, or the like.

[0052] The control unit 507 outputs control instructions to each unit of the control device 40 based on operation information from the user input unit 506. In this embodiment, the user instructs settings for background generation, and the control unit 507 executes control in accordance with the user instruction input via the user input unit 506. At that time, the control unit 507 determines (specifies) the image processing device to be controlled based on information acquired from the camera classification information storage unit 502. As a method for this specification, any of the following five methods can be used.

[0053] (Specification method 1) Designation method 1 is a method for designating a subject to be photographed. In this designation method, for example, a camera is designated to photograph a catcher. In this case, based on the camera classification information 600, cameras with camera IDs 11, 15, and 17 are designated.

[0054] (Specification method 2) Designation method 2 is a method of designating by the elevation angle of the camera. In this designation method, for example, cameras with a high elevation angle are designated. In this case, based on the camera classification information 600, cameras with camera IDs 11, 13, 14, 15, 16, 18, and 19 are designated.

[0055] (Specification method 3) Designation method 3 is a method of designating a combination of the image capture target and the camera's elevation angle. In this designation method, for example, a camera is designated when the image capture target is a catcher and the camera's elevation angle is high. In this case, based on the camera classification information 600, cameras with camera IDs 11 and 15 are designated.

[0056] (Specification method 4) Designation method 4 is a method for designating multiple imaging targets. In this designation method, for example, a camera is designated to image a catcher or pitcher. In this case, based on the camera classification information 600, cameras with camera IDs 11, 12, 15, 17, and 18 are designated.

[0057] (Specification method 5) Designation method 5 is a method of designating by camera group. In this designation method, for example, a camera in camera group 3 is designated. In this case, based on the camera classification information 600, the cameras with camera IDs 13, 14, 16, and 19 are designated.

[0058] Furthermore, the control unit 507 instructs the image processing devices 21 to 29 to start and stop foreground / background separation and to start and stop data transmission.

[0059] The communication unit 508 outputs the control information from the control unit 507 to the outside. The communication unit 508 also receives a notification indicating the result of the control, and outputs the received notification to the control unit 507.

[0060] (Background generation setting process) 8(a) is a flowchart showing the flow of the background generation setting control process. Hereinafter, the background generation setting control process will be described according to the flowchart.

[0061] In S801, the CPU 201 (user input unit 506) of the control device 40 checks (determines) whether there is a control instruction from the user. If the CPU 201 of the control device 40 determines that there is no control instruction from the user (NO in S801), it continues to wait for a control instruction. If the CPU 201 of the control device 40 determines that there is a control instruction from the user (YES in S801), the process of S802 is executed.

[0062] In S802, the CPU 201 (control unit 507) of the control device 40 checks the content of the control instruction input to the user input unit 506. The content of the control instruction is, for example, background generation settings. The control content includes a setting target and setting content. The setting target is either camera identification information in the information about the camera, or an imaging target or camera group in the camera classification information. In the case of camera identification information, it is camera 11 indicating the camera ID. In the case of designation by imaging target in the camera classification information, it is information indicating the imaging target such as a catcher or pitcher. In the case of a camera group in the camera classification information, it is camera group 1. Furthermore, the setting content is a setting related to background generation, as described above. In this case, it is assumed that the setting target is a camera whose imaging target is a catcher, and the setting content is content to set the background generation determination cycle to 32 seconds. Note that, although a case where the setting content is the background generation determination cycle will be described here, it is not limited to this. The setting contents may be the judgment time or the judgment threshold, or any two of the judgment time, the judgment cycle, and the judgment threshold, or all of the judgment time, the judgment cycle, and the judgment threshold.

[0063] In S803, the CPU 201 (control unit 507) of the control device 40 acquires the camera classification information from the camera classification information storage unit 502.

[0064] In S804, the CPU 201 (control unit 507) of the control device 40 extracts image processing devices to be controlled. Here, the cameras 11, 15, and 17 capture images of catchers, and the image processing devices 21, 25, and 27 to which the captured images are input are extracted as control devices.

[0065] In S805, the CPU 201 (control unit 507) of the control device 40 outputs the background generation setting (determination cycle set to 32 seconds) to the image processing devices 21, 25, and 27 via the communication unit 508. Note that the image processing devices receive information regarding the background generation setting in S811, which will be described later.

[0066] In S806, the CPU 201 (communication unit 508) of the control device 40 receives the setting completion notification and outputs it to the control unit 507. The CPU 201 (control unit 507) of the control device 40 determines that the background generation setting is complete, and ends the background generation setting control process shown in FIG. 8(a).

[0067] (Background generation setting process) 8(b) is a flowchart showing the flow of the background generation setting process. The background generation setting process will be explained below according to the flowchart. Note that, since the processes of the image processing devices 21, 25, and 27 are equivalent, the process of the image processing device 21 will be explained as an example.

[0068] In S811, the CPU 201 (setting receiving unit 302) of the image processing device 21 waits for background generation settings to be transmitted from the control device 40, and receives the background generation settings transmitted from the control device 40. The received background generation settings are output to the background generation unit 304.

[0069] In S812, the CPU 201 (background generation unit 304) of the image processing device 21 confirms the received background generation settings and executes the confirmed settings to set the background generation settings, thereby reflecting the background generation settings. For example, the determination cycle for determining whether or not the background is present is set to 32 seconds.

[0070] In S813, the CPU 201 (background generation unit 304) of the image processing device 21 notifies the setting reception unit 302 of the completion of the setting, and the CPU 201 (setting reception unit 302) of the image processing device 21 transmits a setting completion notification to the control device 40.

[0071] (Background generation settings) Here, a case will be described in which a UI screen including buttons associated with background update modes that combine several pre-prepared settings is used to enable faster background generation settings. That is, a case will be described in which a UI screen is displayed that shows the update method in a manner that the user can specify.

[0072] First, there are five modes available, which change the speed at which the background updates: stop, slow, medium, fast, and instant. Each of these five modes has its own associated settings.

[0073] The "Stop" mode is a mode for setting a state in which the background is not updated at all. The "Slow" mode is a mode for setting a state in which the background is updated slowly, at a speed slower than the normal speed. The "Medium" mode is a mode for setting a state in which the background is updated at a normal speed. The "Fast" mode is a mode for setting a state in which the background is updated quickly, at a speed faster than the normal speed. The "Instant" mode is a mode for setting a state in which the background is updated immediately, at which the image captured by the camera is reflected directly in the background.

[0074] (Setting parameters for each background update mode) 9 is a diagram showing example setting parameters for realizing each background update mode. The setting parameters 900 have information on the following items: background update mode 910, background update cycle (seconds) 920, determination time (seconds) 930, determination cycle (seconds) 940, and threshold 950. That is, for each of the background update modes 910, "stop," "low speed," "medium speed," "high speed," and "instant," information on the following items is linked: background update cycle (seconds) 920, determination time (seconds) 930, determination cycle (seconds) 940, and threshold 950 for determining that the background is included.

[0075] For example, if the "Stop" mode is selected for the background update mode 910, the "Background update cycle (seconds)" 920 is set to ∞ (infinity) and the background update is stopped. Since ∞ (infinity) cannot actually be set as a numerical value, for example, the background update is controlled to be stopped by setting a specific value such as 65535.

[0076] When the "slow" mode is selected for the background update mode 910, the "judgment time (seconds)" 930 is set to 64 seconds, and the background is updated at a slow speed. In other words, when the "slow" mode is selected, a judgment is made 16 times, once every four seconds, using a threshold (RGB pixel value), before a final judgment is made as to whether the background is included. This prevents players or referees who do not move much during a game from being judged as background. When the "medium" mode is selected for the background update mode 910, the "judgment time (seconds)" 930 is set to 32 seconds, and the background is updated at a medium speed. In other words, when the "medium" mode is selected, a judgment is made 8 times, once every four seconds, using a threshold (RGB pixel value), before a final judgment is made as to whether the background is included. When the "high" mode is selected for the background update mode 910, the "judgment time (seconds)" 930 is set to 16 seconds, and the background is updated at a high speed. That is, when the "high speed" mode is selected, a threshold value (RGB pixel value) is used to repeat the determination four times, once every four seconds, before a final determination is made as to whether the object is a background.

[0077] If the "instant" mode is selected for the background update mode 910, the "determination time (seconds)" 930 is set to 1 second, the "determination cycle (seconds)" 940 is set to 1 second, and the "threshold value" 950 for determination is set to 1023, and the background is updated immediately. In other words, all pixels in the captured image are determined to be background and updated.

[0078] The following describes the application of the above background update mode settings to a baseball game. For example, while a player is playing at a designated position on the field, the background is slowly updated by setting the background update mode 910 to "slow" mode. When a player leaves his designated position on the field, such as during a changeover, the background is quickly updated by setting the background update mode 910 to "fast" mode or "instant" mode. Note that although five modes have been described above, this is not a limitation. As long as the update speeds differ, there may be fewer or more than five modes. Furthermore, the values ​​set for the background update cycle (seconds) 920, determination time (seconds) 930, determination cycle (seconds) 940, and threshold 950 for the above five modes are merely examples, and other values ​​may be set.

[0079] (UI screen for background update settings) 10 is a diagram showing an example of a UI screen including a setting button that combines a setting target and a background update mode. Note that in S801 shown in FIG. 8(a), it may be determined whether or not a control instruction has been received from the user via the UI screen 1000.

[0080] The UI screen 1000 includes various buttons for setting the target camera and background update mode. That is, the UI screen 1000 selectively displays a speed designation button for designating the speed at which the background image is updated and a partial update designation button for designating a predetermined area as the target for updating the background image. By clicking buttons 1001 to 1011 on the UI screen 1000 with a mouse, the user can execute settings related to background generation associated with the clicked button. That is, the update method is set based on a user operation via the UI screen 1000.

[0081] Buttons 1001 to 1005 are buttons for setting the background generation mode for all cameras. That is, the "All Cameras Immediate" button 1001 is a button for setting the "Immediate" mode for the background update mode 910 for all cameras. The "All Cameras Fast" button 1002 is a button for setting the "Fast" mode for the background update mode 910 for all cameras. The "All Cameras Medium Speed" button 1003 is a button for setting the "Medium Speed" mode for the background update mode 910 for all cameras.

[0082] The "All Cameras Slow" button 1004 is a button for setting the "Slow" mode for the background update mode 910 for all cameras. The "All Cameras Stop" button 1005 is a button for setting the "Stop" mode for the background update mode 910 for all cameras.

[0083] The UI screen 1000 includes buttons for setting each mode for the background update mode 910 for all cameras, so that a desired background update mode can be easily set for all cameras by operating one button, thereby controlling the background generation speed for all cameras.

[0084] Buttons 1006 to 1009 are buttons for setting the background generation mode for a specific camera. That is, the "Catcher Immediate" button 1006 is a button for setting the background update mode 910 to "Immediate" mode for a camera that is set to capture the catcher. The "Catcher Slow" button 1007 is a button for setting the background update mode 910 to "Slow" mode for a camera that is set to capture the catcher. When a player disappears from near home base during a change of pitches, for example, by quickly changing the background generation mode with these buttons, it is possible to update the background even in games where there are few scenes without players.

[0085] "Pitcher Immediate" button 1008 is the same as "Catcher Immediate" button 1006, except that it is for a camera that is set to capture the pitcher. "Pitcher Slow" button 1009 is the same as "Catcher Slow" button 1007, except that it is for a camera that is set to capture the pitcher.

[0086] Buttons 1010 and 1011 are buttons for setting the background partial update function for all cameras. That is, the "All Cameras Background Partial Update ON" button 1010 is a button for enabling the background partial update function for all cameras. The "All Cameras Background Partial Update OFF" button 1011 is a button for disabling the background partial update function for all cameras.

[0087] Any one of buttons 1001-1005, any one of buttons 1006-1007, any one of buttons 1008-1009, and any one of buttons 1010-1011 may be set simultaneously. For example, "All Cameras Immediate" 1001, "Catcher Slow" 1007, "Pitcher Slow" 1009, and "All Cameras Background Partial Update ON" 1010 may be set simultaneously. In this case, the background update mode is set to "Immediate" for all cameras other than the camera capturing the catcher and pitcher, and the background update mode is set to "Slow" for the camera capturing the catcher. In addition, the background partial update function is set for all cameras. In other words, the update method, in which a partial area of ​​the captured image obtained by the camera is used to update the background image, can be set based on a user operation via the UI screen 1000.

[0088] It should be noted that the buttons 1001 to 1011 shown on the UI screen 1000 in FIG. 10 are merely an example, and buttons corresponding to imaging targets and update speeds other than the pitcher and catcher may be present on the UI screen 1000.

[0089] (UI screen for background update settings) 11 is a diagram showing an example of a UI screen including setting buttons that combine the game progress, setting target, and background update mode. Note that in S801 shown in FIG. 8(a), it may be determined via the UI screen 1100 whether a control instruction has been received from the user.

[0090] The UI screen 1100 includes various buttons for setting the target camera and background update mode as the baseball game progresses. That is, the UI screen 1100 displays selectable speed designation buttons for designating the speed at which the background image is updated as the game progresses as captured by the camera. As with the UI screen 1000, the user can execute settings related to background generation associated with the clicked button by clicking buttons 1101 to 1112 on the UI screen 1100 with the mouse. That is, the update method is set based on a user operation via the UI screen 1100.

[0091] In this embodiment, it is assumed that ground preparations are performed in the bottom of the fifth inning. Therefore, buttons 1103 to 1107 are provided on the UI screen 1100 so that the backgrounds near home base and the pitcher's mound can be updated immediately at the bottom of the fifth inning. Also, it is assumed that there is a cheering time in the bottom of the seventh inning, during which the lighting fluctuates drastically. Therefore, button 1110 is provided on the UI screen 1100 so that the background updates for all cameras can be stopped at the bottom of the seventh inning.

[0092] That is, on the UI screen 1100, an "All Cameras Fast" button 1101 and an "All Cameras Slow" button 1102 are provided for the top of the first through fifth innings. The "All Cameras Fast" button 1101 and the "All Cameras Slow" button 1102 are buttons for setting the background update mode for all cameras from the top of the first through fifth innings. The "All Cameras Fast" button 1101 is a button for setting the background update mode to "fast" mode, and the "All Cameras Slow" button 1102 is a button for setting the background update mode to "slow" mode.

[0093] The UI screen 1100 provides a "Catcher Immediate" button 1103 and a "Pitcher Immediate" button 1104 for the bottom of the fifth inning. Additionally, a "All Camera Background Partial Update ON" button 1105, a "All Camera Slow" button 1106, and a "All Camera Background Partial Update OFF" button 1107 are provided. The "Catcher Immediate" button 1103 is a button for setting the background update mode for cameras whose imaging target is the catcher during the bottom of the fifth inning to "Immediate" mode. The "Pitcher Immediate" button 1104 is a button for setting the background update mode for cameras whose imaging target is the pitcher during the bottom of the fifth inning to "Immediate" mode. The "All Camera Background Partial Update ON" button 1105 is a button for enabling background partial updates for all cameras during the bottom of the fifth inning. The "All Camera Slow" button 1106 is a button for setting the background update mode for all cameras during the bottom of the fifth inning to "Slow" mode. The "All Cameras Background Part Update OFF" button 1107 is a button for disabling background part updates for all cameras in the bottom of the fifth inning.

[0094] On the UI screen 1100, an "All Cameras Fast" button 1108 and an "All Cameras Slow" button 1109 are provided for the top of the sixth to seventh innings. The "All Cameras Fast" button 1108 and the "All Cameras Slow" button 1109 are buttons for setting the background update mode for all cameras from the top of the sixth to seventh innings. The "All Cameras Fast" button 1108 is a button for setting the background update mode to "fast" mode, and the "All Cameras Slow" button 1109 is a button for setting the background update mode to "slow" mode.

[0095] The UI screen 1100 provides a "Stop all cameras" button 1110 for the bottom of the seventh inning. The "Stop all cameras" button 1110 is a button that sets the background update mode for all cameras to "Stop" mode for the bottom of the seventh inning.

[0096] Furthermore, the UI screen 1100 provides an "All Cameras Fast" button 1111 and an "All Cameras Slow" button 1112 for the top of the eighth inning through the bottom of the ninth inning. The "All Cameras Fast" button 1111 and the "All Cameras Slow" button 1112 are buttons for setting the background update mode for all cameras from the top of the eighth inning through the bottom of the ninth inning. The "All Cameras Fast" button 1111 is a button for setting the background update mode to "fast," and the "All Cameras Slow" button 1112 is a button for setting the background update mode to "slow."

[0097] As described above, it is possible to arrange multiple buttons to which the same setting content is assigned on UI screen 1100. Also, buttons 1101 to 1112 shown on UI screen 1100 are merely examples, and buttons corresponding to imaging targets and update rates other than the pitcher and catcher may be provided on UI screen 1100.

[0098] (Background update process) 12 is a flowchart showing the flow of background update processing. For example, when no object exists in the imaging area of ​​a camera (imaging device), an image captured by the camera (imaging device) is acquired as a background image.

[0099] First, in S1201, timing is performed, that is, the elapsed time from the start of the background image update process is measured.

[0100] Next, in S1202, the CPU 201 of the image processing apparatus determines whether the determination time has elapsed, and repeats the determination in S1202 until the determination time corresponding to the set update speed (low speed, medium speed, high speed, immediate) has elapsed.

[0101] In S1203, the CPU 201 of the image processing device determines whether the background image can be updated according to the set background update method. For example, if the "high-speed" mode is set, a determination is made four times, once every four seconds, using a threshold value, before a final determination is made as to whether the image is background. If it is determined to be background, it is determined that the background image can be updated. If the CPU 201 of the image processing device determines that the background image cannot be updated (NO in S1203), it ends the background update process shown in FIG. 12. If the CPU 201 of the image processing device determines that the background image can be updated (YES in S1203), the process proceeds to S1204.

[0102] In S1204, the CPU 201 of the image processing apparatus updates the background image. When the updating of the background image is completed, the flow of the background update process shown in FIG.

[0103] Although the present embodiment has been described using baseball as an example, the present invention can be applied to other sports and other imaging subjects, and is not particularly limited to such sports as American football, rugby, basketball, soccer, cricket, racing (horse, boat, bicycle), badminton, tennis, etc.

[0104] As described above, in this embodiment, multiple cameras are classified into multiple groups based on the target of interest in the images captured by each camera, and a background generation method is set individually for each classified group. For each classified group, multiple background images are updated using images captured by one or more cameras included in the group according to the set background image update method. This makes it possible to update the background the moment there is no foreground object in the imaging area of ​​one or more cameras included in the group, and the background can be updated even if an object is present in the imaging area for a long time.

[0105] (Other embodiments) The present disclosure can also be realized by executing the following process. That is, a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or various storage media, and one or more processors in a computer of the system or device read and execute the program. The present disclosure can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0106] The disclosure of this embodiment includes the following configuration examples.

[0107] (Configuration 1) an acquisition means for acquiring a plurality of captured images obtained by imaging with a plurality of imaging means; an extraction means for extracting a foreground region from the plurality of captured images by a background subtraction method; a generating means for generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; and The generating means For each group into which the plurality of imaging means are classified based on an object of interest in imaging by each of the plurality of imaging means, the background image is updated using an image acquired by one or more imaging means included in the group in accordance with a set background image update method. 1. An image processing device comprising:

[0108] (Configuration 2) The update method is set based on a user operation via a UI screen. 2. The image processing device according to configuration 1,

[0109] (Configuration 3) As the updating method, a speed at which the background image is updated is set based on the user operation via the UI screen. 3. The image processing device according to configuration 2.

[0110] (Configuration 4) The speed at which the background image is updated is set for each of the classified groups. 4. The image processing device according to configuration 3.

[0111] (Configuration 5) The speed at which the background image is updated is set in units of progress of the game captured by the plurality of imaging means. 5. The image processing device according to configuration 3 or 4.

[0112] (Configuration 6) The speed at which the background image is updated is set to any one of a stop mode in which updating of the background image is stopped, an immediate mode in which the background image is immediately updated at a first speed, a high-speed mode in which the background image is updated at a second speed slower than the first speed, a medium-speed mode in which the background image is updated at a medium speed at a third speed slower than the second speed, and a low-speed mode in which the background image is updated at a low speed at a fourth speed slower than the third speed. 6. The image processing device according to any one of configurations 3 to 5,

[0113] (Configuration 7) When the immediate mode is set, The generating means reflects the captured image obtained by the imaging means as it is in the background image. 7. The image processing device according to configuration 6,

[0114] (Configuration 8) As the updating method, a partial area of ​​the captured image obtained by the imaging means is used as an update target for the background image, and this is set based on the user operation via the UI screen. 8. The image processing device according to any one of configurations 2 to 7,

[0115] (Configuration 9) The plurality of imaging means are classified into the groups corresponding to the elevation angles at the time of imaging of the plurality of imaging means. 9. The image processing device according to any one of configurations 1 to 8,

[0116] (Configuration 10) As the update method, any one of an update cycle for updating the background image, a determination time for determining that the image is a background image, a determination cycle for determining that the image is a background image, and a determination threshold for the background image is set. 10. The image processing device according to any one of configurations 1 to 9,

[0117] (Configuration 11) a classification means for classifying the plurality of imaging means into a plurality of groups based on an object of interest in imaging of each of the plurality of imaging means; a setting means for setting, for each of the classified groups, a method of updating a background image used to extract a foreground region by a background subtraction method from an image captured by one or more of the imaging means included in the group; and A control device comprising:

[0118] (Configuration 12) A system having an image processing device and a control device that controls the image processing device, The image processing device includes: an acquisition means for acquiring a plurality of captured images obtained by imaging with a plurality of imaging means; an extraction means for extracting a foreground region from the plurality of captured images by a background subtraction method; a generating means for generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; and The control device a setting means for setting a method for updating the background image; The setting means classifying the plurality of imaging means into a plurality of groups based on an object of interest in imaging each of the plurality of imaging means; a method for updating the background image for each of the classified groups; The generating means For each of the classified groups, the background image is updated using the captured images obtained by one or more imaging means included in the group in accordance with the set update method. A system characterized by:

[0119] (Configuration 13) an acquisition step of acquiring a plurality of captured images obtained by imaging with a plurality of imaging means; an extraction step of extracting a foreground region from the plurality of captured images by a background subtraction method; a generating step of generating a background image used in the background subtraction method while updating the background image based on the plurality of captured images; Including, In the producing step, For each group into which the plurality of imaging means are classified based on an object of interest in imaging by each of the plurality of imaging means, the background image is updated using an image acquired by one or more imaging means included in the group in accordance with a set background image update method. 2. A method for controlling an image processing apparatus comprising:

[0120] (Configuration 14) A program for causing a computer to function as the image processing device according to any one of configurations 1 to 10.

Claims

1. a plurality of imaging means for capturing images by imaging; an extraction means for extracting a foreground region based on a difference between the captured image and a background image; updating means for updating the background image based on the captured image; and The update means When an instruction to update a background image for an update target in an imaging area is received, a partial area of ​​the background image corresponding to the update target is updated based on a captured image including the update target. An image processing system comprising:

2. The update instruction is given based on a user operation via a UI screen.

2. The image processing system according to claim 1.

3. The update instruction sets how often the background image is updated.

2. The image processing system according to claim 1.

4. The frequency of updating the background image is set for each group into which the imaging means are classified, which is determined based on the update target.

4. The image processing system according to claim 3.

5. The frequency of updating the background image is set in units of progress of the game captured by the plurality of imaging means.

4. The image processing system according to claim 3.

6. The frequency of updating the background image is set to any one of a stop mode in which updating of the background image is stopped, an immediate mode in which the background image is immediately updated at a first frequency, a high-speed mode in which the background image is updated at a second frequency slower than the first frequency, a medium-speed mode in which the background image is updated at a third frequency slower than the second frequency, and a low-speed mode in which the background image is updated at a fourth frequency slower than the third frequency.

4. The image processing system according to claim 3.

7. When the immediate mode is set, The updating means updates the background image by directly reflecting the captured image obtained by the imaging means.

7. The image processing system according to claim 6.

8. 2. The image processing system according to claim 1, wherein the update instruction sets a threshold value used to determine that pixels of the captured image belong to a background region.

9. The threshold value is set for each group into which the imaging means are classified, which is determined based on the update target.

9. The image processing system according to claim 8.

10. The update means determines, based on the threshold value, an area in which there is little change in the color of pixels of the captured image for a predetermined time period as a background area.

9. The image processing system according to claim 8.

11. The update instruction sets either the determination time for determining that the image is a background image or the determination cycle for determining that the image is a background image.

2. The image processing system according to claim 1.

12. an imaging step of acquiring captured images by imaging with a plurality of imaging means; an extraction step of extracting a foreground region based on a difference between the captured image and a background image; an updating step of updating the background image based on the captured image; Including, In the updating step, When an instruction to update a background image for an update target in an imaging area is received, a partial area of ​​the background image corresponding to the update target is updated based on a captured image including the update target.

2. A control method for an image processing system comprising:

13. A program for causing a computer to function as the image processing system according to claim 1.

Images