Image processing apparatus, image processing method, and program

Abstract

To when a background image is created from a plurality of picked-up images obtained within a certain period of time, if a stationary object is present in the picked-up images, it is captured in the background image.SOLUTION: An image processing apparatus executes update processing on an update area of a created background image corresponding to a plurality of picked-up images obtained within a certain period of time, the update area specified by area information, on the basis of the plurality of picked-up images, and thereby creates a new background image. The area information is information indicating, of the background image, the update area subjected to the update processing and a non-update area not subjected to the update processing.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to a technique for generating a background image excluding a foreground from a captured image.

Background Art

[0002] There are many use cases where it is desired to faithfully extract and separate only the subject (object) of interest from a captured image. As an example, there is a use case for generating a virtual viewpoint image. A virtual viewpoint image is obtained by generating data (3D model) representing the three-dimensional shape of an object using a plurality of captured images obtained by synchronously capturing images with a plurality of different imaging devices, and imaging the 3D model with an imaging device (virtual camera) virtually arranged in the imaging space. When generating a virtual viewpoint image in this way, in order to faithfully reproduce the 3D model of the object, it is necessary to accurately separate the area of the object reflected in each captured image from the background along its contour.

[0003] As one of the methods used for this separation, there is a background difference method in which a background image in which no object exists (that is, only the background is reflected) is compared with a captured image in which an object exists (that is, the foreground and the background are reflected), and the difference between the two is extracted. In order to implement the background difference method, a background image is required. However, in a case where an object always exists within the angle of view of the imaging device, it is not possible to use a captured image at a certain moment as the background image, and it is generated by combining a plurality of captured images obtained over a certain period of time.

[0004] Here, in the generation method disclosed in Patent Document 1, first, the absolute value of the difference between a captured image at a certain time and the captured image one time before that is binarized, and a mask image is created in which the value of a pixel in which a change is detected is "0" and the value of a pixel in which no change is detected is "1". Then, for a pixel in which no change is detected, the value in the count buffer is incremented by 1, and for a pixel in which a change is detected, the value in the count buffer is reset to "0". As a result, when the value in the count buffer is greater than the threshold value, the pixel value of the captured image at that time is copied to the background image to update the background image.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0006] In the method disclosed in Patent Document 1 above, when the object to be extracted as the foreground remains stationary for a certain period of time, the position where it appears does not change in a plurality of captured images obtained during that certain period of time. As a result, the area of the object is incorporated into the background image. Then, the area of the object cannot be separated and extracted as the foreground from subsequent captured images. The present disclosure has been made in view of the above problems.

Means for Solving the Problems

[0007] An image processing apparatus according to the present disclosure is an image processing apparatus that generates a background image from a plurality of captured images obtained during a certain period of time, and includes an image acquisition unit that acquires the plurality of captured images, an information acquisition unit that acquires region information indicating an update region that is a target of update processing and a non-update region that is not a target of update processing for the background image, and an image generation unit that performs update processing on the update region specified by the region information among the generated background images corresponding to the plurality of captured images based on the plurality of captured images to generate a new background image. When the image generation means is not set to perform an update process using the area information by the setting means, the image generation means performs an update process on the entire area of the generated background image based on the plurality of captured images to generate a new background image. It is characterized by this.

Effects of the Invention

[0008] According to the present disclosure, when generating a background image from a plurality of captured images obtained during a certain period of time, it is possible to generate a background image in which the area of the object is not incorporated even when the object remains stationary for a certain period of time.

Brief Description of Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Modes for Carrying Out the Invention

[0010] Hereinafter, embodiments for carrying out the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all combinations of the features described in the present embodiment are essential for the solution means of the present invention.

[0011] [Embodiment 1] [Device Configuration] FIG. 1 is a diagram showing a logical configuration (functional configuration) of an image processing apparatus 100 that generates a background image according to the present embodiment. The image processing apparatus 100 includes an image acquisition unit 101, a region information acquisition unit 102, a background image generation unit 103, and a background image holding unit 104. The image processing apparatus 100 generates and updates a background image by using, as inputs, a plurality of captured images 110 captured by the same imaging device (not shown) over a certain period of time and update region information 111 prepared in advance, and outputs the latest background image 112. Here, the update region information is information indicating a region (update region) to be updated and a region (non-update region) not to be updated among the generated background images corresponding to the captured images. Hereinafter, each functional unit will be described.

[0012] The image acquisition unit 101 receives a plurality of captured images 110 and outputs them to the background image generation unit 103. The number of captured images used by the background image generation unit 103 for generating the background image will be described later. The region information acquisition unit 102 receives the update region information 111 and outputs it to the background image generation unit 103. The background image generation unit 103 performs an update process on the captured image input from the image acquisition unit 101, the update region information 111 input from the region information acquisition unit 102, and the background image held by the background image holding unit 104. The background image generation unit 103 outputs the updated background image to the background image holding unit 104 and also outputs the updated background image as the latest background image 112 from the image processing apparatus 100. The background image holding unit 104 holds the background image input from the background image generation unit 103 and outputs it to the background image generation unit 103 as a reference background image used in the above update process. Then, when the update process is performed by the background image generation unit 103, it receives and holds the new updated background image obtained by the update process.

[0013] Next, the hardware configuration of the image processing apparatus 100 will be described with reference to FIG. 2. The image processing apparatus 100 includes a CPU 201, a ROM 202, a RAM 203, an auxiliary storage device 204, a display unit 205, an operation unit 206, a communication I / F 207, and a bus 208. The CPU 201 controls the entire image processing apparatus 100 using computer programs and data stored in the ROM 202 and the RAM 203, and realizes the above-described respective functional units. Note that the image processing apparatus 100 may have one or more dedicated hardware different from the CPU 201, and at least a part of the processing by the CPU 201 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The ROM 202 stores programs and the like that do not require modification. The RAM 203 temporarily stores programs and data supplied from the auxiliary storage device 204, data supplied from the outside via the communication I / F 207, and the like. The auxiliary storage device 204 is configured by, for example, a hard disk drive or the like, and stores various data. In the RAM 203 and the auxiliary storage device 204, in addition to various data such as a plurality of captured images 110, update area information 111, the latest background image 112, and the generated background image held for updating in FIG. 1, intermediate data generated at the time of update, various setting information, and the like are stored. The display unit 205 is configured by, for example, a liquid crystal display LED or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the image processing apparatus 100. The operation unit 206 is configured by, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and receives an operation by the user and inputs various instructions to the CPU 201. The CPU 201 operates as a display control unit that controls the display unit 205 and an operation control unit that controls the operation unit 206. The communication I / F 207 is used for communication with a device external to the image processing apparatus 100. For example, when the image processing apparatus 100 is connected to an external device by wire, a communication cable is connected to the communication I / F 207. When the image processing apparatus 100 has a function of performing wireless communication with an external device, the communication I / F 207 includes an antenna.The bus 208 connects each part of the image processing apparatus 100 and transmits information. In the present embodiment, it is assumed that the display unit 205 and the operation unit 206 are inside the image processing apparatus 100. However, at least one of the display unit 205 and the operation unit 206 may exist outside the image processing apparatus 100 as another device.

[0014] <Operation of the Image Processing Apparatus> Subsequently, the process of generating and updating the background image according to the present embodiment will be described with reference to FIGS. 3 to 5. FIG. 3 is a flowchart showing the flow of the background image generation process, and FIGS. 4 and 5 are diagrams for explaining the background image generation method. Before entering the detailed description of the flowchart of FIG. 3, the background image generation method in the present embodiment will be described using the explanatory diagrams of FIGS. 4 and 5.

[0015] ≪Background Image Generation Method> FIG. 4(a) shows a generated background image 400 in the case where a soccer game is an imaging scene. In the background image 400, a field 4001, a goal 4002, and a spectator stand 4003 (hereinafter, these are collectively referred to as “background components”) that are treated as the background are shown. FIG. 4(b) shows an imaging image 410 corresponding to a certain time t during the game. In the imaging image 410, in addition to the background components shown in FIG. 4(a), each object such as a goalkeeper 4011, a field player 4021, a ball 4031, and a field player 4041 that is in the foreground is shown. FIG. 4(c) shows an imaging image 420 corresponding to time t+1, which is captured after a certain period of time has elapsed from the imaging time t of the imaging image 410. The same objects as those in the imaging image 410 are shown in the imaging image 420, and the reference numerals 4021′, 4031′, and 4041′ with “′” attached indicate that the objects are not shown at the same positions (moved from time t to time t+1). That is, it means that the goalkeeper 4011 has not moved during the period from time t to time t+1, but the two field players 4021 and 4041 and the ball 4031 have moved. The background image can be obtained by comparing pixels in corresponding relationships for a plurality of imaging images with different imaging timings, determining a pixel region where the change in pixel values is equal to or less than a threshold value as the background, and updating the background image held by the background image holding unit 104. Note that, for example, the background image 400 can also be obtained by imaging the field 4001 in a state where there are no players or balls to be treated as the foreground, such as before the start of the game. The flowchart of FIG. 3 described later may be started using, as an initial background image, an image captured when there are no foreground objects such as before the start of the game and held by the background image holding unit 104. When generating (updating) a new background image by inputting the generated background image 400, and two imaging images 410 and 420 corresponding to a certain time t and a time t+1 that has advanced temporally from there, the following procedures 1 to 2 are followed.

[0016] <<Procedure 1>> First, between the captured image 410 and the captured image 420, the pixel values at corresponding pixel positions are compared, and portions where the difference in pixel values is less than a certain threshold and those where it is not are identified. FIG. 5(a) shows a binary difference image obtained based on the result of such comparison. In the difference image 510 of FIG. 5(a), a pixel region where the difference in pixel values is less than the threshold is shown in black (0), and a pixel region where it is equal to or greater than the threshold is shown in white (1). Now, among all the pixel regions of the captured image 420, in the pixel regions where the field player 4021, the ball 4031, and the field player 4041 existed in the captured image 410, these objects do not exist. Therefore, for these pixel regions, it is determined that the difference in pixel values is equal to or greater than the threshold. Then, among all the pixel regions of the captured image 420, the pixel regions for which it is determined that the difference in pixel values is equal to or greater than the threshold become the difference regions 5021, 5031, and 5041 represented in white (1) in the difference image 510. Similarly, among all the pixel regions of the captured image 420, the pixel regions of the field player 4021', the ball 4031', and the player 4041' do not have these objects in the captured image 410. Therefore, these pixel regions become the difference regions 5022, 5032, and 5042 represented in white (1) in the difference image 510.

[0017] <<Procedure 2>> Next, using the difference image 510 in Fig. 5(a), the generated background image 400 in Fig. 4(a) is updated. Specifically, the portion other than the difference region in the difference image 510 (= the region where the pixel value difference is less than a certain value) is determined as the background region, and the background image 400 is updated using the pixel values of the pixel region on the captured image corresponding to the black pixel region in the difference image 510. Here, either the captured image 410 or the captured image 420 may be used for the update, but it is usually desirable to use the captured image corresponding to a later time. Alternatively, the average value of the pixel values of the corresponding pixels in both captured images may be used. Fig. 5(b) shows the background image obtained by updating the background image 400 using the difference image 510. In the updated background image 520 shown in Fig. 5(b), the keeper 4011 that did not change its position in the captured images 410 and 420 has been incorporated into a part of the background. That is, in this case, the generation (update) of the background image has failed.

[0018] The present disclosure aims to prevent an object that should originally be treated as a foreground from being incorporated into the background image as a result of the object remaining stationary for a certain period of time, as in the above-described case.

[0019] Note that the method for generating the background image is not limited to the above. For example, each pixel value of the captured image may be acquired and accumulated for a certain period, and the pixel value of the background image may be updated by statistical processing. That is, the distribution of pixel values is obtained for a plurality of captured images captured during a certain period, classified into a plurality of groups consisting of similar pixel values, and the average value or the most frequent value of the group with the highest appearance frequency may be used as the representative value and the pixel value of the background image. Alternatively, an image at a moment when no object exists may be held in advance as a reference background image, and only the pixel values that approximate the pixel values in the reference background image among the pixel values of a plurality of captured images captured during a certain period may be used as the pixel values of the updated background image. Furthermore, these methods may be used in combination according to the characteristics of the object to be imaged.

[0020] ≪Operation flow≫ Next, along with the flowchart of FIG. 3, the flow of the background image generation process in this embodiment will be described in detail. The flowchart shown in FIG. 3 shows the specific process flow when updating the background image at a certain time (frame). Each time the process of this flowchart is executed, one background update is completed. The update frequency is determined according to the specifications of the system side (the system that performs foreground-background separation or color-coding of the background model) that uses the latest background image 112, and is executed at a predetermined update frequency. Also, before the execution of this flow starts, it is assumed that the update area information prepared in advance by the user is read from the auxiliary storage device 204 by the area information acquisition unit 102 and expanded in the RAM 203 so that the CPU 201 can refer to it. In the following description, the symbol "S" means step.

[0021] In S301, the image acquisition unit 101 acquires a plurality of captured images captured within a certain period of time. The plurality of captured images to be acquired are images obtained by capturing with a certain imaging device at different times under the same imaging conditions, such as the captured image 410 in FIG. 4(b) and the captured image 420 in FIG. 4(c) described above. Here, the "certain period of time" is, for example, about 0.5 seconds to 5 seconds in the case of a sports game such as soccer. Now, assume that the number of captured images used for the update process is 2. Here, if the frame rate of the imaging device is 60 fps and the "certain period of time" is 5 seconds, then out of the 300 frames of captured images obtained in 5 seconds, for example, two captured images, namely the 0th frame and the 299th frame, are acquired. Note that the number of captured images to be acquired may be 3 or more. The larger the number, the more effectively it is possible to prevent the situation where an object that happened to be stationary for a certain period of time is incorporated into the background. Also, when determining the "certain period of time", the moving speed of the object to be imaged may be considered. For example, based on the average size and moving speed of the object, a time interval may be set such that the objects do not overlap. However, even if the "certain period of time" is appropriately set in this way, it is impossible to completely prevent an object that continues to be stationary for a long time from being incorporated into the background image. In addition, since structures or the like existing in the imaging space may be deformed or moved during the imaging period, it is necessary to periodically update the entire background image at a time interval longer than the above "certain period of time".

[0022] The following steps from S302 to S307 are executed by the background image generation unit 103. First, in S302, the background image to be updated is read from the background image holding unit 104. As described above, the background image to be updated is the current background image (the one generated by the most recent update process) held by the background image holding unit 104. Here, it is assumed that the background image 400 showing only the background structure shown in FIG. 4(a) is read out.

[0023] In S303, depending on whether a setting is made to perform partial update (hereinafter referred to as "partial update") targeting only a part of the background image, S304 or S307 is executed (branch step). Here, whether to perform partial update is set based on, for example, a user instruction via a UI screen (not shown) where execution / non-execution of partial update can be selected. And in this step, control is performed such that it is branched by referring to the flag information indicating the presence or absence of execution of partial update set based on the user instruction. If partial update is to be performed, S304 is executed next, and if partial update is not to be performed, S307 is executed next.

[0024] In S304 when partial update is to be performed, an area to be updated (processing target area) in the current background image read from the background image holding unit 104 is determined based on the update area information. Here, taking the case of the background image 400 as an example, the manner in which the processing target area is determined will be described. As described above, in the captured image 410 corresponding to the time t shown in FIGS. 4(b) and 4(c) and the captured image 420 corresponding to the time t + 1, the moving field players 4021, 4041 and the ball 4031 and the stationary keeper 4011 are mixed. Therefore, update area information is created in advance so that background update is not performed for an area where the stationary keeper 4011 is likely to continue to be stationary so that the stationary keeper 4011 is not captured in the background image, and background update is performed for an area not including the goal 4002. FIG. 4(d) is a diagram showing an example of a binary mask image as the update area information. The mask image 430 in FIG. 4(d) is composed of an update area 431 shown by hatching and a non-update area 432 without hatching. And FIG. 4(e) is a diagram showing the area in the captured image 410 corresponding to the update area 431 with a solid line frame 411, and FIG. 4(f) is a diagram showing the area in the captured image 410 corresponding to the non-update area 432 with a broken line frame 412.

[0025] Note that the update area information only needs to be able to identify the updated area and the non-updated area in the background image, and its format is not limited to the mask image. For example, it may be two-dimensional coordinate information that can identify the updated area and the non-updated area respectively. Also, in the example of the mask image 430 in FIG. 4(d), the non-updated area is one, but it may also be multiple. As a case where there are multiple non-updated areas, for example, in a baseball game, a case where places with relatively little movement such as the umpire's position and the catcher's position are scattered in the captured image can be considered. In this step, based on such update area information, the processing target area for updating the pixel values in the current background image is determined.

[0026] In S305, background update is performed on the processing target area determined in S304 by applying the aforementioned background image generation method. As a result, only the pixel values of the pixels belonging to the processing target area of the current background image are updated.

[0027] In S306, it is determined whether the update process has been completed for all the updated areas specified by the update area information. If there are unprocessed updated areas, the process returns to S304, the next processing target area is determined, and the process continues. On the other hand, if the update process has been completed for all the updated areas indicated by the update area information, then S308 is executed next.

[0028] In S307 when partial update is not performed, the entire area of the current background image read from the background image holding unit 104 is used as the processing target area, and update processing is performed by applying the aforementioned background image generation method. As a result, the pixel values of each pixel constituting the entire area of the current background image are updated as necessary.

[0029] In S308, the background image generation unit 103 outputs the updated background image obtained through the processing so far as the latest background image 112. Also, the updated background image is output to the background image holding unit 104 and held in the background image holding unit 104 in preparation for the next update process. The above is the content of the background image generation process of this embodiment.

[0030] <Modification Example 1> If there is no timing when a stationary object exists, it is appropriate to perform an update process on the entire area of the background image. For example, in the aforementioned soccer example, when the goalkeeper 4011 is playing while moving away from the goal 4002 and is not stationary in front of the goal 4002, the goalkeeper 4011 will not be captured in the background even without performing partial updates. Therefore, for example, the timing when a specific object that may be stationary within a certain period of time is not within the update area can be detected by image analysis such as object detection, or if it is real-time processing, the operator may also judge by looking at the imaging situation. And when there is no specific object within the update area, the setting may be dynamically changed so as to perform an overall update instead of a partial update.

[0031] <Modification Example 2> For an image area where there is a high possibility that an object moving during a certain period of time exists, background updating may be performed in different ways according to the type of the object. FIGS. 6(a) to 6(f) are diagrams for explaining an application example of this modification. FIG. 6(a) is the same as FIG. 4(a) described above, and shows a background image in which only background components (4001 to 4003) are shown and no foreground object is shown. FIG. 6(b) shows a captured image 610 corresponding to a certain time t during the game. Now, in the captured image 610, each object such as a goalkeeper 6011, a field player 6021, a ball 6031, and a field player 6041 that are in the foreground are shown. Further, in the spectator stand 4003 in the captured image 610, four spectators 6051 are shown. FIG. 6(c) shows a captured image 620 corresponding to time t+1, which is captured after a certain period of time has elapsed since the capture of the captured image 610. The same objects as those in the captured image 610 are shown in the captured image 620, and the reference numerals 6021', 6031', and 6041 with '' are all moving by more than one body length from time t to time t+1. Also, 6011'' with "" has moved by about half of the body from time t to time t+1. And the spectator 6051 is shown at the same position in the captured image 610 and the captured image 620. That is, it has been stationary from time t to time t+1.

[0032] In this modified example, based on the type of object reflected in the captured image, an update method for each region in the background image is determined. First, for a region where there is a high possibility that only objects of a type with a high moving speed exist, it is set as a region (short-time update region) where background update is performed at a relatively short time (e.g., 0.5 seconds) as the "fixed time" in S301. On the other hand, for a region where there is a high possibility that objects of a type with a low moving speed exist (long-time update region), it is set as a region where background update is performed at a relatively long time (e.g., 5.0 seconds) as the "fixed time" in S301. Further, for a region where there is a high possibility that stationary or objects that can be regarded as the same (≒ moving speed zero) exist, it is set as a region where background update is not performed (non-update region). In the above application example, the solid-line frame 631 in FIG. 6(d) indicates the region in the captured image 610 corresponding to the short-time update region, and the dashed-line frame 632 in FIG. 6(e) indicates the region in the captured image 610 corresponding to the long-time update region. Also, the dashed-dotted-line frame 633 in FIG. 6(f) indicates the region in the captured image 610 corresponding to the non-update region.

[0033] In this modified example, update region information that can identify these three types of regions is prepared in advance and read before the start of the background image update process. FIG. 7 is a flowchart showing the flow of the background image generation process according to this modified example. Hereinafter, the background image generation process according to this modified example will be described with reference to the flowchart of FIG. 7.

[0034] In S701, a plurality of captured images for the short-time update region and a plurality of captured images for the long-time update region are acquired at the timings of the fixed time set for the short-time update region and the fixed time set for the long-time update region, respectively.

[0035] Since each of steps S702 to S704 is the same as S302 to S304 in the flowchart of FIG. 3 described above, the description thereof will be omitted. When the update target area in the background image is determined based on the update area information in S704, in the next S705, a process of allocating an update method is performed according to whether the determined update target area is a short-time update area or a long-time update area (branch step). Specifically, if the update target area is a short-time update area, S706 will be executed next, and if the update target area is a long-time update area, S707 will be executed next.

[0036] In S706, using a plurality of captured images acquired at short time intervals in S701, background update is performed for the processing target area determined in S704 by applying a background image generation method capable of relatively quickly capturing changes in the background part due to the passage of time and deformation into the background image. In the case of a short-time update area, since the assumed object is an object with a high moving speed and / or a large moving amount, it is also less likely that a part of the object overlaps between the captured images to be compared. Therefore, for example, a generation method in which a difference value (threshold value) for detecting the presence or absence of a change in pixel value is set relatively small is applied.

[0037] In S707, using a plurality of captured images acquired at long time intervals in S701, background update is performed for the processing target area determined in S704 by applying a background image generation method capable of relatively slowly capturing changes in the background part due to the passage of time and deformation into the background image. In the case of a long-time update area, since the assumed object is an object with a low moving speed and / or a small moving amount, it is also less likely that a part of the object overlaps between the captured images to be compared. Therefore, for example, a generation method in which a difference value (threshold value) for detecting the presence or absence of a change in pixel value is set relatively large is applied. Each of steps S708 to S710 is the same as S306 to S308 in the flowchart of FIG. 3 described above, and thus the description thereof will be omitted. Note that this modified example and the above-described modified example 1 may be combined.

[0038] <Modified Example 3> In the above-described embodiments, the update region information has been described as being prepared in advance by a user or the like according to the imaging scene. However, for example, it may be dynamically generated based on the change in the pixel value of each pixel constituting the captured image. The method in the case of dynamic generation will be described below.

[0039] FIGS. 8(a) and 8(b) are graphs with the pixel value (in the case of 8 bits, “0 to 255”) on the vertical axis and time (seconds) on the horizontal axis, showing the change in the pixel value of a certain pixel in the captured image over a predetermined time (for example, several tens of seconds). Here, the pixel value is, for example, each component value (R value, G value, B value) in the captured image expressed in the RGB color space or their average value. FIG. 8(a) corresponds to an object with a relatively slow moving speed, and FIG. 8(b) corresponds to an object with a relatively fast moving speed. And, the double-headed arrow 802 in FIG. 8(a) and the double-headed arrow 812 in FIG. 8(b) indicate the range of pixel values representing the color of the original background, and the double-headed arrow 803 in FIG. 8(a) and the double-headed arrows 813 / 814 in FIG. 8(b) indicate the range of pixel values representing colors other than the background.

[0040] When the moving speed of the object is slow, as shown by the solid line 801 in FIG. 8(a), the period during which the pixel value represents a color other than the background becomes relatively long, and the number of appearances per unit time during that period decreases. On the other hand, when the moving speed of the object is fast, as shown by the solid line 811 in FIG. 8(b), the period during which the pixel value represents a color other than the background becomes relatively short, and the number of appearances per unit time during that period increases. Therefore, analyze the tendency of pixel value change per unit time for the captured images obtained in time series, determine the region composed of pixel groups close to the solid line 801 in FIG. 8(a) as the long-time update region, and determine the region composed of pixel groups close to the solid line 811 in FIG. 8(b) as the short-time update region. Thereby, the update region information can be obtained automatically.

[0041] In the case of this modification example, regardless of the type of the object, a background image in which an object that remains stationary for a certain period of time is not captured can be generated. Needless to say, the color space of the captured image is not limited to the RGB color space, and other image formats may also be used.

[0042] As described above, according to this embodiment including each modification example, even when an object has been stationary for a certain period of time in a partial area of a captured image, a background image can be generated and updated without the object being incorporated into the background.

[0043] (Other Embodiments) The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiment to a system or apparatus via a network or a storage medium, and having one or more processors in a computer of the system or apparatus read and execute the program. It can also be realized by a circuit (for example, ASIC) that realizes one or more functions.

[0044] Further, the present disclosure includes the following configurations and methods.

[0045] [Configuration 1] An image processing apparatus that generates a background image from a plurality of captured images obtained during a certain period of time, image acquisition means for acquiring the plurality of captured images; information acquisition means for acquiring region information indicating an update region that is a target of update processing and a non-update region that is not a target of update processing for the background image; image generation means for generating a new background image by performing update processing on the update region specified by the region information among the generated background images corresponding to the plurality of captured images based on the plurality of captured images; An image processing apparatus characterized by comprising:

[0046] [Configuration 2] further comprising setting means for setting whether to perform update processing using the region information, wherein, when the setting means sets to perform update processing using the region information, the image generation means performs update processing on the update region to generate a new background image; The image processing apparatus according to Configuration 1, characterized by the above.

[0047] [Configuration 3] It further has display means for displaying a UI screen for the user to instruct the above settings, The setting means performs the setting based on a user instruction via the UI screen. The image processing apparatus according to Configuration 2, characterized in that.

[0048] [Configuration 4] The image processing apparatus according to Configuration 3, characterized in that the setting means performs the setting based on a user instruction performed via the UI screen before the image acquisition means starts acquiring the plurality of captured images.

[0049] [Configuration 5] The image processing apparatus according to Configuration 3, characterized in that the setting means performs the setting based on a user instruction via the UI screen on which the plurality of acquired captured images are displayed after the image acquisition means starts acquiring the plurality of captured images.

[0050] [Configuration 6] It further has detection means for detecting whether there is a specific object in the update area specified by the area information of the plurality of captured images acquired by the image acquisition means, When the specific object is not detected by the detection means, the setting means performs a setting not to perform an update process using the area information. The image processing apparatus according to Configuration 2, characterized in that.

[0051] [Configuration 7] When the setting for performing the update process using the area information is not set by the setting means, the image generation means performs an update process targeting the entire area of the generated background image based on the plurality of captured images to generate a new background image. The image processing apparatus according to any one of Configurations 2 to 6, characterized in that.

[0052] [Configuration 8] It further has holding means for holding the generated background image. The image generation means generates the new background image from the generated background image held by the holding means. The image processing apparatus according to any one of Configurations 1 to 7, characterized in that.

[0053] [Configuration 9] The update process is a process of comparing pixels in a corresponding relationship included in the target area of the update process for the plurality of captured images, and changing the pixel value of the corresponding pixel in the generated background image for pixels whose pixel value difference is equal to or less than a threshold value. The image processing apparatus according to any one of Configurations 1 to 8, characterized in that.

[0054] [Configuration 10] The area information includes a first update area corresponding to a first type of object with a relatively large moving speed and a second update area corresponding to a second type of object with a relatively small moving speed. The fixed time for acquiring the plurality of captured images used for the update process for the first update area is shorter than the fixed time for acquiring the plurality of captured images used for the update process for the second update area. The image processing apparatus according to Configuration 9, characterized in that.

[0055] [Configuration 11] The threshold value used for the update process for the first update area is smaller than the threshold value used for the update process for the second update area. The image processing apparatus according to Configuration 10, characterized in that.

[0056] [Configuration 12] The update process is a process of acquiring and accumulating each pixel value of the plurality of captured images for a certain period of time, and changing the pixel value in the generated background image by statistical processing. The image processing apparatus according to any one of Configurations 1 to 8, characterized in that.

[0057] [Configuration 13] The update process is a process of obtaining the distribution of pixel values for the plurality of captured images, classifying them into a plurality of groups consisting of similar pixel values, and setting the average value or the mode value of the group with the highest occurrence frequency as the pixel value of the new background image. The image processing apparatus according to any one of Configurations 1 to 8, characterized in that.

[0058] [Configuration 14] The update process is a process of previously holding an image at a moment when no object exists as a reference background image, and using only the pixel values in the plurality of captured images that approximate the pixel values in the reference background image as the pixel values of the updated background image. The image processing apparatus according to any one of Configurations 1 to 8, characterized in that.

[0059] [Configuration 15] The image processing apparatus further includes generation means for generating the region information based on changes in pixel values of each pixel of the plurality of captured images. The information acquisition means acquires the region information generated by the generation means. The image processing apparatus according to any one of Configurations 1 to 9, characterized in that.

[0060] [Configuration 16] The region information is a mask image representing the updated region and the non-updated region in binary values. The image processing apparatus according to any one of Configurations 1 to 12, characterized in that.

[0061] [Method 1] An image processing method for generating a background image from a plurality of captured images obtained during a certain period of time, An image acquisition step of acquiring the plurality of captured images, An information acquisition step of acquiring region information indicating an updated region that is a target of an update process and a non-updated region that is not a target of the update process for the background image, An image generation step of generating a new background image by performing an update process on the updated region specified by the region information among the generated background images corresponding to the plurality of captured images based on the plurality of captured images. An image processing method characterized by including

[0062] [Configuration 17] A program for causing a computer to function as the image processing apparatus according to any one of Configurations 1 to 16.

Claims

1. An image processing apparatus for generating a background image from a plurality of captured images obtained over a certain period of time, comprising: image acquisition means for acquiring the plurality of captured images; information acquisition means for acquiring region information indicating an update region to be subject to an update process and a non-update region not to be subject to the update process for the background image; setting means for setting whether to perform an update process using the region information; image generation means for, when the setting means is set to perform an update process using the region information, performing an update process on the update region specified by the region information among the generated background images corresponding to the plurality of captured images based on the plurality of captured images to generate a new background image; characterized by comprising: when the setting means is not set to perform an update process using the region information, the image generation means performs an update process on the entire region of the generated background image based on the plurality of captured images to generate a new background image. An image processing apparatus characterized by the above.

2. further comprising display means for displaying a UI screen for a user to instruct the setting, wherein the setting means performs the setting based on a user instruction via the UI screen. The image processing apparatus according to claim 1, characterized by the above.

3. The image processing apparatus according to claim 2, characterized in that the setting means performs the setting based on a user instruction performed via the UI screen before the image acquisition means starts acquiring the plurality of captured images.

4. The image processing apparatus according to claim 2, characterized in that the setting means performs the setting based on a user instruction via the UI screen on which the plurality of acquired captured images are displayed after the image acquisition means starts acquiring the plurality of captured images.

5. further comprising detection means for detecting whether a specific object is present in the update region specified by the region information of the plurality of captured images acquired by the image acquisition means, wherein the setting means performs a setting not to perform an update process using the region information when the detection means does not detect the specific object. The image processing apparatus according to claim 1, characterized by the above.

6. further comprising holding means for holding the generated background image. The image generation means generates the new background image from the generated background image held by the holding means. The image processing apparatus according to claim 1, characterized in that.

7. The update process compares pixels in a corresponding relationship included in the target area for the plurality of captured images, and changes the pixel value of the corresponding pixel in the generated background image for pixels whose pixel value difference is equal to or less than a threshold value. The image processing apparatus according to claim 1, characterized in that it is a process.

8. The area information includes a first update area corresponding to a first type of object with a relatively high moving speed and a second update area corresponding to a second type of object with a relatively low moving speed. The fixed time for acquiring the plurality of captured images used in the update process for the first update area is shorter than the fixed time for acquiring the plurality of captured images used in the update process for the second update area. The image processing apparatus according to claim 7, characterized in that.

9. The threshold value used in the update process for the first update area is smaller than the threshold value used in the update process for the second update area. The image processing apparatus according to claim 8, characterized in that.

10. The update process acquires and accumulates the pixel values of each pixel of the plurality of captured images for a certain period, and changes the pixel value in the generated background image by statistical processing. The image processing apparatus according to claim 1, characterized in that it is a process.

11. The update process acquires the distribution of pixel values for the plurality of captured images, classifies them into a plurality of groups consisting of similar pixel values, and sets the average value or the most frequent value of the group with the highest appearance frequency as the pixel value of the new background image. The image processing apparatus according to claim 1, characterized in that it is a process.

12. The update process holds an image at a moment when no object exists in advance as a reference background image, and only pixel values approximated to the pixel values in the reference background image among the pixel values of the plurality of captured images are used as the pixel values of the updated background image. The image processing apparatus according to claim 1, characterized in that it is a process.

13. The image processing apparatus further includes a generation means for generating the area information based on the change in the pixel value of each pixel of the plurality of captured images. The information acquisition means acquires the area information generated by the generation means. The image processing apparatus according to claim 1, characterized in that.

14. The area information is a mask image representing the updated area and the non-updated area in binary values. The image processing apparatus according to claim 1, characterized in that.

15. An image processing apparatus that generates a background image from a plurality of captured images obtained over a certain period of time, Image acquisition means for acquiring the plurality of captured images, Information acquisition means for acquiring area information indicating an updated area that is a target of update processing and a non-updated area that is not a target of update processing for the background image, Setting means for setting whether to perform update processing using the area information, When the setting means is set to perform update processing using the area information, image generation means for generating a new background image by performing update processing on the updated area specified by the area information among the generated background images corresponding to the plurality of captured images based on the plurality of captured images, Detection means for detecting whether a specific object exists in the updated area specified by the area information of the plurality of captured images acquired by the image acquisition means, having, When the specific object is not detected by the detection means, the setting means performs a setting not to perform update processing using the area information The image processing apparatus characterized by that.

16. An image processing method for generating a background image from a plurality of captured images obtained over a certain period of time, An image acquisition step of acquiring the plurality of captured images, An information acquisition step of acquiring area information indicating an updated area that is a target of update processing and a non-updated area that is not a target of update processing for the background image, A setting step of setting whether to perform update processing using the area information, When the setting step is set to perform update processing using the area information, an image generation step of generating a new background image by performing update processing on the updated area specified by the area information among the generated background images corresponding to the plurality of captured images based on the plurality of captured images, including, In the image generation step, when the setting step is not set to perform update processing using the area information, update processing for the entire area of the generated background image is performed based on the plurality of captured images to generate a new background image. The image processing method characterized by that.

17. A program for causing a computer to execute the image processing method according to claim 16.

Images