Surveillance image generation system, image processing device, image processing method, and program

Abstract

An image processing device (100) comprises: an acquisition unit (102) that acquires a plurality of images of the same location taken at different timings; a selection unit (104) that compares at least two of the plurality of images to select regions of interest, which are regions where a mutual difference satisfies a criterion; and a processing unit (106) that performs an averaging process to average the regions of interest contained in each of the at least two images.

Description

[Technical Field]

[0001] The present invention relates to a monitoring image generating system, an image processing device, an image processing method, and a program. [Background technology]

[0002] There are various technologies for removing people (or objects) other than the target of surveillance from images captured by surveillance cameras. In particular, when images captured by surveillance cameras are stored for a certain period of time, it is often desirable to remove people from the images from the perspective of personal privacy.

[0003] For example, Patent Document 1 describes that in an image processing device for a surveillance system, in order to accurately capture the appearance of a monitored object, images of moving objects such as passersby and short-term residents are removed from multiple still images taken of the surveillance area in chronological order, and the presence or absence of changes in long-term residents within the surveillance area is determined.

[0004] Patent Document 2 describes a device for detecting differences between images that improves the accuracy of determining whether or not there is a difference between a target image and a reference image. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] Japanese Patent Application Laid-Open No. 2010-278963 [Patent Document 2] JP 2018-78454 A Summary of the Invention [Problem to be solved by the invention]

[0006] Generally, when a plurality of time-series images are averaged and portions where movement has occurred are blurred, a faint human figure often remains in the image after averaging.

[0007] The present invention has been made in view of the above circumstances, and an object of the present invention is to provide an image processing technique that makes it difficult for people to remain in an image. [Means for solving the problem]

[0008] In order to solve the above-mentioned problems, each aspect of the present invention employs the following configuration.

[0009] The first aspect relates to an image processing device. An image processing device according to a first aspect includes: an acquisition means for acquiring a plurality of images taken at different times of the same location; a selection means for comparing at least two of the plurality of images and selecting a target region where the difference between the images satisfies a criterion; and a processing means for performing an averaging process for averaging the target regions included in each of the at least two images.

[0010] A second aspect relates to an image processing method implemented by at least one computer. An image processing method according to a second aspect includes: The image processing device Acquire multiple images taken at different times of the same location, comparing at least two of the plurality of images and selecting a target region where the difference between the images satisfies a criterion; performing an averaging process for averaging the target regions included in each of the at least two images.

[0011] Another aspect of the present invention may be a program that causes at least one computer to execute the method of the second aspect, or a computer-readable recording medium on which such a program is recorded. This recording medium includes a non-transitory tangible medium. The computer program comprises computer program code which, when executed by a computer, causes the computer to perform the image processing method on an image processing device.

[0012] Any combination of the above components, and any transformation of the present invention into a method, device, system, recording medium, computer program, etc., are also valid aspects of the present invention.

[0013] Furthermore, the various components of the present invention do not necessarily have to be independent entities, but may be formed as a single member by multiple components, one component may be formed from multiple components, one component may be part of another component, or part of one component may overlap with part of another component, etc.

[0014] Furthermore, although the method and computer program of the present invention describe a number of steps in a sequential order, the order in which the steps are described does not limit the order in which the steps are executed. Therefore, when implementing the method and computer program of the present invention, the order of the steps can be changed as long as it does not cause any problems in terms of the content.

[0015] Furthermore, the multiple steps of the method and computer program of the present invention are not limited to being executed at different times, and therefore, a step may occur while another step is being executed, or the execution timing of a step may partially or completely overlap with the execution timing of another step, etc. [Effects of the Invention]

[0016] According to the above aspects, it is possible to provide an image processing technique that makes it difficult for people to appear in an image. [Brief explanation of the drawings]

[0017] [Figure 1] 1 is a diagram conceptually illustrating a system configuration of a monitoring image generating system according to an embodiment of the present invention. [Figure 2] 2 is a block diagram illustrating an example of the hardware configuration of a computer that realizes the image processing device of the monitoring image generation system shown in FIG. 1. FIG. [Figure 3] 1 is a functional block diagram logically illustrating the configuration of an image processing apparatus according to an embodiment. [Figure 4] FIG. 10 is a diagram for explaining image averaging processing. [Figure 5] FIG. 10 is a diagram for explaining image averaging processing. [Figure 6] 10 is a flowchart illustrating an example of an operation of the image processing device. [Figure 7] 10A and 10B are diagrams for explaining a process for removing a person area from a monitoring image. [Figure 8] 10 is a flowchart illustrating an example of the operation of the image processing apparatus according to the embodiment. [Figure 9] FIG. 10 is a diagram for explaining image averaging processing. [Figure 10] FIG. 10 is a diagram for explaining weighted averaging processing. [Figure 11] 10A and 10B are diagrams illustrating an example of a data structure of result information and an update status thereof. DETAILED DESCRIPTION OF THE INVENTION

[0018] Hereinafter, embodiments of the present invention will be described with reference to the drawings. In all drawings, similar components are designated by similar reference numerals, and their description will be omitted where appropriate. In addition, in each drawing, configurations of parts that are not related to the essence of the present invention are omitted and are not shown.

[0019] In the embodiments, "acquisition" includes at least one of the following: a device going to retrieve data or information stored in another device or storage medium (active acquisition), and inputting data or information output from another device into the device (passive acquisition). Examples of active acquisition include making a request or inquiry to another device and receiving a reply, and accessing and reading information from another device or storage medium. An example of passive acquisition is receiving information that is distributed (or transmitted, pushed, etc.). Furthermore, "acquisition" may also mean selecting and acquiring data or information from received data or information, or selecting and receiving distributed data or information.

[0020] (First embodiment) <System configuration> FIG. 1 is a diagram conceptually showing the system configuration of a monitoring image generating system 1 according to an embodiment of the present invention. The surveillance image generation system 1 aims to generate surveillance images of a store or the like that do not capture people such as customers. The surveillance image generation system 1 includes a camera 5 that captures images of a location to be monitored, and an image processing device 100. The image processing device 100 has a storage device 110. The storage device 110 is, for example, a hard disk, a solid state drive (SSD), or a memory card. The storage device 110 may be a device included inside the image processing device 100, a device separate from the image processing device 100, or a combination of these. The storage device 110 may be, for example, so-called online storage.

[0021] The storage device 110 stores images captured by the camera 5, monitoring images generated by the image processing device 100, and various information generated in the process of generating the monitoring images.

[0022] 1, the surveillance image generation system 1 generates surveillance images captured inside a store such as a convenience store. For example, the camera 5 captures images of areas such as a cash register counter area where a POS register 10 is installed, and a product display area where display shelves 20 on which products are displayed are installed.

[0023] The generated monitoring images are preferably used to monitor, for example, the increase or decrease in the number of products on the display shelves 20, and therefore do not include people such as customers or store clerks. However, the purpose of using the generated monitoring images is not limited to this. For example, the monitoring images may be used to identify the display status of products on the display shelves 20 or to monitor the freshness of food and ingredients.

[0024] The POS register 10 is a device where at least one of a customer and a store clerk performs product registration processing and / or transaction processing. The display shelf 20 is a fixture having at least one shelf or surface on which products are placed, a fixture that displays products by hanging them, a refrigerated or frozen showcase, a gondola, etc., but is not limited to this. Although only one POS register 10 and one display shelf 20 are shown in Fig. 1, there may be multiple of each.

[0025] The camera 5 includes an imaging element such as a lens and a CCD (Charge Coupled Device) image sensor. The camera 5 may be a network camera that communicates with the image processing device 100 via the communication network 3, or may be a camera that is not connected to the communication network 3.

[0026] 1 shows only one camera 5, but there may be provided a plurality of cameras 5. The images generated by the camera 5 are at least one of moving images, still images, and frame images at predetermined intervals.

[0027] The image generated by the camera 5 may be transmitted directly to the image processing device 100, or may not be transmitted directly from the camera 5. The image generated by the camera 5 may be temporarily stored in a storage device (which may be the storage device 110 or another storage device (including a recording medium)), and the image processing device 100 may read the image from the storage device sequentially or at predetermined intervals. Furthermore, the image transmitted to the image processing device 100 may be a moving image, a frame image at predetermined intervals, or a still image sampled at predetermined intervals.

[0028] <Hardware configuration example> FIG. 2 is a block diagram illustrating an example of the hardware configuration of a computer 1000 that realizes the image processing device 100 of the monitoring image generating system 1 shown in FIG.

[0029] The computer 1000 includes a bus 1010 , a processor 1020 , a memory 1030 , a storage device 1040 , an input / output interface 1050 , and a network interface 1060 .

[0030] The bus 1010 is a data transmission path for transmitting and receiving data among the processor 1020, memory 1030, storage device 1040, input / output interface 1050, and network interface 1060. However, the method of connecting the processor 1020 and the like to each other is not limited to bus connection.

[0031] The processor 1020 is implemented by a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), or the like.

[0032] The memory 1030 is a main storage device realized by a RAM (Random Access Memory) or the like.

[0033] The storage device 1040 is an auxiliary storage device realized by an HDD (Hard Disk Drive), an SSD (Solid State Drive), a memory card, a ROM (Read Only Memory), or the like. The storage device 1040 stores program modules that realize each function of the image processing device 100 of the surveillance image generation system 1 (for example, the acquisition unit 102, the selection unit 104, and the processing unit 106 in FIG. 3 , which will be described later). The processor 1020 loads each of these program modules into the memory 1030 and executes them, thereby realizing each function corresponding to the program module. The storage device 1040 also functions as a memory unit (not shown) that stores various information used by the image processing device 100. The storage device 110 may also be realized by the storage device 1040.

[0034] The program module may be recorded on a recording medium. The recording medium on which the program module is recorded may include a non-transitory, tangible medium usable by the computer 1000, and the program code readable by the computer 1000 (processor 1020) may be embedded in the medium.

[0035] The input / output interface 1050 is an interface for connecting the computer 1000 to various input / output devices.

[0036] The network interface 1060 is an interface for connecting the computer 1000 to a communication network 3. This communication network 3 is, for example, a LAN (Local Area Network) or a WAN (Wide Area Network). The method for connecting the network interface 1060 to the communication network 3 may be a wireless connection or a wired connection. However, the network interface 1060 may not be used in some cases.

[0037] The computer 1000 is connected to necessary devices (for example, a camera 5, a display (not shown), an operation unit (not shown), etc.) via an input / output interface 1050 or a network interface 1060.

[0038] The monitoring image generating system 1 may be realized by a plurality of computers 1000 that constitute the image processing device 100.

[0039] Each component of the image processing device 100 of this embodiment shown in Fig. 3, which will be described later, is realized by any combination of hardware and software of the computer 1000 shown in Fig. 2. Those skilled in the art will understand that there are various variations in the realization method and device. The functional block diagram showing the image processing device 100 of each embodiment shows logical functional blocks rather than a hardware-based configuration.

[0040] <Example of functional configuration> FIG. 3 is a functional block diagram logically showing the configuration of the image processing device 100 of this embodiment. The image processing device 100 includes an acquisition unit 102, a selection unit 104, and a processing unit . The acquisition unit 102 acquires multiple images taken at the same location at different times. The selection unit 104 compares at least two of the multiple images and selects a target region where the difference between them satisfies a criterion. The processing unit 106 performs averaging processing to average the target regions included in each of the at least two images.

[0041] The locations to be photographed include the product display area, the area around the cash register, etc. For example, the captured images can be used to detect if a product is out of stock or if the product display state is disordered, and the store clerk can be instructed to replenish the products or organize the products on the display shelves 20.

[0042] The timing of the photographing is a predetermined sampling interval, such as one minute, five minutes, or ten minutes, and may be set depending on the subject to be photographed. This is because the length of time customers stay in front of a product varies depending on the type of store, its location, the area within the store, the types of products displayed, and the like. The length of time customers stay in front of a product varies depending on the type of store, such as a convenience store, a department store, or a bookstore. Generally, customers stay shorter in a convenience store than in a department store, and longer in a bookstore than in a department store. Alternatively, the length of time customers stay may vary depending on the store's location, such as in front of a station, along a main road, in a busy shopping district, a tourist spot, or in a residential area. For example, customers are more likely to stay shorter in a store near a station than in other stores.

[0043] In addition, the amount of time customers spend in a store varies between the area where products are displayed and the area in front of the cash register, and also varies depending on the type of product (sales area) displayed. For example, in a convenience store, the area for magazines is often used for other products (e.g., groceries). Area Furthermore, whether or not a cash register is crowded varies depending on the store and the area within the store, and even within the same store or area, it may differ depending on the time of day.

[0044] Furthermore, since there are places (areas) within a single image where people tend to stay and places (areas) where people do not tend to stay, it may be possible to set the sampling interval according to the area within the image. This configuration will be described in detail in the embodiment below.

[0045] The area to be compared by the selection unit 104 unit is, for example, 1 pixel. However, pixels unit For example, the comparison may be performed in an area including the surrounding pixels. unit This can prevent small noises from occurring compared to the processing in

[0046] 4 and 5 are diagrams for explaining image averaging. FIG. 4(a) shows an example of a surveillance image of a POS register 10 in a store. FIG. 4(b) shows a customer approaching the POS register 10 and operating it. FIG. 4(c) compares the images of FIG. 4(a) and FIG. 4(b), and areas where the difference does not meet the criteria (non-target areas) are shown in black. FIG. 4(d) shows the result of combining the images of FIG. 4(a) and FIG. 4(b), and areas where the difference between the two images does not meet the criteria (non-target areas) remain black without being averaged.

[0047] The manner in which the above image processing is performed on a pixel-by-pixel basis will be explained using FIG. 5(a) shows the latest image P1 and an image P2 taken one minute earlier, which are two adjacent pixels A and B. Pixel A in image P1 and pixel A' in image P2 are in the same region, and pixel B in image P1 and pixel B' in image P2 are in the same region. The selection unit 104 compares pixel A in image P1 with pixel A' in image P2, and also compares pixel B in image P1 with pixel B' in image P2 (step S1).

[0048] In this embodiment, each pixel is represented by an RGB value. For example, the selection unit 104 compares each value and determines whether the difference in at least one of the values satisfies a criterion. For example, an area where the difference in at least one of the values is equal to or less than the criterion may be selected as the target area. The criterion may be, for example, a difference of 100 or less. This criterion is an example and is not limited to this. The criterion may be set according to the monitoring target. For example, the criterion may be a value that allows the difference between the color of the product and the color of the product's background to be detected with a predetermined accuracy or higher. Alternatively, the criterion may be that the distribution range (or distance) of the two RGB values is within a predetermined range (predetermined distance).

[0049] FIG. 5(b) shows an image obtained by combining the regions of images P1 and P2. In this example, the difference between pixel A in image P1 and pixel A' in image P2 is less than 100, which satisfies the criterion. Therefore, pixel A in image P1 and pixel A' in image P2 are selected and added as a target region. Specifically, the RGB values of pixel A in image P1 and pixel A' in image P2 are added together. On the other hand, the difference between pixel B in image P1 and pixel B' in image P2 is greater than 100 in terms of both the R and B values. Therefore, the region of pixel B is not selected (non-target region) and is excluded from the combined image (step S3). To exclude pixel B from the addition, in the example of FIG. 5(b), the RGB values of pixel B are set to 0 (shown as (0,0,0) in the figure) and added together.

[0050] Figure 5(c) shows each region (pixel A and pixel B) of image Ps1 after averaging. The RGB values added in step S3 are divided by the number of images to which they were added (here, 2 for images P1 and P2) to determine the average value of each value (step S5). The region of pixel A (target region) in image Ps1 after averaging has been subjected to averaging, but the region of pixel B (non-target region) has been excluded from the averaging.

[0051] In addition, in the embodiment, the averaging process is performed using two images, but the present invention is not limited to this, and the averaging process may be performed using two or more images.

[0052] <Example of operation> The operation of the image processing device 100 configured as above will be described below. Fig. 6 is a flowchart showing an example of the operation of the image processing device 100. First, the image processing device 100 sets a counter i to 1 (step S101). Then, the acquisition unit 102 acquires the latest image P1 (Pi) and the image P2 (Pi+1) one minute before the latest image P1 (Pi) (step S103).

[0053] The selection unit 104 compares the two images P1 and P2 (step S105). Here, the processes of steps S107 to S109 are performed for each of the multiple regions in the image. For each region, the selection unit 104 determines whether the difference satisfies a criterion, in this case, whether the difference is equal to or less than the criterion (step S107). The selection unit 104 selects the region where the difference satisfies the criterion, in this case, the region where the difference is equal to or less than the criterion, as the target region (YES in step S107), and the processing unit 106 adds the selected target regions, that is, the region of image P1 and the region of image P2, and performs averaging (step S109). Of the multiple regions in images P1 and P2, the region where the difference does not satisfy the criterion, in this case, the region where the difference exceeds the criterion (NO in step S107), becomes a non-target region, is not selected, and the process bypasses step S109 and proceeds to step S111.

[0054] 7A and 7B are diagrams illustrating the process of removing a person region from a surveillance image. For example, as shown in Fig. 7A, among multiple surveillance images P1 to Pn (n is a natural number), moving object regions R1 and R2 exist in the center between image P2 and image P3. These moving object regions R1 and R2 represent, for example, customers moving around in a store.

[0055] Figure 7(b) shows the images after excluding areas where the difference between the two images does not meet the criteria, and Figure 7(c) shows the combined image after averaging.

[0056] As shown in Figure 7(b), when images P1 and P2 are compared, moving object regions R1 and R2 are excluded as regions whose difference does not satisfy the criteria, resulting in image P2', in which the unselected regions (non-target regions) are shown in black. In composite image Ps1, which is obtained by adding the selected regions of images P1 and P2' as target regions and averaging them, black regions remain that have not been averaged.

[0057] Returning to FIG. 6, in step S111, counter i is incremented, and it is determined whether counter i exceeds a predetermined number N (step S113). Here, the predetermined number N is the number of times that image averaging is performed, and is preset to, for example, 10. However, the number of times N that averaging is performed is not limited to this. If counter i exceeds N (YES in step S113), this processing ends. If counter i does not exceed N (NO in step S113), the processing returns to step S103, and the acquisition unit 102 acquires image P2 from one minute ago and image P3 from two minutes ago.

[0058] Then, the selection unit 104 compares the image P2 with the image P3 (step S105). Here, the processes of steps S107 to S109 are performed for each of the multiple regions in the image. For each region, the selection unit 104 determines whether the difference satisfies a criterion, in this case, whether the difference is equal to or smaller than the criterion (step S107). The selection unit 104 selects the region where the difference satisfies the criterion, in this case, the region where the difference is equal to or smaller than the criterion, as the target region (YES in step S107), and the processing unit 106 adds the region of image P2 and the region of image P3, which are the selected target regions, and performs averaging (step S109).

[0059] As a result, as shown in Figure 7(b), when images P2 and P3 are compared, areas whose difference does not meet the criterion are excluded from image P3', and the non-selected non-target areas are shown in black. As shown in Figure 7(c), in composite image Ps2 obtained by adding and averaging the selected target areas of images P2 and P3', black areas remain that have not been averaged. On the other hand, target areas whose difference meets the criterion have been averaged.

[0060] Returning to Fig. 6, counter i is further incremented (step S111), and the process returns to step S103 to repeat the process, resulting in composite images Ps3 and Ps4, as shown in Fig. 7(c). In this way, in images P1 to P5, the moving object regions R1 and R2 that existed between images P2 and P3 are no longer present in image Ps4 generated by the averaging process. In other words, an image is generated in which the moving object customer that was captured in the image has been erased.

[0061] As described above, in this embodiment, the selection unit 104 compares multiple images captured at different times at the same location and acquired by the acquisition unit 102, selects areas whose differences satisfy a criterion as target areas, and the processing unit 106 performs averaging processing to average the target areas included in each of the two images. As a result, according to this embodiment, areas with large differences in the images can be excluded from the averaging processing, making it possible to remove customers who appear temporarily from the image. Furthermore, because the image obtained as a result of the averaging processing does not include areas with large differences, it is possible to prevent noise (temporarily present objects or people) from being introduced into the generated image.

[0062] (Second embodiment) This embodiment is the same as the above embodiment except that a termination criterion for the averaging process is set. The image processing device 100 of this embodiment has the same configuration as the above embodiment, and will be described using FIG. 3. This embodiment can also be combined with other embodiments described later.

[0063] In the image processing device 100, the selection unit 104 compares at least two images by changing the combination of images to be compared, and the processing unit 106 repeats the averaging process until averaging is performed on an area in the image that is equal to or larger than the reference range.

[0064] The reference range may be, for example, a predetermined percentage (e.g., 90%) of the entire image area, or a predetermined percentage (e.g., 90%) of a predetermined area within the image, such as the area in front of the POS register 10 or the display shelf 20, or a specific area therein (e.g., the area of a specific product). Also, different criteria may be set for each predetermined area within the image. For example, the display shelf and product areas may be 99%, and the aisles and background may be 80%.

[0065] 8 is a flowchart showing an example of the operation of the image processing device 100 of this embodiment. The processing procedure of this embodiment further includes step S121 in addition to steps S101 to S113 of the flowchart of FIG. 6 of the above embodiment.

[0066] Figure 8 If the counter i does not exceed the predetermined number N (NO in step S113), the image processing device 100 determines whether the averaging process has been completed for the area equal to or greater than the reference range (step S121). This determination process may be performed by at least one of the acquisition unit 102, the selection unit 104, and the processing unit 106, and may be performed by any of the acquisition unit 102, the selection unit 104, and the processing unit 106.

[0067] If the averaging process has not been completed for the area above the reference range (NO in step S121), the process returns to step S103 and is repeated. If the averaging process has been completed for the area above the reference range (YES in step S121), the process ends.

[0068] A specific example will be described using FIG. 9. A case will be described in which an image is divided into an area of the display shelf 20 and an area of two aisles (first and second aisles) and processed. In this way, the image processing device 100 may perform image analysis processing on the image, distinguishing areas within the image into people, background, display shelves, and merchandise, and processing each area separately. The image analysis processing may be performed by an image analysis processing device (not shown), which may be included in the image processing device 100, may be a device separate from the image processing device 100, or may be a combination of these.

[0069] Figure 9 shows the state of each area in the most recent image up to the image taken eight minutes ago. Products were present on the display shelves 20 in the images taken four minutes ago, but the products have disappeared since three minutes ago. Also, people are sometimes visible in the display shelves 20 and in the areas of each aisle in the images. When no people are present in the display shelves 20 or in the areas of each aisle in the images, the background or the display shelves 20 is visible.

[0070] First, in the latest image, no products are visible in the area of display shelf 20, but a person is visible in the second aisle. In the image one minute ago, a person is visible in the area of display shelf 20, but no people are visible in the first or second aisle. Therefore, in the comparison result between the latest image and the image one minute ago, the area of display shelf 20 and the area of the second aisle are excluded, and the area of the first aisle is averaged as the target area.

[0071] In the image taken two minutes ago, no products are visible in the area of display shelf 20, but a person is visible in the first aisle. Therefore, when comparing the image taken one minute ago with the image taken two minutes ago, the area of display shelf 20 and the area of the first aisle are excluded, and the area of the second aisle is treated as the target area and averaged.

[0072] In the image taken three minutes ago, no products are visible in the area of display shelf 20, but a person is visible in aisle 2. Therefore, when comparing the image taken two minutes ago with the image taken three minutes ago, the areas of aisles 1 and 2 are excluded, and the area of display shelf 20 is averaged as the target area.

[0073] As a result, averaging has been completed for all three regions in the image, and the image processing device 100 ends the averaging process. Processing of images from four minutes ago and thereafter can be omitted. As a result, in this example, images from four minutes ago and thereafter, in which products are present on the display shelves 20, are not added to the averaging process, making it possible to generate an image that shows the latest state in which no products are present on the display shelves 20, and also reducing the processing load.

[0074] Furthermore, if processing of the area above the reference range is not completed even after averaging processing has been performed a predetermined number of times (for example, 10 times), image generation at that time may be deemed to have failed, and an image at another time may be acquired and processed again. Furthermore, the image processing device 100 may further include means (not shown) for recording or outputting (notifying) that image generation has failed.

[0075] According to this embodiment, the same effects as those of the above embodiment are achieved, and since the processing is terminated once the averaging process has been performed on the area above the reference range, even if the averaging process has not been performed on the entire area of the image, the averaging process can be terminated once the processing has been completed on the necessary areas, thereby reducing the processing load. Furthermore, when using images to check the display status, it is desirable that no afterimages of the products remain, and this embodiment is also effective in this regard.

[0076] (Third embodiment) This embodiment is the same as the first and second embodiments except that it has a configuration in which images are weighted in averaging processing. The image processing device 100 of this embodiment has the same configuration as the embodiment in Fig. 3, so it will be described using Fig. 3. This embodiment will be described using an example of a configuration combined with the second embodiment, but it may also be combined with other embodiments.

[0077] When performing the averaging process, the processing unit 106 weights each image using the difference on the time axis from the most recent image.

[0078] 10 is a diagram illustrating the averaging process when weighting according to this embodiment is performed. In this example, averaging is performed using images taken every minute. In this example, the weighting coefficients are set to decrease from 10, 9, 8, . . . , 2, and 1 for images taken from the most recent image up to 9 minutes ago, going back in time.

[0079] In other words, by performing image processing with a high degree of trust (weighting) in more recent information (images), the current situation can be more accurately reflected in the image. For example, in the case of an image of the display shelf 20 after a customer has taken out an item to purchase, an image that accurately shows the current situation after the item has been removed can be generated by weighting the new image after the item has been removed and performing averaging processing, rather than adding to the averaging processing the past image in which the item was present.

[0080] As shown in FIG. 10, the weighted result is closer to the latest image than the unweighted result.

[0081] Furthermore, the selection unit 104 repeatedly selects two images that are adjacent to each other in time series, and the processing unit 106 performs averaging processing every time the selection unit 104 selects two images. Here, the averaging processing by the processing unit 106 is expressed by equation (1).

number

[0082] In this embodiment, every time two images are selected, averaging is performed using equation (1). Therefore, the processing unit 106 stores the calculation results up to the previous time in the storage device 110 as result information 120, and updates the result information 120 stored in the storage device 110 every time averaging is performed.

[0083] As shown in FIG. 11, the result of the averaging process (result information 120) includes, for each target region, information indicating the first term (numerator of formula (1)) indicating the sum of values obtained by multiplying the value ci of the target region by a weighting coefficient ki, and the second term (denominator of formula (1)) indicating the sum of the weighting coefficients ki used in the multiplication. Here, i is a natural number, and i=1 for the most recent image in chronological order. ki is a weighting coefficient, and the coefficient ki used for the most recent image in chronological order has a larger value. N is the number of samples of images to be averaged. If averaging is completed for regions equal to or larger than the reference range before the number of samples N, the averaging is terminated even if i is smaller than the number of samples N.

[0084] When performing the averaging process on the next two images, the processing unit 106 adds the first and second terms of the target area of the current image to the result of the averaging process (result information 120) stored in the storage device 110.

[0085] For example, when averaging is performed on images from the most recent image to those five minutes before, each term is added to and updated in the result information 120 each time calculation is performed, as shown in FIG. The result of comparing the latest image with the image from one minute ago is X1 = (10 × c1 + 9 × c2) / (10 + 9) (Figure 11(a)). The image comparison results from 1 minute and 2 minutes ago are added to X1, resulting in X2 = (10 × c1 + 9 × c2 + 8 × c3) / (10 + 9 + 8) (Figure 11(b)). The image comparison results from 2 minutes and 3 minutes ago are added to X2, resulting in X3 = (10 × c1 + 9 × c2 + 8 × c3 + 7 × c4) / (10 + 9 + 8 + 7) (Figure 11(c)). In the comparison of the images from 3 and 4 minutes ago, the area in the image from 4 minutes ago is excluded because the difference exceeds the standard, so the corresponding term is not added and the previous value is maintained (Figure 11(d)). X4=(10×c1+9×c2+8×c3+7×c4) / (10+9+8+7) The image comparison results from 4 minutes and 5 minutes ago are added to X4, resulting in X5 = (10 × c1 + 9 × c2 + 8 × c3 + 7 × c4 + 5 × c6) / (10 + 9 + 8 + 7 + 5) (Figure 11(e)).

[0086] Here, the values stored in the result information 120 are the position information of the target region of each image Pi and the sum of the first and second terms for the numerator and denominator, but they may also be the values of the individual terms before the sum of the first and second terms. Alternatively, the result information 120 may store the position information of the region of each image Pi, the RGB values ci, the weighting coefficients ki, and information indicating whether or not to add, in association with each other.

[0087] According to this embodiment, the same effects as those of the above embodiment are achieved, and since averaging is performed by assigning a larger weight to newer images and a smaller weight to images with larger differences, the current situation of the monitored object can be accurately reflected in the generated image. However, it does not have to be the "current" situation; when processing past images, the situation of the image at the time processing began will be used.

[0088] (Fourth embodiment) This embodiment differs from the above-described embodiments in that it has a configuration for setting a sampling interval for an image to be processed. The image processing device 100 of this embodiment has the same configuration as the embodiment of Fig. 3, and will be described using Fig. 3. This embodiment will be described using an example of a configuration combined with the third embodiment, but it can be combined with other embodiments as long as no contradictions arise.

[0089] The processing unit 106 sets the sampling interval of the image depending on the region and performs averaging. The sampling interval may be a predetermined value or may be dynamically changed.

[0090] Furthermore, the processing unit 106 processes past images to calculate the time until a change equal to or greater than a reference value occurs in the region, and sets the calculated time as the sampling interval for each region.

[0091] In this way, the sampling interval may be set for each region in the image. For example, the frequency with which moving objects (customers or store clerks) appear, the duration of their stay, and the timing of their appearance may differ depending on the location, and the frequency and timing of replacement (sales of products) of monitored objects (for example, specific products) may differ depending on the object and time of day, so the accuracy of image processing can be improved by setting an appropriate sampling interval according to the conditions for each object.

[0092] Furthermore, the frequency of appearance of moving objects and the product sales situation also change depending on the time of day, such as weekdays and holidays, whether there is an event (campaign, sale), working hours, daytime, nighttime, etc. Therefore, the sampling interval may be set according to the time of day, such as weekdays and holidays, whether there is an event (campaign, sale), working hours, daytime, nighttime, etc.

[0093] Although the embodiments of the present invention have been described above with reference to the drawings, these are merely examples of the present invention, and various other configurations can also be adopted. For example, in the above embodiment, the weighting coefficient is set depending on time factors, but in another example, if the difference in change between images is large, for example, if it exceeds a predetermined standard, the weighting coefficient may be set small (for example, 0.1, etc.) The weighting coefficient according to the time series may be further multiplied by this coefficient, or only this coefficient may be used without using the weighting coefficient according to the time series.

[0094] This configuration ensures that highly variable image conditions do not affect the average image.

[0095] In the above embodiment, the processing is performed using RGB values, but the hue and brightness of the image may also be used. If the change in hue of the image is below a reference value and the change in brightness is above a reference value, the selection unit 104 determines that the difference is below the reference value.

[0096] For example, there may be cases where the RGB values do not accurately represent the difference, such as when an image region includes a location where sunlight enters from outdoors. For this reason, depending on the conditions, the selection unit 104 may perform the determination process using hue and brightness instead of RGB values. Furthermore, the processing unit 106 may also perform averaging process using hue and brightness instead of RGB values. Alternatively, both processing using RGB values (determination or averaging process) and processing using hue and brightness (determination or averaging process) may be performed. For example, the selection unit 104 may select the target region by excluding regions where the difference does not satisfy the criterion in at least one of the determination results.

[0097] The conditions may be, for example, the time of day when sunlight is shining in, the season, or the weather. For example, under conditions such as a sunny afternoon, hue and brightness may be used instead of RGB values.

[0098] According to this configuration, even if it is difficult to detect the difference between images using RGB values due to lighting conditions, the accuracy of the difference detection can be improved by using hue and brightness. improvement It can be done.

[0099] Note that values expressed by a color representation method other than the RGB values or hue and brightness may also be used. For example, color spaces such as YUV, YCbCr, and YPbPr may also be used. These color spaces allow color information to be represented using a reduced number of bits per pixel, thereby reducing the amount of data in the image to be processed. Furthermore, when using an image to check the display status of products, if it is known that there is a high contrast between the product display location and the product, the selection unit 104 may determine whether the criterion is met based on whether the difference in luminance (Y signal) is below a certain level, rather than using color difference signals (U signal and V signal in the case of YUV).

[0100] Other color representation methods, such as the CMYK (Cyan Magenta Yellow Key plate) color model, the CIE (Commission Internationale de l'Eclairage) XYZ color space, the xyY color system, the L*u*v* color system, and the L*a*b* color system, may also be used to determine differences or perform averaging. The color representation method to be used may be selected appropriately depending on the color properties of the monitored object in the image. Furthermore, the color representation method used may be changed depending on the object (product, background, person) in the image area.

[0101] In the above embodiment, the averaging process is performed using two chronologically adjacent images, but this is not limiting. For example, for a region for which averaging has not been completed after averaging the latest image with the image one minute ago and the image one minute ago with the image two minutes ago, the latest image may be compared with the image three minutes ago and the resulting region may be averaged.

[0102] This configuration makes it possible to generate an image that is closer to the most recent state.

[0103] Although the present invention has been described above with reference to the embodiments and examples, the present invention is not limited to the above-described embodiments and examples. Various modifications that can be understood by those skilled in the art can be made to the configuration and details of the present invention within the scope of the present invention. In the present invention, when information about a user is acquired and used, it shall be done lawfully.

[0104] Some or all of the above-described embodiments can be described as, but are not limited to, the following supplementary notes. Below, examples of reference forms are given. 1. An acquisition means for acquiring multiple images taken at different times of the same location; a selection means for comparing at least two of the plurality of images and selecting a target region where the difference between the images satisfies a criterion; and processing means for performing averaging of the target regions included in each of the at least two images. 2. In the image processing device described in 1., until the averaging process is performed on an area in the image that is equal to or larger than the reference range. the selection means compares the at least two images by changing the combination of the images to be compared; The processing means repeats the averaging process. 3. In the image processing device according to 1. or 2., The image processing device, wherein the unit of the area is one pixel. 4. In the image processing device according to any one of 1. to 3., The image processing device wherein the processing means weights the image using a difference on a time axis from the latest image when performing the averaging process. 5. In the image processing device described in 4., the selecting means repeatedly selects two images that are adjacent to each other in time series; the processing means performs the averaging process each time the selection means selects the two images; the result of the averaging process includes, for each of the target regions, information indicating a first term indicating a value obtained by multiplying the value of the target region by a weighting coefficient, and a second term indicating the weighting coefficient used for the multiplication, and is stored in a storage means; When performing the averaging process on the next two images, the processing means adds the first term and the second term of the target region of the current image to the result of the averaging process stored in the storage means. Image processing device. 6. In the image processing device according to any one of 1. to 5., The processing means sets a sampling interval for the image according to the area and performs the averaging process. 7. In the image processing device described in 6., The processing means processes past images to calculate the time until a change greater than a reference value occurs in the region, and sets the calculated time as the sampling interval for each region. 8. In the image processing device according to any one of 1. to 7., The image processing device, wherein the sampling intervals of the plurality of images differ depending on the subject being photographed. 9. In the image processing device according to any one of 1. to 8., The image processing device, wherein the selection means determines that the difference is equal to or smaller than the standard when the change in hue of the image is equal to or smaller than the standard and the change in brightness is equal to or larger than the standard.

[0105] 10. An image processing device; a surveillance camera that captures images of the same location at different times and generates multiple images; The image processing device includes: an acquisition means for acquiring the plurality of images generated by the surveillance camera; a selection means for comparing at least two of the plurality of images and selecting a target region where the difference between the images satisfies a criterion; and a processing means for performing an averaging process to average the target regions included in each of the at least two images. Surveillance image generation system. 11. In the monitoring image generating system according to 10., until the averaging process is performed on an area in the image that is equal to or larger than the reference range. In the image processing device, the selection means compares the at least two images by changing the combination of the images to be compared; The processing means repeats the averaging process. 12. In the monitoring image generating system according to 10. or 11., A monitoring image generating system, wherein the unit of the area is one pixel. 13. In the monitoring image generating system according to any one of 10. to 12., A surveillance image generating system, wherein the processing means of the image processing device weights the image using a difference on the time axis from the latest image when performing the averaging process. 14. In the monitoring image generating system according to 13., In the image processing device, the selecting means repeatedly selects two images that are adjacent to each other in time series; the processing means performs the averaging process each time the selection means selects the two images; the result of the averaging process includes, for each of the target regions, information indicating a first term indicating a value obtained by multiplying the value of the target region by a weighting coefficient, and a second term indicating the weighting coefficient used for the multiplication, and is stored in a storage means; A surveillance image generation system in which, when performing the averaging process on the next two images, the processing means adds the first and second terms of the target area of the current image to the result of the averaging process stored in the storage means. 15. In the monitoring image generating system according to any one of 10. to 14., In the image processing device, A monitoring image generating system, wherein the processing means sets a sampling interval for the image according to the area and performs the averaging process. 16. In the monitoring image generating system according to 15., In the image processing device, The processing means processes past images to calculate the time until a change greater than a reference value occurs in the area, and sets the calculated time as the sampling interval for each area. 17. In the monitoring image generating system according to any one of items 10 to 16, A monitoring image generating system, wherein the sampling intervals of the plurality of images vary depending on the subject being photographed. 18. In the monitoring image generating system according to any one of items 10 to 17, In the image processing device, The selection means determines that the difference is below the standard when the change in hue of the image is below the standard and the change in brightness is above the standard.

[0106] 19. An image processing device Acquire multiple images taken at different times of the same location, comparing at least two of the plurality of images and selecting a target region where the difference between the images satisfies a criterion; performing an averaging process for averaging the target regions included in each of the at least two images; Image processing methods. 20. In the image processing method according to 19., The image processing device until the averaging process is performed on an area in the image that is equal to or larger than the reference range. comparing the at least two images by changing the combination of the images to be compared; The image processing method further comprises repeating the averaging process. 21. In the image processing method according to 19. or 20., An image processing method in which the unit of the area is one pixel. 22. In the image processing method according to any one of items 19 to 21, The image processing device An image processing method, wherein, when performing the averaging process, the image is weighted using a difference on the time axis from the latest image. 23. In the image processing method according to 22., The image processing device Repeatedly select two images that are adjacent to each other in time sequence, The averaging process is performed each time the two images are selected; the result of the averaging process includes, for each of the target regions, information indicating a first term indicating a value obtained by multiplying the value of the target region by a weighting coefficient, and a second term indicating the weighting coefficient used for the multiplication, and is stored in a storage means; The image processing device An image processing method, wherein when performing the averaging process on the next two images, the first term and the second term of the target area of the current image are added to the result of the averaging process stored in the storage means. 24. In the image processing method according to any one of items 19 to 23, The image processing device an image processing method in which a sampling interval of the image is set according to the region, and the averaging process is performed; 25. In the image processing method according to 24, The image processing device An image processing method comprising: processing past images to calculate a time until a change equal to or greater than a reference value occurs in the region; and setting the calculated time as the sampling interval for each region. 26. In the image processing method according to any one of 19. to 25., An image processing method, wherein the sampling intervals of the plurality of images vary depending on the subject being photographed. 27. In the image processing method according to any one of 19. to 26., The image processing device When the change in hue of the image is equal to or less than a standard and the change in brightness is equal to or more than a standard, the difference is determined to be equal to or less than a standard.

[0107] 28. To the computer, A procedure for acquiring multiple images taken at different times of the same location; comparing at least two of the plurality of images and selecting a target region where the difference between the two images satisfies a criterion; a program for executing a procedure for performing an averaging process for averaging the target regions included in each of the at least two images. 29. In the program described in 28., until the averaging process is performed on an area in the image that is equal to or larger than the reference range. a step of comparing the at least two images by changing the combination of the images to be compared; A program for causing a computer to execute the procedure of repeating the averaging process. 30. In the program described in 28. or 29., The unit of the area is one pixel, program. 31. In the program according to any one of 28. to 30., A program for causing a computer to execute a procedure for weighting the image using a difference on the time axis from the latest image when performing the averaging process. 32. In the program described in 31., Repeatedly selecting two images that are adjacent to each other in time sequence; performing the averaging process each time the two images are selected; the result of the averaging process includes, for each of the target regions, information indicating a first term indicating a value obtained by multiplying the value of the target region by a weighting coefficient, and a second term indicating the weighting coefficient used for the multiplication, and is stored in a storage means; A program for causing a computer to execute the procedure of adding the first and second terms of the target area of the current image to the result of the averaging stored in the storage means when performing the averaging process on the next two images. 33. In the program according to any one of 28. to 32., a program for causing a computer to execute a procedure for setting a sampling interval of the image according to the area and performing the averaging process. 34. In the program described in 33., A program for causing a computer to execute a procedure of calculating the time until a change greater than a reference value occurs in the region by processing past images, and setting the calculated time as the sampling interval for each region. 35. In the program according to any one of 28. to 34., The sampling intervals of the plurality of images vary depending on the subject being photographed. 36. In the program according to any one of 28. to 35., a program for causing a computer to execute a procedure for determining that the difference is below a standard when the change in hue of the image is below a standard and the change in brightness is above a standard; [Explanation of symbols]

[0108] 1. Surveillance image generation system 3. Communication Network 5. Camera 10 POS register 20 display shelves 100 Image processing device 102 Acquisition Department 104 Selection section 106 Processing section 110 Storage device 120 Results information 1000 computers 1010 Bus 1020 processor 1030 memory 1040 Storage Device 1050 Input / Output Interface 1060 Network Interface

Claims

1. an acquisition means for acquiring a plurality of images taken at different times of the same location; a selection means for comparing at least two of the plurality of images and selecting a target region that is a region where the difference between the images is equal to or smaller than a reference value; and processing means for performing averaging of the target regions included in each of the at least two images.

2. 2. The image processing device according to claim 1, until the averaging process is performed on an area in the image that is equal to or larger than the reference range. the selection means compares the at least two images by changing the combination of the images to be compared; The processing means repeats the averaging process.

3. 3. The image processing device according to claim 1, The image processing device, wherein the unit of the area is one pixel.

4. 4. The image processing device according to claim 1, The image processing device wherein the processing means weights the image using a difference on a time axis from the latest image when performing the averaging process.

5. 5. The image processing device according to claim 4, the selecting means repeatedly selects two images that are adjacent to each other in time series; the processing means performs the averaging process each time the selection means selects the two images; the result of the averaging process includes, for each of the target regions, information indicating a first term indicating a value obtained by multiplying the value of the target region by a weighting coefficient, and a second term indicating the weighting coefficient used for the multiplication, and is stored in a storage means; When performing the averaging process on the next two images, the processing means adds the first and second terms of the target region of the current image to the result of the averaging process stored in the storage means. Image processing device.

6. 6. The image processing device according to claim 1, The processing means sets a sampling interval for the image according to the area and performs the averaging process.

7. 7. The image processing device according to claim 6, The processing means processes past images to calculate the time until a change greater than a reference value occurs in the region, and sets the calculated time as the sampling interval for each region.

8. an image processing device; a surveillance camera that captures images of the same location at different times and generates multiple images; The image processing device includes: an acquisition means for acquiring the plurality of images generated by the surveillance camera; a selection means for comparing at least two of the plurality of images and selecting a target region that is a region where the difference between the images is equal to or smaller than a reference value; and a processing means for performing an averaging process to average the target regions included in each of the at least two images. Surveillance image generation system.

9. The image processing device Acquire multiple images taken at different times of the same location, comparing at least two of the plurality of images and selecting a target region where the difference between the images is equal to or less than a reference value; performing an averaging process for averaging the target regions included in each of the at least two images; Image processing methods.

10. On the computer, A procedure for acquiring multiple images taken at different times of the same location; a step of comparing at least two of the plurality of images and selecting a target region which is a region where the difference between the images is equal to or smaller than a reference value; a program for executing a procedure for performing an averaging process for averaging the target regions included in each of the at least two images.

Images