Image processing apparatus, image processing method, and computer program

Abstract

To provide an image processing device for selecting a warning device based on a relation between a position of an object and a position of the warning device.SOLUTION: An image processing device executes processing to determine a warning device to issue warning to a target person, and includes: image acquisition means for acquiring an image; first acquisition means S801 for acquiring warning device positions which indicate positions of a plurality of warning devices in the image; second acquisition means S802 for acquiring an object position which indicates a position of an object designated by a user in the image; selection means S803, S804 for selecting some warning devices among the plurality of warning devices based on the respective warning device positions acquired by the first acquisition means and the object position acquired by the second acquisition means; and control means S805 for issuing warning by using the some warning devices selected by the selection means.SELECTED DRAWING: Figure 8

Description

Technical Field

[0001] The present invention relates to an image processing apparatus, an image processing method, and a computer program.

Background Art

[0002] In recent years, shoplifting damage in various facilities such as retail stores and large-scale commercial stores has been worsening, and there is a need on the part of the store to reduce the damage caused by shoplifting. In response to this need, there is a technology that analyzes the video captured by a surveillance camera in real time to detect suspicious behavior that may be shoplifting, and automatically issues an audio warning in place of store employees or security guards to prevent shoplifting.

[0003] As a technology for detecting suspicious behavior, for example, there is a method described in Patent Document 1. In Patent Document 1, feature values such as the position, orientation, and line-of-sight direction of a person are extracted from the image data obtained from the video captured by a camera, and these feature values are analyzed to detect a person suspected of shoplifting. Also, when a person suspected of shoplifting is detected, the store employees or security guards of the store are notified, and the store employees or security guards talk to the target person to suppress shoplifting. Patent Document 2 describes a method of determining a warning device from a plurality of warning devices according to the position within the image of the detected suspicious person.

[0004] Patent Document 3 describes an example of notifying the store employees of the store when a person suspected of shoplifting is detected. In this example, the detection information is notified to the mobile terminal carried by the store employees. Also, when the store employee who received the notification on the mobile terminal completes the response to the suspicious person, the response completion information is transmitted to the management server.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

Patent Document 2

[0006] As illustrated in Patent Documents 1 to 3, it is common practice to notify store employees of a person suspected of shoplifting. Upon receiving the notification, the employee is required to approach the suspicious person to observe them or speak to them directly in order to prevent shoplifting. On the other hand, store employees are supposed to be performing other tasks such as stocking shelves and operating the cash register, so it is desirable to minimize the time and burden they spend dealing with suspicious individuals.

[0007] Therefore, one objective of the present invention is to provide an image processing apparatus that selects a warning device based on the relationship between the position of the object and the position of the warning device. [Means for solving the problem]

[0008] One aspect of the present invention is an image processing apparatus, Image acquisition means for acquiring images, A first acquisition means for acquiring the warning device positions, which indicate the positions of multiple warning devices in the aforementioned image, In the aforementioned image, a second acquisition means for acquiring the target position indicating the location of an object specified by the user, A selection means for selecting some of the warning devices from the plurality of warning devices based on the respective warning device positions acquired by the first acquisition means and the target positions acquired by the second acquisition means, The system is characterized by comprising a control means that provides a warning using some of the warning devices selected by the selection means. [Effects of the Invention]

[0009] According to one aspect of the present invention, an image processing device can be realized that selects a warning device based on the relationship between the position of an object and the position of the warning device. [Brief explanation of the drawing]

[0010] [Figure 1] This is a block diagram showing an example of the hardware configuration of the image processing device according to Example 1. [Figure 2] This is a functional block diagram of the image processing apparatus according to Example 1. [Figure 3] This is a schematic diagram showing an example of a store layout to be monitored using the image processing device 100 of Example 1. [Figure 4] This figure shows an example of a screen displayed on the display device 107 of the image processing device 100 according to Embodiment 1. [Figure 5] This figure shows an example of selecting a target person through user interaction. [Figure 6] This figure shows an example of determining which warning device to use based on the distance to the target person determined by user input. [Figure 7] This figure shows an example of determining which warning device to issue a warning from a warning area linked to a target person and warning device determined by user operation. [Figure 8] This flowchart shows the process flow of the image processing device 100 according to Example 1, which determines a warning device to warn the target person. [Figure 9] This figure shows an example in Example 2 where a sound source is selected from multiple sound sources registered in the warning device. [Figure 10] This is a functional block diagram of the image processing device 100 according to Example 3. [Modes for carrying out the invention]

[0011] Embodiments of the present invention will be described below with reference to the drawings. However, the present invention is not limited to the following embodiments. In each drawing, the same reference numeral is used for the same member or element, and redundant explanations are omitted or simplified.

[0012] (Example 1) In an embodiment of the present invention, a plurality of warning devices are installed in advance to effectively issue an audible warning to a suspicious person captured by a surveillance camera. Also, the positions of the warning devices installed in advance are displayed on the image data captured by the surveillance camera. Then, the user designates a suspicious person from the people in the image data, determines a warning device that issues a warning to the suspicious person based on the coordinate information of the warning device set in advance and the coordinate information of the suspicious person determined by the user, and issues a warning.

[0013] FIG. 1 is a block diagram showing a hardware configuration example of an image processing apparatus according to Embodiment 1. The image processing apparatus 100 includes a CPU 101 as a computer, a ROM 102 and a RAM 103 as storage media, a hard disk drive (HDD) 104, a communication unit 105, an input device 106, a display device 107, and the like.

[0014] The communication unit 105 is connected to a wired or wireless network 108 and can perform two-way communication with an external server or the like of the image processing apparatus. The CPU 101 reads a computer program recorded in the ROM 102 and executes various processes. The RAM 103 is used as a temporary storage area such as a main memory or a work area. The HDD (Hard Disk Drive) 104 is used for storing image data and the like.

[0015] Incidentally, the CPU 101 can acquire image data from a video acquisition device such as a surveillance camera via the network 108. Also, the image data can be acquired by downloading the image data stored in an external server. Also, the image data can be acquired by reading the image data stored in the HDD 104.

[0016] The display device 107 is a display device such as a liquid crystal display that displays image data and the results of suspicious person detection. The input device 106 includes a keyboard, mouse, touch panel, and other devices for the user to manually input information such as selecting image data, setting the location of sound sources on the image data, and various instructions. Note that the hardware shown in Figure 1 does not have to be built into the same enclosure, and may be composed of separate devices connected to each other via signal paths.

[0017] As described above, the hardware configuration of the image processing device 100 has hardware components similar to those found in a personal computer (PC). Therefore, the various functions realized by the image processing device 100 can be implemented as software that runs on a PC. The image processing device 100 can realize various functions of the image processing device 100 as shown in Figure 2 and the processes shown in the flowchart of Figure 8, which will be described later, by having the CPU 101 execute a program.

[0018] Figure 2 is a functional block diagram of the image processing apparatus according to Embodiment 1. Note that some of the functional blocks shown in Figure 2 are implemented by having the CPU 101, which acts as a computer included in the image processing apparatus 100, execute computer programs stored in memory such as ROM 102, which acts as a storage medium. However, some or all of these may be implemented in hardware. As hardware, dedicated circuits (ASICs) or processors (reconfigurable processors, DSPs) can be used.

[0019] Furthermore, when each of the functional blocks shown in Figure 2 is implemented in hardware, the hardware components do not necessarily have to be housed in the same enclosure; they may be configured as separate devices connected to each other via signal paths. Furthermore, the above explanation regarding Figure 2 also applies to Figure 10, which will be discussed later.

[0020] The image processing device 100 includes an image acquisition unit 202, a setting unit 203, an operation input unit 204, a display unit 205, a warning device determination unit 206, a warning instruction unit 207, and the like. The image processing device 100 is also connected to a video acquisition device 201 such as a surveillance camera and a warning device 208 via a network 108 such as the Internet. Note that the number of video acquisition devices 201 and warning devices 208 is not limited to the example in Figure 2.

[0021] Next, each part will be described. The image acquisition unit 202 functions as an image acquisition means for acquiring images, and performs an image acquisition step to acquire image data obtained from video sent from the video acquisition device 201. Here, the video may be a video file in an image format such as an AVI file or a MOV file, or it may be real-time video sent from a webcam. Also, there may be one or more types. The video acquisition device 201 includes, for example, multiple surveillance cameras 303, 304, 305, etc., as shown in Figure 3.

[0022] Figure 3 is a schematic diagram showing an example of a store layout monitored using the image processing device 100 of Embodiment 1. The store 300 has a terminal 301 installed in the back room and an employee 302 working in the back room. Multiple surveillance cameras 303, 304, and 305 as image acquisition means, multiple warning devices 402 to 404 (for example, speakers in this embodiment), and product shelves 405 to 409 are arranged.

[0023] The setting unit 203 sets the coordinates of the warning device on the image displayed by the display unit 205. The coordinates can be specified by accepting numerical input from the user, or by using a UI, for example, by allowing the user to specify the coordinates by dragging and dropping an icon of the warning device.

[0024] Figure 4 shows an example of a screen displayed on the display device 107 of the image processing device 100 according to Embodiment 1. The setting screen 400 includes an image 401 acquired by the image acquisition unit 202 and a warning device list 412. Note that image 401 is an enlarged view of a part of the store 300 layout. The warning device list 412 displays several warning devices 402-404 installed in store 300, which have been pre-registered by the user, as SP1-SP3.

[0025] The setting unit 203 selects a warning device from the warning device list 412 based on user instructions. Upon selection of a warning device, it displays icons 4021, 4031, and 4041 corresponding to the selected warning devices 402, 403, and 404. Furthermore, icons 4021, 4031, and 4041 corresponding to warning devices 402, 403, and 404 are placed on the image data, for example, by dragging and dropping them according to user instructions. This sets the coordinates of each of the warning device icons 4021, 4031, and 4041. Note that in Figure 4, icon 4041 corresponding to warning device 404 is omitted.

[0026] In this embodiment, assuming that a warning device 402 is installed on product shelf 406 and a warning device 403 is installed on product shelf 407 in the image data, icons 4021 and 4031 for the warning devices are placed at the corresponding positions in the image data, and their coordinates are set. When setting the coordinates for multiple warning devices, tags such as SP1 and SP2 may be displayed next to each warning device icon.

[0027] The warning device determination unit 206 determines a warning device to warn a person based on the coordinate information of a person specified by the user using the operation input unit 204, among the people included in the image data displayed by the display unit 205.

[0028] Figure 5 shows an example of selecting a target person through user operation, where the monitor screen 500 displays image 501 acquired by the image acquisition unit 202. Image 501 displays person 503, and user operation 502 allows the user to select person 503 by touching the location of person 503 on the touch panel of the monitor screen 500.

[0029] In Example 1, the system determines which warning device to warn person 503 based on the coordinate information of the person entered by user operation 502 and the coordinate information associated with (associated with) multiple warning devices set in the setting unit 203.

[0030] Figure 6 shows an example of determining the warning device to be issued based on the distance to the target person determined by user operation. The warning device is selected from the coordinate information of the person entered in user operation 502 and the coordinate information associated with multiple warning devices set in the setting unit 203. Line 601 is a straight line connecting the coordinate information of person 503 entered in user operation 502 and the coordinate information of warning device 402 set in the setting unit 203. Similarly, line 602 is a straight line connecting the coordinate information of the person entered in user operation 502 and the warning device 403 set in the setting unit 203. The lengths of lines 602 and 603 correspond to the distance.

[0031] The warning device determination unit 206 calculates the coordinate information of the person entered by the user operation 502 and the distance to the multiple warning devices set by the setting unit 203. That is, it compares the distances corresponding to the respective lengths of the two straight lines 601 and 602 and determines the warning device at the shorter distance to be the warning device that will warn the person 503. In the example in Figure 6, since the straight line 601 is shorter, the warning device 402 is determined to be the warning device that warns person 503.

[0032] Furthermore, the method for selecting one warning device from multiple warning devices is not limited to the method described above. For example, Figure 7 shows an example of determining which warning device to issue a warning from the warning area associated with the target person and warning device determined by user operation. In Figure 7, when setting coordinate information associated with multiple warning devices in the setting unit 203, the warning area 701 associated with warning device 402 and the warning area 702 associated with warning device 403 are set.

[0033] Then, depending on which warning area the coordinate information of the person entered in user operation 502 falls into, a warning is issued to the person from the warning device associated with that warning area. In other words, based on the relationship between the area set based on the warning device position and the target position, some warning devices corresponding to the area containing the target position are selected, and a warning is issued from the selected warning devices. In the example in Figure 7, the coordinate information of person 503 entered by user operation 502 is included in the warning area 701, so the warning device 402 associated with the warning area 701 is determined to be the warning device that warns person 503.

[0034] Figure 8 is a flowchart showing the process flow of the image processing device 100 according to Embodiment 1, which determines a warning device to warn the target person. Note that the CPU 101, acting as a computer, executes a computer program stored in memory such as ROM 102, thereby performing the operation of each step in the flowchart in Figure 8.

[0035] In step S801, when the user uses the input device 106 to set the locations where the warning devices 402 and 403 are installed on the image, the setting unit 203 determines the coordinates of the warning devices 402 and 403. Here, step S801 functions as a first acquisition step (first acquisition means) for acquiring warning device positions that indicate the locations of multiple warning devices on the image. Step S801 also functions as a setting means for pre-setting warning device positions that indicate the locations of multiple warning devices.

[0036] Next, in step S802, when the user selects a person on the image using the input device 106, the coordinates of the person selected by the user are determined. Step S802 functions as a second acquisition step (second acquisition means) for acquiring the position (target position) of an object specified by the user in the image.

[0037] Subsequently, in step S803, the distance between the selected person's coordinates and each warning device is calculated based on the coordinates of the person selected by the user and the coordinates of the pre-registered warning devices 402 and 403. Then, in step S804, the warning device closest to the selected person's coordinates, i.e., the shortest distance, is determined to be the warning device for issuing a warning to the target person. In other words, some warning devices are selected based on the distance between each warning device's position and the target position.

[0038] Here, steps S803 and S804 function as selection steps (selection means) to select some of the warning devices from among the multiple warning devices based on the respective warning device positions obtained in step S801 and the target positions obtained in step S802. Next, in step S805, a warning instruction is issued to the determined warning device, thereby issuing a warning to the selected person via that warning device. Here, step S805 functions as a control step (control means) that issues a warning using some of the warning devices selected in step S804.

[0039] Next, in step S806, it is determined whether the system has terminated or not. If the answer is No, the process returns to step S802, and steps S802 to S806 are repeated. If the answer in step S806 is Yes, the flow shown in Figure 8 is terminated. Thus, according to this embodiment 1, the image processing device 100 can issue a warning to the person selected by the user from an appropriate warning device.

[0040] (Example 2) In Example 1, in steps S804 and S805 of Figure 8, the system determines which warning device to use to warn the target person based on the user's input coordinates and the coordinates of warning devices 402 and 403, and then issues a warning instruction. However, Example 1 does not consider the case where multiple sound sources are registered to the warning device.

[0041] Therefore, in Example 2, the warning device is determined from the user's input coordinates and the coordinates of the warning device, and the sound source registered to the determined warning device is made selectable. Figure 9 shows an example in Embodiment 2 where a sound source is selected from multiple sound sources registered in the warning device. As shown in Figure 9, sound sources A, B, and C are registered in the warning device 402, and sound sources D, E, and F (not shown) are registered in the warning device 403 and are selectable.

[0042] In Figure 9, when the user operation 502 has determined that the warning device 402 should issue a warning, a selection icon 901 is displayed for selecting a sound source that can be played by the warning device 402. If the display unit 205 (operation input unit 204) is a touch panel, the selection icon can be selected using flick input or similar methods. However, the method of displaying and selecting sound sources that can be played by the warning device is not limited to the example shown in Figure 9.

[0043] By using a GUI operation screen like the one shown in Figure 9, not only is the selection of a warning device for the target person performed automatically, but the user can also easily select the sound source for that warning device, enabling more effective warnings. Note that the type of sound source in Example 2 is not limited to the type of sound (e.g., male voice or female voice), but also includes selecting the type of warning sound (e.g., sound pattern such as beep or chime) and the type of warning message.

[0044] (Example 3) In Examples 1 and 2, the warning device that would warn the target person was determined based on the distance between the coordinates of the person entered by the user and the coordinates of multiple warning devices. However, Example 3 assumes a situation where there are obstacles such as walls or product shelves in the store that would obstruct the audio warning. Specifically, in Example 2, it is determined whether or not there are obstacles between the location of the warning device and the target location, and some of the multiple warning devices are selected according to the result of that determination.

[0045] Figure 10 is a functional block diagram of the image processing apparatus 100 according to Embodiment 3. Here, only the functions and configurations that differ from the functional block diagram in Figure 2 will be explained. In Example 2, the occluding object information acquisition unit 1001 acquires information about occluding objects such as shelves and walls included in the image data. The occluding object information includes at least the coordinates in the image data.

[0046] Known methods can be used to obtain occluding information. For example, occluding information can be obtained by detecting occluding objects in an image using image recognition. Alternatively, the coordinate information of the occluding object can be numerically input using the operation input unit 204, or the coordinates of the occluding object can be specified using a UI, for example, by drawing a cursor around the occluding object. However, the method is not limited to these.

[0047] The warning device determination unit 1002 calculates the distance between a warning device and a person, taking into account the influence of an obstacle, based on the coordinates of multiple warning devices set in the setting unit 203, the coordinates of a person entered by user operation 502, and the coordinates of an obstacle acquired by the obstacle information acquisition unit 1001. In other words, from among the warning devices 402 and 403, the warning device that has the shortest distance between the warning device and the specified person, taking into account the influence of the obstacle, is determined to be the warning device that issues the warning.

[0048] For example, if there is an obstruction between the coordinates of a specified person and the warning device, the distance information is multiplied by a coefficient corresponding to the thickness of the obstruction in that straight line direction. For example, if the thickness of the obstruction in the straight line direction connecting two points is 0m, the coefficient multiplied by the distance between the two points is 1; if it is 0.5m, the coefficient multiplied by the distance between the two points is 1.5; and if the thickness is 1m, the coefficient is 3. In short, the larger the thickness, the larger the coefficient should be.

[0049] Then, the distance between each warning device and the person's coordinates is weighted by multiplying it by a coefficient corresponding to the thickness of the obstruction. After that, the distances between multiple warning devices and the person's coordinates are compared, and the closest warning device is determined. In other words, depending on the thickness of the obstruction between the warning device's position and the target position, some of the multiple warning devices can be selected.

[0050] The above explanation described the distance between the warning device and the person on a 2D map, but it is desirable to calculate the distance on a 3D map. In that case, the height of the person's ears should be set to, for example, 1.5m, and the height of the obstruction should also be registered on the 3D map. Then, the thickness of the obstruction along the line between the coordinates of the warning device and the coordinates of the person's ear height on that 3D map should be found, and the distance should be multiplied by a coefficient corresponding to that thickness.

[0051] In this embodiment, an obstruction refers to something that obstructs the space between the warning device and the object in three-dimensional space. For example, if the warning device is installed on the ceiling and is sufficiently higher than a product shelf, it may not obstruct the space between the object and the warning device. The same applies if the product shelf is lower than, for example, the height of the object's ear. In such cases, the product shelf will not be considered an obstruction in the three-dimensional map, so it is desirable to calculate the distance, obstruction, and thickness of the obstruction in the three-dimensional map in this embodiment.

[0052] In Example 3, if the distance between multiple warning devices, taking into account the effect of obstacles, and the suspicious person is the same, the warning device that has no obstacle between it and the suspicious person, or has a thinner obstacle, may be selected as the warning device that issues the warning. Alternatively, if the distances are the same, the warning device that is not weighted based on the information acquired by the obstacle information acquisition unit 1001, or has a small weight, may be selected as the warning device that issues the warning. As described above, according to Example 3, it is possible to provide an effective warning to a designated person detected from an image, taking into account any obstructions.

[0053] In the above embodiment, one warning device is selected from among multiple warning devices, but this is not limited to one. For example, two or three warning devices may be selected from among multiple warning devices. In other words, it is sufficient to select some of the warning devices from among multiple warning devices. This allows for efficient warnings to be given to targets such as people, eliminating the need for unnecessary warnings from warning devices located far from the target. Consequently, unnecessary warnings to people other than the designated target will not startle customers or others.

[0054] Furthermore, when selecting some of several warning devices, it is not necessary to completely turn off the other warning devices; it is sufficient to simply lower their volume relatively. In other words, embodiments of the present invention include reducing the volume of the unselected warning devices compared to the volume of the selected warning devices. Conversely, they also include increasing the volume of the selected warning devices compared to the volume of the unselected warning devices. For example, it is possible to continuously play background music from multiple warning devices installed in a store while issuing warnings from some of the selected warning devices.

[0055] In the above embodiment, we described an example where the user specifies a suspicious person or similar object on the screen, but the object does not have to be a human; for example, it could be an animal. In other words, the configuration of the above embodiment may be applied to a system for monitoring or guiding the behavior of livestock, for example.

[0056] Although the present invention has been described in detail above based on preferred embodiments, the present invention is not limited to the above embodiments, and various modifications are possible based on the spirit of the present invention, and these modifications are not excluded from the scope of the present invention. Furthermore, in this embodiment, some or all of the control may be performed by supplying a computer program that realizes the functions of the embodiment described above to an image processing device, etc., via a network or various storage media. The computer (or CPU, MPU, etc.) in the image processing device, etc., may then read and execute the program. In that case, the program and the storage medium storing the program constitute the present invention. [Explanation of symbols]

[0057] 100 Image Processing Devices 201 Video Acquisition Device 203 Settings Section 206 Warning device determination section

Claims

1. Image acquisition means for acquiring images, A first acquisition means for acquiring the warning device positions, which indicate the positions of multiple warning devices in the aforementioned image, In the aforementioned image, a second acquisition means for acquiring the target position indicating the location of an object specified by the user, A selection means for selecting some of the warning devices from the plurality of warning devices based on the respective warning device positions acquired by the first acquisition means and the target positions acquired by the second acquisition means, An image processing apparatus comprising control means for issuing a warning using some of the warning devices selected by the selection means.

2. The image processing apparatus according to claim 1, further comprising setting means for pre-setting the warning device positions that indicate the positions of each of the plurality of warning devices.

3. The image processing apparatus according to claim 1 or 2, characterized in that the selection means selects some of the warning devices based on the distance between each of the warning device locations and the target location.

4. The image processing apparatus according to claim 1 or 2, wherein the selection means selects some of the warning devices based on the relationship between the region set based on the warning device position and the target position.

5. The image processing apparatus according to claim 4, characterized in that the selection means selects a portion of the warning devices corresponding to the region in which the target position is included.

6. The image processing apparatus according to any one of claims 1 to 5, characterized in that the selection means determines whether or not an obstruction exists between the warning device position and the target position, and selects some of the warning devices from the plurality of warning devices according to the result of the determination.

7. The image processing apparatus according to claim 6, characterized in that the selection means selects some of the plurality of warning devices according to the thickness of an obstruction present between the warning device position and the target position.

8. Image acquisition step to obtain an image, A first acquisition step of acquiring the warning device positions, which indicate the positions of multiple warning devices in the aforementioned image, In the aforementioned image, a second acquisition step is to acquire the target position indicating the location of an object specified by the user, A selection step in which a portion of the plurality of warning devices are selected based on the respective warning device positions obtained in the first acquisition step and the target positions obtained in the second acquisition step, An image processing method characterized by comprising a control step of issuing a warning using some of the warning devices selected in the selection step.

9. A computer program for controlling each means of the image processing apparatus described in any one of claims 1 to 7 by a computer.

Images