Information processing apparatus, information processing method, and program

Abstract

To reduce erroneous detection of a single state.SOLUTION: The information processing apparatus detects a person from an image, and among the detected persons, detects a person who is not within a predetermined area set in the image as a solitary state, if no other person exists within a threshold distance.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing technique for detecting a person's state by image analysis. [Background technology]

[0002] In recent years, shoplifting damages have become serious in various facilities such as retail stores and large commercial stores, and there is a need for stores to reduce these damages. In response to such needs, there is a technology that prevents shoplifting by analyzing images captured by a surveillance camera in real time to detect suspicious behavior that may be shoplifting and notifying store staff or security guards of the detection results. Patent Document 1 discloses a technology that extracts feature values ​​such as a person's position, orientation, and line of sight from image data captured by a camera, and detects a person suspected of shoplifting by analyzing these feature values.

[0003] On the other hand, when we look at the circumstances in which shoplifting occurs, shoplifting is often done by a single person with no other people around, except when the perpetrator is a gang or a group of shoplifters. For this reason, there is a technology that can accurately detect people suspected of shoplifting by determining whether each person in an image is alone or not. Patent Document 2 discloses a technology that detects people who are "alone" with no other people around them by detecting the positions of people from image data captured by a camera and measuring the distance between people. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2017-076171 A [Patent Document 2] Patent Publication No. 2021-117768 Summary of the Invention [Problem to be solved by the invention]

[0005] However, the above-mentioned method for detecting a person in a solitary state may erroneously detect a person as being solitary when in fact the person is not solitary.

[0006] Therefore, an object of the present invention is to make it possible to reduce erroneous detection of the single state. [Means for solving the problem]

[0007] The information processing device of the present invention is characterized by having a person detection means for detecting a person from an image, and a single state detection means for detecting, among the detected people, a person who has no other people within a threshold distance and is not within a specified area set in the image as a single person. Effect of the Invention

[0008] According to the present invention, it is possible to reduce erroneous detection of the single state. [Brief description of the drawings]

[0009] [Figure 1] FIG. 2 is a diagram illustrating an example of a functional configuration of an information processing device. [Diagram 2] FIG. 2 is a diagram illustrating an example of a hardware configuration of an information processing device. [Diagram 3] FIG. 13 is a diagram showing an example of a camera image taken inside a store. [Figure 4] FIG. 1 is a diagram showing an example of an actual store space and a person not captured in a camera image. [Diagram 5] 13 is a flowchart of information processing. [Figure 6] 13 is a flowchart of an image analysis process. [Figure 7] FIG. 13 is a diagram showing an example of person detection and ID. [Figure 8] FIG. 11 is an explanatory diagram of an example of residence time measurement. [Figure 9] 13 is a flowchart of a single state detection process. [Figure 10] FIG. 11 is an explanatory diagram of an example of detection of a single state. [Figure 11]FIG. 13 is an explanatory diagram of another example of additional determination. [Figure 12] FIG. 11 is an explanatory diagram of an example of a priority person determination process. [Figure 13] 11 is a diagram showing an example of a human detection area during a posture estimation process. FIG. [Figure 14] FIG. 11 is an explanatory diagram of an example of a posture estimation processing result. [Figure 15] FIG. 11 is an explanatory diagram of an example of a behavior pattern. [Figure 16] FIG. 13 is a diagram showing an example of a setting screen when a boundary area is directly specified. [Figure 17] FIG. 13 is a diagram showing an example of a setting screen when a boundary line to an image edge is specified. [Figure 18] 13 is a diagram showing an example of a setting screen when a border line to another image edge is specified. FIG. [Figure 19] 13 is a diagram showing an example of a setting screen when a boundary line is specified other than to an image edge; FIG. [Figure 20] 13 is an explanatory diagram of an example in which the distance from the boundary line to the boundary area is changed. FIG. [Figure 21] FIG. 13 is a diagram showing an example of a setting screen when a boundary area is automatically set. [Figure 22] FIG. 11 is an explanatory diagram of a boundary area automatic setting process. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present invention will be described with reference to the drawings. The following embodiments do not limit the present invention, and not all of the combinations of features described in the present embodiments are necessarily essential to the solution of the present invention. The configurations of the embodiments may be appropriately modified or changed depending on the specifications of the device to which the present invention is applied and various conditions (conditions of use, environment of use, etc.). In the following embodiments, duplicated descriptions of the same or similar configurations and processing steps will be omitted.

[0011] FIG. 1 is a diagram showing an example of the functional configuration of an information processing device 101 according to this embodiment. The imaging device 102 is, for example, a surveillance camera installed in various facilities such as retail stores and large commercial stores. The image acquisition unit 103 acquires image data from the imaging device 102. In the following description, image data handled in the information processing device 101 will simply be referred to as an image. In the example of FIG. 1, the image acquisition unit 103 acquires an image captured by the imaging device 102, but may acquire the image indirectly via a relay device (not shown), for example. In the example of FIG. 1, the imaging device 102 is connected as an external device to the information processing device 101, but the information processing device 101 itself may have an imaging device, or conversely, the imaging device may have the function of an information processing device. In the example of this embodiment, the imaging device 102 is a network camera connected as an external device to the information processing device 101.

[0012] The image analysis unit 104 detects a person from the image acquired by the image acquisition unit 103, and performs image analysis processing to detect suspicious behavior patterns and predetermined behavior of the detected person. That is, the image analysis unit 104 detects a person with a suspicious behavior pattern or behavior by executing various information processing described later on the image acquired by the image acquisition unit 103. In the case of this embodiment, the suspicious behavior pattern or predetermined behavior analyzed by the image analysis unit 104 is, for example, a behavior pattern or behavior of shoplifting. Details of the image analysis processing in the image analysis unit 104 will be described later. Hereinafter, a person with a suspicious behavior pattern or behavior will be appropriately called a suspicious person.

[0013] When a suspicious person is detected by the image analysis unit 104, the audio playback unit 105 plays audio from an externally connected speaker 109. In the present embodiment, the audio playback unit 105 outputs a notification audio for notifying, for example, a store clerk or a security guard that a suspicious person has been detected, and a warning audio such as calling out to the suspicious person, from the speaker 109. Note that the notification audio and the warning audio may be output separately, or only one of them may be output, or both may be output.

[0014] The display unit 107 displays, on a display device, images acquired by the image acquisition unit 103 from the imaging device 102, a setting screen (to be described later) in the information processing device 101, images indicating a processing status, text, and the like. The operation acquisition unit 108 acquires an input operation by a user. The setting unit 106 sets various operations and the like in the information processing device 101. Although details will be described later, the settings made by the setting unit 106 include various settings related to image analysis processing, which will be described later, in the image analysis unit 104. For example, when the operation acquisition unit 108 acquires an input operation performed by a user while viewing a display on the display unit 107, the setting unit 106 sets the image analysis unit 104 and the audio playback unit 105 in accordance with the input operation.

[0015] FIG. 2 is a diagram illustrating an example of a hardware configuration of the information processing device 101. As shown in FIG. The information processing device 101 includes a CPU 201, a ROM 202, a RAM 203, a large-capacity memory 204, a disk drive 205, a network IF (interface) 206, an input device 207, a display device 208, etc. The network IF 206 is connected to a network 211, and the imaging device 102 and the speaker 109 described above are connected to the information processing device 101 via the network 211.

[0016] The CPU 201 is a control device that controls the information processing device 101. The ROM 202 stores a control program that the CPU 201 uses to control the information processing device 101, an information processing program for performing information processing related to each functional unit of the information processing device 101 shown in FIG. 1, and the like. A secondary storage device may be provided instead of the ROM 202. The RAM 203 is a memory for the CPU 201 to expand the program read from the ROM 202 and execute the processing. The RAM 203 is also used as a temporary storage area for temporarily storing data to be subjected to various processes.

[0017] The large-capacity memory 204 is an HDD, an SSD, or the like, and stores images captured by the imaging device 102. The CPU 201 reads out the images stored in the large-capacity memory 204 and performs processing such as image analysis. The CPU 201 may acquire images from a recording disk such as a CD, a DVD, a Blu-ray disk, or a flexible disk loaded in the disk drive 205. The CPU 201 may also acquire images received by the network IF 206 from the outside via the network 211. The network IF 206 is a circuit that performs communication via the network 211. When the CPU 201 acquires images from the disk drive 205 or the network IF 206, the large-capacity memory 204 may not necessarily be provided. When the CPU 201 acquires images from the large-capacity memory 204 or the network IF 206, the disk drive 205 may not necessarily be provided. Similarly, when the CPU 201 acquires images from the large-capacity memory 204 or the network IF 206, the network IF 206 may not necessarily be provided.

[0018] The display device 208 is a display device that displays images, text, etc. The display unit 107 in FIG. The input device 207 is a device equipped with at least one of a keyboard for input, a pointing device for indicating a display position in the display area of ​​the display device 208, a mouse, a touch panel, etc. The operation acquisition unit 108 in FIG. 1 acquires an input operation by a user using the input device 207.

[0019] As described above, the hardware configuration of the information processing device 101 has components similar to those of hardware components mounted on a general PC (personal computer) or the like. Therefore, various functions realized by the information processing device 101 can be implemented as software (information processing program) that runs on the PC. The CPU 201 executes the information processing program according to this embodiment, thereby realizing processes related to each functional unit of the information processing device 101 shown in FIG. 1.

[0020] Before explaining the details of information processing in the information processing device 101 according to this embodiment shown in Figures 1 and 2, we will now explain, with reference to Figures 3 and 4, what needs to be considered when detecting a suspicious person from an image captured by a surveillance camera. FIG. 3 is a diagram showing an example of an image (assumed to be a camera image) captured by a surveillance camera installed in a store. In the camera image 301, a person 302 is captured on the left side of the screen, and a person 303 is captured near a product shelf 304. In the case of the existing single person determination described above, the two people 302 and 303 are sufficiently far apart, so each person is determined to be in a single state. However, in an actual store space, there is space outside the camera image 301 (outside the angle of view of the surveillance camera) and behind the product shelf 304. For example, as shown in FIG. 4, a person 404 may exist outside an area 401 captured by the surveillance camera (an area corresponding to the camera image 301), and for example, a person 405 may exist behind the product shelf 304. In this case, the person 302 captured in the camera image 301 is close to the person 404 in the store space, and the person 303 in the camera image 301 is close to the person 405 in the store space, so it cannot necessarily be said that each person is in a single state.

[0021] In this way, when making a judgment as to whether or not a person 302 near the edge of a camera image or a person 303 near an obstruction such as a product shelf 304 is alone, there is a possibility that another person may be present in a position that is a blind spot of the surveillance camera, that is, in a position that is not captured in the camera image. In this case, there is a risk that the person being judged as being alone may be erroneously judged to be alone even though he or she is not actually alone. For this reason, the information processing device 101 of this embodiment enables independent determination that takes into consideration the possibility that another person may be present in a position not captured in the captured image of the imaging device 102. Hereinafter, information processing according to this embodiment that enables independent determination that takes into consideration the possibility that another person may be present in a position not captured in the captured image will be described.

[0022] FIG. 5 is a flowchart showing the flow of information processing performed in each functional unit of the information processing device 101 shown in FIG. First, in step S501, the image analysis unit 104 executes image analysis processing on the image acquired by the image acquisition unit 103 from the imaging device .

[0023] 6 is a diagram showing the flow of the image analysis process in step S501. That is, in step S501, the image analysis unit 104 processes the input image in the order shown in FIG. The image analysis unit 104 first executes person detection processing 601 on the input image to detect people in the image. Then, in the person detection processing 601, images of consecutive frames are sequentially analyzed, identification information (hereinafter, referred to as ID) is assigned to each person detected, and the person for each ID is tracked between frames. Note that methods for linking and tracking people between consecutive frames include, for example, a method using the amount of change or direction of a person's position, and a method using features of a person's hairstyle or clothing, and the person detection processing 601 uses any one of these methods or a combination of these methods.

[0024] FIG. 7 is a diagram showing an example in which an ID is assigned to each person detected by sequentially analyzing images of consecutive frames in the person detection process 601. FIG. 7(a) shows an example in which person 701 and person 702 are detected from an image 700 of an n-th frame, and ID:001 is assigned to person 701, and ID:002 is assigned to person 702. FIG. 7(b) shows an example in which person 711, person 712, and person 713 are detected from an image 710 of the next n+1-th frame. The image analysis unit 104 links corresponding people between these frames based on the positions and movement amounts of people in the images of the n-th frame and the following n+1-th frame. Then, the image analysis unit 104 assigns the same ID assigned to the corresponding person in the image 500 of the n-th frame to the people linked between the n+1-th frame and the n-th frame. 7(b) shows an example in which ID: 001 assigned to person 701 in the nth frame is assigned to person 711 in the n+1th frame, and ID: 002 assigned to person 702 in the nth frame is assigned to person 712. On the other hand, the image analysis unit 104 assigns a new ID: 003 to person 713, who cannot be linked to the person between the nth frame and the n+1th frame.

[0025] Following the person detection process 601 in Fig. 6, the image analysis unit 104 performs a staying time measurement process 602 for each detected person. In this embodiment, the staying time is defined as the time during which a detected person continues to exist in images of multiple frames. Using the ID assigned in the person detection process 601, the image analysis unit 104 measures the staying time during which a person assigned the same ID continues to exist in images of multiple frames.

[0026] Fig. 8 is a diagram showing an example of the dwell time measured for each of the persons with ID: 001 to ID: 003. In Fig. 8, for example, the person with ID: 001 continues to be present in multiple frames from the n-2th frame onwards, and if the frame interval is 200 milliseconds, the dwell time of the person with ID: 001 at the n+1th frame is 600 milliseconds.

[0027] The image analysis unit 104 performs a solo state detection process 603 for each detected person following the stay time measurement process 602 in Fig. 6. That is, the solo state detection process 603 detects a person who is in a solo state from among the detected people.

[0028] Fig. 9 is a flowchart showing the flow of the solo state detection process 603. In the solo state detection process 603, the image analysis unit 104 executes the loop process of Fig. 9 for each person detected in the image, thereby determining that each person is in one of the states of "solo state", "non-solo state", or "quasi-solo state".

[0029] First, in the process of step S901, the image analysis unit 104 focuses on a person with ID:x in the frame, and measures the distance between the person of interest with ID:x and other people other than ID:x. Furthermore, the image analysis unit 104 compares the distance measured between the person of interest and each of the other people with a preset judgment threshold distance to determine whether or not other people exist within the judgment threshold distance. If at least one other person exists within the judgment threshold distance, the image analysis unit 104 proceeds to the process of step S903, and determines that other people exist near the person of interest with ID:x, and that the person of interest with ID:x is in a "non-alone state."

[0030] On the other hand, if there is no other person within the determination threshold distance, it can be said that the person with ID:x is alone at least in the image. However, as described in Fig. 3 and Fig. 4, there is a possibility that other people exist in positions not shown in the captured image of the imaging device 102. For this reason, the image analysis unit 104 advances the process to step S902 and subsequent steps to perform an alone determination that takes into account the possibility that other people exist in positions not shown in the captured image of the imaging device 102.

[0031] When proceeding to step S902, the image analysis unit 104 judges whether or not the position of the person of interest with ID:x is within a predetermined boundary area in the image. Here, the boundary area is an area that is a boundary between an area that is captured in the captured image of the imaging device 102 and an area that is not captured. For example, the outside of the image capture angle of the imaging device 102 is not captured as an image, but the space is actually expanding, so the vicinity of the boundary between the outside and inside of the image capture angle is set as the boundary area. The boundary between the outside and inside of the image capture angle is the edge of the captured image, so the vicinity of the edge of the captured image is set as the boundary area. In addition, if a tall shelf or the like is present within the image capture angle, the rear side of the shelf is not captured as an image, but the space may actually be expanding, so the vicinity of the edge of the shelf is also set as the boundary area. Note that the boundary area may be set in advance or automatically set by image analysis, and a detailed method for setting the boundary area will be described later. In addition, in this embodiment, the head coordinates of the person are used as the detection position of the person. Therefore, the head coordinates of each person are also used when measuring the distance between people. When determining whether a person is inside or outside a boundary area, it is determined whether the head coordinates of the person are included within the boundary area.

[0032] If, in the inside / outside determination of whether or not the person is inside the boundary area, the image analysis unit 104 determines that the detection position of the person of interest with ID: x is outside the boundary area, then, in the processing of step S904, it determines that the person with ID: x is in a "single state." On the other hand, if the detection position of the person of interest with ID:x is within the boundary area, there is no other person in the vicinity within the area shown in the image, but there is a possibility that another person whose distance from the person of interest with ID:x is within the judgment threshold distance is present within the area not shown in the image. Therefore, if the detection position of the person of interest with ID:x is within the boundary area, the image analysis unit 104 determines in step S905 that the person of interest with ID:x is in a "quasi-solo state" that cannot be determined to be in a "solo state" or a "non-solo state".

[0033] Thus, in the solo state detection process 603, the image analysis unit 104 executes the loop process of FIG. 9 for each person detected in the image to determine whether each person is in a "solo state", "non-solo state", or "quasi-solo state".

[0034] The above-mentioned single state detection process 603 will be described with a specific determination example with reference to Fig. 10. Fig. 10(a) shows an example of an image 1000 of the nth frame, and Fig. 10(b) shows an example of an image 1010 of the next n+1th frame. Since the person 1001 in the image 1000 of the nth frame and the person 1011 in the image 1010 of the n+1th frame in Fig. 10(a) are linked persons, the same ID: 001 is assigned to them. Similarly, the person 1002 in the image 1000 of the nth frame and the person 1012 in the image 1010 of the n+1th frame are linked persons, so the same ID: 002 is assigned to them. Meanwhile, a new ID: 003 is assigned to the person 1013 in the image 1010 of the n+1th frame. Additionally, Figures 10(a) and 10(b) show boundary area 1030 as an example of a boundary area that is set near the edge of an image, and boundary area 1031 as an example of a boundary area that is set near the edge of a shelf.

[0035] In the example of the image 1000 of the n-th frame shown in FIG. 10(a), when focusing on the person 1001 with ID:001, the only other person detected is the person 1002 with ID:002, and the distance between these people is smaller (closer) than the judgment threshold distance 1020. That is, in the example of FIG. 10(a), since another person 1002 exists near the person 1001, the image analysis unit 104 judges that the person 1001 is in a "non-solo state". On the other hand, the same is true when focusing on the person 1002 with ID:002, since another person 1001 exists near the person 1002, the image analysis unit 104 also judges that the person 1002 is in a "non-solo state".

[0036] In the example of the image 1010 of the n+1th frame shown in FIG. 10(b), when focusing on the person 1011 of ID:001, the person 1012 of ID:002 and the person 1013 of ID:003 are at a distance from the person 1011 that is greater than the judgment threshold distance 1020 (farther away). Therefore, the person 1011 of ID:001 appears to be alone in the area shown in the image. However, the location of the person 1011 of ID:001 is within the boundary area 1030. Therefore, the image analysis unit 104 judges the person 1011 of ID:001 to be in a "quasi-alone state". Furthermore, when focusing on the person 1012 of ID:002, like the person 1011 of ID:001, there is no other person near the person 1012 in the image, but they are within the boundary area 1031. Therefore, the image analysis unit 104 judges the person 1012 with ID:002 to be in a "semi-alone state". On the other hand, when focusing on the person 1013 with ID:003, there is no other person in the vicinity of the person 1013 in the image, and the person is outside the boundary areas 1030 and 1031. Therefore, the image analysis unit 104 judges the person 1013 with ID:003 to be in a "alone state".

[0037] In this way, the image analysis unit 104 judges whether each person detected from the image of each frame is in a "solo state," "non-solo state," or "quasi-solo state." Note that in this embodiment, a person who is within a boundary area in the image but has no other people nearby, such as the person with ID:001 or ID:002, is determined to be in a "quasi-solo state," but it may also be configured to be determined to be in a "non-solo state." When configured in this way, a person who may have other people nearby can always be determined to be in a "non-solo state."

[0038] In addition, the image analysis unit 104 can also perform various additional determinations in the determination in the single state detection process 603. As an example of the additional determination, when a person who was present in a boundary area in an image of a certain frame disappears from the image in the next frame, the image analysis unit 104 determines that the person has passed through the boundary area and moved to an area not shown in the captured image (outside the captured image angle or behind a shelf, etc.). Here, considering the walking speed of a person and the determination threshold distance, it is considered that a person who has disappeared from a boundary area exists near the boundary area within a predetermined time from the time of disappearance. For example, assuming that the walking speed of a person is 1 meter per second and the determination threshold distance is 2 meters, it is considered that a person who has disappeared from a boundary area exists near the boundary area within 2 seconds from the time of disappearance. Based on this idea, the image analysis unit 104 may perform an additional determination such that, when a person has disappeared from a boundary area, a person who is in the boundary area for 2 seconds after the disappearance is determined to be in a "non-single state" rather than a "quasi-single state".

[0039] Another example of the additional determination will be described with reference to FIG. 11. In the example of FIG. 11, the rear side of the product shelf 1101 is not shown in the captured image, so both the left and right sides of the product shelf 1101 are set as boundary areas 1102. As in the above example, if a person disappears within this boundary area, it is considered that the person has entered the rear side of the product shelf 1101. On the other hand, if a person who did not exist in the previous frame appears within the boundary area 1102 in the image of the subsequent frame, it is considered that the person has exited from the rear side of the product shelf 1101. As in the example of FIG. 11, if all the entrances and exits to the rear of the product shelf 1101 are covered by the boundary area, it is possible to estimate the number of people present behind the product shelf 1101 by comparing the number of people who have entered and exited behind the product shelf 1101. For example, if the number of people who have entered is equal to the number of people who have exited, it is considered that no people are present behind the product shelf 1101. Based on this idea, the image analysis unit 104 estimates the number of people present behind the product shelf 1101, and if the number of people is zero, performs an additional judgment such as judging people within the boundary area to be in a "non-alone state" rather than a "quasi-alone state." As described above, various additional determinations can be added to the single state detection process 603 of this embodiment.

[0040] Following the single state detection process 603 in FIG. 6, the image analysis unit 104 performs a priority person determination process 604, which narrows down people to be targets of more detailed behavior detection processing from among people detected in the image. Here, the detailed behavior detection process may be, for example, a posture estimation process that detects posture and behavior based on the skeleton of a person. However, since such a process requires a large calculation resource, the image analysis unit 104 executes the process by focusing on people who should be analyzed preferentially among the people detected in the image. In the priority person determination process according to this embodiment, an example will be described in which a maximum of two people are determined from among the people present in the image as priority people to be the targets of the posture estimation process 605, that is, the targets of behavior detection.

[0041] FIG. 12 shows the result of determining two people as priority people to be the target of behavior detection by the posture estimation process 605 from three people with ID:001 to ID:003. In this embodiment, the priority people are determined in the order of solo state, semi-solo state, and non-solo state. In addition, when there are multiple people in the same state among the solo state, semi-solo state, and non-solo state, the person who has the longer stay time among the multiple people in the same state is given higher priority as the target person of behavior detection. In the example of FIG. 12, since only the person with ID:003 among the three people is in the "solo state", first, the person with ID:003 is determined as the priority person to be the target of behavior detection. Next, since the person with ID:001 and the person with ID:002 are both in the same "semi-solo state", the person with ID:001 who has the longer stay time among the two is determined as the priority person to be the target of behavior detection. At this point, the upper limit number of priority people, which is two, has been reached, so the person with ID:002 is not the target person of behavior detection by the posture estimation process 605.

[0042] In the above example, the number of priority people to be detected as behaviors by the posture estimation process 605 is limited to a maximum of two people based on the measurement result of the staying time and the result of the single state detection. Such a method of prioritizing is based on the hypothesis that suspicious people such as shoplifters are often alone and often stay for a long time. In this embodiment, the single state detection process 603 is configured to always determine whether or not the person is in a boundary area, but if the number of people detected in the image is equal to or less than the maximum number of priority people (two or less in this example), all of the detected people can be targeted for behavior detection. For this reason, the process may be configured to determine only "single state" and "non-single state" based only on the distance between people in the image, without determining whether or not the person is in a boundary area.

[0043] Following the priority person determination process 604 in Fig. 6, the image analysis unit 104 executes a posture estimation process 605 for each person determined as a target for behavior detection. Generally, a skeleton estimation method or the like is used to estimate the posture of a person (human body), and the skeleton estimation method includes a bottom-up method and a top-down method. In this embodiment, the top-down method is used as an example. First, the image analysis unit 104 cuts out rectangular areas surrounding each person (human body) detected in the image as person detection areas. Fig. 13 shows an example in which a person with ID:001, a person with ID:002, and a person with ID:003 are detected from an image, and rectangular areas surrounding these people are set as person detection areas 1301, 1302, and 1303. Then, as a top-down skeleton estimation process, the image analysis unit 104 inputs the cut-out images of these person detection areas 1301, 1302, and 1303, and estimates the positions of feature points such as joint points, eyes, and nose of each person.

[0044] Fig. 14 shows an example of input images and output information of skeleton estimation processing, and Fig. 14(a) shows an example of an input image 1401, which is an image obtained by cutting out a human detection area from an original captured image. Fig. 14(b) shows an example of output information 1402, which is information consisting of coordinate point groups of each joint, which is a result of skeleton estimation processing for the input image 1401. In the example of Fig. 14(b), an example is shown in which coordinate point groups of the shoulders, elbows, wrists, waists, knees, ankles, nose, and eyes are output as output information 1402 of the skeleton estimation processing result. Note that the skeleton estimation method is not limited to this example, and other known methods may be used.

[0045] The image analysis unit 104 can detect, for example, a person's behavior of looking around restlessly or suddenly crouching down by acquiring a group of coordinate points such as the output information 1402 in time series. For example, when detecting a behavior of looking around restlessly, a method is available in which the direction of the person and head is detected from the posture estimation processing result, the head shaking state is estimated from the change in the direction, and whether the head is shaking so that the estimated value of the head shaking state is equal to or greater than a preset value. It is also possible to detect a person who is crouching down from the coordinate positions and coordinate changes of each joint of the lower body. Furthermore, there is also a method of detecting the extension and contraction of the arm based on the coordinate change of the elbow or wrist, and detecting the action of picking up a product. With such a method, it is possible to detect posture and behavior based on skeletal information. The image analysis unit 104 executes the image analysis process of step S501 in FIG. 5 through the process described above, thereby detecting various actions of each person detected from the captured image.

[0046] Next, in step S502, the image analysis unit 104 determines whether or not there is a suspicious behavior pattern or behavior based on the various actions detected in step S501. The image analysis unit 104 determines whether or not there is a person who matches a pre-registered behavior pattern, for example, and if there is a matching person, it is determined that "a suspicious behavior pattern has been detected."

[0047] In this embodiment, four behavior patterns 1 to 4 as shown in FIG. 15 are registered. For example, pattern 1 is a behavior pattern in which a person in a solitary state who has stayed on the screen for 30 seconds or more looks around restlessly. Pattern 2 is a behavior pattern in which a person in a solitary state who has stayed on the screen for 30 seconds or more looks around restlessly and then stretches out his / her hand. Pattern 3 is a behavior pattern in which a person in a solitary state who has stayed on the screen for 30 seconds or more looks around restlessly and then crouches down. Pattern 4 is a behavior pattern in which a person in a semi-solo state who has stayed on the screen for 60 seconds or more looks around restlessly and then crouches down. The behavior patterns that are determined to be suspicious behavior are not limited to the example in FIG. 15. Behavior patterns for not only solitary and semi-solo states but also non-solo states may be registered, and the stay time may be shorter or longer than 30 or 60 seconds. If there is a person matching any of these preregistered behavior patterns, the image analysis unit 104 determines in step S502 of FIG. 5 that "a suspicious behavior pattern has been detected."

[0048] If a suspicious behavior pattern is detected in step S502, the image analysis unit 104 notifies the audio playback unit 105 of the detection in the next step S503. As a result, the audio playback unit 105 outputs, from the speaker 109, a notification sound for a store clerk or a security guard, or a warning sound for calling out to a suspicious person, as described above. According to the information processing device 101 of this embodiment, by performing the above-described configuration and processing, it becomes possible to detect suspicious behavior patterns of a person detected in an image and automatically play back audio.

[0049] Next, the boundary area setting process in the single state detection process 603 in Fig. 6 will be described. Various methods for setting the boundary area are possible, but in this embodiment, "a method for directly designating the boundary area itself", "a method for designating the boundary area using a boundary line", and "a method for automatically setting" are given as examples. Below, these boundary area setting methods will be described using an example in which an image captured by the imaging device 102 is displayed on the screen of the display device 208, and the boundary area is set on the display screen.

[0050] In this embodiment, the user can set a boundary area by operating a pointing device such as a mouse of the input device 207 while viewing an image displayed on the screen of the display device 208 by the display unit 107. Operation input by the user to the input device 207 is acquired by the operation acquisition unit 108 and sent to the setting unit 106, and the setting unit 106 sets the boundary area based on the operation input.

[0051] 16 to 21 are diagrams showing examples of setting screens when a boundary area is set. The setting screens of FIGS. 16 to 21 have setting items of "specify area", "specify boundary line", and "automatic specification", and the user can select one of these setting items by operating a pointing device of the input device 207 or the like. The setting screens of FIGS. 16 to 21 also display an image display area in which a captured image is displayed, an OK button that the user selects when setting a candidate area as a boundary area, and a cancel button that the user selects when canceling the candidate area. In the information processing device 101 of this embodiment, the setting unit 106 manages each display content in the setting screen, and causes it to be displayed on the display device 208 via the display unit 107.

[0052] FIG. 16 shows a setting screen when the setting item "specify area" is selected. When the setting item "specify area" is selected, the setting unit 106 executes processing related to the "method of directly specifying the boundary area itself". When "specify area" is selected, the setting unit 106 displays the captured image acquired by the image acquisition unit 103 from the imaging device 102 in the image display area 1600. Then, when the user selects (presses) an OK button after a candidate area is designated in the image display area 1600, the setting unit 106 sets the user-designated candidate area as the boundary area. FIG. 16 shows an example in which the user designates a candidate area by clicking a plurality of desired positions in the image display area 1600 with a mouse or the like. For example, when the user designates each position indicated by a plurality of black circles 1601 in the image display area 1600 in sequence, the setting unit 106 sets the area surrounded by the line segments connecting the plurality of black circles 1601 in sequence as the user-designated candidate area 1602. Similarly, the setting unit 106 sets the area surrounded by the line segments connecting the black circles 1601 sequentially specified by the user as a candidate area 1603. Thereafter, when the user selects (presses) the OK button, the setting unit 106 confirms and registers the candidate areas 1602 and 1603 as boundary areas. In the example of FIG. 16, the boundary area corresponding to the candidate area 1602 is the boundary area near the edge of the captured image, and the boundary area corresponding to the candidate area 1603 is the boundary area near the edge of the product shelf. As shown in FIG. 16, when the setting item "specify area" is selected, the setting unit 106 can set the boundary area directly specified by the user.

[0053] 17 to 19 show an example of the transition of the setting screen when the setting item "Specify boundary line" is selected. When the setting item "Specify boundary line" is selected, the setting unit 106 executes processing related to the "method of designating by boundary line". When "Specify boundary line" is selected, the setting unit 106 displays the captured image in the image display area 1700, as in the example of FIG. 16. Then, when the user specifies a boundary line in the image display area 1700, the setting unit 106 sets a candidate area based on the boundary line. After that, when the user selects (presses) the OK button, the candidate area set based on the boundary line is set as the boundary area.

[0054] 17 shows an example in which a boundary line 1703 is specified by a user clicking two positions, a start point and an end point, with a mouse or the like within an image display area 1700. For example, when the user specifies two positions, a start point and an end point indicated by black circles 1701 and 1702 within the image display area 1700, the setting unit 106 sets a line segment connecting the viewpoint and the two end points, black circles 1701 and 1702, as the specified boundary line 1703. Here, when the boundary line 1703 specified by the user is an edge of a captured image, the setting unit 106 sets an area that is a predetermined distance inward from the boundary line 1703 toward the center of the screen as a candidate area 1704.

[0055] 18 shows an example in which a boundary line 1803 is specified by a user clicking two positions, a start point and an end point, in the image display area 1700 with a mouse or the like, following the example in FIG. 17. For example, when the user specifies two positions, a start point and an end point indicated by black circles 1801 and 1802 in the image display area 1700, the setting unit 106 sets a line segment connecting the two black circles 1801 and 1802, which are the viewpoint and the end point, as the specified boundary line 1803. Here, when the boundary line 1803 specified by the user is an end of a captured image, the setting unit 106 sets an area inward from the boundary line 1803 by a predetermined distance toward the center of the screen as a candidate area 1804. Note that when two or more specified candidate areas partially overlap or are adjacent to each other, the setting unit 106 may set the two or more overlapping or adjacent candidate areas as one candidate area.

[0056] 19 shows an example in which the user specifies a boundary line 1903 by clicking two positions, a start point and an end point, in the image display area 1700 with a mouse or the like, following the example in FIG. 18. For example, when the user specifies two positions, a start point and an end point, indicated by black circles 1901 and 1902, in the image display area 1700, the setting unit 106 sets a line segment connecting the two start point and end point black circles 1901 and 1902 as the user-specified boundary line 1903. If the user-specified boundary line 1903 is not an end of the captured image, the setting unit 106 specifies two areas, each of which corresponds to a predetermined distance, with the boundary line 1903 as the candidate areas 1904 and 1905 specified by the boundary line 1903. FIG. 19 shows an example in which the boundary line 1903 is specified to match the end of a product shelf, and the candidate area 1904 is set to the left of the boundary line 1903, and the candidate area 1905 is set to the left of the boundary line 1903. Here, if the user-specified boundary line is not an edge of the captured image, the setting unit 106 allows the user to select the candidate area bordering the boundary line by clicking the mouse or the like. In the example of FIG. 19, the area to the right of the boundary line 1903 is a product shelf, and is an area in which a person cannot exist, and therefore it is considered inappropriate to set it as a boundary area. On the other hand, the area to the left of the boundary line 1903 is a space, and is an area in which a person can exist. For this reason, the user selects the candidate area 1904 to the left of the boundary line 1903. In other words, by selecting the candidate area 1904, the user can specify the boundary area to the left of the product shelf.

[0057] Thereafter, when the user selects the OK button, the setting unit 106 sets candidate area 1704 in Fig. 17 and candidate area 1804 in Fig. 18 as boundary areas, and also confirms and registers candidate area 1904 in Fig. 19 as a boundary area. That is, candidate area 1704 in Fig. 17 and candidate area 1804 in Fig. 18 become boundary areas near the edge of the captured image, and candidate area 1904 in Fig. 19 becomes a boundary area near the edge of the product shelf. As shown in Figs. 17 to 19, when the setting item "Specify boundary line" is selected, the setting unit 106 can set a boundary area based on a boundary line specified by the user.

[0058] As described above, when the "Specify Border Line" setting item is selected, in this embodiment, the area within a specified distance from the border line is set as the border area, but the distance from the border line that defines the border area may be changed depending on the position within the image display area. Fig. 20 is a diagram showing an example in which the distance from the boundary line to the boundary area is changed depending on the position in the image display area. In Fig. 20, the setting items "Specify Area", "Specify Boundary Line", and "Specify Automatically", the image display area 2000, the OK button, and the cancel button are the same as those in the setting screen described above. Although not shown in Fig. 20, it is assumed that a boundary line is specified by the user in the same manner as the boundary line 1703 in Fig. 17, the boundary line 1803 in Fig. 18, and the boundary line 1903 in Fig. 19 described above, and the left side of the product shelf is selected as the boundary area as in the example of Fig. 19.

[0059] In the case of the boundary area setting example of Fig. 20, the distance from the boundary line in boundary area 2001 and boundary area 2002 is changed depending on the position in image display area 2000. That is, in the example of Fig. 20, setting unit 106 decreases the distance from the boundary line from the front side to the back side of the captured image displayed in image display area 2000. This takes into consideration that the real space distance occupied by a unit distance on the captured image increases the further back it is.

[0060] FIG. 21 shows the setting screen when the setting item "Set automatically" is selected. When the setting item "Set automatically" is selected, the setting unit 106 executes processing related to the "method of setting automatically". Even when "Set automatically" is selected, the setting unit 106 displays a setting screen including setting items, an image display area 2100, an OK button, and a Cancel button similar to those in the examples of FIGS. 16 to 20 described above. On the other hand, when "Set automatically" is selected, the user does not specify the setting by operating the mouse as described above, and the setting unit 106 automatically sets the boundary area based on the result of the human detection process performed by the image analysis unit 104.

[0061] An example of automatic setting of a boundary area based on the result of the person detection process will be described with reference to FIG. A method for automatically setting a boundary area will be described with reference to FIG. 22. The dotted rectangle 2200 shown in FIG. 22 represents the imaging angle of view of the imaging device 102. The image analysis unit 104 continues to execute a person detection process for a predetermined time (for example, 24 hours) on the captured image corresponding to this imaging angle of view. The setting unit 106 stores the person detection results detected in time series by the image analysis unit 104 through the person detection process for the predetermined time, and accumulates the appearance position and disappearance position of the stored person. Here, in the space in which the surveillance camera exemplified in this embodiment is installed, for example, real space spreads to the left side of the imaging angle of view and below the imaging angle of view, and real space also spreads behind the product shelf. In this case, it is considered that the positions where a person appears and disappears in the captured image will be concentrated in the shaded areas 2201, 2202, and 2203 shown in FIG. 22. For this reason, the setting unit 106 sets an area at a predetermined distance from the coordinate group of the appearance coordinates where a person appears and the disappearance coordinates where a person disappears in the captured image as the boundary area. That is, since the coordinate group of the appearance coordinates and disappearance coordinates in the shaded areas 2201 and 2202 corresponds to the edge of the captured image, the setting unit 106 sets an area a predetermined distance from these coordinate groups toward the center of the screen as the boundary area. Also, since the coordinate group of the appearance coordinates and disappearance coordinates in the shaded area 2203 corresponds to the edge of the product shelf, the setting unit 106 sets an area a predetermined distance from these coordinate groups toward the spatial direction in which the person appears as the boundary area. When "automatic setting" is selected in this manner, the setting unit 106 can automatically set the boundary area based on the result of human detection processing performed by the image analysis unit 104 for a predetermined period of time.

[0062] The setting unit 106 of this embodiment can use the above-described example setting screen to set the boundary area in the single state detection process 603. Note that the method for setting the boundary area is not limited to the above-described example, and any method that can specify an area may be used.

[0063] As described above, the information processing device 101 of this embodiment is capable of performing an independent determination that takes into consideration the possibility that another person may be present in a position not captured by the captured image of the imaging device 102. Therefore, according to this embodiment, it is possible to reduce the occurrence of erroneous determination in the independent determination. In this embodiment, an example of a system that prevents shoplifting by notifying store clerks or security guards when a suspicious person is detected, or by automatically calling out to people behaving suspiciously, is given; however, the system is not limited to this example, and various modifications and alterations are possible within the scope of the gist.

[0064] The present invention can also be realized by a process in which a program for implementing one or more of the functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. The present invention can also be realized by a circuit (e.g., ASIC) for implementing one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features.

[0065] The disclosure of this embodiment includes the following configuration, method, and program. (Configuration 1) A person detection means for detecting a person from an image; a single person detection means for detecting, among the detected persons, a person who is not within a threshold distance from another person and is not within a predetermined area set in the image as a single person; 13. An information processing device comprising: (Configuration 2) The single state detection means a first determination means for determining whether or not another person is present within the threshold distance from a target person to be processed; A second determination means for determining whether the target person is within the predetermined area; a third determination means for determining that the target person is alone when the first determination means determines that no other person exists within the threshold distance from the target person and the second determination means determines that the target person is not within the predetermined area, and otherwise determining that the target person is not alone; 2. The information processing device according to configuration 1, comprising: (Configuration 3) 3. The information processing device according to configuration 1 or 2, further comprising a setting unit for setting the predetermined area for the image. (Configuration 4) A display means for displaying the image; An operation acquisition means for acquiring an operation of a user; and 4. The information processing apparatus according to configuration 3, wherein the setting means sets an area designated by the user in the image displayed on the display means as the predetermined area. (Configuration 5) A display means for displaying the image; An operation acquisition means for acquiring an operation of a user; and The information processing device according to configuration 3, wherein the setting means sets as the predetermined area at least a part of an area within a predetermined distance from a boundary line specified by the user for the image displayed on the display means. (Configuration 6) The information processing device according to configuration 5, wherein the setting means sets an area within a predetermined distance from the boundary line specified by the user as the predetermined area when the boundary line corresponds to an edge of the image. (Configuration 7) The information processing device described in configuration 5 or 6, characterized in that when the boundary line specified by the user does not correspond to the edge of the image, the setting means sets the area selected by the user out of the two areas identified by the boundary line as the specified area. (Configuration 8) 8. The information processing device according to any one of configurations 5 to 7, wherein the setting means sets the predetermined area by determining the predetermined distance from the boundary line depending on a position within the image. (Configuration 9) 4. The information processing apparatus according to configuration 3, wherein the setting means sets the predetermined area based on positions in the image of people detected in time series by the detection means. (Configuration 10) The information processing device described in configuration 9, characterized in that the setting means acquires coordinates corresponding to at least one of a position where the person is newly detected by the detection means and a position where the person that was detected by the detection means is no longer detected based on a position in the time-series image of the person, and sets the specified area based on the coordinates. (Configuration 11) 11. The information processing apparatus according to configuration 10, wherein the setting means sets at least a part of an area within a predetermined distance from the coordinates as the predetermined area. (Configuration 12) 12. The information processing apparatus according to claim 11, wherein said setting means sets an area within a predetermined distance from said coordinates as said predetermined area when said coordinates correspond to an edge of said image. (Configuration 13) The information processing device according to configuration 11 or 12, characterized in that, when the coordinates do not correspond to the edge of the image, the setting means sets an area within a predetermined distance from the coordinates in which the person is detected as the predetermined area. (Configuration 14) The information processing device described in any one of configurations 1 to 13, characterized in that the solo state detection means determines that a person who is not present within the threshold distance and is within the specified area is in a quasi-solo state, which may be either a solo state or a non-solo state. (Configuration 15) The information processing device described in configuration 14, characterized in that the single state detection means determines that a person in the specified area is in a non-single state for a predetermined time period from the time when a state in which another person is detected by the detection means in the specified area changes to a state in which the other person is no longer detected. (Configuration 16) a measuring means for measuring a residence time of a person detected by the person detection means; a determination means for determining a person to be a target of additional information processing from among the plurality of people detected by the person detection means, based on at least one of the measurement result of the residence time and the detection result of the single state detection means; and The information processing device described in configuration 14 or 15, characterized in that the determination means selects a person to be the target of the additional information processing in the order of the detection results by the solo state detection means: the solo state, the quasi-solo state, and the non-solo state. (Configuration 17) The information processing device described in configuration 16, characterized in that when the number of people in the image detected by the person detection means is equal to or less than a predetermined number, and if no other people are present within a predetermined distance from the target person being processed, the single state detection means determines that the target person is in the single state, and does not determine whether the target person is within the predetermined area. (Configuration 18) 17. The information processing device according to configuration 16, wherein the additional information processing is processing for detecting a person having a predetermined behavior pattern. (Method 1) a person detection step of detecting a person from an image; a single-person state detection step of detecting, among the detected persons, a person who is not within a threshold distance from another person and is not within a predetermined area set in the image as a single person; 13. An information processing method comprising: (Program 1) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 18. [Explanation of symbols]

[0066] 101: information processing device, 102: imaging device, 103: image acquisition unit, 104: image analysis unit, 105: audio playback unit, 106: setting unit, 107: display unit, 108: operation acquisition unit, 109: speaker

Claims

1. A person detection means for detecting a person from an image; a single person detection means for detecting, among the detected persons, a person who is not within a threshold distance from another person and is not within a predetermined area set in the image as a single person; 13. An information processing device comprising:

2. The single state detection means a first determination means for determining whether or not another person is present within the threshold distance from a target person to be processed; A second determination means for determining whether the target person is within the predetermined area; a third determination means for determining that the target person is alone when the first determination means determines that no other person exists within the threshold distance from the target person and the second determination means determines that the target person is not within the predetermined area, and otherwise determining that the target person is not alone; 2. The information processing apparatus according to claim 1, further comprising:

3. 3. The information processing apparatus according to claim 1, further comprising a setting unit for setting the predetermined area for the image.

4. A display means for displaying the image; An operation acquisition means for acquiring an operation of a user; and 4. The information processing apparatus according to claim 3, wherein the setting means sets an area designated by the user on the image displayed on the display means as the predetermined area.

5. A display means for displaying the image; An operation acquisition means for acquiring an operation of a user; and 4. The information processing apparatus according to claim 3, wherein the setting means sets, as the predetermined area, at least a part of an area within a predetermined distance from a border line designated by the user for the image displayed on the display means.

6. 6. The information processing apparatus according to claim 5, wherein said setting means sets, when the boundary line designated by the user corresponds to an edge of the image, an area within a predetermined distance from the boundary line as the predetermined area.

7. The information processing device according to claim 5, characterized in that, when the boundary line specified by the user does not correspond to an edge of the image, the setting means sets the area selected by the user out of the two areas specified by the boundary line as the specified area.

8. 6. The information processing apparatus according to claim 5, wherein said setting means sets said predetermined area by determining said predetermined distance from said boundary line in accordance with a position within said image.

9. 4. The information processing apparatus according to claim 3, wherein the setting means sets the predetermined area based on the positions of people in the image detected in time series by the detection means.

10. The information processing device according to claim 9, characterized in that the setting means acquires coordinates corresponding to at least one of a position where the person is newly detected by the detection means and a position where the person that was detected by the detection means is no longer detected based on a position in the time-series image of the person, and sets the specified area based on the coordinates.

11. 11. The information processing apparatus according to claim 10, wherein the setting means sets at least a part of an area within a predetermined distance from the coordinates as the predetermined area.

12. 12. The information processing apparatus according to claim 11, wherein said setting means sets, when said coordinates correspond to an edge of said image, an area within a predetermined distance from said coordinates as said predetermined area.

13. The information processing device according to claim 11 , characterized in that, when the coordinates do not correspond to an edge of the image, the setting means sets an area within a predetermined distance from the coordinates in which the person is detected as the predetermined area.

14. The information processing device described in claim 1, characterized in that the solo state detection means determines that a person who is not present within the threshold distance and is within the specified area is in a quasi-solo state, which may be either a solo state or a non-solo state.

15. The information processing device according to claim 14, characterized in that the single state detection means determines that a person present within the specified area is in a non-single state for a predetermined time period from the time when a state in which another person is detected by the detection means within the specified area changes to a state in which the other person is no longer detected.

16. a measuring means for measuring a residence time of a person detected by the person detection means; a determination means for determining a person to be a target of additional information processing from among the plurality of people detected by the person detection means, based on at least one of the measurement result of the residence time and the detection result of the single state detection means; and The information processing device according to claim 14 or 15, characterized in that the determination means selects a person to be the target of the additional information processing in the order of the detection results by the solo state detection means: the solo state, the quasi-solo state, and the non-solo state.

17. The information processing device according to claim 16, characterized in that when the number of people in the image detected by the person detection means is equal to or less than a predetermined number, and if there are no other people within a predetermined distance from the target person being processed, the single state detection means determines that the target person is in the single state, and does not determine whether the target person is within the predetermined area.

18. 17. The information processing apparatus according to claim 16, wherein the additional information processing is processing for detecting a person having a predetermined behavior pattern.

19. a person detection step of detecting a person from an image; a single-person state detection step of detecting, among the detected persons, a person who is not within a threshold distance from another person and is not within a predetermined area set in the image as a single person; 13. An information processing method comprising:

20. Computer, A person detection means for detecting a person from an image; a single person detection means for detecting, among the detected persons, a person who is not within a threshold distance from another person and is not within a predetermined area set in the image as a single person; A program that causes the device to function as an information processing device having the above-mentioned configuration.

Images