Image processing device, image processing method, and program

Abstract

To provide an image processing device capable of detecting a suspicious behavior with high accuracy.SOLUTION: An image processing device 200 includes a detection tracking unit 202 that performs detection tracking of a person by using an image obtained by capturing a moving image, a direction detection unit 204 that detects a direction of the face of the person, and a behavior determination unit 205 that detects a suspicious behavior of the person on the basis of an accumulated value of a time-series change amount of the direction of the face.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image processing technique for detecting suspicious behavior of a person from captured video images. [Background technology]

[0002] Shoplifting has become a serious problem in the retail industry. Surveillance camera systems have been introduced for some time, and in recent years, technology has been developed to analyze surveillance camera images and detect suspicious behavior in order to detect shoplifting and its signs and take action before it occurs. The technology described in Patent Document 1 acquires a person's line of sight or face direction and the arrangement of objects around the person, and determines that the person is behaving suspiciously if they turn their head or gaze in the direction of an object. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] International Publication No. 2018 / 008575 Summary of the Invention [Problem to be solved by the invention]

[0004] In the case of the technology described in Patent Document 1, it is possible to detect suspicious behavior to a certain extent, but since a person engaging in suspicious behavior is not necessarily facing the direction of an object, it is difficult to say that suspicious behavior can be detected with a high degree of accuracy.

[0005] Therefore, an object of the present invention is to make it possible to detect suspicious behavior with high accuracy. [Means for solving the problem]

[0006] The image processing device of the present invention includes: a direction detection means for detecting a face direction of a person using an image obtained by capturing a moving image; A possible area where there is no obstruction blocking the line of sight and an impossible area where the obstruction is present are identified in the imaging area of ​​the video, and a determination result as to whether or not the possible area is continuous from the face of the person to the end of the image in the direction of the face direction is obtained. and a behavior detection means for detecting suspicious behavior of the person based on the behavior. Effect of the Invention

[0007] According to the present invention, suspicious behavior can be detected with high accuracy. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing device according to an embodiment; [Diagram 2] FIG. 13 is a diagram showing an example of a possible area, an impossible area, and the direction of a person's face. [Diagram 3] 4 is a flowchart showing the operation of the imaging apparatus. [Figure 4] 4 is a flowchart of an image analysis process executed by the image processing device. [Diagram 5] FIG. 13 is a diagram illustrating an example of a UI for setting a direction. [Figure 6] FIG. 2 illustrates an example of a hardware configuration. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present invention will be described with reference to the drawings. Note that the following embodiment does not limit the present invention, and not all of the combinations of features described in the present embodiment are essential to the present invention. The configurations shown in the following embodiment are merely examples, and the present invention is not limited to the configurations shown in the drawings. In addition, the same components will be described with the same reference numerals.

[0010] <First embodiment> In this embodiment, a surveillance camera system installed in a retail store such as a convenience store will be described as an application example of the image processing device. In the system according to this embodiment, a surveillance camera is installed in a store to take pictures, and the video images obtained by the shooting are analyzed in real time to detect suspicious behavior and notify employees, etc. Here, for example, a person who is planning to shoplift does not necessarily face the direction of an object such as a product, and often checks the surroundings because he or she is concerned about being seen by others. At this time, the direction that the person checks is likely to be the direction of an open space where the person is more likely to be seen by others. Therefore, in the system according to this embodiment, suspicious behavior can be detected with high accuracy by considering the space that the person who is planning to shoplift, etc. checks, that is, the space where the person is more likely to be seen.

[0011] <System configuration> 1 is a block diagram showing an example of the system configuration of a surveillance system according to the present embodiment. This system is roughly divided into an imaging device 100 and an image processing device 200. The imaging device 100 includes an imaging unit 101 and a transmission unit 102 . The imaging unit 101 is composed of an imaging lens, an imaging sensor such as a CCD or CMOS, an imaging signal processing unit, etc. The imaging unit 101 transmits images for each frame constituting a moving image captured at a frame cycle, or images acquired at a predetermined time interval from the moving image, to the transmission unit 102. The transmission unit 102 adds various information related to shooting, such as the angle of view at the time of capturing an image in the imaging device 100, and additional information such as time, to the image, converts it into data that can be transmitted over a network, and transmits it to the image processing device 200.

[0012] The image processing device 200 is configured to have a receiving unit 201, a detection and tracking unit 202, a memory unit 203, an orientation detection unit 204, a behavior determination unit 205, and an output unit 206, as functional units that execute image analysis processing to enable detection of suspicious behavior in this embodiment. The receiving unit 201 extracts an image from the data received from the image capturing device 100, and sends it to a detection and tracking unit 202 at the downstream side.

[0013] The detection and tracking unit 202 detects and tracks people in an image. Any method may be used to detect people from an image. For example, the detection and tracking unit 202 detects people from an image by pattern matching edges in an image with a person's shape, using a CNN (Convolutional Neural Network), or by background subtraction. The detected person is expressed as data that indicates the center coordinates of a rectangle surrounding the person, with the upper left corner of the image as the origin, and the width and height of the rectangle. The tracking process is a process of matching a detected person with multiple images in the time direction. Any tracking process method may be used. For example, the detection and tracking unit 202 predicts the position of a person in the current frame from the center position and movement vector of the person included in the previous frame, and matches the person based on the predicted position of the person and the center position of the person included in the current frame. Then, the detection and tracking unit 202 assigns a person ID (identification information) to the associated person. As a result, people to whom the same person ID is assigned are treated as the same person from now on. The detection and tracking unit 202 repeats this process for each frame in the time direction to achieve tracking of each person.

[0014] The storage unit 203 stores the coordinates of the person in the image for each person ID acquired by the detection and tracking unit 202. Furthermore, the detection and tracking unit 202 acquires trajectory information representing the movement route of each person by chronologically connecting the coordinates of each person for each person ID stored in the storage unit 203. The storage unit 203 also stores trajectory information representing the movement trajectory acquired by the detection and tracking unit 202 for each person ID.

[0015] The orientation detection unit 204 detects the orientation of the face of the same person for each person ID obtained by the detection and tracking unit 202. The orientation detection unit 204 detects the area of ​​the person's face using a method similar to the person detection process in the detection and tracking unit 202, such as a pattern matching method with the face shape or a method such as CNN, and further detects each organ in the face, such as the eyes, and detects the orientation of the person's face from the positional relationship of each organ. The orientation detection unit 204 acquires data represented by the angle of inclination of the face, the angle in the vertical direction of the face, the angle in the horizontal direction of the face, and the like, as the detection result of the face orientation. However, if the person is facing backwards and the face, the organs of the face, or the face orientation cannot be detected, the orientation detection unit 204 sets the angle of inclination of the face, the angle in the vertical direction of the face, and the angle in the horizontal direction of the face as data of the face orientation detection result as data with no value. Note that the method of detecting the face orientation is not limited to the above-mentioned method, and may be, for example, a method using CNN or a template matching method.

[0016] The behavior determination unit 205 uses the face direction detection result by the orientation detection unit 204 and the trajectory information stored in the storage unit 203 to determine whether or not a person to which a person ID has been assigned has engaged in suspicious behavior. In the present embodiment, the behavior determination unit 205 determines a face swing behavior and a visible direction, which will be described later, based on the face direction detection result and the trajectory information stored in the storage unit 203, and performs a process of determining suspicious behavior of a person based on these determination results. Then, when the behavior determination unit 205 detects suspicious behavior of a person, it sends a suspicious behavior detection notice indicating that to the output unit 206. Details of the suspicious behavior determination and detection process in the behavior determination unit 205 will be described later.

[0017] When the output unit 206 receives a suspicious behavior detection notification of a person from the behavior determination unit 205, it displays an image on a monitor screen (not shown) in which the area of ​​the person in which the suspicious behavior was detected is highlighted by adding a predetermined mark such as a rectangle. This makes it possible to inform the person in question of the suspicious behavior detected to the manager of the surveillance camera, the store employee, etc. In addition, an image of the person may be sent to a mobile terminal carried by the manager or the employee, etc., to encourage caution. The actions taken by the output unit 206 when it receives a suspicious behavior detection notification are not limited to these, and can be utilized for various actions related to shoplifting countermeasures.

[0018] Hereinafter, a detailed description will be given of the process performed by the behavior determination unit 205 of this embodiment, in which the face direction detection results and trajectory information are used to determine the face movement behavior and the visible direction for each person ID, and the process of determining and detecting suspicious behavior of a person based on these determination results. <Judgment of face-shaking behavior> First, the determination of facial movement behavior, which is one of the suspicious behaviors of a person, will be described. In this embodiment, a face-turning behavior, which is one of suspicious behaviors, is a behavior in which a person turns his or her face to check the surroundings. In this embodiment, the behavior determination unit 205 obtains the magnitude of the time-series change in the face direction from the face direction detection result, that is, the amount of time-series change in the face direction, and stores the amount of time-series change as a determination score inside. Then, the behavior determination unit 205 determines whether or not a face-turning behavior has been performed based on the determination score. At this time, the behavior determination unit 205 calculates, for each person ID, the absolute value of the difference between the face direction detection results obtained from two images that are consecutive in the time direction. That is, the behavior determination unit 205 calculates the absolute value of the difference between the angle of the face inclination, the absolute value of the angle of the face in the up-down direction, and the absolute value of the angle of the face in the left-right direction in the two images. Then, if one or more of the absolute values ​​of the differences for each angle exceed the face direction threshold, the behavior determination unit 205 increases the determination score by determining that the face direction has changed suddenly. On the other hand, if both absolute difference values ​​are equal to or smaller than the face direction threshold, the behavior determining unit 205 determines that the face direction is stable and decreases the determination score.

[0019] To explain with a specific example, the face direction detection result of a person is expressed by the face inclination angle, the face vertical angle, and the face horizontal angle as described above. For example, the face direction detection result of a person at time t-1 is {face inclination angle, face vertical angle, face horizontal angle}={+5,-10,-15}, the face direction detection result at time t is {+5,-15,+50}, and the face direction threshold is 50. In this example, the absolute difference value between the face direction detection results at time t and time t-1 is {0,0,65}, and any one or more of the items of the face direction detection result exceeds the face direction threshold, so the behavior determination unit 205 increases the determination score as the face direction has changed suddenly. Note that, for example, if all the items of the face direction detection result are equal to or less than the face direction threshold, the behavior determination unit 205 decreases the determination score as the face direction is stable. Then, every time the judgment score is changed, the behavior judgment unit 205 compares it with a predetermined judgment threshold, and if the judgment score exceeds the judgment threshold, it judges that the person has performed a facial movement behavior.

[0020] In the above example, the face direction threshold is set to one value, but it may be set to a different value for each item of the face direction detection result. The face direction threshold, the judgment threshold, and the increase and decrease amounts of the judgment score may be values ​​determined in a prior experiment, values ​​determined by observing the behavior of a person captured on a surveillance camera, or values ​​determined by a simulation using a large number of images labeled with face-waving behavior and non-face-waving behavior.

[0021] <Determining the visible direction> Next, the determination of the visible direction will be described. In this embodiment, a visible direction is a direction in which there is no obstruction within a specific distance from a person in a specific direction. In this embodiment, prior to determining the visible direction, the image is divided into an area where a person can exist (hereinafter, referred to as a possible area) and an area where a person cannot exist (hereinafter, referred to as an impossible area). In this embodiment, the possible area and the impossible area are set using trajectory information of multiple people stored in the storage unit 203. Note that in this embodiment, an example is given in which the possible area and the impossible area are automatically set by the behavior determination unit 205 based on the trajectory information in the storage unit 203, but for example, they may be set by a separately provided setting unit not shown, or may be set by an administrator of the surveillance camera or the like through input or the like. When the administrator of the surveillance camera or the like sets the possible area and the impossible area, for example, an image is displayed on a monitor screen or the like, and the administrator of the surveillance camera or the like manually sets the possible area and the impossible area by, for example, operating a mouse or the like.

[0022] FIG. 2 is a diagram showing an example in which an image is divided into a plurality of small regions in a lattice shape, and a possible region or an impossible region is set for each of the small regions. A possible region is set when there is statistically a person's trajectory information, and an impossible region is set when there is no person's trajectory information statistically. That is, a possible region is a region that indicates that there are no obstacles to the extent that a person can move at least, and that there is nothing blocking the line of sight, while an impossible region is a region that indicates that a person has difficulty moving due to obstacles or the like, and that the line of sight is blocked. In this embodiment, a region in which a predetermined amount or more of a person's trajectory exists is set as a possible region, while a region in which a predetermined amount or less of a person's trajectory exists is set as an impossible region. For example, based on trajectory information for a predetermined period in the past (for example, the past one day), the behavior determination unit 205 sets each small region in which a predetermined percentage or more (for example, 5% or more) of the number of people appearing in the image have movement trajectories, as a possible region. Meanwhile, the behavior determination unit 205 sets the remaining small regions that were not set as possible regions as impossible regions. FIG. 2 shows an example in which an image 300 is divided into multiple grid-like small regions, with each small region 301 indicated by thick lines in the figure being set as a possible region, and each small region 302 indicated by thin lines in the figure being set as an impossible region.

[0023] The predetermined amount for setting the possible area may be a value determined by a prior experiment, a value determined by observing the behavior of a person captured by a security camera, or a value determined by a simulation using a large number of images capturing a moving person. For example, when a security camera manager or the like sets the possible area and the impossible area, the image displayed on a monitor screen or the like may be divided into small areas in a lattice pattern as shown in FIG.

[0024] Furthermore, the behavior determination unit 205 performs a process of determining the viewable direction based on the information on the face direction of the person detected by the direction detection unit 204 and the possible area and the impossible area set as described above. In the case of this embodiment, when the possible area continues from the face center of the person to the image edge in the face direction, the behavior determination unit 205 determines that the line of sight of the person is a viewable direction, that is, determines that the person at this time looks in the viewable direction. For example, in FIG. 2, an arrow 303 indicates the face direction of the person 304, a line segment 305 indicates from the face center of the person 304 to the image edge, and each small area on the line segment 305 all belongs to the possible area. Therefore, the behavior determination unit 205 determines that the line of sight of the person 304 is a viewable direction, that is, determines that the person 304 looks in the viewable direction. On the other hand, for example, when there is one or more small areas belonging to the impossible area among each small area from the face center of the person 304 to the image edge, the behavior determination unit 205 determines that the person does not look in the viewable direction.

[0025] As described above, the possible area is a place where the person's trajectory is more than a predetermined amount, and is an area indicating that there are no obstacles to the extent that the person can move at a minimum and that there is nothing blocking the line of sight. Therefore, as shown in FIG. 2, when the line segment 305 to the image edge in the face direction belongs only to the possible area, it can be treated as the person looking in a direction that is visible. On the other hand, the impossible area is a place where the person's trajectory is less than a predetermined amount, and is an area indicating that the person has difficulty moving due to an obstacle or the like and that something blocking the line of sight is present. Therefore, as shown in FIG. 2, when there is one or more small areas that belong to the impossible area on the line segment to the image edge in the face direction, it can be treated as the person looking in a direction that is difficult to see.

[0026] The behavior determination unit 205 then performs a judgment on suspicious behavior of the person based on the aforementioned judgment result of the face movement behavior and the judgment result of the visible direction. In this embodiment, the behavior determination unit 205 internally holds a suspicious score for each person ID, and in the same manner as in the face-turn determination process described above, if the face direction changes suddenly, the suspicious score corresponding to the person ID is increased. When performing suspicious behavior determination, the behavior determination unit 205 obtains a time-series change amount of the face direction from the face direction detection result in the same manner as in the face-turn determination described above, and further holds the accumulated value of the time-series change amount as the suspicious score. Then, the behavior determination unit 205 determines whether the person has performed suspicious behavior based on the suspicious score. For example, in the same manner as in the face-turn behavior determination process, if the face direction of the person changes suddenly, the behavior determination unit 205 increases the suspicious score corresponding to the person ID. In addition, if the behavior determination unit 205 determines that the person has looked in the visible direction based on the result of the visible direction determination described above, a bonus score is added to the suspicious score, that is, the suspicious score is further increased. On the other hand, similar to the face-shake behavior determination process, when the behavior determination unit 205 determines that the person's face direction is stable, it decreases the suspiciousness score corresponding to the person ID. Then, each time the suspiciousness score changes, the behavior determination unit 205 compares it with the suspiciousness determination threshold, and when the suspiciousness score exceeds the suspiciousness determination threshold, it determines that the person has behaved suspiciously. On the other hand, when the suspiciousness score is equal to or less than the suspiciousness determination threshold, the behavior determination unit 205 determines that the person has not behaved suspiciously.

[0027] The suspiciousness determination threshold, the increase amount of the suspiciousness score, and the decrease amount of the suspiciousness score are set in advance in consideration of the behavior of a person. When setting, values ​​determined by a prior experiment, values ​​determined by observing the behavior of a person captured on a surveillance camera, or values ​​determined by a simulation using a large number of images labeled with face-to-face behavior and non-face-to-face behavior may be used. In addition, it is preferable to set the images used for setting based on multiple images of a person who has committed shoplifting or a person who is suspected of shoplifting and is behaving suspiciously.

[0028] In this embodiment, Fig. 2 shows an example in which image 300 is divided into 10 x 12 small regions, but the number of divisions is preferably adjusted according to the angle of view and image size when image 300 is captured. However, if the image is divided more finely, the density of the trajectory information becomes relatively low, and depending on the way a person moves, an impossible region may occur within a possible region. In this case, it is preferable to, for example, increase the amount of trajectory information or add a process of treating an impossible region surrounded by a possible region as a possible region.

[0029] <System operation explanation> The process performed by the imaging device 100 according to this embodiment will be described with reference to the flowchart of FIG. First, in step S101, the imaging unit 101 of the imaging device 100 captures a moving image and acquires images at a predetermined frame rate. Next, in step S102, the transmission unit 102 assigns the image captured by the image capturing unit 101 with the unique number of the image capturing device 100 and time information, processes the image into a format that can be transmitted over the network, and transmits the image. Then, in step S103, the imaging device 100 determines whether or not a request to stop image transmission has been made, and repeats the image acquisition process in step S101 and the image transmission process in step S102 unless a request to stop is made.

[0030] The suspicious behavior determination process in the image processing device 200 according to this embodiment will be described with reference to the flowchart of FIG. First, in step S201, the receiving unit 201 of the image processing device 200 receives an image sent from the imaging device 100. The receiving unit 201 sends the received image to the detection and tracking unit 202. Next, in step S202, the detection and tracking unit 202 performs a person detection and tracking process to detect and track a person from the image sent from the receiving unit 201. The detection and tracking unit 202 also generates metadata such as rectangular coordinates on the image of the person, which are the person detection results, and a person ID and coordinates on the image, which are the tracking process results. Then, in step S203, the storage unit 203 stores the person detection result, the tracking information, and the metadata generated in step S202.

[0031] Furthermore, in step S204, the direction detection unit 204 performs a face direction detection process for each person detected by the detection and tracking unit 202. Next, in step S205, the behavior determination unit 205 performs a process of detecting suspicious behavior of a person by using the face direction detection result in step S204 and the trajectory information in step S203. Then, when the behavior determination unit 205 detects suspicious behavior, the output unit 206 performs a process of notifying an employee or the like. The above steps are performed each time an image is acquired. The image processing device 200 determines whether or not a request to end the image analysis process has been made, and returns to step S201 unless a request to end the image analysis process has been made, and repeats steps S201 to S205.

[0032] As described above, in the system of this embodiment, the suspicious behavior detection process is performed taking into account the direction of the person's view in addition to the direction of the person's face. For example, the behavior of a person looking at product shelves or the surroundings to search for a desired product is a behavior that a normal shopper would perform, but the technology described in Patent Document 1, for example, may erroneously detect such a shopper's behavior as suspicious. On the other hand, in this embodiment, suspicious behavior is determined taking into account the direction of the person's view, so that suspicious behavior such as checking whether there is another person looking at the person can be determined with high accuracy.

[0033] In this embodiment, an example is described in which a face-turning behavior using face direction detection is determined as a suspicious behavior, but the present invention is not limited to face direction detection, and the same effect can be obtained by using a technology that detects the gaze direction of a person. In addition, the present invention can be applied to cases where a behavior that confirms the presence of another person looking at the person, such as a person staring in a specific direction, is detected as a suspicious behavior, not limited to face-turning behavior.

[0034] In the present embodiment, the image capturing device 100 performs processing according to the frame rate at which the image capturing device 100 captures a moving image, that is, processes all images in the moving image, but the detection and tracking unit 202, the orientation detection unit 204, and the behavior determination unit 205 do not necessarily need to process all images. In general, a high frame rate is preferable for tracking processing, whereas face swing behavior can be determined even at a lower frame rate than tracking processing. For this reason, a sufficient effect can be expected even if the detection and tracking unit 202 processes at a high frame rate and the orientation detection unit 204 and the behavior determination unit 205 process at a low frame rate, for example, by thinning out frames.

[0035] <Second embodiment> In the second embodiment, in addition to the contents described in the first embodiment, a system is described that judges suspicious behavior in consideration of equipment in a line of sight and notifies employees, etc. In the case of the second embodiment, a weight is set for the line of sight, and suspicious behavior is judged based on the weight. In this embodiment, the weight for the line of sight can be set, for example, through a UI (user interface) screen. Also, in this embodiment, the weight for the line of sight is set according to the type of equipment in the line of sight, or based on the characteristics of a person who has engaged in suspicious behavior in the past.

[0036] The configuration of each functional unit of the system according to the second embodiment is the same as the configuration shown in Fig. 1 of the first embodiment, and therefore illustration thereof is omitted. The configuration and operation of each functional unit of the imaging device 100 are the same as those described above, and the receiving unit 201, the detection and tracking unit 202, the storage unit 203, the orientation detection unit 204, and the output unit 206 of the image processing device 200 are the same as those described above, and therefore explanations thereof are omitted. In the case of the second embodiment, the processing in the behavior determination unit 205 of the image processing device 200 is different from that of the first embodiment.

[0037] In the second embodiment, the behavior determination unit 205 has a function as an equipment direction setting unit (not shown) in addition to the functions described in the first embodiment. The equipment direction setting unit sets the direction of the store equipment, etc., with respect to the angle of view captured by the imaging device 100.

[0038] Here, the store facilities are places where store employees may be present, such as cash registers, entrances and exits, employee service entrances, employee offices, warehouses, etc. In this embodiment, importance is set for these store facilities as weights for the visible directions.

[0039] FIG. 5 is a diagram showing an example of a UI screen displayed on a monitor installed in a monitoring room or the like that monitors images from a monitoring camera when the equipment direction is set. In the UI screen shown in FIG. 5, a direction setting list 401 is a list for managing names indicating directions, the importance of the set directions, and areas on an image in association with each other. In the example of FIG. 5, the importance is divided into three categories, high, normal, and low, but this is only an example, and the importance may be further subdivided. The UI screen example of FIG. 5 shows a state in which the cash register direction and the entrance / exit direction are set, and an area 402 is an area on an image corresponding to the cash register direction. The direction is set by operating a mouse cursor 403 around the image. In addition, when increasing or decreasing the direction setting, a direction setting is added by pressing a new creation button 404 in the UI screen of FIG. 5 by clicking the mouse, etc., and a direction setting is deleted by pressing a delete button 405 with one of the direction settings selected.

[0040] In the second embodiment, the behavior determination unit 205 executes the suspicious behavior determination process in consideration of the above-mentioned facility direction setting in addition to the suspicious behavior determination process described in the first embodiment. That is, the behavior determination unit 205 of the second embodiment determines and detects suspicious behavior of a person based on the face swing behavior determination result, the visibility direction determination result, and the facility direction setting result.

[0041] In the second embodiment, as in the first embodiment, the behavior determination unit 205 holds a suspicious score for each person ID, and increases the suspicious score corresponding to the person ID when the face direction of each person changes suddenly. In addition, in the second embodiment, the behavior determination unit 205 adjusts the amount of bonus score to be given to the suspicious score according to the importance of the direction setting ahead of the face direction when the person looks in a direction that the person can see. For example, the behavior determination unit 205 increases the amount of bonus score to be given to the suspicious score by three times when the importance is high, by two times when the importance is normal, and by one time when the importance is low, compared to when there is no direction setting. On the other hand, the behavior determination unit 205 decreases the suspicious score when the face direction of the person determined in the same manner as in the first embodiment is stable. Then, the behavior determination unit 205 compares the suspicious score with the suspicious determination threshold every time the suspicious score changes, and determines that the person has behaved suspiciously when the suspicious score exceeds the suspicious determination threshold, and determines that the person has not behaved suspiciously when the suspicious score is equal to or less than the suspicious determination threshold.

[0042] In the second embodiment, the amount of bonus score is adjusted in advance in consideration of the person's behavior. The amount of bonus score may be set using a value determined by a prior experiment, a value determined by observing the behavior of a person captured on a security camera, or a value determined by a simulation using a large number of images labeled with waving behavior and non-waving behavior. The images used in the setting are preferably set based on a number of images showing, for example, a person who has committed shoplifting or a person who is suspected of shoplifting and is behaving suspiciously.

[0043] As described above, in the system according to the second embodiment, suspicious behavior is determined in consideration of the direction of the face, the direction that the person can see, and also the facilities in the direction that the person can see. For example, for a person attempting shoplifting, the direction of the cash register where employees are waiting and the direction of the entrance / exit of the escape route are important directions to check. According to the second embodiment, suspicious behavior can be determined with high accuracy by considering important directions to check, such as the direction of the cash register and the direction of the entrance / exit of the escape route.

[0044] In the present embodiment, the direction setting is manually set by the system administrator or employee through the UI screen for equipment direction setting, but the setting of the importance for each direction is not limited to the above-mentioned setting example. For example, a video is recorded, and a shoplifter or suspicious person in the video, that is, a person who has behaved suspiciously in the past, is selected by operating the mouse cursor, and the behavior of the selected person is analyzed to extract the direction in which the person gazes and checks. Then, by setting the importance of the extracted direction as high, it becomes possible to judge suspicious behavior with higher accuracy. In other words, by setting the weight of the visible direction based on the characteristics of a person who has behaved suspiciously in the past, it becomes possible to judge suspicious behavior with higher accuracy.

[0045] <Third embodiment> In the third embodiment, in addition to the contents described in the first embodiment, a system will be described that determines suspicious behavior by considering not only the face direction but also the direction the person is facing, and notifies employees, etc. The third embodiment is an example that enables a determination that takes into consideration suspicious behavior of a person facing away from the camera, for example.

[0046] The configuration of each functional unit of the system according to the third embodiment is the same as the configuration shown in Fig. 1 of the first embodiment, and therefore illustration thereof is omitted. The configuration and operation of each functional unit of the imaging device 100 are the same as those described above, and the receiving unit 201, the detection and tracking unit 202, the storage unit 203, the orientation detection unit 204, and the output unit 206 of the image processing device 200 are the same as those described above, and therefore explanations thereof are omitted. In the case of the third embodiment, the processing in the behavior determination unit 205 of the image processing device 200 is different from that of the first embodiment.

[0047] In the third embodiment, the behavior determination unit 205 determines / detects suspicious behavior in consideration of the direction the person is facing, in addition to the functions described in the first embodiment. That is, the behavior determination unit 205 in the third embodiment performs suspicious behavior determination of a person based on the determination result of the face turning behavior, the determination result of the visible direction, and the direction the person is facing.

[0048] Here, the behavior determination unit 205 of the third embodiment acquires the moving direction of the person from the trajectory information of the person stored in the storage unit 203, and determines the direction in which the person faces with respect to the imaging device 100 (i.e., the surveillance camera) based on the information on the moving direction. For example, when the moving direction of the person is a direction approaching the imaging device 100, the behavior determination unit 205 determines that the person faces forward with respect to the imaging device 100. On the other hand, when the moving direction is a direction away from the imaging device 100, the behavior determination unit 205 determines that the person faces backwards (with their back to the surveillance camera).

[0049] The behavior determination unit 205 detects suspicious behavior of a person by changing the combination of the person's face direction and the trajectory information depending on whether the person is facing forward or backward. That is, in the third embodiment, the behavior determination unit 205 holds the suspicious score described above for each person ID, and when the person is facing forward, the behavior determination unit 205 increases the suspicious score when the face direction of the person changes suddenly as in the first embodiment. In addition, when the person is facing backward and the face direction is detected by the direction detection unit 204, that is, when a profile of a person facing backwards is detected when the person turns sideways, the behavior determination unit 205 increases the suspicious score, whereas when the face direction is not detected, the behavior determination unit 205 decreases the suspicious score. In addition, when the same person looks in a direction where the same person can be seen, the behavior determination unit 205 gives a bonus score to the suspicious score and increases the suspicious score. In addition, the behavior determination unit 205 decreases the suspicious score when the face direction is stable, as in the determination process of the face swing behavior. Then, as described above, the behavior determination unit 205 compares the suspiciousness score with the suspiciousness determination threshold each time it changes, and if the suspiciousness score exceeds the suspiciousness determination threshold, it determines that the person has engaged in suspicious behavior, and if the suspiciousness score is equal to or less than the suspiciousness determination threshold, it determines that the person has not engaged in suspicious behavior.

[0050] As described above, in the system according to the third embodiment, in addition to the processing described in the first embodiment, suspicious behavior is determined in consideration of the direction in which the person is facing. This makes it possible to accurately determine suspicious behavior even if, for example, a person attempting shoplifting turns his / her back to the imaging device 100 (surveillance camera) to hide his / her hands. In the third embodiment, an example of estimating the direction in which a person is facing based on trajectory information has been described, but this method is not limited to this, and the same effect can be obtained by having a separate direction estimation unit that estimates the direction in which a person is facing from the appearance of the person in the image.

[0051] Next, an example of a hardware configuration for implementing the processing in each functional unit of the image processing device 200 according to each of the above-described embodiments will be described with reference to Fig. 6. Note that each functional unit of the image processing device 200 may be implemented by software as well as by the hardware shown in Fig. 6. Also, one or more functions of the image processing device 200 according to each of the above-described embodiments may be provided by another device.

[0052] The image processing device 200 in each of the above-described embodiments includes a CPU 600, a RAM 610, a ROM 620, a HDD 630, and an I / F 640. The CPU 600 is a central processing unit that controls the image processing device 200. The RAM 610 temporarily stores computer programs executed by the CPU 600. The RAM 610 also provides a work area used when the CPU 600 executes processes. The RAM 610 also functions as, for example, a frame memory or a buffer memory.

[0053] The ROM 620 stores a program for the CPU 600 to control the image processing device 200. The program stored in the ROM 620 is expanded in the RAM 610, and the CPU 600 executes the program to realize the processing of each functional unit of the image processing device 200 described in each embodiment. The HDD 630 is a storage device that records image data, suspicious behavior detection results, and other data. The I / F 640 communicates with external devices via the network 140 in accordance with TCP / IP, HTTP, or the like.

[0054] 6, an example in which the CPU 600 executes the processes of the above-mentioned functional units has been described, but at least a part of the processes of the CPU 600 may be executed by dedicated hardware. For example, the process of displaying a GUI (GRAPHICAL USER INTERFACE) or image data on the display 130 may be executed by a GPU (GRAPHICS PROCESSING UNIT). Also, the process of reading program code from the ROM 620 and loading it in the RAM 610 may be executed by a DMA (DIRECT MEMORY ACCESS) functioning as a transfer device.

[0055] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0056] 100: imaging device, 101: imaging section, 102: transmission section, 200: image processing device, 201: reception section, 202: detection and tracking section, 203: storage section, 204: orientation detection section, 205: behavior determination section, 206: output section

Claims

1. A direction detection means for detecting a face direction of a person using an image obtained by capturing a moving image; a behavior detection means for identifying a possible area in an imaging area of ​​the video in which there is no obstruction that blocks the line of sight and an impossible area in which the obstruction exists, and detecting suspicious behavior of the person based on a determination result of whether or not the possible area is continuous from the face of the person to an end of the image in the direction of the face orientation; 13. An image processing device comprising:

2. A detection and tracking means for acquiring trajectory information of the movements of a plurality of people, The image processing apparatus according to claim 1 , wherein the behavior detection means identifies the possible area and the impossible area based on the trajectory information.

3. The image processing device described in claim 1, characterized in that the behavior detection means increases the suspiciousness score of the person when a possible area continues from the position of the person's face to the end of the image in the direction of the face direction, and determines that the person has engaged in suspicious behavior when the suspiciousness score exceeds a threshold value.

4. 2. The image processing apparatus according to claim 1, wherein the possible area is an area in which a trajectory of a person's movement exists for a predetermined amount or more.

5. The image processing apparatus according to claim 1 , wherein the possible area is an area in which movement trajectories of a number of people that is equal to or greater than a predetermined ratio out of a plurality of people who appeared during a predetermined period in the past exist.

6. An image processing device as described in any one of claims 1 to 5, characterized in that a weight is set for each specific direction in which the possible area continues from the position of the person's face to the edge of the image, and the behavior detection means detects the suspicious behavior based on the weight set for each of the specific directions.

7. The image processing device according to claim 6 , wherein the weight is set according to a type of facility located ahead in the specific direction.

8. 8. The image processing apparatus according to claim 6, wherein the weight is set based on characteristics of a person who has engaged in suspicious behavior in the past.

9. The image processing device according to any one of claims 1 to 8, characterized in that the behavior detection means detects whether the person's face is facing forward toward the imaging device performing the image capture, or facing backwards toward the imaging device, and uses different methods of detecting suspicious behavior when the person is facing forward and when the person is facing backwards.

10. 10. The image processing device according to claim 1, wherein the behavior detection means detects suspicious behavior of the person based on a comparison result between at least one of the plurality of angles representing the face direction and a threshold value.

11. the detection and tracking means performs a tracking process of the person at a frame rate of the video; the direction detection means performs a process of detecting a facial direction of the person at a low frame rate after frames of the video are thinned out, 3. The image processing apparatus according to claim 2, wherein the behavior detection means performs processing for detecting suspicious behavior of the person at the low frame rate.

12. An image processing method executed by an image processing device, comprising: a direction detection step of detecting a face direction of a person using an image obtained by capturing a moving image; a behavior detection process for identifying a possible area in an imaging area of ​​the video in which there is no obstruction that blocks the line of sight and an impossible area in which the obstruction exists, and detecting suspicious behavior of the person based on a determination result of whether or not the possible area is continuous from the face of the person to an end of the image in the direction of the face orientation; 13. An image processing method comprising:

13. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 11.

Images