Image processing device, image processing method, and program

Abstract

To provide an image processing apparatus for detecting surroundings confirmation action from an image, while suppressing detection errors, an image processing method, and a program.SOLUTION: In an image processing system configured by connecting a plurality of imaging devices for imaging employees or customers, a plurality of output devices for giving notifications to persons, and an image processing apparatus over a communication network, the image processing apparatus includes a detection unit 205 comprising a direction detection unit 401 which acquires directions of a person detected from images captured at different times, and a confirmation action detection unit 403 which detects the person confirming surroundings on the basis of an evaluation of a difference between a direction of the person acquired in a specific image and a direction of the person predicted based on a direction of the person acquired in an image older than the specific image.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] This invention relates to a method for detecting a person's behavior of checking their surroundings through image analysis. [Background technology]

[0002] Technologies exist to detect individuals performing specific actions from images. For example, Patent Document 1 discloses a technology that counts the number of times a person's face changes direction in a video and identifies a suspicious person based on the count results. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2011-004050 [Overview of the Initiative] [Problems that the invention aims to solve]

[0004] When detecting individuals who perform specific actions, such as checking their surroundings, as persons of interest in a store, it is difficult to distinguish these actions from simply searching for nearby products or specific individuals. The method disclosed in Patent Document 1, which detects individuals whose facial orientation changes frequently, may falsely detect individuals who are not persons of interest.

[0005] The present invention aims to detect surrounding awareness behavior from images while suppressing false detections. [Means for solving the problem]

[0006] To achieve the objectives of the present invention, the image processing apparatus according to one embodiment of the present invention has the following configuration. That is, An acquisition means for acquiring the orientation of a person detected from images taken at multiple time points, Detection means for detecting that the person is performing an action of checking the surroundings based on an evaluation of the difference between the orientation of the person obtained for a specific image and the orientation of the person predicted based on the orientation of the person obtained for an image older than the specific image; It is provided with.

Effect of the Invention

[0007] It is possible to detect the action of checking the surroundings from an image while suppressing false detection.

Brief Description of the Drawings

[0008] [Figure 1] A block diagram showing a configuration example of an image processing system according to an embodiment. [Figure 2] A block diagram showing a configuration example of the detection unit 205. [Figure 3] A diagram showing an example of the data structure of the data possessed by the action storage unit 206. [Figure 4] A diagram showing a hardware configuration example of an image processing apparatus according to an embodiment. [Figure 5] A flowchart showing an example of the processing performed by the imaging apparatus 100. [Figure 6] A flowchart showing an example of the processing performed by the image processing apparatus 200. [Figure 7] A flowchart showing an example of the processing performed by the output apparatus 300. [Figure 8] A block diagram showing a configuration example of the detection unit 205. [Figure 9] A flowchart showing an example of the processing performed by the image processing apparatus 200. [Figure 10] A diagram showing an example of the UI in an embodiment.

Mode for Carrying Out the Invention

[0009] The embodiments will be described in detail below with reference to the attached drawings. Note that the following embodiments do not limit the invention as defined in the claims. While the embodiments describe multiple features, not all of these features are essential to the invention, and the features may be combined in any way. Furthermore, in the attached drawings, identical or similar configurations are given the same reference numerals, and redundant descriptions are omitted.

[0010] [Embodiment 1] <Structure> An image processing device according to one embodiment of the present invention can detect from an image of a person that a person is performing an action of checking their surroundings. Embodiment 1 describes a case in which the image processing device according to one embodiment of the present invention is used in an image processing system which is a surveillance system for a retail store. The image processing system in Embodiment 1 is installed in a retail store and includes a camera that takes pictures and records video.

[0011] For example, in a retail store, a person who is checking their surroundings may be looking for an opportunity to shoplift, and detecting such individuals from images is useful for preventing shoplifting. On the other hand, a person who is checking their surroundings may also be looking for products, and assisting such individuals can improve customer satisfaction. This detection of surrounding-checking behavior can also be useful in various environments such as nursing homes.

[0012] On the other hand, for example, in a retail store, if an aisle is curved, the direction of the aisle can change simply by a customer walking along it. Also, a customer might walk along the aisle, slightly turning their head from side to side while looking at products on the shelves on either side. Such customers are not checking their surroundings, but are simply moving around the store or window shopping. When detecting surrounding-checking behavior, it is not necessary to detect such normal behavior.

[0013] In this image processing system, when a specific action, including checking one's surroundings, is detected based on an image obtained from a camera, audio is played to the detected person from a nearby speaker.

[0014] Figure 1 shows the configuration of an image processing system according to Embodiment 1. This image processing system includes an imaging device 100, an image processing device 200, and an output device 300. The imaging device 100 is, for example, a camera and can photograph employees or customers. The image processing system may have multiple imaging devices 100, each of which can be installed in the store. The output device 300 is a device that can notify a person, for example, a speaker. The image processing system may have multiple output devices 300, each of which can be installed, for example, on a product shelf in the store. The imaging device 100, the image processing device 200, and the output device 300 can be connected to each other via a communication network. As the communication network, for example, a wired LAN (Local Area Network) or a wireless LAN or other computer network can be used.

[0015] The imaging device 100 includes an imaging unit 101 and an image transmission unit 102. The imaging unit 101 includes an optical system including an imaging lens, an imaging sensor such as a CCD or CMOS, and a signal processing unit. The imaging unit 101 can capture images of a person at multiple times, for example, at predetermined time intervals, and the captured images are sent to the image transmission unit 102. The image transmission unit 102 transmits the images captured by the imaging unit 101 to the image processing device 200. The image transmission unit 102 can add additional information such as imaging device information and time to the images, and can also convert the images into data that can be transmitted over a network before transmitting them to the image processing device 200.

[0016] The image processing device 200 comprises an image receiving unit 201, a tracking unit 202, a feature database 203, a matching unit 204, a detection unit 205, an action memory unit 206, a determination unit 207, an audio memory unit 208, a position memory unit 209, a selection unit 210, and an audio transmission unit 211. The image receiving unit 201 sends images received from the imaging device 100 to the tracking unit 202. The image receiving unit 201 can send images of a person captured at multiple time points, transmitted by the image transmission unit 102, to the tracking unit 202. Note that the image processing device 200 does not necessarily have to have all of the illustrated configurations, and some of the configurations may be implemented by external devices. For example, the imaging device 100 may perform the processing of the tracking unit 202, feature database 203, matching unit 204, and detection unit 205. In that case, the image processing device 200 obtains the tracking results and face orientation detection results from the imaging device 100. In this configuration, the inability of the imaging device 100 to detect the orientation of a person from the image means that the image processing device 200 cannot acquire the orientation of a person for this image.

[0017] The tracking unit 202 performs detection processing for people in images transmitted by the imaging device 100. The tracking unit 202 also performs tracking processing to associate the detected people with multiple images. In this way, the tracking unit 202 can detect people from each image at multiple time points and associate the detected people in each image through person tracking processing.

[0018] The detection process is not particularly limited, and for example, people can be detected using machine learning or pattern matching. In this embodiment, the tracking unit 202 extracts the position of a person on the image using machine learning. For example, the position of a person can be extracted using a neural network trained to output the position of a person on the input image. The position of a person can be represented, for example, by the center coordinates of a rectangle surrounding the person (referred to as the person's rectangular region), the width of the rectangle, and the height, with the upper left corner of the image as the origin.

[0019] Tracking is the process of associating a person detected in an image with a plurality of images obtained by the imaging device 100. The tracking unit 202 associates a person detected in each successive image with the same person based on a predetermined algorithm. In this embodiment, the tracking unit 202 can perform tracking based on the position of the person. For example, the tracking unit 202 can associate the person detected in each image based on the length of the line segment connecting the center coordinates of the rectangular area in each successive image, the amount of change in the width and height of the rectangular area, and the predicted position of the person obtained from the results of past tracking processing. As another example, the tracking unit 202 may perform the association using the image features of the person.

[0020] The tracking unit 202 also assigns a tracking ID to each person. The tracking unit 202 can assign the same ID to each person detected in each associated image. The tracking ID can be a unique identifier that does not overlap among multiple imaging devices 100. For example, if a tracking ID 10 is assigned to a person detected from an image sequence from one imaging device 100, the tracking unit 202 can control the system so that a tracking ID 10 is not assigned to a person detected from an image sequence from another imaging device 100. Once the detection and tracking processes are complete, the tracking unit 202 sends the image, tracking ID, and person location information (e.g., center coordinates, width, and height of the rectangular area) to the matching unit 204 and the detection unit 205.

[0021] The Feature DB203 manages the facial feature data of individuals used in facial recognition processing. For example, the Feature DB203 may have a table that can contain a facial feature index number, a person ID, and facial feature data. Here, the facial feature index number is the record identifier, the person ID is the person identifier, and the facial feature data is data that represents the characteristics of this person extracted from their facial image.

[0022] The matching unit 204 matches individuals in images obtained at different times or from different imaging devices, and assigns a unique person ID to each individual. In this embodiment, person matching is performed by facial recognition processing using facial feature data stored in the feature DB 203. The matching unit 204 can perform facial recognition processing for each individual whenever it receives location information of a person detected by the tracking unit 202.

[0023] As a specific example, the matching unit 204 can perform face detection processing on the inside of the rectangular area detected by the tracking unit 202. If the matching unit 204 detects a face area from inside this rectangular area, it can extract face feature data (referred to as matching data) from this face area. The matching unit 204 also compares the matching data with all face feature data (referred to as matched data) stored in the feature DB 203. For each matched data, the matching unit 204 can output a likelihood indicating the degree of match between the face feature data. The matching unit 204 then determines that the person corresponding to the matched data that exceeds a threshold and has the highest likelihood output is the same person as the person detected by the tracking unit 202. If a person is identified through face recognition, the matching unit 204 obtains the person ID corresponding to the matched data from the feature DB and stores this person ID in association with the tracking ID received from the tracking unit 202.

[0024] On the other hand, if there is no matching data with a likelihood exceeding the threshold, and the person detected by the tracking unit 202 cannot be identified, the matching unit 204 determines that the person detected by the tracking unit 202 is a new person not registered in the feature database 203. The matching unit 204 then creates a new record for the new person in the feature database 203 and stores the newly issued face feature index number and person ID. Furthermore, the matching unit 204 stores the face feature data (extracted matching data) in association with the issued face feature index number. The matching unit 204 also stores the newly issued person ID in association with the tracking ID received from the tracking unit 202. However, issuing a person ID for a new person is not mandatory; a person ID may be issued according to predetermined conditions (for example, if confirmation behavior is detected).

[0025] Furthermore, if the face detection process fails to detect a face region, the matching unit 204 can refer to the stored correspondence between the person ID and the tracking ID. If the person ID has previously been associated with the tracking ID of the person being processed, the same person ID can be assigned to the same person being processed. On the other hand, if the person ID has not previously been associated with the tracking ID of the person being processed, it is not necessary to assign a person ID.

[0026] By using facial recognition in this way, the matching unit 204 can perform matching between people appearing in images obtained by different imaging devices, and between people appearing in images obtained at different, non-consecutive times. Such processing is effective when the determination unit 207 performs processing according to the detected person, but it is not essential for detecting surrounding awareness behavior and can be omitted depending on the application. The matching unit 204 sends the tracking ID and its corresponding person ID, as well as the person's location information obtained from the tracking unit 202, to the behavior memory unit 206. The matching unit 204 can further send information to the behavior memory unit 206 indicating the time the image in which the person was detected was captured and the imaging device that acquired the image. This information can be obtained from additional information attached to the image.

[0027] The detection unit 205 extracts the behavior of the person detected by the tracking unit 202. In this embodiment, the detection unit 205 can detect surrounding-checking behavior in which a person checks their surroundings. Surrounding-checking behavior includes, for example, the act of a person turning their head to check if there are any employees or other customers nearby, or the act of searching for a specific product.

[0028] Figure 2 shows the configuration of the detection unit 205 involved in detecting surrounding-checking behavior. The detection unit 205 includes a direction detection unit 401, a direction prediction unit 402, and a checking behavior detection unit 403. The direction detection unit 401 detects the direction of a person in an image. In this embodiment, the direction detection unit 401 detects the direction of the person's face as the direction of the person. The direction detection unit 401 can, for example, use machine learning to extract the face direction of each person relative to the imaging device. The direction prediction unit 402 predicts the direction of a person in a specific image based on the direction of the person detected in images prior to the specific image. The specific image is the image that is the target of detection of surrounding-checking behavior, and in this embodiment, it refers to the latest image acquired by the image receiving unit 201. Hereinafter, the specific image may be called the current image, and the face direction detected from the specific image may be called the current face direction. On the other hand, the specific image that is the target of detection of surrounding-checking behavior may be an image taken in the past.

[0029] The orientation prediction unit 402 can predict the orientation of a person in a specific image based on the orientation of a person detected in past images, which is associated with the person in the specific image based on the results of the association made by the tracking unit 202. In this embodiment, the orientation prediction unit 402 predicts the current face orientation for each person based on the face orientation previously detected by the face orientation detection unit 401. The confirmation behavior detection unit 403 detects that a person is performing an action to check their surroundings based on an evaluation of the difference between the orientation of the person detected in the specific image and the orientation of the person predicted based on the orientation of the person detected from past images. In this embodiment, the confirmation behavior detection unit 403 detects the surroundings checking behavior based on the difference between the orientation of the person detected by the orientation detection unit 401 in the specific image and the orientation of the person predicted by the orientation prediction unit 402. Details of the processing performed by the detection unit 205 will be described later.

[0030] The behaviors extracted by the detection unit 205 are not limited to surroundings checking behaviors. For example, behaviors such as reaching for a shelf, picking up an item, returning an item, or putting an item in a bag may be detected. For example, the behavior of reaching for a shelf can be detected by performing pose estimation processing using machine learning on each person's image. According to the pose estimation processing, the positions of the eyes, nose, neck, shoulders, elbows, hands, waist, knees, or feet can be estimated from within the rectangular area of ​​the detected person. If the coordinates of the hand are within the area indicated by coordinates indicating the position of the shelf which have been set in advance, it can be determined that the person has reached for the shelf. The results of such behavior extraction can be referenced, for example, when the determination unit 207 makes a determination. When the detection unit 205 extracts such behavior, it sends the person's tracking ID and information indicating the type of behavior to the behavior memory unit 206. The detection unit 205 may also send information indicating the time the image in which such behavior was detected was captured to the behavior memory unit 206.

[0031] The behavior memory unit 206 stores the information received from the matching unit 204 and the detection unit 205. Figure 3 shows an example of the data structure of the information stored by the behavior memory unit 206. Figure 3(a) is the person list, (b) is the registration list, (c) is the behavior list, and (d) is the behavior list of unmatched persons.

[0032] The person list shown in Figure 3(a) is a list of people to whom the matching unit 204 has assigned a person ID. The person list contains attributes, appearance time, last detection time, registration ID, and face feature index number for each person ID. The person list can be updated when information is obtained from the matching unit 204. Specifically, the behavior memory unit 206 can compare the person IDs contained in the information obtained from the matching unit 204 with the person IDs listed in the person list. If the person ID contained in the information obtained from the matching unit 204 is registered in the person list, the behavior memory unit 206 can update the last detection time corresponding to this person ID to the current time (for example, the time the image was captured).

[0033] Furthermore, if the person ID is not registered in the person list, the behavior memory unit 206 can create a new record in the person list, register the person ID, and save the current time as the appearance time and last detection time. Next, the behavior memory unit 206 uses facial recognition to determine whether the person corresponding to this person ID is registered in the registration list. For example, the behavior memory unit 206 can determine whether this person is registered in the registration list by comparing the facial feature data registered in the feature DB 203 corresponding to the person ID with the facial feature data corresponding to the facial feature index number registered in the registration list. If this person is registered in the registration list, the behavior memory unit 206 can register the attributes and registration ID listed in the registration list as the attributes and registration ID of the person list. On the other hand, if this person is not registered in the registration list, the attributes of the person list can be set to "customer". Note that for persons who have not been assigned a person ID among the information obtained from the matching unit 204, registration in the person list is not required.

[0034] The registration list shown in Figure 3(b) is a list for managing known individuals. The registration list can include store managers, employees, and individuals of interest. The registration list includes a registration ID as an identifier, attributes of the store manager or employee, and a facial feature index number indicating the facial features of the individual as managed in the feature DB203. The registration list can be created in advance and can be updated or deleted as needed when there are changes in employees or when a new request for registration of an individual of interest arises.

[0035] The activity lists shown in Figures 3(c) and (d) are lists that record a person's activity history. The activity lists can be updated when information is obtained from the matching unit 204 or from the detection unit 205. When the matching unit 204 obtains the location information of a person detected along with the person ID, the activity storage unit 206 records the detection time (e.g., the time the image was captured), camera ID, tracking ID, and the person's location information in the activity list shown in Figure 3(c) for each person ID. In Figure 3(c), the center coordinates of the person's rectangular area are recorded as the person's location information. On the other hand, for people who have not been assigned a person ID, the activity storage unit 206 records the detection time, camera ID, tracking ID, and the person's location information in chronological order in the activity list for unmatched people shown in Figure 3(d).

[0036] The behavior memory unit 206 can also register information indicating the area within the store where a person is located in the behavior list shown in Figures 3(c) and (d). For example, the behavior memory unit 206 can refer to a lookup table that shows the correspondence between a previously created camera ID and an area name, and register the area name corresponding to the camera ID from which the image of the person was acquired.

[0037] Furthermore, when the behavior memory unit 206 obtains information from the detection unit 205 indicating the behavior of a person corresponding to a specific tracking ID, it updates the information indicating the behavior of that person in the behavior list. For example, the behavior memory unit 206 may record information indicating the type of behavior in the behavior list for a person corresponding to a specific tracking ID, associating it with the image capture time when the behavior was detected.

[0038] The behavior memory unit 206 can send update information to the determination unit 207 each time the behavior list is updated, indicating that the information has been updated. The behavior memory unit 206 may also send update information each time information indicating a person's behavior is registered.

[0039] The determination unit 207 determines whether or not to issue a notification based on the detection result of a person checking their surroundings. In this embodiment, notification is made by audio playback, but the method of notification is not particularly limited, and image output or light output may also be used. In this embodiment, each time update information is obtained from the determination unit 207 and the action memory unit 206, the system refers to the information in the action memory unit 206 to determine whether or not there is a target for audio playback.

[0040] The determination unit 207 can determine whether or not there is a target for audio playback by comparing the behavior list held by the behavior memory unit 206 with pre-set rules. In this embodiment, if the behavior of a particular person satisfies a predetermined rule, this person is determined to be a target for audio playback, and audio is played for this person. The rules can be determined based on the detection results of surrounding awareness behavior, as well as the time spent (e.g., time spent in a store or the same area), the presence or absence of a specific behavior (e.g., reaching for a product on a shelf), or the attributes of the person.

[0041] Examples of rules include satisfying two conditions: "the behavior in the latest record is checking the surroundings" and "staying in the same area for 5 minutes or more," or satisfying one condition: "the behavior in the record over the past minute includes checking the surroundings and reaching for a product shelf." The rules used for determination are not limited to those above, and any item managed by the person list or behavior list can be used for determination. For example, the above rules may be combined with condition 3, "the attribute in the person list is not an employee." In addition, multiple rules may be set, and different notifications may be issued when each rule is met. The determination unit 207 can perform such determinations for people whose behavior list has been updated.

[0042] In addition to rule-based determination methods, other determination methods may also be used. For example, a classifier created by machine learning from video footage of a person performing an action that should be subject to audio playback may be used to determine whether or not the person is subject to audio playback. In this case, the classifier may be input, for example, an image within the face region, or a value indicating the difference between the detection result and the prediction result of the person's orientation (for example, a score described later).

[0043] If the determination unit 207 determines that a person is the target of audio playback, it transmits the person's ID (or tracking ID if no person ID is assigned) to the selection unit 210. The determination unit 207 also transmits to the selection unit 210 a detection pattern indicating which rule (or determination device) was used to detect the action.

[0044] The audio storage unit 208 stores audio files to be used for playback by the output device 300, associating them with detection patterns output by the determination unit 207. For example, a detection pattern based on two conditions, "the behavior of the latest record is checking the surroundings" and "staying in the same area for 5 minutes or more," can be associated with an audio file that says, "If you are looking for something, please feel free to ask an employee."

[0045] The location memory unit 209 stores information about the output device 300 corresponding to the location of the person whose voice is to be played back. For example, the location memory unit 209 can store information indicating the nearest output device 300 for each area where a person is detected. However, the method of association is arbitrary; instead of associating the output device 300 for each area, the output device 300 may be associated for each camera ID, each sub-region in the captured image, or each pixel in the captured image.

[0046] When the selection unit 210 receives information from the determination unit 207 indicating the person to whom the audio should be played, it selects a method for notifying the person. In this embodiment, the selection unit 210 selects an audio file to be played for the person and an output device 300 to play the audio. Based on the association between the detection pattern received from the determination unit 207 and the audio storage unit 208, the selection unit 210 can select an audio file corresponding to the detection pattern. The selection unit 210 can also obtain the camera ID, location, or area where the person was last detected, which is recorded in the action list of the action storage unit 206. Based on this information, the selection unit 210 can refer to the location storage unit 209 and select the output device 300 closest to the person for audio playback.

[0047] The voice transmission unit 211 can send a notification when it detects that a person is checking their surroundings. The voice transmission unit 211 can select a voice output device corresponding to the person's location and cause the selected voice output device to output audio. In this embodiment, the voice transmission unit 211 transmits the audio file selected by the selection unit 210 and a playback instruction to the output device 300 selected by the selection unit 210. The output device 300 has a voice reception unit 301 and an output unit 302. The voice reception unit 301 receives the audio file from the voice transmission unit 211 and transmits it to the output unit 302. The output unit 302 plays the received audio file.

[0048] Next, an example of the hardware configuration of the image processing device 200 will be described with reference to Figure 4. The image processing device 200 includes a CPU 511, a ROM 512, a RAM 513, a storage 514, a display unit 515, an input I / F 516, and a communication unit 517. The CPU 511 reads the control program stored in the ROM 512 and executes various processes. The RAM 513 is used as the main memory or temporary storage area such as the work area of ​​the CPU 511. The storage 514 stores various data and various programs. The display unit 515 displays various information. The display unit 515 may be a display device integrated with a touch panel. The input I / F 516 is an interface for inputting operation information. The communication unit 517 performs communication processing with external devices such as the imaging device 100 via a network, either by wire or wireless.

[0049] The functions and processing of the image processing device 200 can be realized by the CPU 511 reading a program stored in the ROM 512 or storage 514 and executing this program. Alternatively, the CPU 511 may read a program stored in a storage medium such as an SD card instead of the ROM 512.

[0050] In this embodiment, the image processing device 200 is described in which one processor (CPU 511) uses one memory (ROM 512) to execute each of the processes shown in the flowchart described later, but other configurations can also be adopted. For example, multiple processors, or multiple RAMs, ROMs, or storage devices can cooperate to execute each of the processes shown in the flowchart described later. In addition, some processes may be executed using hardware circuits instead of processors. Furthermore, the functions and processing of the image processing device 200 may be realized using a processor other than a CPU. For example, a GPU (Graphics Processing Unit) can be used instead of a CPU. Moreover, the image processing device according to one embodiment of the present invention may be composed of multiple information processing devices connected via a network, for example.

[0051] <Processing Procedure> Next, an example of the processing performed by the imaging device 100 according to this embodiment will be explained using the flowchart in Figure 5. In S1101, the imaging unit 101 captures a person and creates an image. In S1102, the image transmission unit 102 transmits the created image to the image processing device 200. In S1103, the imaging device 100 determines whether or not it has received a request to stop image transmission. If such a request is received, the processing in Figure 5 ends; otherwise, the process returns to step S1101.

[0052] Figure 6(a) is a flowchart showing an example of processing performed by the image processing device 200. In step S1201, the image receiving unit 201 receives the image transmitted from the imaging device 100 and sends it to the tracking unit 202. In step S1202, the tracking unit 202 performs the person detection and tracking processing described above on the image received from the image receiving unit 201. In step S1203, the matching unit 204 performs the matching processing of the person detected and tracked as described above.

[0053] In step S1204, the detection unit 205 extracts the behavior of the detected and tracked person. The process of detecting surrounding-checking behavior in step S1204 will be explained with reference to the flowchart in Figure 6(b). Note that if behavior other than surrounding-checking behavior is to be extracted, Figure 6(b) can be used for separate extraction processing.

[0054] In step S1301, the orientation detection unit 401 detects the orientation of a person in the image. This process can be performed for every person whenever information about the person detected by the tracking unit 202 (e.g., image and position information) is received. The orientation of a person can be expressed as a three-dimensional rotation (in degrees) of roll, pitch, and yaw, based on the orientation when the person is facing the imaging device.

[0055] In this embodiment, the orientation detection unit 401 detects the orientation of a person's face as the orientation of the person. As an example of the process, the orientation detection unit 401 first extracts a face region by performing face detection processing on the inside of a rectangular region of the person. Then, if a face region is extracted, the orientation detection unit 401 extracts the face orientation from this face region. The orientation detection unit 401 sends the extracted face orientation, the corresponding tracking ID, and detection success / failure information (for example, a flag set to the value 1) indicating that the face orientation was detected to the orientation prediction unit 402. If the face region or face orientation could not be extracted, the orientation detection unit 401 sends the corresponding tracking ID and detection success / failure information (for example, a flag set to the value 0) indicating that the face orientation could not be detected to the orientation prediction unit 402.

[0056] The extraction of face regions and face orientations can be performed using machine learning. For example, these processes can be performed using neural networks trained to output the position of face regions from a person's image, or to output face orientations from a face image. The face detection process by the orientation detection unit 401 and the face detection process by the matching unit 204 may be shared. For example, the orientation detection unit 401 may detect the face orientation of a person using the results of the face detection process by the matching unit 204.

[0057] In step S1302, the orientation prediction unit 402 predicts the orientation of a person in a specific image based on the orientation of the person detected from past images. In this embodiment, the orientation detection unit 401 predicts the current face orientation of each person for each tracking ID based on the face orientation previously detected by the orientation detection unit 401. The current face orientation can be predicted using any model. For example, the face orientation can be predicted assuming that the person's face is performing constant-velocity rotational motion. As a specific example, a Kalman filter assuming constant-velocity rotational motion can be used as the predictor. This predictor can predict the current face orientation based on the state estimate (estimated face orientation and rotational speed) and the face orientation detection result for the time immediately preceding each tracking ID.

[0058] The details of the prediction process will be explained using the flowchart in Figure 6(c). In step S1401, the orientation prediction unit 402 determines whether or not a face orientation has been detected in the past for the tracking ID, which is the target of face orientation prediction. Whether or not a face orientation has been detected can be determined based on the detection success / failure information obtained from the orientation detection unit 401. If a face orientation has been detected in the past, the process proceeds to step S1402; otherwise, the process proceeds to step S1403.

[0059] In step S1402, the orientation prediction unit 402 obtains the current face orientation prediction result from the predictor corresponding to each tracking ID. The orientation prediction unit 402 sends the obtained prediction result, tracking ID, prediction success / failure information (for example, a flag set to the value 1) indicating that face orientation prediction was successful, face orientation detection result, and detection success / failure information to the confirmation action detection unit 403. After that, the process proceeds to step S1404.

[0060] In step S1403, the orientation prediction unit 402 determines whether it was able to detect the current face orientation for the tracking ID, which is the target of face orientation prediction. If face orientation is detected, the process proceeds to step S1404. If it is not detected, the tracking ID and prediction success / failure information (for example, a flag with a value of 0) indicating that face orientation could not be predicted are sent to the confirmation behavior detection unit 403, and the process then proceeds to step S1303 in Figure 6(b).

[0061] In step S1404, the orientation prediction unit 402 updates the predictor corresponding to the target tracking ID in order to predict the face orientation at the next time step. If the current face orientation can be detected, the face orientation detection result is used to update the predictor's estimated state value; otherwise, the prediction result obtained in step S1402 is used. After that, the process proceeds to step S1303 in Figure 6(b).

[0062] In step S1303, the confirmation action detection unit 403 determines whether the person corresponding to each tracking ID is performing a surrounding confirmation action based on the orientation detection result and the orientation prediction result. In the present embodiment, the confirmation action detection unit 403 detects that the person is performing an action of confirming the surroundings based on an evaluation of the difference between the orientation of the person detected in a specific image and the orientation of the person predicted based on the orientation of the person detected from past images. For example, the confirmation action detection unit 403 can obtain a score indicating the possibility that the person is performing a surrounding confirmation action in a specific image based on the difference between the face orientation detection result and the face orientation prediction result. The score can be obtained, for example, using the following equations (1) and (2). D pt ={(m rt -e rt ) 2 +(m pt -e pt ) 2 +(m yt -e yt ) 2} 1 / 2 ··· Equation (1) S t =(D t / th1) 2 ··· Equation (2)

[0063] In the above equation, m rt , m pt , and m yt are the roll, pitch, and yaw of the face orientation detected from the image at time t, respectively. Also, e rt , e pt , and e yt represent the roll, pitch, and yaw of the face orientation prediction result at time t, respectively. Also, D t is the difference between the face orientation detection result and the prediction result at time t, and S tThis is an evaluation value of the difference between the detected face orientation and the predicted face orientation at time t, and is used as a score indicating the possibility that the person is checking their surroundings. th1 is a threshold used to adjust the score calculation. In equation (1), the difference between the detected face orientation result, the predicted result, and the squared error is calculated, and in equation (2), the score is calculated based on this difference. According to equation (2), if the difference is th1 (e.g., 10) or less, the score tends to be small, and if the difference is th1 or greater, the score tends to be large. Note that if, for the target tracking ID, detection success / failure information indicating that face orientation could not be detected or prediction success / failure information indicating that face orientation could not be predicted is received at time t, the score at time t can be set to 0.

[0064] The confirmation behavior detection unit 403 can determine that a person corresponding to a tracking ID is performing a surroundings confirmation behavior if the score obtained for a specific time t is greater than the threshold th2 (for example, 15).

[0065] According to this method, movements that deviate significantly from the constant-velocity rotational motion predicted from past facial movements tend to receive higher scores. For example, if a person frequently turns their head to check their surroundings, the actual direction of the person is likely to deviate from the predicted direction, resulting in a higher score. On the other hand, if a person gradually changes direction as they walk along a walkway, or slowly and gently turns their head while looking at products, the difference between the actual direction and the prediction becomes smaller, resulting in a lower score and reducing the possibility of false detection as a surrounding-checking behavior. Thus, this method of evaluating the difference between a person's detected direction and the predicted direction can reduce false detections of surrounding-checking behavior.

[0066] In this embodiment, in order to detect surrounding-checking behavior, it is possible to detect whether a person is checking their surroundings based on the magnitude of the change in the person's orientation evaluated over multiple time points. For example, the checking behavior detection unit 403 can detect whether a person is checking their surroundings based on an evaluation of the difference between the person's orientation detected in each image at multiple time points and the predicted person's orientation. Specifically, the checking behavior detection unit 403 can calculate the above score indicating the possibility of surrounding-checking behavior at each time point by comparing the face orientation detection result and the face orientation prediction result at each time point. Then, it can determine whether the statistical value of the score within a certain period of time (e.g., average, maximum, or minimum) is above a predetermined threshold, and if it is above the threshold, it can be determined that the person is checking their surroundings. In this embodiment, the checking behavior detection unit 403 detects whether a person is checking their surroundings based on the average of the evaluation values ​​(scores) of the difference between the person's orientation detected in each image at multiple time points and the predicted person's orientation.

[0067] For example, the confirmation behavior detection unit 403 calculates the score for each tracking ID at each time point according to equation (2), and then calculates the average score for each tracking ID within a certain period of time. If the average score within a certain period of time (e.g., 3 seconds) up to time t is greater than the threshold th2 (e.g., 15), the confirmation behavior detection unit 403 can determine that the person corresponding to the tracking ID is performing a surroundings confirmation behavior. In this case, even if the score at a specific time within the certain period of time is greater than the threshold th2, if the average score is less than the threshold th2, the person corresponding to the tracking ID is not determined to be performing a surroundings confirmation behavior. In this way, by using the average score within a certain period of time, it is possible to suppress false detections of surroundings confirmation behavior even if the score temporarily increases at the moment when the person's movement changes, such as when turning along a corridor or making a U-turn. In this embodiment, the difference between the detection result and the prediction result of the person's orientation is used as the magnitude of the change in the person's orientation. However, even when evaluating the magnitude of the change in the person's orientation using other methods, false detections can be similarly suppressed by detecting surroundings confirmation behavior based on the magnitude of the change in the person's orientation evaluated at multiple time points. Furthermore, while the evaluation value here was determined by focusing on the magnitude of the change in face orientation, the evaluation value could also be determined based on the speed and acceleration of face movement.

[0068] If the behavior detection unit 403 determines that a person is performing a behavior to check their surroundings, it sends the tracking ID and information indicating the type of behavior (e.g., a behavior to check their surroundings) to the behavior memory unit 206. After that, the process proceeds to step S1205 in Figure 6(a).

[0069] In step S1205, the behavior memory unit 206 updates each of the lists shown in Figure 3 as described above, based on the information received from the matching unit 204 and the detection unit 205.

[0070] In step S1206, the determination unit 207 determines the person to be played back as described above. In step S1207, the determination unit 207 determines whether or not a person to be played back has been detected. If a person is detected, the process proceeds to 1208; otherwise, the process returns to step S1201.

[0071] In step S1208, the selection unit 210 selects the output device 300 that will play the audio as described above, based on the camera ID of the imaging device 100 that acquired the image in which the person to be played back is detected.

[0072] In step S1209, the selection unit 210 acquires an audio file from the audio storage unit 208 based on the detection pattern of the person whose voice is to be played back.

[0073] In step S1210, the audio transmission unit 211 transmits the audio file selected in step S1209 and a playback instruction to the output device 300 selected in step S1208.

[0074] In step S1211, the image processing device 200 determines whether or not a request to stop processing has been input, and unless such a request has been input, the processing from step S1201 is repeated.

[0075] Figure 7 is a flowchart showing an example of processing performed by the output device 300. In step S1501, the audio receiving unit 301 acquires the audio file transmitted from the image processing device 200. In step S1502, the output unit 302 plays the audio file. In S1503, the output device 300 determines whether a request to stop processing has been input, and unless such a request has been input, the processing from step S1501 is repeated.

[0076] As described above, according to this embodiment, by using a comparison between the direction detection result and the direction prediction result of a person, it is possible to detect the person's surroundings while suppressing false detections. Furthermore, by detecting the surroundings while suppressing false detections in this way, it is possible to more accurately detect customers looking for a specific product or suspicious individuals looking around, and to respond using voice playback.

[0077] <Variation> (Orientation detection unit 401) In Embodiment 1, the orientation detection unit 401 performed face orientation detection processing following face detection processing. However, the orientation detection unit 401 may also use a detector that simultaneously detects the face region and face orientation.

[0078] Alternatively, the orientation detection unit 401 may detect the orientation of a person by detecting other parts of the person instead of detecting the orientation of the face. For example, the orientation detection unit 401 may detect the orientation of the person's head. For example, the orientation detection unit 401 may detect the head and estimate the detected head orientation. The head orientation can be detected, for example, by performing a 3D pose estimation process using machine learning techniques on a rectangular area of ​​the person. Alternatively, the head orientation may be estimated based on the relative positional relationships of various parts of the person (e.g., eyes, nose, ears, and shoulders, etc.) obtained by performing a 2D pose estimation process.

[0079] Furthermore, the orientation detection unit 401 may also detect the orientation of the person's torso (the orientation of the front of the torso). The face and torso often face the same direction, and it is thought that the torso also moves when a person checks their surroundings. The orientation of the torso, like the orientation of the head, can be detected, for example, by performing 3D pose estimation processing using machine learning technology. In addition, the orientation of the person's face, head, or torso can be detected by any other method.

[0080] In Embodiment 1, the orientation of a person was represented by three-dimensional rotations of roll, pitch, and yaw. However, the method of representing the orientation of a person is not limited to this. For example, any representation method can be used, such as representing it by one-dimensional rotation (e.g., yaw only) or by Euler angles.

[0081] (Direction prediction unit 402) In Embodiment 1, the orientation prediction unit 402 used a Kalman Filter as a predictor. However, the method for predicting a person's orientation is not limited to this. For example, the orientation prediction unit 402 may predict a person's orientation by extrapolation using past orientation detection results. Specifically, the orientation prediction unit 402 can predict a person's current orientation by linear extrapolation using a predetermined number of past (e.g., two) face orientation detection results. In this case, in step S1404, the orientation prediction unit 402 can update the predetermined number of past face orientation detection results used for linear extrapolation. Alternatively, the orientation prediction unit 402 may predict a person's orientation using machine learning techniques that can handle time-series information, such as an RNN (Recurrent Neural Network). In addition, a person's orientation can be predicted by any method using past face orientation detection results.

[0082] There are cases where a person's orientation cannot be detected, such as when their face is facing away from the imaging device or when their face is hidden by a product shelf or another person. If this situation persists for a while, using prediction based on constant-velocity rotational motion, as in Embodiment 1, may result in a large discrepancy between the predicted orientation and the actual orientation. Therefore, if the orientation prediction unit 402 has not been able to detect the orientation of a person associated with a person in a particular image for a predetermined period of time, it can correct the prediction result of the person's orientation according to the orientation of the person detected from the particular image. For example, if the orientation prediction unit 402 has not been able to detect the orientation for a certain period of time (e.g., 2 seconds), it may correct the predictor information and the prediction result of the orientation using the orientation detected thereafter. For example, the orientation prediction unit 402 can correct the predictor's state estimate (e.g., face orientation) to match the detection result and perform a prediction using the corrected state estimate. In this case, it can be evaluated that there is no difference between the detected result and the predicted result of the person's orientation.

[0083] Furthermore, if the difference between the orientation of a person detected from a specific image and the predicted orientation of a person in a specific image is greater than or equal to a threshold, the orientation prediction unit 402 may modify the prediction result of the person's orientation according to the orientation of the person detected from a specific image. In addition, if the difference between the orientation of a person in a specific image and the orientation of the most recent person associated with the person in the specific image is greater than or equal to a threshold, the orientation prediction unit 402 may modify the prediction result of the person's orientation according to the orientation of the person detected from a specific image. For example, if the difference between the orientation detection result and the prediction result, or the difference between the orientation detection result and the last detected orientation is greater than or equal to a threshold, the orientation prediction unit 402 may similarly modify the predictor's state estimate (e.g., face orientation) and the orientation prediction result to match the detection result. For example, the difference D between the detection result and the prediction result according to equation (1) t If the value is 60 or higher, the information in such predictors can be modified.

[0084] (Confirmation behavior detection unit 403) The method for calculating the score in step S1303 is not limited to the method described above. For example, the difference between the orientation detection result and the prediction result may be expressed as a rotation around an arbitrary axis instead of a squared error as in equation (1), and the score may be calculated from the magnitude of the rotation angle. Also, when calculating the score according to equation (2), D t When it is less than th1, S t =(D t Alternatively, we can use / th1). With this configuration, if slight head movements occur frequently, the average score tends to increase more easily than when using equation (2). Furthermore, by employing the Huber loss function, even if the difference becomes extremely large due to misdetection of orientation, the score can be prevented from increasing too much. In addition, the score may be calculated using a score calculation process that uses the difference between the face orientation detection result and the prediction result, or the time-series information of the face orientation detection result and the prediction result, as input, using machine learning techniques. Otherwise, the score can be calculated using any other method.

[0085] Furthermore, when the confirmation behavior detection unit 403 detects surrounding confirmation behavior using the average score over a certain period of time, once the score exceeds the threshold th2, the average score may continue to exceed the threshold th2 for a while. In such a situation, the amount of data may increase rapidly as information indicating surrounding confirmation behavior is continuously added to the behavior memory unit 206 for the same tracking ID. Therefore, once the confirmation behavior detection unit 403 has notified the behavior memory unit 206 of surrounding confirmation behavior information after the score has exceeded the threshold th2, it does not need to notify of surrounding confirmation behavior related to the same tracking ID for a certain period of time (for example, 10 seconds).

[0086] (Example of using the results of detecting surrounding awareness behavior) Embodiment 1 described an example of detecting suspicious individuals in a store by detecting surrounding-checking behavior, but the applications of surrounding-checking behavior detection are not limited to the above. For example, a person performing surrounding-checking behavior may be detected from surveillance camera footage in a residence or private property, and the person may be notified as suspicious. It may also be used to film the driver's seat of a car and detect whether the driver is performing surrounding-checking behavior while driving. In addition, surrounding-checking behavior detection can be used for any other purpose.

[0087] [Embodiment 2] In Embodiment 2, the detection results of the orientation of multiple parts of a person are combined to detect the behavior of checking one's surroundings. Below, an example of an image processing system that detects the behavior of checking one's surroundings by combining the orientation of the face and the orientation of the torso is described. In this way, by combining the orientation of multiple parts (for example, the orientation of the face and the orientation of the torso), the behavior of checking one's surroundings can be detected with greater accuracy.

[0088] The configuration and processing of the image processing system according to Embodiment 2 are the same as those of Embodiment 1, except for the detection unit 205 and the processing in step S1204. The following description will focus on the differences from Embodiment 1. The detection unit 205 according to this embodiment can detect the orientation of multiple parts of a person in an image. It can then detect the surroundings checking behavior based on an evaluation of the difference between the orientation detected in a specific image and the orientation predicted based on the orientation detected from past images for each of the multiple parts. As shown in Figure 8, the detection unit 205 according to this embodiment includes a face orientation detection unit 801, a face orientation prediction unit 802, a checking behavior detection unit 803, a torso orientation detection unit 804, and a torso orientation prediction unit 805. The operation of the face orientation detection unit 801 and the face orientation prediction unit 802 is the same as the operation of the orientation detection unit 401 and the orientation prediction unit 402 in Embodiment 1 in detecting and predicting the face orientation of a person.

[0089] The torso orientation detection unit 804 detects the torso orientation of a person in an image. The torso orientation detection unit 804 can perform 3D pose estimation processing using machine learning to extract the torso orientation of each person relative to the imaging device. The torso orientation detection unit 804 can obtain the relative positions of the neck, shoulders, and waist, etc., through 3D pose estimation processing, and can determine the torso orientation (the orientation of the front of the torso) according to these relative positions. The torso orientation can be expressed in terms of roll, pitch, and yaw, similar to the face orientation. If the torso orientation is detected, the torso orientation detection unit 804 sends the extracted torso orientation, the corresponding tracking ID, and detection success / failure information (for example, a flag set to the value 1) to the torso orientation prediction unit 805. If the torso orientation cannot be detected, the torso orientation detection unit 804 sends the corresponding tracking ID and detection success / failure information (for example, a flag set to the value 0) to the torso orientation prediction unit 805.

[0090] The torso orientation prediction unit 805 can predict the current torso orientation from past torso orientation detection results by applying the same method as the orientation prediction unit 402 to the torso orientation detection results. For example, the torso orientation prediction unit 805 can predict the torso orientation by assuming that the torso is rotating at a constant velocity and using a Kalman filter as a predictor. If a prediction is made, the torso orientation prediction unit 805 sends the torso orientation prediction result, the corresponding tracking ID, prediction success / failure information indicating that the torso orientation was predicted (for example, a flag set to the value 1), the torso detection result, and the torso orientation detection success / failure information to the confirmation behavior detection unit 803. If a prediction is not made, the torso orientation prediction unit 805 sends the tracking ID and prediction success / failure information indicating that the torso orientation could not be predicted (for example, a flag set to the value 0) to the confirmation behavior detection unit 403.

[0091] The confirmation behavior detection unit 803 detects surrounding confirmation behavior based on an evaluation of the difference between the orientation detected in a specific image and the orientation predicted based on orientations detected from past images, for both the person's face and torso. As described below, the confirmation behavior detection unit 803 can detect surrounding confirmation behavior using the face orientation detection result and prediction result, as well as the torso orientation detection result and prediction result.

[0092] Figure 9 is a flowchart showing the detection process of surrounding confirmation behavior performed by the detection unit 205 of this embodiment. Steps S901 and S902 are performed in the same manner as steps S1301 and S1302 of Embodiment 1. In step S903, the torso orientation detection unit 804 detects the torso orientation of each person as described above. In step S904, the torso orientation prediction unit 805 predicts the current torso orientation from past torso orientation detection results as described above.

[0093] In step S905, the confirmation behavior detection unit 803 combines the detection and prediction results for face orientation and torso orientation to determine whether each person is performing a surroundings confirmation behavior. In this embodiment as well, similar to Embodiment 1, the surroundings confirmation behavior is determined based on an evaluation of the difference between the orientation of the person detected in a specific image and the predicted orientation of the person, but the method of calculating the score at each time point differs from Embodiment 1. On the other hand, similar to Embodiment 1, the surroundings confirmation behavior is determined using a statistical value (e.g., the average value) of the score within a certain period of time.

[0094] In this embodiment, the confirmation behavior detection unit 803 refers to the success / failure information of face orientation detection and the success / failure information of prediction to determine whether it is possible to calculate a score using face orientation according to equations (1) and (2). If the confirmation behavior detection unit 803 is unable to detect or predict face orientation, it determines that it is not possible to calculate a score using face orientation; otherwise, it determines that it is possible to calculate a score using face orientation.

[0095] Similarly, the confirmation behavior detection unit 803 refers to the detection success / failure information and prediction success / failure information for torso orientation to determine whether it is possible to calculate a score using torso orientation according to equations (1) and (2). If the confirmation behavior detection unit 803 was unable to detect or predict torso orientation, it determines that it is not possible to calculate a score using torso orientation; otherwise, it determines that it is possible to calculate a score using torso orientation. Note that the calculation of the score according to equations (1) and (2) is performed by m rt , m pt , m yt , e rt , e pt , and e yt This can be done by using the detection and prediction results of the torso orientation.

[0096] In this embodiment, the confirmation behavior detection unit 803 can select a method for determining surrounding confirmation behavior depending on whether it was able to detect or predict the orientation of the face or the orientation of the torso. That is, the confirmation behavior detection unit 803 can select whether to detect whether a person is performing a surrounding confirmation behavior based on the orientation of the face, or whether to detect whether a person is performing a surrounding confirmation behavior based on the orientation of the torso. If it is possible to evaluate the difference between the detected orientation and the predicted orientation for only one of the orientations of multiple parts of the human body, the confirmation behavior detection unit 803 can use the orientations available for calculation to detect whether or not surrounding confirmation behavior is occurring. For example, if it is possible to calculate a score using only one of the orientations of the face or the torso, the confirmation behavior detection unit 803 uses the available orientations to calculate the score for the target's tracking ID at the current time according to equations (1) and (2).

[0097] On the other hand, if it is possible to evaluate the difference between the detected orientation and the predicted orientation for two or more of the orientations of multiple parts of the human body, the confirmation behavior detection unit 803 can select the parts to be used to detect the surrounding confirmation behavior according to a predetermined algorithm. For example, if scores can be calculated for both face orientation and torso orientation, the confirmation behavior detection unit 803 can calculate the score using the orientation of the part for which a score could be calculated for a longer period of time in the most recent moment. Also, if the time for which a score could be calculated is the same, face orientation can be prioritized because it is expected to be less affected by being hidden by product shelves or other customers. Furthermore, if a score cannot be calculated based on either face orientation or torso orientation, the score for the current moment is set to 0.

[0098] However, the method for calculating the score by combining the detection and prediction results of face orientation and torso orientation is not limited to this method. For example, if it is possible to calculate a score using face orientation and torso orientation separately, either the score based on face orientation or the score based on torso orientation alone may always be used, or the average of the scores calculated from each may be used. In this way, the confirmation behavior detection unit 803 can detect surrounding confirmation behavior using the orientation of multiple body parts. In addition, the score can be detected by any other method.

[0099] As described above, in this embodiment, by combining the orientation detection results and prediction results for two or more parts of a person, even if it is not possible to evaluate the difference in the person's orientation based on the orientation of one part, the difference can be evaluated based on other parts. Therefore, it is expected that the person's orientation can be evaluated more stably and the surrounding awareness behavior can be detected more accurately.

[0100] <Variation> As a further method, the surrounding awareness behavior may be detected by considering the difference between the detected results of face orientation and body orientation. For example, if both face orientation and body orientation are detected, a score calculated based on the difference between the detected results of face orientation and body orientation may be added to the score for that time. For example, e in equation (1) rt , e pt , and e yt By replacing each with the detection result for the torso orientation, the difference between the detection result for the face orientation and the detection result for the torso orientation can be calculated. Then, D in equation (2) t By replacing the calculated difference with the calculated difference, a score is calculated based on the difference between the detected face orientation and body orientation. The score thus obtained can be added to the score for each time point (for example, based on the difference between the detected and predicted face orientation or body orientation). With this configuration, the score will be higher when the face orientation and body orientation are facing different directions. When the face orientation and body orientation are facing different directions, it is considered highly likely that the animal is performing a surrounding-checking behavior, so this method is expected to detect surrounding-checking behavior with greater accuracy. Any other arbitrary score calculation method can be used.

[0101] [Embodiment 3] Embodiment 3 relates to an example of a UI that notifies a user (employee, manager, or security guard, etc.) of detected information when the detection unit 205 according to Embodiment 1 or 2 detects behavior such as checking one's surroundings. By displaying the detected information on the UI, the user can be accurately notified of the person in question. With such a configuration, it becomes easy for the user to take action, such as reporting to the police, etc., if they determine that the person in question is suspicious, or dispatching an employee to the location of the person in question if the user determines that the person is looking for a specific product or employee. The configuration that notifies on the screen when behavior such as checking one's surroundings is detected, as described below, can be used in combination with the configuration that notifies a person by voice as described in Embodiment 1 or 2, but the configuration that notifies on the screen may also be used on its own.

[0102] The processing according to this embodiment can be performed, for example, by the display control unit 212 of the image processing device 200, and the UI can be displayed on the display unit 515. The display control unit 212 can provide notification when it detects that a person is checking their surroundings. The following description will focus on aspects that differ from Embodiment 1, but a similar UI display can be performed in Embodiment 2 as well.

[0103] In this embodiment, in addition to the processing described in Embodiment 1, the detection unit 205 stores the face orientation detection results for each person for a certain period of time (for example, 10 seconds).

[0104] When the image receiving unit 201 receives an image from the imaging device 100, the display control unit 212 displays the image on the display unit 515. Furthermore, when behavior is extracted by the detection unit 205, the display control unit 212 can overlay information about the target person onto the image displayed on the display unit 515 by referring to the information stored in the behavior memory unit 206 and the face orientation detection results of each person stored in the detection unit 205.

[0105] Examples of the UI displayed on the display unit 515 and the information superimposed on the image will be explained with reference to Figure 10. Figure 10 shows a case where a person 602 behind a product shelf 603 is detected checking their surroundings. In Figure 10, the display unit 515 displays an image 601 acquired from the imaging device 100.

[0106] Figure 10(a) shows an example of information indicating the orientation of person 602 when surrounding-checking behavior is detected. In this example, the display control unit 212 acquires the orientation detection result stored by the detection unit 205 and overlays information indicating the range of directions that person 602 faced within a certain period of time onto image 601. In this example, the display control unit 212 calculates, for example, the maximum and minimum values ​​of yaw from the acquired orientation detection result and displays a sector figure 604 centered on person 602's head, indicating the range of the calculated values. Note that the display control unit 212 may generate information indicating the range of directions based on orientation prediction results instead of orientation detection results. Alternatively, 202 may generate information indicating the range of directions based on both orientation detection results and prediction results. For example, information indicating the range of directions may be generated based on the maximum and minimum values ​​of the set of orientation detection results and prediction results. In this way, when surrounding-checking behavior is detected, the UI can show which direction the target person was facing within a predetermined time, indicating which direction the target person was paying attention to.

[0107] Figure 10(b) is another example showing information indicating the orientation of person 602, in which a surrounding-checking behavior was detected. In this example, instead of showing the range of person 602's orientation with a sector shape 604, graph 605 is superimposed near person 602. Graph 605 shows the change in the yaw value of person 602's orientation over time, with the vertical axis representing the yaw value and the horizontal axis representing time.

[0108] Figure 10(c) shows an example of a notification that identifies person 602, who has been detected to be checking their surroundings. In Figure 10(c), a rectangle 606 is superimposed on person 602.

[0109] Alternatively, instead of overlaying the information onto the image 601 as shown in Figures 10(a) to (c), the information may be overlaid onto a map (for example, a floor plan). In the example in Figure 10(d), the display unit 515 shows a map 607 of the store, and information about the person 602 whose surroundings-checking behavior was detected is overlaid on the map 607.

[0110] As shown in Figure 10(d), icons can be displayed on the map 607 to indicate the location of the detected or tracked person, or the location of the imaging device 100. By measuring the position and orientation of the imaging device 100 within the store in advance, the locations of the imaging device 100 and the person can be shown on the map.

[0111] In Figure 10(d), similar to Figure 10(a), information indicating the orientation of person 602 is shown for person 602 in which surrounding-checking behavior was detected. The sector-shaped figure 608, which shows the direction that person 602 faced within a certain period of time, can be generated by converting the orientation detection result stored in the detection unit 205 into an orientation on the map based on the position and orientation of the imaging device 100.

[0112] The notification methods shown in Figures 10(a) to (d) are merely examples; the user can be notified of the person who has detected the surroundings checking behavior using any display method. Furthermore, if other behaviors besides surroundings checking are also to be notified, rectangles with different colors for each behavior can be superimposed as shown in Figure 10(c), or a string indicating the extracted behavior can be superimposed, so that it is clear which behavior has been detected.

[0113] As shown in Figures 10(a) to (d), the display control unit 212 can display on the screen an image obtained by the imaging device 100, or a map of the location where a person is present, and information indicating that the person is checking their surroundings. Also, as shown in Figures 10(a) to (b), the display control unit 212 can display on the screen information indicating the range of the direction of the detected or predicted person within a predetermined time.

[0114] In this embodiment, we have described a case in which the detection unit 205 notifies information about a person when it detects a person's behavior, such as checking their surroundings. On the other hand, the detection unit 205 may also notify information about a person when the determination unit 207 detects a person who is the target of audio playback.

[0115] The processing flow in this embodiment will now be described. This embodiment can be realized by having the display control unit 212 perform the above processing between steps S1206 and S1207 in the processing of the image processing device 200 in Embodiment 1 or 2 shown in Figure 6(a). That is, the display control unit 212 displays the image acquired from the imaging device 100 on the display unit 515. Furthermore, if the detection unit 205 detects the behavior of a person, the display control unit 212 superimposes information about this person onto the image.

[0116] Furthermore, the display control unit 212 may notify the user by displaying a thumbnail image of the person performing the action of checking their surroundings on the screen. For example, instead of presenting the user with information about the detected person at the time the behavior is detected, the information can be displayed all at once at a later time, for example, after the store's closing time for the day. In this case, video footage of 10 seconds before and after the time the behavior was extracted can be recorded, and thumbnail images can be created for each video. These thumbnail images can be displayed on the display unit 515 after the closing time. When the user selects a thumbnail image using the input I / F 516, the corresponding video can be displayed on the display unit 515. In this case, the above-mentioned information about the detected person can be superimposed on the video or thumbnail image displayed on the display unit 515. With this configuration, the user can check the customers who were there throughout the day. Also, if the user finds a suspicious person based on the video footage checked here, they can register this person as a person of interest in the registration list.

[0117] Furthermore, the display control unit 212 may notify the user of locations where surrounding-checking behavior has been detected frequently. For example, the display control unit 212 can display on the screen an image obtained by the imaging device 100, or a map of the locations where people are present, and information indicating the number or frequency of times surrounding-checking behavior has been detected at each location in the image or map. As an example, the display control unit 212 may overlay information indicating locations where surrounding-checking behavior has been detected frequently throughout the day onto a map as shown in Figure 10(d). Such information could be, for example, a heat map. With such a configuration, it becomes easy to understand locations where surrounding-checking behavior occurs frequently. Based on the notified information, the user can take measures such as improving signage within the store or encouraging employees to actively pass through these locations.

[0118] As described above, in this embodiment, by notifying the user via the UI of a person who has performed a surrounding-checking action or other behavior, it becomes easier for the user to identify the person in question.

[0119] (Other examples) The present invention can also be realized by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be realized by a circuit (e.g., an ASIC) that implements one or more functions. The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention. Accordingly, claims are attached to disclose the scope of the invention. [Explanation of Symbols]

[0120] 201: Image receiving unit, 202: Tracking unit, 205: Detection unit, 207: Judgment unit, 210: Selection unit, 211: Audio transmission unit, 212: Display control unit

Claims

1. an acquisition means for acquiring the orientation of a person detected from images captured at a plurality of times; a detection means for detecting whether the person is checking their surroundings based on an evaluation of a difference between a direction of the person acquired for a specific image and a direction of the person predicted based on a direction of the person acquired for an image older than the specific image; and An image processing device comprising:

2. 2. The image processing device according to claim 1, wherein the detection means detects that the person is checking their surroundings based on an evaluation of a difference between the direction of the person acquired for each image at a plurality of times and the predicted direction of the person.

3. 3. The image processing device according to claim 1, wherein the detection means detects that the person is checking their surroundings based on an average evaluation value of the difference between the person's orientation acquired for each image at a plurality of times and the predicted person's orientation.

4. 4. The image processing device according to claim 1, further comprising: a prediction means for predicting the orientation of a person in the specific image based on the orientation of a person detected in a past image, the orientation being associated with the person in the specific image according to the results of detecting a person from each image at a plurality of times and associating the people detected in each image by person tracking processing.

5. If the orientation of the person associated with the person in the specific image has not been detected within a recent predetermined time period, 5. The image processing apparatus according to claim 4, wherein said prediction means corrects the predicted result of the person's orientation in accordance with the orientation of the person acquired for said specific image.

6. When a difference between the orientation of the person in the specific image acquired by the acquisition means and the orientation of the nearest person associated with the person in the specific image is equal to or greater than a threshold value, 5. The image processing apparatus according to claim 4, wherein said prediction means corrects the predicted result of the person's orientation in accordance with the orientation of the person acquired for said specific image.

7. If a difference between the orientation of the person acquired for the specific image and the predicted orientation of the person in the specific image is equal to or greater than a threshold, 5. The image processing apparatus according to claim 4, wherein said prediction means corrects the predicted result of the person's orientation in accordance with the orientation of the person acquired for said specific image.

8. The image processing device according to claim 1 , wherein the acquiring means acquires a facial orientation of the person as the orientation of the person.

9. the acquiring means acquires orientations of a plurality of body parts of the person in the image; 8. The image processing device according to claim 1, wherein the detection means detects that the person is checking their surroundings based on an evaluation of a difference between a direction of each of the plurality of body parts acquired in the specific image and a direction predicted based on a direction of each of the plurality of body parts acquired in a past image.

10. the acquiring means acquires a facial orientation of the person in the image and a torso orientation of the person in the image; 8. The image processing device according to claim 1, wherein the detection means detects that the person is checking their surroundings based on an evaluation of a difference between the orientation of the person's face and torso acquired for the specific image and the orientation predicted based on the orientation acquired for past images.

11. 11. The image processing device according to claim 10, wherein the detection means selects whether to detect that the person is performing the action of checking their surroundings based on the facial direction or to detect that the person is performing the action of checking their surroundings based on the torso direction, depending on whether the facial direction has been acquired or predicted or whether the torso direction has been acquired or predicted.

12. The image processing device according to claim 1 , further comprising a notification unit that notifies the user when it is detected that the user is checking the surroundings.

13. 13. The image processing device according to claim 12, wherein the notification means selects an audio output device corresponding to the position of the person and causes the selected audio output device to output audio.

14. 13. The image processing device according to claim 12, wherein the notification means displays on a screen the image or a map of a location where the person is present, and information indicating that the person is checking their surroundings.

15. 15. The image processing device according to claim 14, wherein the notification means displays, on the screen, information indicating a range of orientations of the person detected or predicted within a predetermined time period.

16. 13. The image processing device according to claim 12, wherein the notification means displays on a screen a map of the image or the location of the person, and information indicating the number or frequency at which behavior of checking the surroundings was detected at each position in the image or on the map.

17. 13. The image processing device according to claim 12, wherein the notification means displays a thumbnail image of the person who is checking the surroundings on a screen.

18. an acquisition means for acquiring the orientation of a person detected from images captured at a plurality of times; evaluation means for evaluating the magnitude of change in the person's orientation at a specific time; a detection means for detecting that the person is checking their surroundings based on the magnitude of the change in the person's orientation evaluated at a plurality of times; An image processing device comprising:

19. acquiring the orientation of a person detected from images captured at a plurality of times; detecting whether the person is checking their surroundings based on an evaluation of a difference between a direction of the person captured in a particular image and a predicted direction of the person based on a direction of the person captured in past images; An image processing method comprising:

20. A program for causing a computer to function as the image processing device according to any one of claims 1 to 18.

Images