Image processing device, image processing method, and program

Abstract

To improve detection accuracy in detecting the action of a person by using a tracking result of the person in an image.SOLUTION: An image processing apparatus is provided with capturing means for capturing an image of a person, tracking means for detecting and tracking the person in the image, and control means for controlling processing for detection of the action of the person based on a tracking result of the tracking means, wherein the tracking result includes status information indicating a tracking status of the detected person, and the control means determines whether to use the tracking result based on a type of the processing and the status information.SELECTED DRAWING: Figure 7

Description

[Technical Field]

[0001] The present invention relates to techniques for analyzing objects in images. [Background technology]

[0002] One method for analyzing people's behavior and attributes involves detecting and tracking people from surveillance camera images and analyzing the behavior and attributes of each tracked person. When people overlap in an image, there is a possibility that the extraction of person features may not be performed accurately, so the analysis is performed taking into account the overlapping of people. Patent Document 1 discloses that in person search, if a person overlaps with another subject, the feature amount of the person is not updated. Patent Document 2 discloses that in estimating a person's attributes such as age and gender, the estimation score is lowered while the person overlaps with another person. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2016-197345 [Patent Document 2] Japanese Patent Application Publication No. 2019-197353 Summary of the Invention [Problem to be solved by the invention]

[0004] When people overlap, the image features of the obscured people are reduced, which generally reduces the accuracy of person detection and tracking. However, because behavior detection aims to continuously track the behavior patterns of each person within the shooting range, uniformly ignoring tracking results where detection and tracking accuracy has decreased due to overlapping people reduces the amount of information used in behavior detection, increasing the probability of false positives and negative detections. On the other hand, uniformly using tracking results where detection and tracking accuracy has decreased due to overlapping people reduces the accuracy of the information used in behavior detection, increasing the probability of false positives due to switching of tracked people, etc.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to improve the detection accuracy when detecting human behavior using the tracking results of a person in an image. [Means for solving the problem]

[0006] The image processing device of the present invention comprises an acquisition means for acquiring an image of a person, a tracking means for detecting and tracking the person in the image, and a control means for controlling processing for detecting the behavior of the person based on the tracking result of the tracking means, wherein the tracking result includes status information representing the status of tracking of the detected person, and the control means controls whether to use the tracking result based on the type of processing and the status information. [Effects of the Invention]

[0007] According to the present invention, it is possible to improve the detection accuracy when detecting the behavior of a person using the tracking results of the person in an image. [Brief explanation of the drawings]

[0008] [Figure 1] 1 is a diagram showing the configuration of an image processing system according to a first embodiment. [Figure 2] FIG. 4 is an image diagram showing an example of a tracking result according to the first embodiment. [Figure 3] FIG. 4 is a diagram showing an example of a tracking result according to the first embodiment. [Figure 4] FIG. 10 is a diagram for explaining the independent behavior determination process according to the first embodiment. [Figure 5] 1 is a diagram illustrating an example of the hardware configuration of an image processing device according to a first embodiment. [Figure 6] 4 is a flowchart showing processing of the imaging device according to the first embodiment. [Figure 7] 4 is a flowchart showing processing performed by the image processing device according to the first embodiment. [Figure 8] FIG. 10 is a diagram showing the configuration of an image processing system according to a second embodiment. [Figure 9] FIG. 10 is a diagram showing an example of a posture estimation result according to the second embodiment. [Figure 10] FIG. 10 is a diagram for explaining detection of a residence time according to the second embodiment. [Figure 11] 10 is a flowchart showing processing performed by an image processing device according to a second embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present invention will be described with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations.

[0010] <Embodiment 1> In this embodiment, as an example of an image processing system, a surveillance system will be described that analyzes images output from cameras installed in retail stores such as convenience stores, and transmits a detection event to another system when suspicious behavior of a person in the image is detected. In this embodiment, a case where solitary behavior is detected as suspicious behavior of a person will be described.

[0011] FIG. 1 is a diagram showing the configuration of an image processing system according to the first embodiment. The image processing system is composed of an imaging device 100 and an image processing device 200. The imaging device 100 is a camera, and multiple imaging devices 100 are installed in a store to capture images of employees and customers. The imaging device 100 and the image processing device 200 are connected via a communication network. Specifically, they are connected by a computer network such as a wired LAN (Local Area Network) or a wireless LAN.

[0012] The imaging device 100 is composed of an imaging unit 101 and an image transmission unit 102. The imaging unit 101 is composed of an imaging lens, an imaging sensor such as a CCD or CMOS, a signal processing unit, etc. The captured images are sent to the image transmission unit 102 at predetermined time intervals. The image transmission unit 102 adds additional information such as imaging device information and time to the images acquired from the imaging unit 101 and converts the images into data that can be transmitted over a communication network. The image transmission unit 102 then transmits the converted data to the image processing device 200.

[0013] Next, we will explain the functional configuration of the image processing device 200. The image processing device 200 has the functions of an image receiving unit 202, a person tracking unit 203, a detection control unit 204, a tracking result storage unit 205, a parameter calculation unit 206, a parameter storage unit 207, an independent behavior determination unit 208, and a detection result transmission unit 209.

[0014] The image receiving unit 202 receives data from the image capturing device 100, acquires images from the received data, and sequentially provides the images to the person tracking unit 203. The person tracking unit 203 detects people in an image and performs tracking processing to match the detected people between images. The person tracking unit 203 first uses machine learning to detect the position of the person in the image. The position of the person in the image is expressed by the center coordinates of a rectangle surrounding the person, with the upper left corner of the image as the origin, and the size of the rectangle (width and height). Next, the detected people are matched between consecutive images. Specifically, the unit matches each person detected in each frame using the length of the line segment connecting the center coordinates of the rectangle between consecutive frames, the amount of change in the size of the rectangle, and the predicted position of the person obtained by previous tracking processing. The person tracking unit 203 assigns a tracking ID to the detected person so that the same person can be identified in consecutive images. The tracking ID is an identifier that indicates the identity of the person with a person detected previously. A person already detected in a previous frame is assigned the same tracking ID as the previous frame. On the other hand, a newly detected person is assigned a new tracking ID. Furthermore, the person tracking unit 203 assigns tracking suspension information, which is status information indicating the tracking status, along with the tracking ID to the detected person. In this embodiment, the tracking suspension information has a suspended state and a normal state. When the probability that the tracking IDs have been swapped is equal to or greater than a predetermined value, such as when people are overlapping in the image, the tracking suspension information becomes suspended. On the other hand, when the probability that the tracking IDs have been swapped is less than the predetermined value, the tracking suspension information becomes normal. The suspended state occurs when there are multiple candidates for a tracking ID to be assigned to the detected person, that is, when there are multiple tracking IDs whose tracking likelihood is equal to or greater than a predetermined value. The tracking likelihood is a scalar value indicating the reliability of tracking, and its value range is from 0.0 to 1.0. Furthermore, the tracking suspension information may become suspended when the degree of overlap between people is equal to or greater than a predetermined value.

[0015] The tracking process will be specifically described with reference to FIG. Fig. 2(a) is an image diagram showing the tracking results in an image at a certain point in time. In Fig. 2(a), three people 401 to 403 are detected, and tracking processing is performed on each person. Person 401 is assigned a tracking ID of "1," person 402 is assigned a tracking ID of "2," and person 403 is assigned a tracking ID of "3." In addition, the three people 401 to 403 are assigned a normal state as tracking suspension information. Figures 2(b), 2(c), and 2(d) are conceptual diagrams showing tracking results in images taken a certain amount of time after the time shown in Figure 2(a). The images shown in Figures 2(b), 2(c), and 2(d) show a state in which person 402 has moved to the right of the image and is overlapping with person 403. Using Figures 2(b), 2(c), and 2(d), we will explain the tracking result patterns that can occur when people overlap each other.

[0016] FIG. 2(b) shows a case where three people 401 to 403 are detected, and a tracking ID of "1" is assigned to person 401, a tracking ID of "2" is assigned to person 402, and a tracking ID of "3" is assigned to person 403. In this case, the images of the detected people are correctly associated with each other. However, people 402 and 403 are assigned a pending status as tracking pending information. This is because the tracking likelihoods of "2" and "3" as candidates for the tracking ID of person 402 are each equal to or greater than a predetermined value. Similarly, the tracking likelihoods of "3" and "2" as candidates for the tracking ID of person 403 are each equal to or greater than a predetermined value. Hereinafter, a rectangle surrounding a person to which a tracking ID has been assigned is referred to as a tracking rectangle.

[0017] FIG. 2(c) shows a case where three people 401 to 403 are detected, and the person 401 is assigned a tracking ID of "1," the person 402 is assigned a tracking ID of "3," and the person 403 is assigned a tracking ID of "2." In this case, the tracking IDs of people 402 and 403 are swapped. If tracking rectangles with swapped tracking IDs are used as they are to detect the behavior of people in this way, information about other people will be mixed in with the correct information, resulting in noise. For example, the tracking information related to a series of tracking rectangles with a tracking ID of "2" is linked to person 402 at the time of FIG. 2(a), and linked to person 403, a different person from person 402, at the time of FIG. 2(c).

[0018] This will be explained in more detail below. In reality, person 402 is moving to the right of the image. Here, if the time-series changes in the tracking information including the tracking results of FIG. 2(c) are used, it would appear that the person is moving in the direction from person 402 in FIG. 2(a) toward person 403 in FIG. 2(c), that is, toward the upper right of the image. Furthermore, due to differences in the positions, physiques, and clothing of person 402 and person 403, the size of the tracking rectangle changes differently from when tracking the same person. Therefore, there is a possibility that false detections or non-detections may occur in the behavior detection process. However, in FIG. 2(c), tracking suspension information indicating a suspension state is attached to each tracking result for person 402 and person 403. Therefore, in this embodiment, the behavior detection process is performed taking into account the tracking suspension state, i.e., the high possibility that the tracking IDs have been swapped. This makes it possible to reduce false detections and non-detections.

[0019] 2(d) shows a case where, due to the influence of overlapping people, the person 402 in the foreground has been detected, but the person 403 in the background has not been detected. Even in this case, the tracking likelihoods of "2" and "3" as tracking ID candidates for person 402 are each equal to or greater than a predetermined value, so tracking suspension information indicating a suspension state is assigned to person 402.

[0020] Upon completion of the tracking process, the person tracking unit 203 provides the image that was the processing target and the tracking result for that image to the detection control unit 204. FIG. 3 shows an example of the tracking result. The example in FIG. 3 shows the tracking result for each person detected from the image. As shown in FIG. 3, the tracking result is information including a tracking ID, tracking rectangle center coordinates (x, y), tracking rectangle size (width, height), and tracking suspension information. The person tracking unit 203 sequentially performs tracking process on the images received from the image receiving unit 202, and sequentially provides the obtained tracking results to the detection control unit 204.

[0021] The detection control unit 204 receives tracking results from the person tracking unit 203 and controls multiple processes related to behavior detection. Depending on the type of process to be executed, the detection control unit 204 controls whether to use a tracking result list including tracking results for both normal and suspended states when tracking suspension information is in the normal state, or to create and use a tracking result list including only tracking results for normal states when tracking suspension information is in the normal state. Specifically, for processes that are not based on time-series changes in the tracking rectangle, a tracking result list that also includes suspended states is used, and for processes that are based on time-series changes in the tracking rectangle, a tracking result list that excludes suspended states is used.

[0022] In this embodiment, there are two types of behavior detection processing: parameter calculation processing for solitary behavior determination, which is executed by the parameter calculation unit 206, and solitary behavior determination processing, which is executed by the solitary behavior determination unit 208. For the parameter calculation processing, which is processing based on time-series changes in the tracking rectangle, the detection control unit 204 controls to create and use a tracking result list that excludes pending states. Furthermore, for the solitary behavior determination processing, which is processing not based on time-series changes in the tracking rectangle, the detection control unit 204 controls to use a tracking result list that also includes pending states.

[0023] The parameter calculation unit 206 performs parameter calculation processing to calculate position estimation parameters. The position estimation parameters are parameters for converting image coordinates into three-dimensional position information and are used in the independent behavior determination processing. First, the parameter calculation unit 206 creates a tracking result list excluding pending statuses. Specifically, of the tracking results acquired from the detection control unit 204, those whose tracking pending information is in a pending state are excluded, and the tracking result storage unit 205 sequentially stores the results. As a result, time-series data including only tracking results whose tracking pending information is in a normal state is created as a tracking result list in the tracking result storage unit 205. Next, the parameter calculation unit 206 reads past tracking results from the tracking result list stored in the tracking result storage unit 205, and calculates position estimation parameters using the read tracking results and the currently acquired tracking results. The calculated position estimation parameters are then stored in the parameter storage unit 207.

[0024] The method for calculating the parameters for position estimation is explained below. Regarding the size of an object on an image, the relation between the position information on the detection plane (the virtual plane on which the center coordinates of the object to be detected move) and the object size can be defined by the following equation (1). W1=a(x-xm)+b(y-ym)+wm...(1) However, each symbol represents the following. W1: Human body size x,y: Tracking rectangle center coordinates xm, ym: Average value of the center coordinates of the tracking rectangle of the read tracking results wm: Average tracking rectangle size (width) of the read tracking results a, b: Estimated parameters

[0025] The tracking rectangle size (width) is used as W1. While the tracking rectangle size (height) can also be used, using the tracking rectangle size (width) is more suitable for stable calculations. A single pixel movement in the vertical (y) direction of the image has a greater impact on the depth position than the horizontal (x) direction, making it more susceptible to misalignment of the tracking rectangle. The estimated parameters a and b can be calculated from the acquired tracking rectangle using the least squares method. The parameter calculation unit 206 calculates the estimated parameters a and b using tracking rectangles that have moved a predetermined distance or more within a predetermined time, rather than using all tracking rectangles. This is to remove tracking rectangles that have erroneously tracked posters, etc., and to prevent bias in the tracking rectangle coordinates. If the tracking IDs are swapped, there is a risk of erroneous determination when determining whether the object has moved a predetermined distance or more within a predetermined time. In this embodiment, tracking rectangles with tracking suspended are excluded, reducing the possibility of erroneous determination. This improves the accuracy of the calculation of the estimated parameters. The parameter calculation unit 206 stores a, b, and xm, ym, and wm calculated using the above equation (1) in the parameter storage unit 207 as parameters for position estimation.

[0026] The independent behavior determination unit 208 performs independent behavior determination processing for each person being tracked, based on the tracking results received from the detection control unit 204 and the position estimation parameters read from the parameter storage unit 207. The independent behavior determination unit 208 determines that the behavior of the person to be detected is independent behavior if the person to be detected is farther away from all other people than a predetermined threshold, or if there is a nearby person but an obstruction is present in a position that blocks the view of the person to be detected from all nearby people.

[0027] First, the independent behavior determination unit 208 converts the image coordinates of each person's tracking rectangle into three-dimensional position information using the center coordinates x, y of the tracking rectangle and the read-out position estimation parameters. Specifically, the independent behavior determination unit 208 substitutes the center coordinates x, y of the tracking rectangle and the read-out a, b, xm, ym, and wm into the above formula (1) to obtain W2 as the estimated human body size. That is, W2 is expressed as a(x-xm)+b(y-ym)+wm.

[0028] Next, the solitary behavior determining unit 208 calculates three-dimensional position information X, Y, Z using the center coordinates x, y of the tracking rectangle, the estimated object size W2, and the following equations (2) to (4). Z=focal×B / W2 (2) X = Z × (x - cx) / focal (3) Y = Z × (y-cy) / focal (4) However, each symbol represents the following. X,Y,Z: 3D position information W2: Estimated human body size x,y: Center coordinates of the tracking rectangle B: Average human body size focal: camera focal length cx, cy: Image center coordinates For the above B, if shoulder width is assumed as the tracking rectangle size (width), an average value such as 0.43 m can be used. For the above focal, a value written in the extended area of ​​the image may be used, or a value obtained as imaging device information from the imaging unit 101 may be used. For the above cx and cy, the coordinates of the center of the screen obtained from the screen size of the display unit 415 (FIG. 5) may be used.

[0029] Next, the independent behavior determination unit 208 estimates the three-dimensional position of each person being tracked, and then creates pairs of two people from all the people. Then, it calculates the inter-person distance for each created pair using the following equation (5).

number

[0030] The independent behavior determination unit 208 processes each person being tracked as a detection target one by one, calculates the inter-person distance between the detection target person and all other people, and determines whether the inter-person distance is less than a predetermined threshold. If there are no other people whose inter-person distance is less than the predetermined threshold, there are no nearby people, and therefore it is determined that the detection target person is acting alone. Furthermore, the independent behavior determination unit 208 estimates the position of an obstructing object on the camera image in advance. Then, even if there are nearby people, if there is an obstructing object that blocks the view of the detection target person from all nearby people, the detection target person is determined to be acting independently. The independent behavior determination unit 208 performs independent behavior determination processing with all people being tracked as detection targets, and provides the independent behavior determination result to the detection result transmission unit 209.

[0031] The detection result transmission unit 209 receives the result of the independent behavior determination executed by the independent behavior determination unit 208, and transmits the result of the independent behavior determination to a destination registered in advance.

[0032] As mentioned above, the parameter calculation process and the sole behavior determination process differ in whether or not the tracking results from the tracking suspended state are used. The parameter calculation process calculates position estimation parameters without using the tracking rectangle from the tracking suspended state. On the other hand, the sole behavior determination process also uses the tracking rectangle from the tracking suspended state. The reason why the sole behavior determination process also uses the tracking rectangle from the tracking suspended state is explained below using the example in Figure 4.

[0033] FIG. 4(a) is an image diagram in which tracking IDs and tracking suspension information are assigned to people in an image at a certain point in time. Person 601 is assigned a tracking ID of "1," person 602 is assigned a tracking ID of "2," and person 603 is assigned a tracking ID of "3." Person 601's tracking suspension information is in a normal state, but persons 602 and 603 are in a tracking suspension state due to overlapping. Persons 601 and 602 are close to each other, and persons 602 and 603 are also close to each other. Therefore, none of the three people are acting independently. In this state, as shown in FIG. 4(b), the tracking rectangle in the tracking suspension state is not used in the independent behavior determination process. In this case, only the tracking rectangle of person 601 with tracking ID "1" remains, and the tracking rectangles of the other people with tracking ID "1" are not used in the independent behavior determination process. As a result, the independent behavior determination unit 208 determines that there is no nearby person with tracking ID "1" and erroneously determines that person 601 is acting independently. In this embodiment, the independent behavior determination process also uses the tracking rectangle in the tracking suspension state, making it possible to avoid such erroneous determination. Note that, in determining whether there is a nearby person when viewed from tracking ID "1," it is not important whether the nearby person has tracking ID "2" or "3." What is important is that a person is detected in a nearby position, regardless of whether tracking IDs "2" and "3" are swapped.

[0034] Next, the hardware configuration of the image processing device 200 will be described with reference to FIG. 5. The image processing device 200 has a CPU 411, a ROM 412, a RAM 413, a storage 414, a display unit 415, an input I / F 416, and a communication unit 417. The CPU 411 reads out control programs stored in the ROM 412 and executes various processes. The RAM 413 is used as a temporary storage area such as the CPU 411's main memory or work area. The storage 414 stores various data, various programs, and the like. The display unit 415 displays various information under the control of the CPU 411. Note that the display unit 415 may be a display device integrated with a touch panel. The input I / F 416 is an interface for inputting operation information. The communication unit 417 performs communication processing with external devices such as the imaging device 100 via a wired or wireless communication network under the control of the CPU 411.

[0035] The functions of the image processing device 200 and each process shown in the flowchart are realized by the CPU 411 reading out a program stored in the ROM 412 or the storage 414 and executing this program. As another example, the CPU 411 may read out a program stored in a recording medium such as an SD card instead of the ROM 412. The ROM 412 or the storage 414 may provide a storage area for storing data held by the tracking result storage unit 205 and the parameter storage unit 207 shown in FIG. 1.

[0036] In this embodiment, the image processing device 200 is configured such that one processor (CPU 411) executes each process shown in the flowcharts described below using one memory (ROM 412), but other configurations are also possible. For example, each process shown in the flowcharts described below can be executed by using multiple processors, RAMs, ROMs, and storages in cooperation with each other. Also, some processes may be executed using hardware circuits. Furthermore, functions and processes of the image processing device 200 described below may be realized using a processor other than a CPU (for example, a GPU (Graphics Processing Unit) may be used instead of a CPU).

[0037] Next, the processing of the imaging device 100 of this embodiment will be described using the flowchart in Fig. 6. In the following description of the flowchart, each process (step) is denoted by adding an S to the beginning, and the process (step) will not be described. The processing of this flowchart is realized by the CPU of the imaging device 100 executing a program stored in a storage device of the imaging device 100. In the flowchart of Fig. 6 below, the imaging device 100 will be described as the entity that executes the processing of each step, but specifically, it is the CPU of the imaging device 100 that executes the processing of each step. In S101, the imaging device 100 uses the imaging unit 101 to acquire an image. In S102, the imaging device 100 transmits the image acquired in S101 to the image processing device 200 using the image transmission unit . In S103, unless there is a request to stop image transmission, the imaging device 100 repeatedly executes image acquisition (S101) and image transmission (S102) at predetermined time intervals. If there is a request to stop image transmission, the series of steps in the flowchart shown in FIG. 6 ends.

[0038] 7 is a flowchart showing the processing executed by the image processing device 200 according to this embodiment. This flowchart starts when reception of an image from the imaging device 100 starts.

[0039] First, in S201, the image receiving unit 202 receives an image from the imaging device 100. In S202, the person tracking unit 203 performs person detection and tracking processing on the image received in S201, and generates a tracking result that represents the result of the tracking processing. Since this flowchart is executed repeatedly, tracking results are continuously generated by this step. In S203, the detection control unit 204 determines whether the parameter calculation mode is currently set. If the detection control unit 204 determines that the parameter calculation mode is currently set, the process proceeds to S204, and if the detection control unit 204 determines that the parameter calculation mode is not currently set, the process proceeds to S210. In S204, the parameter calculation unit 206 excludes the tracking results obtained in S202 whose tracking suspension status is in a suspended state. In S205, the parameter calculation unit 206 stores the remaining tracking results that were not excluded in S204 in the tracking result storage unit 205. This flowchart is executed repeatedly, and therefore the tracking results are accumulated in chronological order by this step. In S206, the parameter calculation unit 206 determines whether a predetermined time has elapsed since the previous parameter calculation time. If the parameter calculation unit 206 determines that the time elapsed since the previous parameter calculation time is equal to or greater than the predetermined time, the process proceeds to S207, and if it determines that the time is less than the predetermined time, the process proceeds to S210.

[0040] In S207, the parameter calculation unit 206 reads out from the tracking result storage unit 205 past tracking results including the current tracking result obtained in S202. In S208, the parameter calculation unit 206 calculates parameters for position estimation using the tracking results read out in S207. When calculating the parameters for position estimation, tracking rectangles that have moved a predetermined distance or more in a predetermined time are used from among the tracking results read out, but in this step, a tracking result list is used that excludes tracking results in a tracking suspension state. This prevents the mixing of tracking rectangles of different people, and suppresses the generation of noise. In S209, the parameter calculation unit 206 stores the position estimation parameters calculated in S208 and the parameter calculation time (current time) in the parameter storage unit 207.

[0041] In S210, the solitary behavior determination unit 208 reads out parameters for position estimation from the parameter storage unit 207. In S211, the independent behavior determination unit 208 performs independent behavior determination processing for each person in the tracking result using the read position estimation parameters and the tracking result obtained in S202. Specifically, the inter-person distance is estimated for each person, and if there are no other people whose inter-person distances are less than a predetermined threshold, the person is determined to be acting independently. In this step, a tracking result list is used that does not exclude tracking results in which tracking suspension information is suspended, so information on whether or not there are other people in the vicinity of a person can be used. This reduces the risk of erroneously determining that a person is acting independently even when there are nearby people. In S212, the detection result transmission unit 209 transmits the result of the independent behavior determination in S211 to a predetermined destination. In S213, unless there is a request to stop image reception, the image processing device 200 repeatedly executes the processes of S201 to S212. If there is a request to stop image reception, the series of steps in the flowchart shown in FIG.

[0042] According to the flowchart shown in Fig. 7, in the sole behavior determination process, when estimating the inter-person distance from the positions of multiple people being tracked, position estimation parameters calculated without using the tracking results in the tracking suspended state are used, thereby improving the accuracy of sole behavior determination.

[0043] As described above, according to this embodiment, it is possible to switch whether to use tracking results in behavior detection when the tracking person may have been switched due to the influence of overlapping people, etc. As a result, when the tracking information is based on time-series changes, tracking results excluding tracking suspended states can be used, and when the tracking information is not based on time-series changes, tracking results including tracking suspended states can be used. Therefore, since the tracking results can be used for behavior detection without excess or deficiency, the probability of false positives and negatives in behavior detection can be reduced.

[0044] <Embodiment 2> In the first embodiment, the case where the image processing device 200 detects solitary behavior has been described. In the present embodiment, the case where suspicious behavior other than solitary behavior is detected will be described. The following will mainly describe the differences from the first embodiment.

[0045] FIG. 8 is a diagram showing the configuration of an image processing system according to this embodiment. Compared to FIG. 1, FIG. 8 differs mainly in two points. The first point is that a human posture estimation unit 1205 is added between the human tracking unit 203 and the detection control unit 204. The second point is that the tracking result storage unit 205, the parameter calculation unit 206, the parameter storage unit 207, and the independent behavior determination unit 208 are not included, since the processing controlled by the detection control unit 204 is different from that of the first embodiment. Instead, the image processing device 200 according to this embodiment has the functions of a dwell time detection unit 1206, a face direction estimation unit 1207, a face trembling detection unit 1208, and a behavior detection unit 1209. Each of these components 1206 to 1209 is a behavior detection unit that detects a different behavior. In this embodiment, the functions of the tracking result storage unit 205 are included in the face trembling detection unit 1208 and the behavior detection unit 1209.

[0046] First, the person pose estimation unit 1205 will be described. Based on the tracking result of the person tracking unit 203, the person pose estimation unit 1205 uses machine learning to detect the positions of the person's key points on the image from a full-body image of the person being tracked, and outputs the detected coordinates and detection likelihood as a pose estimation result. A person's key points are components of a person, such as major organ points and joints, and include, for example, both eyes, both ears, nose, both shoulders, both hips, both elbows, both wrists, both knees, and both ankles. Upon completing processing, the person pose estimation unit 1205 provides the tracking result and pose estimation result to the detection control unit 204. FIG. 9 shows an example of a pose estimation result. As shown in FIG. 9, the pose estimation result is information including a tracking ID, coordinates of each organ point and joint, and detection likelihood.

[0047] Next, the dwell time detection unit 1206, face direction estimation unit 1207, face vibration detection unit 1208, and behavior detection unit 1209 will be described. The detection control unit 204 provides a tracking result list and a posture estimation result list to each of the behavior detection units 1206 to 1209, and controls the behavior detection process executed by each of the behavior detection units 1206 to 1209. In this case, the detection control unit 204 controls the use of a tracking result list containing tracking results for both the normal state and the suspended state when the tracking suspension information is in the suspended state for detection processes not based on time-series changes. In other words, the control unit 204 controls the use of the tracking result list received from the human posture estimation unit 1205 as is. Furthermore, the detection control unit 204 controls the use of a tracking result list containing only tracking results for the normal state when the tracking suspension information is in the suspended state for detection processes based on time-series changes.

[0048] The staying time detection unit 1206 performs a staying time detection process. The staying time detection process is a process of measuring the time that a person being tracked stays within the imaging range of the imaging unit 101. The staying time detection process is a process that is not based on time-series changes. The staying time detection unit 1206 calculates the current staying time using the tracking result list received from the detection control unit 204 and the staying time list up to the previous time stored internally.

[0049] The residence time detection process will be specifically described with reference to FIG. Figure 10(a) is an image diagram in which tracking IDs, tracking suspension information, and residence times are added to people in an image at a certain point in time. At the time of Figure 10(a), person 701 is assigned a tracking ID of "2" and its residence time is counted as 50 seconds. Also, person 702 is assigned a tracking ID of "3" and its residence time is counted as 80 seconds. The images shown in Figures 10(b) and 10(c) are images taken 10 seconds after the time shown in Figure 10(a). In Figure 10(b), the same tracking IDs as in Figure 10(a) are assigned, and tracking is suspended due to the overlapping of people, adding 10 seconds to the dwell time. Meanwhile, in Figure 10(c), the tracking IDs assigned to people 701 and 702 are swapped. However, similar to Figure 10(b), tracking ID "2" is counted as having a dwell time of 60 seconds, and tracking ID "3" is counted as having a dwell time of 90 seconds. Therefore, even if the tracking IDs are swapped as in Figure 10(c), if the overlapping of people is later resolved and the tracking IDs return to their pre-swap state, the dwell time can be counted up in the same way as in Figure 10(b), where the tracking IDs were not swapped. For this reason, the tracking results in the tracking suspension state are also used in the dwell time detection process. The staying time detection unit 1206 provides the detection result transmission unit 209 with the staying time detected for each person being tracked.

[0050] The face direction estimation unit 1207 performs processing to estimate the face direction of each person in the image. The face direction estimation processing is processing that is not based on time-series changes. The face direction estimation unit 1207 estimates the face direction of each person using the tracking result list and posture estimation result received from the detection control unit 204. Specifically, the face direction estimation unit 1207 estimates the degree direction in which the person's face faces along each of the three axes of up / down, left / right, and in-screen rotation from the positional relationship of the facial features among the key point coordinates included in the posture estimation result. The face direction estimation unit 1207 provides the face direction estimation results of each person being tracked to the face vibration detection unit 1208 and the detection result transmission unit 209.

[0051] The face trembling detection unit 1208 performs processing to detect the surroundings checking behavior of each person in the image. The face trembling detection processing is processing based on time-series changes. The face trembling detection unit 1208 creates a tracking result list excluding pending results. The creation method is the same as in the first embodiment. Next, the face trembling detection unit 1208 detects the surroundings checking behavior of the person based on the created tracking result list, the face direction estimation result received from the face direction estimation unit 1207, and the processing results up to the previous time stored internally. Specifically, of the face direction estimation results received from the face direction estimation unit 1207, only data corresponding to the created tracking result list is used. The surroundings checking behavior of the detection target is the behavior of the detection target person shaking their head to check the presence or absence of surrounding people and their positions. The method of determining whether or not the surroundings checking behavior is performed is to first predict the face direction from changes in the face direction of the person, and then calculate the difference between this predicted value and the face direction included in the face direction estimation result. If the sum of the differences over a predetermined time period is equal to or greater than a predetermined amount, it is determined that the change in face direction is large and that the person is checking their surroundings. On the other hand, if the sum is less than the predetermined amount, it is determined that the person is not checking their surroundings. The face vibration detection unit 1208 calculates the difference between the face direction included in the face direction estimation result and the predicted value for each person, and determines whether or not the person is checking their surroundings. The face vibration detection unit 1208 provides the result of determining whether or not the person is checking their surroundings for each person being tracked to the detection result transmission unit 209.

[0052] The behavior detection unit 1209 performs processing to detect the behavior of each person in the image. The behavior detection processing is processing based on time-series changes. The behavior detection unit 1209 creates a tracking result list excluding pending results. The creation method is the same as in the first embodiment. Next, the behavior detection unit 1209 detects the behavior of the person based on the created tracking result list, the posture estimation results, and the processing results up to the previous time stored internally. Specifically, of the posture estimation results received from the detection control unit 204, only data corresponding to the created tracking result list is used. The behavior of the detection target is the person extending their hand, crouching, etc. The behavior detection unit 1209 acquires keypoint coordinates of each person from the previous posture estimation results and the current posture estimation results, and detects the behavior of each person from the time-series changes in the keypoint coordinates acquired using machine learning. The behavior detection unit 1209 provides the behavior detection results of each person being tracked to the detection result transmission unit 209.

[0053] FIG. 11 is a flowchart showing the processing executed by the image processing device 200 according to this embodiment.

[0054] S301 and S302 are similar to S201 and S202 in Fig. 7, and therefore description thereof will be omitted. When S302 is executed, in S303, the human posture estimation unit 1205 performs human posture estimation using the image received in S301 and the tracking result obtained in S302. In S304, the detection control unit 204 determines whether all behavior detection processes have been executed. If the detection control unit 204 determines that all behavior detection processes have been executed, the process proceeds to S310. If the detection control unit 204 determines that all behavior detection processes have been executed, the process proceeds to S305. In S305, the detection control unit 204 selects the next behavior detection process to be executed. In S306, the detection control unit 204 determines whether the behavior detection process selected in S305 is a process based on time-series changes. If the detection control unit 204 determines that the process is based on time-series changes, the process proceeds to S307. If the detection control unit 204 determines that the process is not based on time-series changes, the process proceeds to S308. In this embodiment, if a process executed by either the face tremble detection unit 1208 or the behavior detection unit 1209 is selected, the process proceeds to S307. If a process executed by either the stay time detection unit 1206 or the face direction estimation unit 1207 is selected, the process proceeds to S308.

[0055] In S307, the behavior detection unit corresponding to the selected behavior detection process (either the face vibration detection unit 1208 or the behavior detection unit 1209) creates a tracking result list excluding the tracking suspended state. In S308, the behavior detection unit corresponding to the selected behavior detection process (one of the staying time detection unit 1206, face direction estimation unit 1207, face trembling detection unit 1208, and behavior detection unit 1209) executes the behavior detection process. In S309, the detection result transmission unit 209 transmits the behavior detection result to a predetermined destination, after which the process proceeds to S304. In S310, unless there is a request to stop image reception, the image processing device 200 repeatedly executes the processes of S301 to S309. If there is a request to stop image reception, the series of steps in the flowchart shown in FIG.

[0056] As described above, according to this embodiment, it is possible to switch whether to use tracking results for behavior detection in which the tracked person may have switched positions due to the influence of overlapping people, etc. As a result, behavior detection based on time-series changes can use tracking rectangle and posture estimation information that excludes tracking suspended states, and behavior detection not based on time-series changes can use tracking rectangle and posture estimation information that includes tracking suspended states. Therefore, since the tracking results can be used for behavior detection without excess or deficiency, the probability of false positives and negatives in behavior detection can be reduced.

[0057] (Other embodiments) Although the exemplary embodiments have been described above in detail, the present invention can be embodied as, for example, a system, a device, a method, a program, a recording medium (storage medium), etc. Specifically, the present invention may be applied to a system made up of multiple devices (for example, a host computer, an interface device, an imaging device, a web application, etc.), or may be applied to an apparatus made up of a single device.

[0058] Needless to say, the object of the present invention can be achieved by the following: A recording medium (or storage medium) on which program code (computer program) of software that realizes the functions of the above-described embodiments is recorded is supplied to a system or device. The storage medium is, of course, a computer-readable storage medium. The computer (or CPU or MPU) of the system or device then reads and executes the program code stored on the recording medium. In this case, the program code itself read from the recording medium realizes the functions of the above-described embodiments, and the recording medium on which the program code is recorded constitutes the present invention. [Explanation of symbols]

[0059] 100: Imaging device, 200: Image processing device

Claims

1. an acquisition means for acquiring an image of a person; a tracking means for detecting and tracking a person in the image; a control means for controlling a process for detecting a person's behavior based on a tracking result of the tracking means; and the tracking result includes status information indicating a tracking status of the detected person; The control means controls whether to use the tracking result based on the type of processing and the state information.

1. An image processing device comprising:

2. The image processing device according to claim 1 , wherein the tracking means sets the status information to a pending state when association with a person detected in the past is to be suspended.

3. 3. The image processing device according to claim 1, wherein the tracking means sets the status information to a hold state when the degree of overlap of people is equal to or greater than a predetermined value.

4. The image processing device described in any one of claims 1 to 3, characterized in that the tracking means assigns an identifier to the person indicating its identity with a person detected in the past, and if the probability that the identifier has been assigned to another person is greater than or equal to a predetermined value, puts the status information into a pending state.

5. 3. The image processing device according to claim 2, wherein the tracking means sets the status information to a pending state when there are a plurality of candidates for the association.

6. The control means If the type of the process is a type that is not based on a time-series change, control is performed so that the tracking result in the pending state is used when the process is executed; If the type of the process is based on a time-series change, control is performed so that the tracking result in the pending state is not used when the process is executed.

6. The image processing device according to claim 2, wherein the image processing device is a computer.

7. the acquisition means acquires the images in time series, the tracking means generates the tracking results in time series, When the type of the processing is a type based on the time-series change, the control means performs control to create time-series data excluding the tracking result in the suspended state.

7. The image processing device according to claim 6,

8. 7. The image processing device according to claim 6, wherein the processing based on the time series change is processing for calculating parameters for estimating position information of a person based on the position and size of a rectangle surrounding the person.

9. The image processing device according to claim 8, characterized in that the processing of a type not based on time-series changes is processing for calculating the distance between multiple people based on position information of the people estimated based on the parameters and the position and size of a rectangle surrounding the people.

10. The image processing apparatus according to claim 6 , wherein the processing not based on time-series changes is processing for measuring a time period during which a person stays within an imaging range.

11. 7. The image processing apparatus according to claim 6, wherein the processing not based on time-series changes is processing for estimating a face direction of a person.

12. The method further comprises estimating means for estimating position information of a component of a person in the image, The processing based on the time series change is processing for detecting a person's behavior based on the time series change of the position information of the component.

7. The image processing device according to claim 6,

13. The image processing device according to claim 6, characterized in that the processing of the type based on the time series change is processing that detects a person's surrounding confirmation behavior based on the difference between the person's facial direction and a predicted value predicted from the facial direction.

14. an acquisition step of acquiring an image of a person; a tracking step of detecting and tracking a person in the image; a control step of controlling a process for detecting a behavior of a person based on a tracking result obtained by the tracking step; Including, the tracking result includes status information indicating a tracking status of the detected person; In the control step, whether to use the tracking result is controlled based on the type of processing and the state information. An image processing method comprising:

15. A program for causing a computer to function as each of the means of the image processing apparatus according to any one of claims 1 to 13.

Images