Image recognition apparatus

Abstract

To suppress deterioration in accuracy of group action recognition caused by temporary absence or non-detection of a person.SOLUTION: An image recognition device includes one or more processors that perform group action recognition of a group from a result of person action recognition of a plurality of persons based on time-series frame images generated by a camera that images a space in which the group is present. The person action recognition can be performed using machine learning. The one or more processors execute a person action recognition determination process of determining that a person action recognition of a person who is a determination target of a first duration time is impossible in a case where the first duration time during which a state where a person present in the past frame image is not present in the latest frame image is continued is less than a first time threshold value, and a group action recognition determination process of determining that a group action recognition is impossible in a case where a second duration time during which a state where the number of persons determined that the person action recognition is impossible is equal to or greater than a number-of-persons threshold value is continued is equal to or greater than a second time threshold value.SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present disclosure relates to an image recognition device that performs group behavior recognition. [Background technology]

[0002] Patent Document 1 discloses a learning device that performs learning for group behavior recognition. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-187870 Summary of the Invention [Problem to be solved by the invention]

[0004] When detecting people who make up a group for group behavior recognition, people may be temporarily absent or undetected due to occlusion of other people, etc. In group behavior recognition, it is common to exclude absent or undetected people from group behavior recognition. However, simply excluding absent or undetected people from group behavior recognition may result in a decrease in the accuracy of group behavior recognition. [Means for solving the problem]

[0005] The image recognition device according to the present disclosure includes one or more processors that perform collective behavior recognition of a group from the results of human behavior recognition of a plurality of people based on a time-series of frame images generated by a camera that captures a space in which the group exists. The one or more processors execute a human behavior recognition determination process that determines that human behavior recognition of a person to be determined as a first duration is impossible when a first duration during which a person that existed in a previous frame image continues to be absent in a latest frame image is less than a first time threshold, and a collective behavior recognition determination process that determines that collective behavior recognition is impossible when a second duration during which a state continues in which the number of people for which human behavior recognition is determined to be impossible is equal to or greater than a number-of-people threshold is equal to or greater than a second time threshold. [Effects of the Invention]

[0006] According to the present disclosure, when the number of people for whom human behavior recognition is determined to be impossible continues to be equal to or greater than the number threshold for a second duration equal to or greater than a second time threshold, group behavior recognition is determined to be impossible. This prevents group behavior recognition from being immediately determined to be impossible when a person is temporarily absent or undetected. This leads to the prevention of a decrease in the accuracy of group behavior recognition. [Brief explanation of the drawings]

[0007] [Figure 1] 1 is a diagram schematically illustrating an example of a configuration of an image recognition device according to an embodiment. [Figure 2] 10 is a flowchart illustrating an example of a processing flow of an image recognition device according to an embodiment. [Figure 3] FIG. 10 is a diagram illustrating an example of a method for setting a reliability threshold. [Figure 4] 3 is a diagram for explaining an example of a method for obtaining a time threshold Tth3 used in step S120 in FIG. 2. FIG. [Figure 5] 3 is a table showing examples of behavior recognition results for each person collected by the processing in step S126 in FIG. 2. DETAILED DESCRIPTION OF THE INVENTION

[0008] Embodiments of the present disclosure will be described with reference to the accompanying drawings.

[0009] 1. Image recognition device configuration FIG. 1 is a diagram schematically illustrating an example of a configuration of an image recognition device 10 according to an embodiment.

[0010] The image recognition device 10 acquires time-series frame images F generated by a camera 3 that captures an image of a space S in which a group 1 exists. The group 1 is made up of a plurality of people 2 (2-1 to 2-N: N is an integer equal to or greater than 2). FIG. 1 shows an example of the frame image F. The space S is, for example, a passenger compartment of a moving object such as a train. The frame image F shows four passengers 2-1 to 2-4 as an example of the plurality of people 2. The camera 3 is mounted on the moving object, for example. The image recognition device 10 is configured to perform collective behavior recognition of the group 1 from the results of human behavior recognition of the plurality of people 2 based on the time-series frame images F.

[0011] The image recognition device 10 includes a communication device 11, one or more processors 12 (hereinafter simply referred to as processors 12), and one or more storage devices 13 (hereinafter simply referred to as storage devices 13). The communication device 11 communicates with a mobile object via a communication network and acquires a time series of frame images F generated by a camera 3.

[0012] The processor 12 executes various processes. Examples of the processor 12 include a general-purpose processor, a specific-purpose processor, a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), and a field-programmable gate array (FPGA). The storage device 13 stores various information required for various processes. Examples of the storage device 13 include a volatile memory, a non-volatile memory, a hard disk drive (HDD), and a solid state drive (SSD).

[0013] The processor 12 executes a computer program. The computer program is stored in the storage device 13. The computer program may be recorded on a computer-readable recording medium. The functions of the image recognition device 10 are realized by cooperation between the processor 12 executing the computer program and the storage device 13. The functions thus realized include functions as a person detector (see step S102), a same person determiner (see step S104), and a behavior recognizer (see step S108), which will be described later.

[0014] A person list is stored in the storage device 13. The person list has various types of person information about each person 2 detected from the frame image F. The person information includes, for example, person ID (Identification) information, information about the frame number of the frame image F in which the person 2 is detected, and person-specific behavior type information, which will be described later.

[0015] 2. Image recognition device processing When detecting people who make up a group for group behavior recognition, people may be temporarily absent or undetected due to people hiding from each other, etc. In group behavior recognition, it is common to exclude absent or undetected people from group behavior recognition. However, simply excluding absent or undetected people from group behavior recognition may reduce the accuracy of group behavior recognition. Note that the "absence" of a person here refers to the fact that the person is present in the space captured by the camera but is not detected because the person is not captured by the camera due to reasons such as being hidden by other people, and "undetected" refers to the fact that a person captured by the camera is not detected.

[0016] FIG. 2 is a flowchart showing an example of the processing flow of image recognition device 10 according to an embodiment. The processing of this flowchart is executed by image recognition device 10 (processor 12) in response to a request to obtain a group behavior recognition result. In an example where group 1 is passengers of a moving object (see FIG. 1), this request is transmitted from the moving object to image recognition device 10. More specifically, camera 3 obtains frame images F at a predetermined frame rate (e.g., 2 fps or higher). Upon receiving the above-mentioned request, image recognition device 10 receives frame images F in order from camera 3 and stores the received frame images F in storage device 13. Image recognition device 10 (processor 12) repeatedly executes the processing of this flowchart for each individual frame image F in the time series.

[0017] In step S100, the processor 12 inputs the latest frame image F (i.e., the current frame image F) to the person detector. Next, in step S102, the processor 12 (person detector) executes person detection processing to detect a person 2 present (shown) in the current frame image F. The person detection processing uses a machine learning model that has been trained in advance and stored in the storage device 13. More specifically, the processor 12 detects a rectangular image area (see FIG. 1) that surrounds the person 2 present in the current frame image F. Such person detection processing is a well-known technique, and the method is not particularly limited. The processor 12, for example, stores in the storage device 13 a partial image obtained by cutting out the current frame image F from the detected rectangular image area.

[0018] After step S102, the process proceeds to step S104. The processes of steps S104 to S124 are executed one by one in order for all persons 2 registered in the person list, including persons 2 that may be newly detected by the current process of step S102. In the following description, the person 2 currently being processed in steps S104 to S124 is also referred to as the "person 2 to be recognized."

[0019] In step S104, the processor 12 (same person determiner) executes a same person determination process. Specifically, the processor 12 compares a predetermined number (e.g., 10) of previous frame images F going back from the current frame image F with the current frame image F to determine (identify) the person 2 that also exists in the current frame image F among the people 2 that exist in the previous frame images F. This same person determination process can be performed, for example, by utilizing a well-known person re-identification (Person ReID (Re-Identification)) technology.

[0020] More specifically, the storage device 13 stores partial images of each person 2 included in the current frame image F and a predetermined number of previous frame images F, and a ReID model based on machine learning. The processor 12 executes the same person determination process using the partial image of the person 2 to be recognized included in the current frame image F, the partial image of each person 2 included in each of the predetermined number of previous frame images F, and the ReID model. In addition, if a new person 2 is detected as a result of the same person determination process, the processor 12 adds the detected person 2 to the person list (see FIG. 1).

[0021] In step S106 following step S104, the processor 12 determines whether or not an absence determination condition is met, for example, using information from the person list. The absence determination condition is that the recognition target person 2 is detected in at least one of a predetermined number of past frame images F, and is not detected in the current frame image F. If the recognition target person 2 is detected in the current frame image F, that is, if the absence determination condition is not met (step S106; No), the process proceeds to step S108.

[0022] In step S108, the processor 12 (behavior recognizer) executes human behavior recognition of the recognition target person 2. Human behavior recognition can be performed using a well-known behavior recognition technology that uses a machine learning model. Specifically, the storage device 13 also stores a behavior recognition model based on machine learning. The processor 12 executes human behavior recognition of the recognition target person 2 using a partial image of the recognition target person 2 included in the current frame image F, a partial image of the recognition target person 2 included in the past frame image F, and the behavior recognition model.

[0023] More specifically, the human behavior recognition result obtained by the processing in step S108 includes a reliability R output from the behavior recognition model for each "human behavior type" that identifies the behavior of the person 2 to be recognized. The reliability R indicates the reliability (in other words, likelihood) of the recognition result for each human behavior type, and is output as a numerical value between 0 and 100, for example (see, for example, FIG. 5A described later). In an example in which person 2 is a passenger on a moving object, the individual human behavior types are, for example, "sitting," "hanging strap," "handrail," "walking," "falling," and "other," as shown in FIG. 5A. "sitting" indicates that person 2 is sitting in a seat. "hanging strap" and "handrail" indicate that person 2 is standing while holding onto a strap and a handrail, respectively. "walking" indicates that person 2 is walking in the passenger compartment. "falling" indicates that person 2 has fallen in the passenger compartment. "Other" comprehensively indicates the behavior of person 2 that does not fall into any of the five behaviors from "sitting" to "falling."

[0024] The processor 12 stores the human behavior recognition result obtained in step S108 (i.e., information on the reliability R for each human behavior type based on the current frame image F) as human behavior type information in the storage device 13. The human behavior type information is included in, for example, a person list.

[0025] In step S110 following step S108, the processor 12 sets the human behavior recognition give-up flag to 0. Information indicating the state (0 or 1) of the set human behavior recognition give-up flag is stored in the storage device 13, for example, as information included in the person-by-person behavior type information.

[0026] On the other hand, if the absence determination condition is met (step S106; Yes), the process proceeds to step S112. In step S112, the processor 12 determines whether the absence duration T1 (first duration) is equal to or greater than a predetermined time threshold Tth1 (first time threshold). The absence duration T1 corresponds to the time during which the recognition target person 2 does not exist in the latest frame image F in the past frame image F (i.e., the absence determination condition is met). The absence duration T1 is counted, for example, by a timer in the processor 12, but may also be calculated using the number of frames during which the absence of the recognition target person 2 continues and the frame rate. If the absence duration T1 is less than the time threshold Tth1 (step S112; No), the process proceeds to step S114.

[0027] In step S114, the processor 12 acquires the human behavior recognition result (past value) of the past frame. Specifically, the processor 12 reads out the relevant human behavior type information, that is, the information on the reliability R for each human behavior type based on the most recent past frame image F in which the recognition target person 2 exists, from the storage device 13. Thereafter, the process proceeds to step S116.

[0028] In step S116, the processor 12 determines whether the human behavior recognition result of the read previous frame is reliable. This determination condition is met when at least one of the reliabilities R of the human behavior recognition result for each person behavior type based on the previous frame image F is equal to or greater than a predetermined reliability threshold Rth, and the human behavior recognition give-up flag of the previous value is 0. In the following description, the reliability R of the human behavior recognition result based on the previous frame image F will also be simply referred to as the "past value of reliability R."

[0029] 3(A) and 3(B) are diagrams illustrating an example of a method for setting the reliability threshold Rth used in step S116. Fig. 3(A) shows the relationship between the reliability threshold Rth for past values ​​of reliability R and the absence duration T1. Fig. 3(B) shows an example of calculating the numerical value of the reliability threshold Rth when the absence duration T1 is a certain value (e.g., 500 ms) together with examples of past values ​​of reliability R for each person and behavior type.

[0030] As shown in Fig. 3(A), the reliability threshold Rth is set to differ depending on the person's behavior type (sitting, walking, falling, etc.). The reason is that the variability of person 2's behavior differs depending on the person's behavior type. According to this setting example, the past value of reliability R can be appropriately evaluated taking into account the variability of person 2's behavior.

[0031] The reliability threshold Rth is set to vary depending on the absence duration T1. More specifically, as shown in FIG. 3A, the reliability threshold Rth for each person-specific behavior type is set to increase as the absence duration T1 passes. This is because the tendency for a person 2's behavior to change over time differs depending on the person-specific behavior type. For example, it takes a certain amount of time for a person who has fallen to get up. In contrast, it does not take much time for a person to sit down or stand up. This setting example allows the past value of the reliability R to be appropriately evaluated taking into account the tendency for a person 2's behavior to change over the absence duration T1. In addition, the reliability threshold Rth for a person-specific behavior type (e.g., walking) that requires greater attention to ensure the safety of person 2, who is a passenger of a moving object, may be set as follows. That is, the reliability threshold Rth may be set so that the increase in the reliability threshold Rth over the absence duration T1 is greater than the reliability threshold Rth for a person-specific behavior type that does not require greater attention.

[0032] Information indicating the above-mentioned relationship between the behavior type by person, the absence duration T1, and the reliability threshold Rth is stored in the storage device 13. Based on the information, the processor 12 calculates the reliability threshold Rth for each behavior type by person according to the absence duration T1 when executing the process of step S116.

[0033] If the past value is reliable (step S116; Yes), the process proceeds to step S118. In step S118, the processor 12 substitutes the human behavior recognition result of the past frame as the human behavior recognition result of the current frame. As a result, the past value is stored in the storage device 13 as human behavior type information indicating the human behavior recognition result of the current frame. Also, in step S118, the processor 12 sets the human behavior recognition give-up flag to 0, similar to step S110.

[0034] On the other hand, if the past value is not reliable (step S116; No), that is, if all of the reliabilities R of the human behavior recognition results for each person-specific behavior type are less than the reliability threshold Rth or the human behavior recognition give-up flag is 1, the process proceeds to step S120. In step S120, the processor 12 determines whether the absence duration T1 is equal to or greater than a predetermined time threshold Tth3. This time threshold Tth3 corresponds to the time (forgetting time) for determining whether to delete the registration of a person 2 with a long absence duration T1 from the person list.

[0035] 4(A) and 4(B) are diagrams illustrating an example of a method for obtaining the time threshold Tth3 used in step S120. FIG. 4(A) shows the relationship between past values ​​of reliability R (more specifically, reliability R of person 2's final behavior before he / she became absent) and "threshold candidate" for each person-specific behavior type (e.g., walking, sitting, falling). The threshold candidate here is a candidate for the time threshold Tth3 (forgetting time). FIG. 4(B) shows an example of calculating the numerical value of the threshold candidate based on the past values ​​of reliability R for each person-specific behavior type.

[0036] The candidate threshold for each person-specific behavior type is determined taking into consideration the difference in the likelihood of a person 2 going out of frame depending on the person-specific behavior type and the magnitude of the past value of the reliability R. Specifically, for example, a walking person 2 is more likely to go out of frame than a person who has fallen. Therefore, if the past value of the reliability R for "walking" is high, it can be considered that the person 2 is likely to be absent because he or she has left the space S. Therefore, as shown in FIG. 4A, the candidate threshold for "walking" is set to be shorter when the past value of the reliability R is high. On the other hand, if the past value of the reliability R for "falling" is high, it can be considered that the person 2 is likely to remain immobile and remain visible on the camera 3 but not be detected by the person detection process. Therefore, as shown in FIG. 4A, the candidate threshold for "falling" is set to be longer when the past value of the reliability R is high. On the other hand, if the past value of the reliability R for each person-specific behavior type is low, it is difficult to definitively determine the human behavior recognition result. For this reason, as shown in FIG. 4(A), each threshold candidate for the person-by-person behavior type is set to become longer as the reliability R decreases.

[0037] The storage device 13 stores information indicating the above-described relationship between the behavior type by person, the past value of the reliability R, and the threshold candidate. Based on the information, the processor 12 calculates a threshold candidate corresponding to the past value of the reliability R for each behavior type by person. Then, the processor 12 acquires the smallest value among the calculated threshold candidate values ​​as the time threshold Tth3. In the example shown in FIG. 4(B), the processor 12 acquires 4000 ms as the time threshold Tth3.

[0038] If the absence duration T1 is less than the time threshold value Tth3 (step S120; No), the process proceeds to step S122. In step S122, the processor 12 determines that human behavior recognition of the current recognition target person 2 (i.e., the person 2 whose absence duration T1 is to be determined) is impossible, and sets the human behavior recognition give-up flag to 1. Note that in FIG. 2, the processes of steps S106-S122 correspond to the "human behavior recognition determination process" according to the present disclosure.

[0039] On the other hand, if the absence duration T1 is equal to or greater than the time threshold Tth1 (step S112; Yes), or if the absence duration T1 is equal to or greater than the time threshold Tth3 (step S120; Yes), the process proceeds to step S124. In step S124, the processor 12 deletes the registration of the current recognition target person 2 from the person list. That is, the person information regarding the person 2 whose absence duration T1 is long and therefore the need for the person to be a recognition target has sufficiently decreased is deleted from the storage device 13. This reduces the processing load on the image recognition device 10. In addition, according to the processes of steps S112-S116 and S120, when the absence duration T1 is less than the time threshold Tth1, it is possible to appropriately determine the time to delete the person information from the person list, taking into account the past value of the reliability R.

[0040] In Fig. 2, when the processes of steps S104 to S124 for all persons 2 registered in the person list are completed, the process proceeds to step S126. The processes from step S126 onwards are executed once for each frame image F. First, in step S126, the processor 12 reads out the behavior recognition results of each person 2 from the storage device 13. Thereafter, the process proceeds to step S128.

[0041] In step S128, the processor 12 determines whether the give-up presence / absence determination and the special behavior presence / absence determination are satisfied. The give-up presence / absence determination is satisfied when the number of persons 2 whose human behavior recognition give-up flag is 1 (i.e., the number of persons 2 for whom human behavior recognition is determined to be impossible) is equal to or greater than a predetermined number threshold Nth. The special behavior presence / absence determination is satisfied when there are no persons 2 performing special behavior. An example of a special behavior is "falling down."

[0042] FIG. 5A is a table showing an example of the behavior recognition results for each person 2 collected by the processing in step S126. Here, this table is used to explain the determination of whether or not a person has given up and the determination of whether or not a special behavior has occurred. As an example, this table shows the person behavior recognition results (i.e., the numerical values ​​of reliability R) for four people 2 (No. 1 to No. 4) for each person-specific behavior type. This table also shows the status (0 or 1) of the person behavior recognition give-up flag. In addition, since person No. 4 has the person behavior recognition give-up flag set to 1, it does not have a numerical value of reliability R for each person-specific behavior type. Furthermore, this table also shows a behavior recognition threshold value set in advance for each person-specific behavior type. Among the person behavior recognition results shown in this table, those with a numerical value equal to or greater than the behavior recognition threshold value may be the target of group behavior recognition, which will be described later. For example, for person No. 1, the recognition result of "sitting" (numerical value: 100) may be the target. For person No. 2, the recognition results for "sitting" (numeric value: 70), "hanging strap" (numeric value: 90), and "walking" (numeric value: 60) are possible targets. For person No. 3, the recognition results for "handrail" (numeric value: 80), "walking" (numeric value: 50), and "other" (numeric value: 60) are possible targets.

[0043] The number of people threshold Nth used to determine whether or not a person has given up is not particularly limited, and may be calculated to be, for example, ¼ of the number of detected people 2 (including absent people 2 but excluding people 2 to be forgotten). In the example shown in FIG. 5(A), the number of detected people is four, so the number of people threshold Nth is 1. And there is one person 2 (No. 4) whose human behavior recognition give-up flag is 1. Therefore, in the example shown in FIG. 5(A), the processor 12 determines that the determination of whether or not a person has given up is successful.

[0044] Furthermore, with regard to the determination of the presence or absence of special behavior, the processor 12 identifies a person 2 having a reliability R for "falling" that is equal to or greater than the behavior recognition threshold (fall threshold) as a person 2 having special behavior. In the example shown in FIG. 5(A), there is no person 2 having a reliability R for "falling" that is equal to or greater than the behavior recognition threshold (e.g., 55), so the processor 12 determines that the determination of the presence or absence of special behavior is not valid.

[0045] If at least one of the determination of whether or not there is a give-up and the determination of whether or not there is a special action is not established (step S128; No), the process proceeds to step S130. In step S130, the processor 12 performs group behavior recognition based on the current frame image F.

[0046] FIG. 5(B) is a table showing another example of the behavior recognition results of each person 2 collected by the processing of step S126, and is used here to explain group behavior recognition. Compared to the table shown in FIG. 5(A), the table shown in FIG. 5(B) is the same in the person behavior recognition results of three people (No. 1 to No. 3), but is different in the person behavior recognition result (reliability R and give-up flag) of the fourth person 2 (No. 4). According to the person behavior recognition results shown in the table shown in FIG. 5(B), the give-up presence / absence determination is not established, and the fall determination is also not established. Note that in FIG. 5(B), part of the reliability R of each person 2 is omitted.

[0047] In group behavior recognition in step S130, processor 12 selects the recognition result with the highest reliability R from the human behavior recognition results for each person 2. In the example shown in FIG. 5(B), for person No. 1, the recognition result of "sitting" (numeric value: 100) is selected. For person No. 2, the recognition result of "hanging strap" (numeric value: 90) is selected. For person No. 3, the recognition result of "handrail" (numeric value: 80) is selected. For person No. 4, the recognition result of "hanging strap" (numeric value: 70) is selected.

[0048] In an example where group 1 is passengers on a moving object, the group behavior types determined in group behavior recognition are, for example, "everyone seated," "everyone seated or holding on to straps or handrails," and "someone has fallen." Additionally, "everyone seated or holding on to straps or handrails" indicates that the individual behavior types of all recognized persons 2 are either "seated," "handrail," or "handrail."

[0049] In the example shown in FIG. 5(B), the person-specific behavior type of the selected recognition result is either "sitting," "hanging strap," or "handrail" for all detected persons 2 (No. 1 to No. 4). Therefore, in this example, processor 12 recognizes "all sitting, or holding onto a strap, or holding onto a handrail" as the behavior of group 1. In other words, processor 12 acquires "all sitting, or holding onto a strap, or holding onto a handrail" as the group behavior recognition result based on the current frame image F.

[0050] In addition, unlike the example shown in Figure 5(B), if the recognition result selected for all detected persons 2 (No. 1 to No. 4) is "sitting," processor 12 acquires "all persons sitting" as the group behavior recognition result. Also, unlike the example shown in Figure 5(B), if there is one or more persons 2 whose reliability R for "falling" is equal to or greater than the behavior recognition threshold (e.g., 55), processor 12 acquires "someone has fallen" as the group behavior recognition result.

[0051] Furthermore, in step S130, processor 12 sets the group behavior recognition give-up flag to 0. Processor 12 stores the group behavior recognition result obtained as described above in storage device 13 together with information indicating the state (0 or 1) of the set group behavior recognition give-up flag.

[0052] On the other hand, if the determination of whether or not a person has given up and the determination of whether or not a special action has been taken are both true (step S128; Yes), the process proceeds to step S132. In step S132, the processor 12 determines whether or not the second duration T2 is equal to or greater than a predetermined time threshold Tth2 (second time threshold). The second duration T2 corresponds to the time during which a state in which the number of people 2 for which it has been determined that human behavior recognition is impossible continues to be equal to or greater than the number threshold Nth. The second duration T2 is counted, for example, by a timer in the processor 12, but may also be calculated using the number of frames during which the state continues and the frame rate.

[0053] If the second duration T2 is less than the time threshold Tth2 (e.g., 2000 ms) (step S132; No), the process proceeds to step S134. In step S134, the group behavior recognition result (i.e., the past value) based on the past frame image F (e.g., the most recent past frame image F) is substituted as the group behavior recognition result for the current frame. As a result, the past value is stored in the storage device 13 as the group behavior recognition result for the current frame. Also, in step S134, the processor 12 sets the group behavior recognition give-up flag to 0, similar to step S130.

[0054] On the other hand, if the second duration T2 is equal to or greater than the time threshold Tth2 (step S132; Yes), the process proceeds to step S136. In step S136, the processor 12 determines that group behavior recognition is impossible, and sets the group behavior recognition give-up flag to 1. Note that in FIG. 2, the processes of steps S126-S136 correspond to the "group behavior recognition determination process" according to the present disclosure.

[0055] After step S130, S134, or S136, the process proceeds to step S138. In step S138, processor 12 outputs (transmits) the group behavior recognition result of the current frame to the destination of use. In an example in which group 1 are passengers on a moving object, the destination of use is, for example, the moving object. In addition, the group behavior recognition result is used, for example, to determine whether or not to depart in an autonomously traveling moving object. The moving object is determined to be able to depart if the group behavior recognition result is "everyone seated" or "everyone seated or holding onto straps or handrails." Furthermore, since the group behavior recognition result to be output to the destination of use includes "someone has fallen," the destination of use can recognize the presence of a person who has fallen and can quickly take measures, such as rescuing the person who has fallen.

[0056] 2 may be simplified as follows: Steps S114-S120 may be omitted, and the process may proceed directly to step S122 if the absence duration T1 is less than the time threshold Tth1 (step S112; No).

[0057] 3.Effects As described above, according to the image recognition device 10 of this embodiment, if the second duration T2 during which the number of people determined to be unable to recognize human behavior continues to be equal to or greater than the number threshold Nth is equal to or greater than the time threshold Tth2, it is determined that group behavior recognition is impossible. This makes it possible to prevent group behavior recognition from being immediately determined to be impossible when a person 2 is temporarily absent or undetected. This leads to the prevention of a decrease in the accuracy of group behavior recognition.

[0058] More specifically, according to this embodiment, when absence duration T1 is less than time threshold Tth1, if the past value of reliability R is high, the human behavior recognition result of the past frame is substituted as the human behavior recognition result of the current frame. On the other hand, if the past value of reliability R is low, it is determined that human behavior recognition is impossible. As a result, when a person 2 is temporarily absent or undetected, group behavior recognition can be appropriately continued by substituting the human behavior recognition result of the past frame.

[0059] Furthermore, according to this embodiment, if the second duration T2 is less than the time threshold Tth2, the group behavior recognition result of the previous frame is substituted as the group behavior recognition result of the current frame. This makes it possible to provide robustness to group behavior recognition by substituting the group behavior recognition result of the previous frame when a person 2 is temporarily absent or undetected. [Explanation of symbols]

[0060] 1 group, 2 multiple people, 3 camera, 10 image recognition device, 11 communication device, 12 processor, 13 storage device

Claims

1. one or more processors that perform collective behavior recognition of a group from a result of human behavior recognition of a plurality of people based on time-series frame images generated by a camera that captures a space in which the group exists; the one or more processors: a human behavior recognition determination process for determining that the human behavior recognition of the person being the determination target for the first duration is impossible when a first duration time during which a person who was present in a past frame image continues to be absent from a latest frame image is less than a first time threshold; a group behavior recognition determination process for determining that the group behavior recognition is impossible when a second duration during which the number of people for whom the human behavior recognition is determined to be impossible continues to be equal to or greater than a number threshold is equal to or greater than a second time threshold; Run Image recognition device.

2. The image recognition device according to claim 1, When the first duration is less than the first time threshold in the human behavior recognition determination process, the one or more processors: If the reliability of the result of the human behavior recognition based on the past frame image is high, the result of the human behavior recognition based on the past frame image is substituted for the result of the human behavior recognition based on the latest frame image; If the reliability is low, it is determined that the human behavior recognition is impossible. Image recognition device.

3. 3. The image recognition device according to claim 1, In the group behavior recognition determination process, if the second duration is less than the second time threshold, the one or more processors substitute a result of the group behavior recognition based on a past frame image for a result of the group behavior recognition based on the latest frame image. Image recognition device.

4. 3. The image recognition device according to claim 2, The reliability threshold for determining whether the reliability is high or low varies depending on the person-specific behavior type that identifies the behaviors of the plurality of people. Image recognition device.

5. 3. The image recognition device according to claim 2, A reliability threshold for determining whether the reliability is high or low varies depending on the first duration. Image recognition device.

Images