Attentional behavior call system

Abstract

To provide an attentional behavior call system that enables appropriate calls to be made according to the attributes, etc. of a behavior object when abnormal behavior or other attentional behavior of the behavior object in a video image is detected.SOLUTION: An attentional behavior call system 1 detects feature points of a target behavior object Z in a time-series image Y, and different calls are made by voice according to the type of attentional behavior when attentional behavior is determined to have been performed based on the displacement of the stored feature points and the detected feature points. However, even when the attentional behavior is determined to have been performed, if the attribute or size of the target behavior object Z satisfies a first predetermined condition, the call is not made or the content or manner of the call is changed.SELECTED DRAWING: Figure 3

Description

[Technical field]

[0001] The present invention relates to a system for calling out attention to an action that is capable of making an appropriate call based on the attributes of an action that is captured on video when attention-grabbing action such as abnormal behavior of the action that is captured on video is detected. [Background technology]

[0002] There has been known a technique for extracting a behavioral entity that has behaved differently from a predetermined "normal behavior" from a time-series image when the behavior of the behavioral entity captured in the time-series image differs from the "normal behavior" (see, for example, Patent Document 1). [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent No. 6525179 Summary of the Invention [Problem to be solved by the invention]

[0004] By using the above technology, it is conceivable that an alarm unit provided near the imaging means could issue a call to an entity that has engaged in abnormal behavior.

[0005] However, for example, children often behave in a way that is judged to be abnormal, and among these behaviors, there are some that would be abnormal if the behavioral subject were an adult, but would not be considered abnormal if the behavioral subject were a child. In such a situation, if calls were made to both adults and children in the same way, the frequent calls would become a noise, and the credibility of the calls would be lost.

[0006] Therefore, the present invention aims to provide a system for calling out attention to behaviors that can make an appropriate call based on the attributes of an action captured in a video when attention-grabbing behavior, such as abnormal behavior, is detected. [Means for solving the problem]

[0007] The present invention provides an attention behavior calling system comprising an acquisition unit that acquires time-series images captured by a photographing means, a detection unit that detects feature points of a target behavior object captured in the time-series images, a memory unit that stores the displacement of the feature points of the behavior object when the behavior object performs an attention behavior, a judgment unit that determines whether the attention behavior has been performed based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image, an estimation unit that estimates the attributes or size of the target behavior object based on the detected feature points or the appearance of the target behavior object, and a calling unit that is provided around the photographing range of the photographing means and that makes a different call by voice depending on the type of the attention behavior when it is determined that the attention behavior has been performed, wherein the calling unit does not make the call or changes the content or manner of the call even when it is determined that the attention behavior has been performed if the attributes or size satisfy a first predetermined condition.

[0008] With this configuration, when a noteworthy behavior such as an abnormal behavior of a target behavior entity is detected in a time-series image, it becomes possible to make an appropriate call according to the attributes of the target behavior entity, etc. For example, it becomes possible to stop the call when the attribute is "child," and to increase the volume of the call when the attribute is "elderly."

[0009] In addition, in another aspect of the present invention, there are provided a program and a method for calling for attention to an action corresponding to the above-mentioned system for calling for attention to an action. Effect of the Invention

[0010] According to the attention behavior calling system of the present invention, when attention behavior such as abnormal behavior of an active subject captured in a video is detected, it is possible to make an appropriate call based on the attributes of the active subject, etc. [Brief description of the drawings]

[0011] [Figure 1]FIG. 1 is an explanatory diagram of a time-series image according to a first embodiment of the present invention; [Diagram 2] Block diagram of an attention behavior calling system according to a first embodiment of the present invention. [Diagram 3] 1 is a flowchart of a system for calling attention to behavior according to a first embodiment of the present invention; [Figure 4] Block diagram of an attention behavior calling system according to a second embodiment of the present invention. [Diagram 5] FIG. 2 is a block diagram of a system for calling attention to behavior according to a modified example of the present invention; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0012] Hereinafter, an attention behavior calling system 1 according to a first embodiment of the present invention will be described with reference to FIGS. 1 to 3. FIG.

[0013] As shown in Fig. 1, the attention behavior calling system 1 is for calling out to a target behavior object Z according to the type of behavior of the target behavior object Z captured in a time-series image Y (in Fig. 1, frames constituting an image) captured by a capture means X. In this embodiment, a human being is used as the target behavior object Z, and for ease of understanding, the target behavior object Z is simply displayed with only a skeleton.

[0014] 2, the attention behavior call system 1 includes an acquisition unit 2, a detection unit 3, a storage unit 4, a determination unit 5, an estimation unit 6, and a call unit 7. In this embodiment, the attention behavior call system 1 is provided integrally with an image capture means X.

[0015] The acquisition unit 2 acquires time-series images Y captured by the imaging means X.

[0016] The detection unit 3 detects feature points of a target action object Z captured in a time-series image Y.

[0017] There are various possible feature points, but in this embodiment, an example will be described in which joints are detected as feature points.

[0018] When detecting joints as feature points, for example, the following method can be considered.

[0019] First, a "joint identification standard" and a "behavior object identification standard" are stored in a storage unit (which may be the storage unit 4 or another storage unit).

[0020] The "joint identification standard" is for identifying multiple human joints, and indicates the shape, direction, size, etc. for identifying each joint.

[0021] The "Behavior Identification Standard" indicates the "basic postures" of various human variations ("walking", "standing upright", etc.), the "range of motion of each joint", and the "distance between each joint" in a single human.

[0022] After detecting multiple joints that meet the above-mentioned "joint identification criteria," it is possible to identify the joints contained in each target action object Z by referring to the "action object identification criteria."

[0023] Note that feature points may be detected individually for each time-series image Y, or may be detected collectively from a plurality of time-series images Y if the time-series order can be specified.

[0024] The storage unit 4 stores the displacements of the feature points of the action object when the action object performs the action of interest. When joints are detected as feature points, the displacements of a plurality of joints are stored.

[0025] Examples of the behavior of interest include falling, punching, kicking, etc., and a plurality of behaviors may be stored without being limited to one. Also, the behavior of interest may be a "premonition behavior" such as a preparatory movement for shoplifting. Note that the memory unit 4 may store the displacement of the feature point information when behaviors other than the behavior of interest (walking, stopping, etc.) occur in order to estimate such behaviors.

[0026] The determination unit 5 determines whether or not attention behavior has been performed based on the stored displacements of the feature points and the displacements of the feature points detected from each of the time-series images Y.

[0027] For example, when the displacement of the detected feature point matches the displacement of the feature point of the attention behavior for a predetermined period or more, it may be determined that "the attention behavior has been performed."

[0028] The estimation unit 6 estimates the attributes or size of the target action object Z based on the detected feature points or the appearance of the target action object Z.

[0029] The attributes or size of the target action object Z may be estimated by a known method.

[0030] For example, when identifying the target action object Z by joints as described above, it is possible to estimate the size of the target action object Z by determining the distance between the head joint point and the ankle joint point.

[0031] In addition to the size described above, attributes can be estimated from characteristic points such as posture and bone structure, and external appearance such as clothing, hair length, and hair color.

[0032] The calling unit 7 is provided around the photographing range of the photographing means X, and when it is determined that an attention behavior has been performed, it calls out differently by voice according to the type of attention behavior.

[0033] Different appeals depending on the type of behavior in question could be, if the behavior in question is a criminal act (shoplifting, violent acts, etc.), then, "Shoplifting (violent acts) is a crime," and if the behavior in question is an accident (falling, etc.), then, "Are you OK?", "Can you get up?", "Are you hurt?", etc.

[0034] In the case of a criminal act, the above call is expected to stop the criminal act. In addition, in the case of an accident, it is possible to determine whether or not rescue is required based on the action or gesture of the target actor Z in response to the call.

[0035] For example, children often perform behaviors that are judged to be attention-grabbing behaviors (abnormal behaviors), and among these, there are some behaviors that would be abnormal if the target behavior entity Z were an adult, but would not be considered abnormal if it were a child. In such a situation, if we were to call out to both adults and children in the same way, the frequent calls would become a noise, and we think that the credibility of the calls would be lost.

[0036] Therefore, in this embodiment, even if it is determined that attention behavior has been performed, if the attribute or size satisfies a first predetermined condition, the call unit 7 does not make a call or changes the content or manner of the call.

[0037] For example, if the attribute is "child," it may be possible to not call out to the user. Also, if the attribute is "elderly," it may be possible to increase the volume of the call (changing the manner of the call) or change the call to a request for help from people around the user (changing the content of the call), taking into consideration that the user is hard of hearing.

[0038] In addition, similar control may be performed simply when the size of the target action object Z satisfies a first predetermined condition (for example, when "the size (height) of the target action object Z is smaller than a predetermined value").

[0039] In this way, in the attention behavior calling system 1 of this embodiment, when attention behavior such as abnormal behavior of a target behavior entity Z is detected in a time-series image Y, it is possible to make an appropriate call based on the attributes, etc. of the target behavior entity Z.

[0040] Next, the flow of a call according to this embodiment will be described with reference to the flowchart of FIG.

[0041] First, when the acquisition unit 2 acquires a time-series image Y (S1), the detection unit 3 detects feature points of a target action object Z captured in the time-series image Y (S2).

[0042] In addition, the attributes or size of the target action object Z are estimated based on the feature points detected in S2 or the appearance of the target action object Z (S3).

[0043] Next, based on the displacement of the feature points stored in the storage unit 4 and the displacement of the feature points detected in S2, it is determined whether or not attention behavior has been performed (S4). S3 and S4 may be performed in the reverse order.

[0044] If it is determined that attention behavior has been performed (S4: YES), it is determined whether or not the attribute or size estimated in S3 satisfies a first predetermined condition (S5).

[0045] If the first predetermined condition is not met (S5: NO), a different call is made by voice depending on the type of attention behavior (S6).

[0046] On the other hand, if the first predetermined condition is met (S5: YES), the call is not made, or the content or manner of the call is changed (S7).

[0047] As described above, in the attention behavior call system 1 of this embodiment, even if it is determined that attention behavior has been performed, if the attribute or size satisfies a first predetermined condition, no call is made, or the content or manner of the call is changed.

[0048] With this configuration, when a noteworthy behavior such as an abnormal behavior of the target behavior entity Z is detected in the time-series images Y, it becomes possible to make an appropriate call according to the attributes of the target behavior entity Z. For example, it is possible to stop the call when the attribute is "child," and to increase the volume of the call when the attribute is "elderly."

[0049] Next, an attention behavior calling system 100 according to a second embodiment of the present invention will be described with reference to Fig. 4. Note that the same members as those in the first embodiment are given the same reference numerals, and the description thereof will be omitted.

[0050] In this embodiment, the attention behavior stored in the storage unit 4 is determined by learning.

[0051] In detail, the attention behavior call system 100 includes a learning-side acquisition unit 11, a learning-side detection unit 12, and a determination unit 13 in addition to the configuration of the first embodiment.

[0052] The learning side acquisition unit 11 acquires sample videos (a plurality of sample time-series images) captured by imaging means X that is installed so as to capture a predetermined range.

[0053] The learning side detection unit 12 detects the behavior of the sample behaving object captured in the sample video.

[0054] The behavior of the sample behavior unit can be detected by storing the displacement of feature points (such as the movement of each joint) when the sample behavior unit performs a specified behavior in a memory unit (which can be memory unit 4 or another memory unit), detecting the feature points in the same manner as detection unit 3, and detecting that "the specified behavior has been performed" if the displacement of the detected feature point information matches the displacement of the stored feature points by a specified amount or more.

[0055] The determination unit 13 determines one or more "normal behaviors" within a predetermined range based on the multiple behaviors detected by the learning-side detection unit 12. In this embodiment, the determined "normal behaviors" are stored in the storage unit 4 as attention behaviors.

[0056] "Normal behavior" can be determined based on various criteria, but for example, it is possible to determine that behavior that has a predetermined (threshold) or higher proportion among all detected behaviors is "normal behavior."

[0057] Then, the determination unit 5 determines that an attention behavior has been performed when the displacement of the feature point detected from each time-series image Y does not correspond to "normal behavior."

[0058] In detail, if the displacement of the feature points detected from each time-series image Y does not correspond to “normal behavior”, i.e., if it is determined that attention behavior has occurred (S3 in FIG. 3: YES), the operations of S4-S7 in FIG. 3 are performed as in the first embodiment.

[0059] As described above, in the attention behavior calling system 100 according to this embodiment, if the displacement of feature points detected from each time-series image Y does not correspond to the “normal behavior” determined by learning, it is determined that attention behavior has been performed.

[0060] According to such a configuration, even if the behavior is not obviously abnormal such as violent behavior or falling, it is possible to call out to the behavior that is inappropriate for the place (predetermined range). However, in the case of such a configuration, for example, the behavior of a child is more likely to be determined as attention-grabbing behavior (abnormal behavior), but even in such a case, if the attribute or size satisfies the first predetermined condition, the call is not made or the content or manner of the call is changed, thereby suppressing frequent calls due to attention-grabbing behavior (abnormal behavior) such as "children."

[0061] The attention behavior call system of the present invention is not limited to the above-described embodiment, and various modifications and improvements are possible within the scope of the claims.

[0062] For example, even if a place is not appropriate for adults to "run," a child may run without malicious intent, but it is not appropriate to uniformly call out to the behavior of "running" even if it is unintended and low risk. Therefore, depending on the malicious intent and the degree of danger of the attention behavior, a call may not be made, or the content or manner of the call may be changed.

[0063] In this case, as shown in FIG. 5, the attention behavior calling system 1 may further include a risk determination unit 8 that refers to a plurality of time-series images Y showing the attention behavior determined to have been performed and individually determines the risk level of the attention behavior determined to have been performed.

[0064] In this modified example, "whether the attribute or size satisfies a first predetermined condition" as in the embodiment is also used to determine malice. For example, if the attribute is "child" or "elderly", it can be considered that the malice is "low". It is also possible to simply consider that the malice is "low" if the size is below a predetermined value.

[0065] For example, when the behavior of interest is "running," it is possible to judge the degree of danger as "high danger" if the speed of the target behavior object Z is above a predetermined value, and "low danger" if it is below the predetermined value.

[0066] Also, for example, when the attention behavior is "lying down", the risk is low to begin with, so it is possible to determine the risk as "low risk" regardless of the speed, etc. However, when the attention behavior is "falling", it is preferable to determine the risk taking into account the speed of falling, etc.

[0067] Then, even if it is determined that attention behavior has occurred, if the attribute or size satisfies a first predetermined condition and the degree of danger satisfies a second predetermined condition set for each attribute or size, the call unit 7 does not make a call or changes the content or manner of the call.

[0068] For example, if the attribute is "child," it can be estimated that the vehicle is "low malicious intent," and if the second predetermined condition is "driving at a low speed," the vehicle is also low in danger, so no call will be made.

[0069] On the other hand, if the attribute is "adult," it can be inferred that the person has "highly malicious intent," and it is conceivable that a call will be made regardless of the level of danger.

[0070] With this configuration, when a notable behavior such as an abnormal behavior of the target behavior subject Z is detected in the time-series images Y, it becomes possible to make a more appropriate appeal depending on the malice and danger level of the notable behavior.

[0071] Additionally, the call may be further controlled depending on the location.

[0072] In this case, it is possible to further include a background detection unit 9 that detects the background captured in the time-series image Y in the attention behavior calling system 1, as shown in FIG.

[0073] Then, depending on the detected background, call unit 7 either does not make a call or changes the content or manner of the call.

[0074] For example, if the background is a staircase, the target entity Z may be startled by the call and may risk falling, so the decision may be to "not make a call." If the background is an elevator, the distance between the call unit 7 and the target entity Z is close, so the decision may be to "turn down the volume."

[0075] With this configuration, when a noteworthy behavior such as an abnormal behavior of the target behavior object Z is detected in the time-series images Y, it becomes possible to make a more appropriate appeal depending on the background.

[0076] In this case, it is even more effective if the above-mentioned risk is also taken into consideration.

[0077] For example, even if the person is a "child" and "running slowly," if there are stairs in the background, the danger level is high, so it is conceivable to issue a call.

[0078] In addition, some users may wish to "call out all abnormal behavior, even that of children."

[0079] Therefore, as shown in FIG. 5, the attention behavior call system 1 may further include a setting unit 10 that allows the user to set whether or not to make a call or to change the content or manner of the call when the age or size of the target behavior entity Z meets a predetermined condition.

[0080] Furthermore, in the above embodiment, the image capturing means X is provided integrally with the interpellation unit 7, but it is sufficient that the image capturing means X is provided around the image capturing range so that the target action object Z can be interpellated.

[0081] In addition, in the above embodiment, all components are provided integrally, but for example, this does not exclude some components (memory unit 4, judgment unit 5, estimation unit 6, etc.) being provided in different locations (cloud, etc.).

[0082] The present invention can also be applied to a program and method corresponding to the processing performed by each component as a controller, and to a recording medium storing the program. In the case of a recording medium, the program is installed in a computer or the like. Here, the recording medium storing the program may be a non-transient recording medium. A CD-ROM or the like is conceivable as a non-transient recording medium, but is not limited thereto. [Explanation of symbols]

[0083] 1.100 Attention call to action system 2 Acquisition part 3. Detection section 4 Storage section 5 Judgment section 6 Estimation part 7. Calling Section 8. Risk Assessment Section 9 Background detection section 10 Setting section 11 Learning side acquisition unit 12 Learning side detector 13 Decision Section X Shooting method Y Time series images Z action body

Claims

1. an acquisition unit for acquiring time-series images captured by an imaging means; A detection unit that detects feature points of a target action object captured in the time-series images; A storage unit that stores a displacement of a feature point of a behavior object when the behavior object performs a behavior of interest; a determination unit that determines whether or not the attention behavior has been performed based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image; an estimation unit that estimates an attribute or a size of the target action object based on the detected feature points or an appearance of the target action object; a calling unit that is provided around a photographing range of the photographing means and that, when it is determined that the attention behavior has been performed, issues a different call by voice according to the type of the attention behavior; Equipped with The attention behavior calling system is characterized in that the call unit does not make the call or changes the content or manner of the call when the attribute or size satisfies a first predetermined condition, even if it is determined that the attention behavior has been performed.

2. A learning side acquisition unit that acquires a sample video captured by the imaging means installed to capture a predetermined range; A learning side detection unit that detects the behavior of a sample behavior object captured in the sample video; A determination unit that determines one or more normal behaviors in the predetermined range based on the multiple behaviors detected by the learning side detection unit; Further comprising: The system for calling for attention to an action as described in claim 1, characterized in that the determination unit determines that the attention behavior has been performed when the displacement of feature points detected from each time-series image does not correspond to the normal behavior.

3. a risk determination unit that determines a risk of each of the attention behaviors determined to have been performed by referring to a plurality of time-series images showing the attention behaviors determined to have been performed; The attention behavior calling system of claim 1, characterized in that the call unit does not make the call or changes the content or manner of the call when the attribute or size satisfies the first predetermined condition and the degree of danger satisfies a second predetermined condition set for each attribute or size, even if it is determined that the attention behavior has been performed.

4. A background detection unit that detects a background shown in the time-series images, The attention behavior calling system according to claim 1 , wherein the calling unit does not make the call or changes the content or manner of the call depending on the detected background.

5. The call system of claim 1, further comprising a setting unit that enables a user to set whether or not to not make the call or to change the content or manner of the call when the age or size of the behavioral entity meets a predetermined condition.

6. A program executed by a computer that stores a displacement of a feature point of a behavior object when the behavior object performs a target behavior, A step of acquiring time-series images captured by an imaging means; detecting feature points of a target action object captured in the time-series images; determining whether or not the attention behavior has been performed based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image; estimating an attribute or a size of the target action based on the detected feature points or an appearance of the target action; a step of providing a voice message corresponding to a type of the attention behavior when the attention behavior is determined to have been performed, the voice message being provided around a shooting range of the shooting means; Equipped with A program for calling out attention behavior, characterized in that in the step of making the call, even if it is determined that the attention behavior has been performed, if the attribute or size satisfies a first predetermined condition, the call is not made or the content or manner of the call is changed.

7. A method executed by a computer that stores a displacement of a feature point of a behavior object when the behavior object performs a target behavior, comprising: A step of acquiring time-series images captured by an imaging means; detecting feature points of a target action object captured in the time-series images; determining whether or not the attention behavior has been performed based on the displacement of the stored feature points and the displacement of the feature points detected from each time-series image; estimating an attribute or a size of the target action based on the detected feature points or an appearance of the target action; a step of providing a voice message corresponding to a type of the attention behavior when the attention behavior is determined to have been performed, the voice message being provided around a shooting range of the shooting means; Equipped with A method for calling for attention behavior, characterized in that in the step of making the call, even if it is determined that the attention behavior has been performed, if the attribute or size satisfies a first predetermined condition, the call is not made or the content or manner of the call is changed.

Images