Dead angle determination device, method, program, action evaluation device, and system

Abstract

To accurately determine existence of a dead angle of a camera.SOLUTION: State detection information (D[t]) on an object is derived on the basis of an input picture (In[t]) from a camera for imaging the object. Current action information (A[t]) is derived by recognizing a current action of the object on the basis of the state detection information (D[t]) and prediction action information (FA[t+1]) is derived by predicting a future action of the object. An error (ALoss[t]) between the current action information (A[t]) at an object time and prediction action information (FA[t]) that is information derived by prediction before the object time and indicates a result of prediction of action of the object at the object time is derived. On the basis of the error (ALoss[t]) and a confidence value (CS[t]) indicating accuracy of the state detection information at the object time, existence of a dead angle of the camera in imaging the object is determined.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to a blind spot determination device, method, and program, and a behavior evaluation device and system. [Background technology]

[0002] There are devices equipped with algorithms or AI (artificial intelligence) that detect a person's state (position, posture, etc.) based on images captured by a camera and recognize (understand) and predict the person's behavior from the detection results. For example, when a camera captures a factory worker's work, the algorithm or AI is used to evaluate the worker's work safety or risk. Or, when a camera captures a vehicle driver's work, the algorithm or AI is used to predict whether the driver will fall asleep at the wheel.

[0003] However, an obstacle (any object or person) may come between the camera and the target (such as a worker or driver), and in this case, a blind spot occurs when the camera captures the target. The occurrence of a blind spot can lead to a decrease in detection accuracy in behavior prediction and malfunction. Patent Document 1 listed below discloses a technology for assisting aircraft piloting in situations where visibility is obstructed. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-55024 Summary of the Invention [Problem to be solved by the invention]

[0005] It is possible to ensure the reliability of the system by taking measures such as masking behavior predictions when blind spots occur. However, to achieve this, technology to detect the presence or absence of blind spots is required. There is a need to develop technology that can accurately determine the presence or absence of blind spots.

[0006] Furthermore, if the position and shape of an obstacle that causes a blind spot are known, it is possible to identify the area blocked by the obstacle within the camera's field of view using geometric calculations. Then, by determining whether an object (such as a worker or driver) is located within the blocked area, it is possible to detect whether a blind spot exists in the image of the object. However, this detection method (hereinafter referred to as the "reference method") requires that information such as the position and shape of the obstacle relative to the camera's installation position and orientation be input into the system or detected by the system. Therefore, when applied to products that require versatility, there are issues with the work involved, such as setup, adjustment, and information input, making the reference method less practical. Furthermore, the reference method is less practical because it is difficult to deal with cases where the position of the obstacle may fluctuate.

[0007] An object of the present invention is to provide a technology that can accurately determine whether or not there is a blind spot for a camera. [Means for solving the problem]

[0008] A blind spot determination device according to the present invention derives state detection information by detecting the state of an object based on an input image from a camera that captures the object. The blind spot determination device derives current behavior information by recognizing the current behavior of the object based on the state detection information, and derives predicted behavior information by predicting the future behavior of the object based on the state detection information. The blind spot determination device derives an error between the current behavior information at a target time and predicted behavior information, which is information derived in a prediction before the target time and indicates a predicted result of the object's behavior at the target time. The blind spot determination device determines whether there is a blind spot for the camera when capturing the object based on the error and a reliability value indicating the accuracy of the state detection information at the target time. [Effects of the Invention]

[0009] Because the state of an object is detected based on an input image from a camera capturing the object, if the camera has a blind spot when capturing the object, the reliability of the state detection information decreases compared to when there is no blind spot, and this decrease in reliability is reflected in the reliability value. However, even if a blind spot exists, if the size of the blind spot is small, the reliability is unlikely to decrease. Therefore, the presence or absence of a blind spot may not be accurately determined based solely on the reliability value indicating the accuracy of the state detection information. On the other hand, if a blind spot occurs at a target time, a discrepancy occurs between the current behavior information and the predicted behavior information derived based on the state detection information of the object, increasing the above-mentioned error. Taking these factors into consideration, the presence or absence of a blind spot is determined based on both the reliability value and the error. This allows the presence or absence of a blind spot to be determined with high accuracy. [Brief explanation of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating the overall configuration of a system according to an embodiment of the present invention. [Figure 2] 1 is an internal block diagram of a behavior evaluation device according to an embodiment of the present invention. [Figure 3] 1 is a diagram illustrating the relationship between a camera and a subject (subject) photographed by the camera according to an embodiment of the present invention. [Figure 4] 1 is a diagram showing the relationship between a camera, a subject (subject person), and an obstacle, as well as a blind spot area, according to an embodiment of the present invention. [Figure 5] 1 is a functional block diagram of a controller in a behavior evaluation device according to a first example of an embodiment of the present invention. FIG. [Figure 6] 3 is an operational flowchart of the behavior evaluation device according to the first example belonging to the embodiment of the present invention. [Figure 7] FIG. 2 is a diagram showing an example of notification content in a notification device according to a first example of an embodiment of the present invention. [Figure 8] FIG. 10 is a partial functional block diagram of a controller in a behavior evaluation device according to a second example of an embodiment of the present invention. [Figure 9] FIG. 10 is a diagram showing time series changes in a blind spot determination value and a comparison result signal according to a second example belonging to an embodiment of the present invention. [Figure 10] FIG. 10 is a diagram illustrating a state in which the position of a subject periodically changes according to a third example of an embodiment of the present invention. [Figure 11] FIG. 10 is a partial functional block diagram of a controller in a behavior evaluation device according to a third example of an embodiment of the present invention. [Figure 12] FIG. 10 is a diagram showing time series changes in a blind spot determination value and a comparison result signal according to a third example belonging to an embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, examples of embodiments of the present invention will be described in detail with reference to the drawings. In each of the drawings, the same parts are designated by the same reference numerals, and duplicate descriptions of the same parts will be omitted as a general rule. In this specification, for the sake of simplicity, symbols or signs referring to information, signals, physical quantities, functional units, circuits, elements, or components may be used, and the names of the information, signals, physical quantities, functional units, circuits, elements, or components corresponding to the symbols or signs may be omitted or abbreviated.

[0012] FIG. 1 shows the overall configuration of the system SYS according to this embodiment. The system SYS has a function of evaluating the behavior of the subject TG. When focusing on this function, the system SYS can be called a behavior evaluation system. The system SYS comprises a behavior evaluation device 10, an evaluation utilization device 20, and a camera CM, which are the main components of this function. The camera CM and the behavior evaluation device 10 are connected to each other by wire or wirelessly. The behavior evaluation device 10 and the evaluation utilization device 20 are connected to each other by wire or wirelessly.

[0013] The camera CM photographs the subject TG. The image obtained by photographing the subject TG is referred to as the input image IN. All or part of the subject TG's body is located within the photographing area of the camera CM (i.e., within the field of view of the camera CM), and therefore the image of the subject TG is included in the input image IN. The image information representing the input image IN is referred to as the image data of the input image IN. The input image IN is supplied to the behavior evaluation device 10. That is, the image data of the input image IN is supplied to the behavior evaluation device 10. In this specification, the acquisition, supply, and recording of the input image IN specifically refer to the acquisition, supply, and recording of the image data of the input image IN, and the processing or operation based on the input image IN specifically refer to the processing or operation based on the image data of the input image IN. The same applies to other similar expressions, and the same interpretation applies to images other than the input image IN.

[0014] The camera CM photographs the subject TG repeatedly at a predetermined frame rate. Therefore, the camera CM sequentially acquires input images IN at the predetermined frame rate, and the image data of the sequentially acquired input images IN are sequentially supplied to the behavior evaluation device 10.

[0015] The behavior evaluation device 10 is any device capable of performing calculations. The behavior evaluation device 10 may be a general-purpose computer device, and a portable information terminal such as a smartphone may also serve as the behavior evaluation device 10. When the behavior evaluation device 10 is installed in a vehicle such as an automobile, the behavior evaluation device 10 may be an on-board device belonging to an ECU (Electronic Control Unit), or may be a device incorporated into a drive recorder, car navigation system, or the like. The behavior evaluation device 10 evaluates the behavior of the subject TG based on the input image IN, and generates behavior evaluation information EV as information indicating the evaluation result of the behavior of the subject TG. The behavior evaluation device 10 can output (transmit) the behavior evaluation information EV to the evaluation utilization device 20.

[0016] The evaluation utilization device 20 is any device that uses the behavioral evaluation information EV. The evaluation utilization device 20 also includes a notification device 21 that can notify the subject TG or other persons of any information. Although the behavioral evaluation device 10 and the evaluation utilization device 20 are shown as separate devices in Fig. 1, the behavioral evaluation device 10 and the evaluation utilization device 20 may be included in a single electronic device.

[0017] 2 shows a schematic internal configuration diagram of the behavior evaluation device 10. The behavior evaluation device 10 includes a controller 11, a memory 12, and a communication unit 13.

[0018] The controller 11 comprehensively controls the operation of each part of the behavior evaluation device 10. The controller 11 includes a processing unit including a CPU (Central Processing Unit) and a GPU (Graphics Processing Unit) as hardware resources. The controller 11 may realize any of the functions to be realized by the controller 11 by executing a program recorded in the memory 12 or any other recording medium (not shown).

[0019] The memory 12 is configured to include a non-volatile memory such as a ROM (Read Only Memory) or a flash memory, and a volatile memory such as a RAM (Random Access Memory). The memory 12 stores various data referenced by the controller 11 as well as various programs to be executed by the controller 11.

[0020] The communication unit 13 transmits and receives any signal between the behavior evaluation device 10 and a different counterpart device. The counterpart device for the communication unit 13 includes the camera CM and the evaluation utilization device 20, as well as any computer device connected to a communication network including the Internet and an intranet. Note that the controller 11 can transmit and receive any information to and from the counterpart device using the communication unit 13, but hereinafter, the description of the communication unit 13 may be omitted.

[0021] In this embodiment, it is primarily assumed that the subject TG is located in a factory that manufactures or ships a given product, and that the subject TG is a worker (a production line worker) at the factory, as shown in FIG. 3. The camera CM is installed in a position that allows it to capture the subject TG working. The work refers to the actions that the subject TG should perform in the factory. The behavior evaluation device 10 evaluates the behavior of the subject TG based on an input image IN that shows the subject TG working.

[0022] The evaluation content by the behavior evaluation device 10 (and therefore the behavior evaluation information EV) includes, for example, a safety level and a risk level. The safety level is an index that indicates the degree of safety of the subject TG while working. The risk level is an index that indicates the degree of danger to the subject TG while working. If the subject TG takes dangerous actions while working, the safety level in the evaluation will decrease and the risk level will increase compared to when the subject TG does not take dangerous actions. The evaluation utilization device 20 can perform risk response processing upon receiving an evaluation result that indicates a high risk level.

[0023] For example, in the risk response process, the evaluation utilization device 20 notifies the factory manager of warning information. The evaluation utilization device 20 may be a portable information terminal such as a smartphone carried by the factory manager. In this case, the warning information is displayed on the display screen of the portable information terminal serving as the evaluation utilization device 20. This display may be accompanied by vibration of the portable information terminal or audio output from the portable information terminal. Upon receiving the warning information, the factory manager can check the safety of the target person TG or provide work guidance, etc.

[0024] Furthermore, for example, in the danger response processing, the evaluation utilization device 20 may blink a warning lamp installed near the subject TG in the factory, or may output a warning sound from a speaker installed near the subject TG in the factory. The subject TG who recognizes the blinking of the warning lamp or the warning sound will recognize that there is a risk in his or her work and can improve the dangerous work.

[0025] In addition, the evaluation content by the behavior evaluation device 10 (and therefore the behavior evaluation information EV) may include various work-related details, such as work accuracy or work speed. Work accuracy is an index that indicates the degree of accuracy of the work performed by the subject TG. Work speed indicates the speed at which the subject TG performs the work. For example, if the subject TG is responsible for a task that involves repeatedly performing steps 1 to 3, the reciprocal of the repetition period of steps 1 to 3 corresponds to the work speed.

[0026] The camera CM is installed in a position that allows it to adequately capture the state of the subject TG working. However, after the camera CM is installed, as shown in FIG. 4, an obstacle 610 may enter between the camera CM and the subject TG. The obstacle 610 may be any equipment or object in the factory, or may be a person other than the subject TG. The obstacle 610 is located within the shooting area of the camera CM and in the area between the camera CM and the subject TG. Therefore, the presence of the obstacle 610 causes a blind spot for the camera CM when shooting the subject TG. Unless otherwise specified, the blind spot in the following description refers to the blind spot of the camera CM when shooting the subject TG, and refers to a situation in which part or all of the subject TG's body is blocked by the obstacle 610 when shooting with the camera CM.

[0027] In Fig. 4, the hatched area represents a blind spot area 620, which is an area where a blind spot occurs. The blind spot area 620 would be a part of the shooting area of the camera CM if there were no obstacle 610. When the obstacle 610 is present, the blind spot area 620 is blocked by the obstacle 610 as seen from the camera CM, and as a result, image data of the image within the blind spot area 620 is not included in the input image IN. In the example of Fig. 4, when the obstacle 610 is present, a part of the body of the subject TG is located in the blind spot area 620, and therefore that part of the body of the subject TG is not shot by the camera CM (image data of that part of the body of the subject TG is not included in the input image IN).

[0028] When a blind spot occurs, depending on the location and size of the blind spot, the behavior evaluation device 10 may not be able to correctly evaluate the behavior of the subject TG. The occurrence of a blind spot can reduce the reliability of the system SYS or cause it to malfunction. The behavior evaluation device 10 has a blind spot determination function that determines whether or not there is a blind spot, which contributes to improving the reliability of the system SYS.

[0029] Below, specific configuration examples, operation examples, application techniques, modified techniques, etc. related to the system SYS will be described in multiple embodiments. The matters described above in this embodiment are applied to each of the following embodiments unless otherwise specified and unless there is a contradiction. If there are any matters in each embodiment that contradict the matters described above, the description in each embodiment may take precedence. Furthermore, unless there is a contradiction, matters described in any of the multiple embodiments shown below can also be applied to any of the other embodiments (i.e., any two or more of the multiple embodiments can be combined).

[0030] <<First Example>> A first embodiment will be described. FIG. 5 is a functional block diagram of a controller 11 according to the first embodiment. The controller 11 includes a behavior evaluation block 111 and a blind spot determination block 112. The behavior evaluation block 111 is a functional block for evaluating the behavior of the target person TG. The behavior evaluation block 111 includes functional blocks F1 to F4. The blind spot determination block 112 is a functional block for determining whether or not a blind spot exists. The blind spot determination block 112 includes functional blocks F5 to F8. All or part of the functions of the functional blocks F1 to F8 may be realized by the controller 11 executing a program recorded in the memory 12 or any other recording medium (not shown). The functional blocks F1, F2, F3, and F4 are a state detection unit, a behavior recognition unit, a behavior prediction unit, and a behavior evaluation unit, respectively. The functional blocks F5, F6, F7, and F8 are a reliability identification unit, a delay unit, an error calculation unit, and a blind spot determination unit, respectively.

[0031] Image data of the input image IN is supplied from the camera CM to the state detection unit F1. The state detection unit F1 performs state detection processing. In the state detection processing, the state detection unit F1 detects the state of the subject TG based on the input image IN (more specifically, based on the image data of the input image IN), and generates and outputs state detection information D indicating the detection result of the state of the subject TG. The state detection unit F1 detects the position and posture of the subject TG as the state of the subject T. The position of the subject TG is the position of the subject TG within the factory where the subject TG works. The state detection unit F1 can detect the posture of the subject TG using a well-known posture detection AI. AI is an abbreviation for artificial intelligence. Note that with respect to any information, the generation of information and the derivation of information are synonymous and can be read interchangeably.

[0032] The state detection unit F1 sequentially generates state detection information D based on input images IN that are sequentially supplied. A plurality of input images IN arranged in time series form a moving image. An input image IN at a certain time t is specifically referred to as input image IN[t], and state detection information D generated by state detection processing based on input image IN[t] is specifically referred to as state detection information D[t]. State detection information D[t] represents the detection result of the state of the subject TG at time t.

[0033] State detection information D is input from the state detection unit F1 to the behavior recognition unit F2. The behavior recognition unit F2 performs behavior recognition processing. In the behavior recognition processing, the behavior recognition unit F2 recognizes the current behavior of the subject TG based on the state detection information D, and generates and outputs current behavior information A indicating the results of the recognition. The recognition in the behavior recognition unit F2 corresponds to understanding the current behavior of the subject TG, and therefore the behavior recognition unit and behavior recognition processing may be read as behavior understanding unit and behavior understanding processing.

[0034] The current behavior information A generated based on the state detection information D[t] is particularly referred to as current behavior information A[t]. The present with respect to the current behavior information A[t] refers to the time t. That is, the current behavior information A[t] represents the recognition result of the behavior of the subject TG at time t (recognition result by behavior recognition processing). The behavior recognition unit F2 may generate the current behavior information A[t] by referring not only to the state detection information D at time t (i.e., the state detection information D[t]), but also to the state detection information D before time t.

[0035] The state detection information D is input to the behavior prediction unit F3 from the state detection unit F1. The behavior prediction unit F3 performs a behavior prediction process. In the behavior prediction process, the behavior prediction unit F3 predicts the future behavior of the subject TG based on the state detection information D, and generates and outputs predicted behavior information FA indicating the result of the prediction. Here, the future refers to a unit time T from the present. UNIT It refers to a point in time after time t. UNIT The time after that is called time (t+1). UNIT is a time period predetermined in the behavior prediction unit F3, for example, 1 second.

[0036] The predicted behavior information FA generated based on the state detection information D[t] is particularly referred to as predicted behavior information FA[t+1]. The predicted behavior information FA[t+1] represents the prediction result of the behavior prediction process performed at time t, and is the prediction of the behavior of the subject TG at time (t+1) at time t. That is, at time t, the behavior prediction unit F3 predicts the behavior of the subject TG at time (t+1) for a unit time T UNIT The behavior prediction unit F3 generates predicted behavior information FA[t+1] by predicting the behavior of the subject TG in the future. The behavior prediction unit F3 may generate predicted behavior information FA[t+1] by referring not only to the state detection information D at time t (i.e., the state detection information D[t]) but also to the state detection information D before time t. The behavior prediction unit F3 may be formed by an existing behavior prediction AI. Note that the behavior prediction AI may also have the functions of the state detection unit F1 and the behavior recognition unit F2, in which case the state detection unit F1, the behavior recognition unit F2, and the behavior prediction unit F3 may be configured by an existing behavior prediction AI.

[0037] In the behavior recognition process, the current behavior of the subject TG is classified into one of a plurality of behavior classes, and current class information indicating the behavior class classified in the behavior recognition process is included in the current behavior information A. Similarly, in the behavior prediction process, the future behavior of the subject TG is classified into one of a plurality of behavior classes, and predicted class information indicating the behavior class classified in the behavior prediction process is included in the predicted behavior information FA. The multiple behavior classes may be defined in any manner. For example, the multiple behavior classes include first to tenth behavior classes, which respectively represent the behaviors of turning right, turning left, looking down, looking back, crouching, holding a soldering iron, holding a cutter, holding scissors, walking, and running.

[0038] In addition, coordinate information (numerical information) indicating the current position of the subject TG is also included in the current behavior information A, and similarly, coordinate information (numerical information) indicating the future position of the subject TG is also included in the predicted behavior information FA. Furthermore, when the entire body of the subject TG moves, trajectory information indicating the trajectory of the movement during a period including time t may be included in the current behavior information A[t], and trajectory information indicating the trajectory of the movement during a period including time (t+1) may be included in the predicted behavior information FA[t+1]. When a part of the body of the subject TG moves, the trajectory information in the current behavior information A[t] indicates the trajectory of the movement of that part during a period including time t, and the trajectory information in the predicted behavior information FA[t+1] indicates the trajectory of the movement of that part during a period including time (t+1). In addition, various information indicating the content of the current movement of the subject TG may be included in the current behavior information A. Similarly, various information indicating the content of the future movement of the subject TG may be included in the predicted behavior information FA.

[0039] The behavior recognition unit F2 may be an AI formed using machine learning, and depending on the method of forming the AI, the current behavior of the subject TG may be quantified in the form of a regression value and included in the current behavior information A. Similarly, the behavior prediction unit F3 may be an AI formed using machine learning, and depending on the method of forming the AI, the future behavior of the subject TG may be quantified in the form of a regression value and included in the predicted behavior information FA.

[0040] The behavior evaluation unit F4 receives current behavior information A from the behavior recognition unit F2 and predicted behavior information FA from the behavior prediction unit F3. The behavior evaluation unit F4 performs behavior evaluation processing. In the behavior evaluation processing, the behavior evaluation unit F4 generates the above-mentioned behavior evaluation information EV by evaluating the behavior of the subject TG based on the current behavior information A and the predicted behavior information FA. The behavior evaluation unit F4 can output (transmit) the behavior evaluation information EV to the evaluation utilization device 20.

[0041] Note that the generation or output of behavior evaluation information EV may be stopped depending on the processing result of the blind spot determination block 112 (details will be described later). The behavior evaluation information EV generated based on the current behavior information A[t] and the predicted behavior information FA[t+1] is specifically referred to as behavior evaluation information EV[t].

[0042] The reliability identification unit F5 performs a reliability identification process. When the condition detection unit F1 detects the condition of the subject TG, it derives a reliability value CS indicating the accuracy (degree of certainty) of its own detection, or derives information that serves as the basis for the reliability value CS. The reliability value CS can also be said to indicate the accuracy (degree of certainty) of the condition detection information D. In the reliability identification process, the reliability identification unit F5 identifies the reliability value CS. The reliability identification unit F5 identifies the reliability value CS by extracting the reliability value CS derived in the condition detection unit F1 from the condition detection unit F1. Alternatively, the reliability identification unit F5 extracts information that serves as the basis for the reliability value CS derived in the condition detection unit F1 from the condition detection unit F1, and derives and identifies the reliability value CS based on the extracted information.

[0043] The reliability value CS when generating the state detection information D[t] based on the input image IN[t] is specifically referred to as the reliability value CS[t]. The reliability value CS[t] indicates the accuracy of the detection when detecting the state of the subject TG based on the input image IN[t], in other words, the accuracy of the state detection information D[t]. The higher the accuracy of the state detection information D[t], the larger the reliability value CS[t]. The reliability value CS[t] has a value between 0 and 1, for example.

[0044] The state detection unit F1 is an inference model obtained by training a learning model consisting of a DNN (Deep Neural Network) using supervised machine learning. The inference model generates state detection information D by inferring (i.e., detecting) the state of the subject TG based on the input image IN. A confidence score is defined during inference in the inference model. This confidence score is used as the above-mentioned confidence value CS. In other words, the confidence score when generating state detection information D[t] from input image IN[t] is the confidence value CS[t]. Learning of the learning model proceeds with the elements necessary for calculating the confidence score set in the terms of the loss function. Therefore, during inference, the inference model can calculate the confidence score or the information that serves as the basis for the confidence score.

[0045] The delay unit F6 delays the predicted behavior information FA output from the behavior prediction unit F3 for a unit time T UNIT The delay unit F6 can be configured with a delay memory. As can be understood from the above explanation, the difference between time t and time (t+1) is the unit time T UNIT Also, from time t to unit time T UNIT At time (t-1), the behavior prediction unit F3 calculates the time T UNIT The predicted behavior information FA[t] is generated by predicting the behavior of the subject TG in the future. The predicted behavior information FA[t] represents the prediction result of the behavior prediction process performed at time (t-1), and is the behavior of the subject TG at time t predicted at time (t-1). The predicted behavior information FA[t] output from the behavior prediction unit F3 at time (t-1) is output from the delay unit F6 to the error calculation unit F7 at time t.

[0046] The error calculation unit F7 receives the current behavior information A from the behavior recognition unit F2 and the predicted behavior information FA from the delay unit F6. The error calculation unit F7 performs an error calculation process to calculate the error ALoss between the input current behavior information A and the predicted behavior information FA. At time t, the current behavior information A[t] is input to the error calculation unit F7 from the behavior recognition unit F2, and the predicted behavior information FA[t] is input to the error calculation unit F7 from the delay unit F6. Therefore, at time t, the error calculation unit F7 calculates the error ALoss[t], which is the error ALoss between the current behavior information A[t] and the predicted behavior information FA[t]. The current behavior information A[t], which is the source of the error ALoss[t], is the current behavior information A at time t. The predicted behavior information FA[t], which is the source of the error ALoss[t], is information derived by prediction in the behavior prediction process at time (t-1) and indicates the predicted result of the behavior of the subject TG at time t.

[0047] In the error calculation process, the error calculation unit F7 compares the behavior of the subject TG indicated in the current behavior information A[t] (hereinafter referred to as the first comparison behavior) with the behavior of the subject TG indicated in the predicted behavior information FA[t] (hereinafter referred to as the second comparison behavior). The first comparison behavior is the behavior of the subject TG at time t recognized by the behavior recognition unit F2 at time t. The second comparison behavior is the behavior of the subject TG at time t predicted by the behavior prediction unit F3 at time (t-1). When the first contrast behavior and the second contrast behavior are perfectly consistent, the error ALoss[t] is zero. When the first contrast behavior and the second contrast behavior are different, ALoss[t]>0, and the error ALoss[t] increases as the degree of difference between the first contrast behavior and the second contrast behavior increases.

[0048] A loss function L having the current behavior information A[t] and the predicted behavior information FA[t] as variables may be set in the error calculation unit F7. Then, the value of the loss function L obtained by substituting the current behavior information A[t] and the predicted behavior information FA[t] into the loss function L can be calculated as the error ALoss[t]. The current behavior information A[t] and the predicted behavior information FA[t] may be regarded as first and second vectors in a multidimensional vector space, and the error ALoss[t] may be calculated based on the distance between the first and second vectors in the multidimensional vector space. The current behavior information A[t] and the predicted behavior information FA[t] may be scalar quantities. In this case, the absolute value of the difference between the current behavior information A[t] and the predicted behavior information FA[t] is the error ALoss[t].

[0049] The reliability value CS from the reliability specification unit F5 and the error ALoss from the error calculation unit F7 are input to the blind spot determination unit F8. The blind spot determination unit F8 performs blind spot determination processing to determine whether or not there is a blind spot based on the input reliability value CS and error ALoss. In the blind spot determination processing, the blind spot determination unit F8 generates blind spot determination information BS that indicates the determination result of whether or not there is a blind spot. The blind spot determination information BS has a value of "0" or "1". Blind spot determination information BS of "0" indicates that there is no blind spot, and blind spot determination information BS of "1" indicates that there is a blind spot.

[0050] The blind spot determination information BS based on the reliability value CS[t] and the error ALoss[t] is specifically referred to as blind spot determination information BS[t]. The blind spot determination information BS[t] indicates whether there was a blind spot at the camera CM when photographing the subject TG at time t (i.e., when the input image IN[t] was acquired by the camera CM). Therefore, blind spot determination information BS[t] of "0" indicates that it was determined that there was no blind spot at the camera CM when photographing the subject TG at time t. However, blind spot determination information BS[t] of "0" may be derived even if there were actually some blind spots when photographing the subject TG at time t. Blind spot determination information BS[t] of "1" indicates that it was determined that there was a blind spot at the camera CM when photographing the subject TG at time t.

[0051] The dead angle determination unit F8 determines the presence or absence of a dead angle by comparing a dead angle determination value Q corresponding to the reliability value CS and the error ALoss with a determination threshold TH. The dead angle determination value Q corresponding to the reliability value CS[t] and the error ALoss[t] is particularly referred to as the dead angle determination value Q[t]. The dead angle determination unit F8 can derive a dead angle determination value Q1 or Q2 as the dead angle determination value Q. The dead angle determination value Q1 corresponding to the reliability value CS[t] and the error ALoss[t] is particularly referred to as the dead angle determination value Q1[t], and the dead angle determination value Q2 corresponding to the reliability value CS[t] and the error ALoss[t] is particularly referred to as the dead angle determination value Q2[t].

[0052] The dead angle determination value Q1 increases as the reliability value CS increases and decreases as the error ALoss increases. Specifically, focusing on the time t, the dead angle determination value Q1[t] increases as the reliability value CS[t] increases and decreases as the error ALoss[t] increases. The dead angle determination value Q1[t] follows, for example, Equation (1A). At this time, the dead angle determination unit F8 determines the validity of Equation (1B). Q1[t]=CS[t]-γ·ALoss[t] ···(1A) CS[t]-γ·ALoss[t]<TH ···(1B)

[0053] When "Q[t]=Q1[t]", when "Q1[t]<TH" holds, the dead angle determination unit F8 determines that there is a dead angle in the camera CM in the shooting of the subject TG at time t, and sets "1" in the dead angle determination information BS[t]. Conversely, when "Q1[t]>TH" holds, the dead angle determination unit F8 determines that there is no dead angle in the camera CM in the shooting of the subject TG at time t, and sets "0" in the dead angle determination information BS[t]. When "Q1[t]=TH" holds, the dead angle determination unit F8 sets "1" or "0" in the dead angle determination information BS[t].

[0054] The dead angle determination value Q2 decreases as the reliability value CS increases and increases as the error ALoss increases. Specifically described by focusing on time t, the dead angle determination value Q2[t] decreases as the reliability value CS[t] increases and increases as the error ALoss[t] increases. The dead angle determination value Q2[t] follows, for example, Equation (2A), and at this time, the dead angle determination unit F8 determines the validity of Equation (2B). Q2[t]=γ·ALoss[t]-CS[t] ···(2A) γ·ALoss[t]-CS[t]>TH ···(2B)

[0055] When “Q[t]=Q2[t]”, when “Q2[t]>TH” holds, the dead angle determination unit F8 determines that there is a dead angle in the camera CM in the shooting of the subject TG at time t, and sets “1” in the dead angle determination information BS[t]. Conversely, when “Q2[t]<TH” holds, the dead angle determination unit F8 determines that there is no dead angle in the camera CM in the shooting of the subject TG at time t, and sets “0” in the dead angle determination information BS[t]. The dead angle determination unit F8 sets “1” or “0” in the dead angle determination information BS[t] when “Q2[t]=TH” holds.

[0056] γ and TH are preset coefficients and determination thresholds. Both the coefficient γ and the determination threshold TH have positive values. The coefficient γ and the determination threshold TH can be set through experiments or the like in the calibration process so that the presence or absence of a dead angle can be correctly classified. After going through the calibration process, it reaches the actual operation process, and only in the actual operation process can significant action evaluation information EV be obtained.

[0057] In the calibration process, after the installation of the camera CM, under the first calibration environment, the controller 11 is caused to derive the reliability value CS[t A and the error ALoss[t A , and under the second calibration environment, the controller 11 is caused to derive the reliability value CS[t B and the error ALoss[t B . t A represents an arbitrary time under the first calibration environment within the execution period of the calibration process. t Brepresents an arbitrary time under the second calibration environment during the execution period of the calibration process.

[0058] The first calibration environment is an environment in which an obstacle 610 is placed between the subject and the camera CM. In the first calibration environment, there is a blind spot for the camera CM when shooting the subject, and part or all of the subject's body is blocked by the obstacle 610 when shooting with the camera CM. The second calibration environment is an environment in which there is no obstacle 610 placed between the subject and the camera CM. In the second calibration environment, there is no blind spot for the camera CM when shooting the subject, and the entire subject's body is within the shooting area of the camera CM when shooting with the camera CM. The subject may be the subject TG, or may be a person other than the subject TG.

[0059] In the calibration process, the reliability value CS[t A ] and error ALoss[t A ] based on the blind spot judgment value Q1[t A ] and the confidence value CS[t B ] and error ALoss[t B ] based on the blind spot judgment value Q1[t B ] is set to the determination threshold TH. In practice, the position of the obstacle 610 or the position of the subject is varied in various ways in each calibration environment to obtain a plurality of blind spot determination values Q1[t A ] and multiple blind spot judgment values Q1[t B ] is derived. Then, the derived multiple blind spot judgment values Q1[t A ] and multiple blind spot judgment values Q1[t B ] can be statistically processed to set the coefficient γ and the decision threshold TH. Alternatively, in the calibration process, the reliability value CS[t A ] and error ALoss[t A ] based on the blind spot judgment value Q2[t A ] and the confidence value CS[t B ] and error ALoss[t B ] based on the blind spot judgment value Q2[t B] is set to the judgment threshold TH. In practice, the position of the obstacle 610 or the position of the subject is changed in various ways in each calibration environment to obtain a plurality of blind spot judgment values Q2[t A ] and multiple blind spot judgment values Q2[t B ] is derived. Then, the derived multiple blind spot judgment values Q2[t A ] and multiple blind spot judgment values Q2[t B ] can be statistically processed to set the coefficient γ and the decision threshold TH.

[0060] The behavior evaluation unit F4 performs behavior evaluation processing as a rule during the period when the blind spot determination information BS has a value of "0," generates behavior evaluation information EV, and outputs the behavior evaluation information EV to the evaluation utilization device 20. During the period when the blind spot determination information BS has a value of "1," the behavior evaluation unit F4 stops the behavior evaluation processing, and in this case, behavior evaluation information EV is not generated. Alternatively, during the period when the blind spot determination information BS has a value of "1," the behavior evaluation unit F4 may perform behavior evaluation processing, but stops outputting the behavior evaluation information EV to the evaluation utilization device 20. That is, with respect to time t, when the blind spot determination information BS[t] has a value of "0," the behavior evaluation unit F4 generates behavior evaluation information EV[t] and outputs it to the evaluation utilization device 20. When the blind spot determination information BS[t] has a value of "1," the behavior evaluation unit F4 stops generating the behavior evaluation information EV[t]. If the generation of the behavior evaluation information EV[t] is stopped, the output of the behavior evaluation information EV[t] to the evaluation utilization device 20 is naturally stopped. Alternatively, when the blind spot judgment information BS[t] has a value of "1", the behavior evaluation unit F4 generates the behavior evaluation information EV[t] but stops (in other words, prohibits) outputting the behavior evaluation information EV[t] to the evaluation utilization device 20.

[0061] The time t is the unit time T UNITAlthough the time period is sufficiently shorter than the time period of the time (t-1), it is understood to be a concept having a certain time length. The same applies to time (t-1) and time (t+1), etc. Strictly speaking, after an input image IN[t] is obtained by shooting with a camera CM at a first time, state detection information D[t] is generated by a state detection process at a second time. Thereafter, current behavior information A[t] is generated at a third time based on the state detection information D[t], and predicted behavior information FA[t+1] is generated at a fourth time (the third and fourth times may be in any order or may be the same time). Further, at a fifth time, under the assumption that "BS[t]=0", behavior evaluation information EV[t] is generated and output. The reliability value CS[t], error ALoss[t], and blind spot determination information BS[t] are derived after the second time and before the fifth time. The time difference between the first and fifth times is the unit time T UNIT and the first to fifth times are all understood to belong to time t.

[0062] FIG. 6 shows an operational flowchart of the behavior evaluation device 10. For the sake of specificity and clarity of explanation, FIG. 6 shows the flow of operation of the behavior evaluation device 10 at time t. The operation of the behavior evaluation device 10 at time t is composed of the processing of steps S11 to S20. In practice, a series of processing made up of steps S11 to S20 is executed every time image data of the input image IN is supplied to the controller 11, and this series of processing is executed repeatedly. The flow of operation of the behavior evaluation device 10 at time t will be described with reference to FIG. 6.

[0063] First, in step S11, image data of an input image IN[t] is supplied from the camera CM to the controller 11, and the image data of the input image IN[t] is acquired by the state detection unit F1. In the following step S12, the state detection unit F1 derives state detection information D[t] based on the image data of the input image IN[t]. In the following step S13, the behavior recognition unit F2 derives current behavior information A[t] based on the state detection information D[t], and the behavior prediction unit F3 derives predicted behavior information FA[t+1] based on the state detection information D[t]. Furthermore, in step S14 after step S12, the reliability identification unit F5 identifies a reliability value CS[t] of the state detection information D[t]. The order of the processes of steps S13 and S14 is arbitrary, and these processes may be performed simultaneously.

[0064] After the current behavior information A[t] is derived, the process of step S15 is executed. In step S15, the error calculation unit F7 derives the error ALoss[t] between the current behavior information A[t] and the predicted behavior information FA[t] obtained in the past prediction. The predicted behavior information FA[t] obtained in the past prediction is supplied from the delay unit F6 to the error calculation unit F7. The order of the processes of steps S14 and S15 is arbitrary, and these processes may be performed simultaneously. However, the process of step S15 is performed at least after the current behavior information A[t] is derived. After the processes of steps S11 and S12 and steps S13 to S15 are performed, the process proceeds to step S16.

[0065] In step S16, the blind spot determination unit F8 determines whether or not there is a blind spot based on the reliability value CS[t] and the error ALoss[t], and derives blind spot determination information BS[t] indicating the determination result. After step S16, in step S17, the behavior evaluation unit F4 checks whether the value of the blind spot determination information BS[t] is "0". If the value of the blind spot determination information BS[t] is "0" (Y in step S17), the process proceeds to step S18, and if the value of the blind spot determination information BS[t] is "1" (N in step S17), the process proceeds to step S19.

[0066] In step S18, the behavior evaluation unit F4 generates behavior evaluation information EV[t] by evaluating the behavior of the subject TG based on the current behavior information A[t] and the predicted behavior information FA[t+1], and outputs it to the evaluation utilization device 20. When proceeding to step S18, the operation of the behavior evaluation device 10 at time t ends upon completion of the processing of step S18.

[0067] In step S19, the behavior evaluation unit F4 stops generating or outputting the behavior evaluation information EV[t]. If the process proceeds to step S19, the processing of step S18 is not executed. In step S20 following step S19, the controller 11 (e.g., the behavior evaluation unit F4) outputs a blind spot detection signal to the evaluation utilization device 20. If the process proceeds to step S19, the operation of the behavior evaluation device 10 at time t ends upon completion of the processing of step S20.

[0068] The blind spot detection signal related to step S20 will now be described. As described above, the evaluation utilization device 20 includes a notification device 21 capable of notifying the subject TG or another person of any information. The notification device 21 is connected to the behavior evaluation device 10 via a wired or wireless connection, and any signal can be sent from the behavior evaluation device 10 to the notification device 21. The evaluation utilization device 20 itself may be considered to be the notification device 21. The notification device 21 may also be a device provided in the system SYS separately from the evaluation utilization device 20. In step S20, the controller 11 outputs (transmits) a blind spot notification signal to the notification device 21. The notification device 21, having received the blind spot notification signal, notifies the subject TG or another person of alert information indicating the presence of a blind spot. In other words, the blind spot notification signal is a signal for causing the notification device 21 to notify the subject TG or another person of alert information indicating the presence of a blind spot.

[0069] FIG. 7(a) shows an information terminal 210 as an example of the notification device 21. The information terminal 210 is a portable information terminal (such as a smartphone) carried by a factory manager who is different from the target person TG. The information terminal 210 has a display screen and a speaker. When the information terminal 210 receives the blind spot notification signal, as shown in FIG. 7(a), the information terminal 210 displays alert information 211 indicating the existence of a blind spot on its display screen and outputs the alert information 211 as audio from its speaker. However, the information terminal 210 may only display or output the alert information 211 as audio. The alert information 211 includes, for example, a message such as "A blind spot has been detected by the camera in area AA" (here, area AA represents an area in the factory where the camera CM is installed). The alert information 211 is notified to the factory manager, who can then take appropriate action, such as eliminating the cause of the blind spot. The information terminal 210 may be a portable information terminal (smartphone or the like) carried by the target person TG, and in this case, the alert information 211 is notified to the target person TG.

[0070] FIG. 7(b) shows an information terminal 220 as an example of the notification device 21. The information terminal 220 is installed in a factory in an area where the target person TG works. The information terminal 220 has a display screen and a speaker. When the information terminal 220 receives the blind spot notification signal, as shown in FIG. 7(b), the information terminal 220 displays alert information 221 indicating the existence of a blind spot on its display screen and outputs the alert information 221 as audio from its speaker. However, the information terminal 220 may only display or output the alert information 221 as audio. The alert information 221 may include, for example, a message such as "A blind spot has occurred in the camera. Please remove any obstacles." The alert information 221 is notified to the target person TG or a person located near the information terminal 220, and upon receiving the notification, the target person TG or the like can take action such as removing the cause of the blind spot.

[0071] The technology described above in the first embodiment will be considered. Because the state of the subject TG is detected based on the input image IN[t] from the camera CM capturing the subject TG, if there is a blind spot in the camera CM capturing the subject TG, the reliability of the state detection information D[t] decreases compared to when there is no blind spot. A decrease in the reliability of the state detection information D[t] corresponds to a decrease in the reliability value CS[t]. However, even if there is a blind spot, if the size of the blind spot is small, the reliability is unlikely to decrease (the reliability value CS[t] is unlikely to decrease). This is because, when blind spots are considered noise, if the size of the blind spot is small, the signal-to-noise ratio of the input image IN[t] in the state detection process does not change significantly compared to when there is no blind spot. Therefore, the presence or absence of a blind spot may not be accurately determined based on the reliability value CS[t] alone. On the other hand, if a blind spot occurs at time t, a discrepancy occurs between the current behavior information A[t] and the predicted behavior information FA[t], increasing the error ALoss[t] between them. Taking these into consideration, the controller 11 determines whether or not there is a blind spot based on both the reliability value CS[t] and the error ALoss[t], thereby enabling the presence or absence of a blind spot to be determined with high accuracy.

[0072] Furthermore, unlike the above-mentioned reference method, the presence or absence of a blind spot can be determined without requiring information such as the position and shape of an obstacle that may cause a blind spot, making it highly versatile. In addition, the configuration of FIG. 5 can be realized simply by adding a blind spot determination function to an existing behavior evaluation AI (corresponding to the behavior evaluation block 111 in FIG. 5) that evaluates the behavior of the subject TG from an input image. Therefore, no additional special device is required to realize the technology of the present disclosure, which is also beneficial in terms of miniaturization and cost reduction of the device.

[0073] Specifically, as described above, the presence or absence of a blind spot is determined by comparing the blind spot determination value Q[t] corresponding to the reliability value CS[t] and the error ALoss[t] with the determination threshold TH. The presence or absence of a blind spot can be determined with high accuracy by using the blind spot determination value Q[t] determined depending on both the reliability value CS[t] and the error ALoss[t]. Whether the blind spot determination value Q[t] is the blind spot determination value Q1[t] or Q2[t] (see equations (1A) and (2A)), the presence or absence of a blind spot can be determined with high accuracy.

[0074] It is difficult to ensure the accuracy of the behavior evaluation information EV when there is a blind spot. For this reason, when it is determined that there is a blind spot, the generation or output of the behavior evaluation information EV is stopped (step S19). This ensures the reliability of the behavior evaluation device 10.

[0075] Furthermore, when it is determined that a blind spot exists, a blind spot detection signal is output to the notification utilization device 20 (notification device 21) (step S20), so that the occurrence of a blind spot can be notified to necessary persons (target person TG, administrator, etc.). Upon receiving this notification, it becomes possible to take appropriate measures, such as removing the cause of the blind spot.

[0076] <<Second Example>> A second embodiment will now be described. FIG. 8 shows a blind spot determination unit F20 according to the second embodiment. In the second embodiment, the blind spot determination unit F20 in FIG. 8 is used as the blind spot determination unit F8 in FIG. 5. Except for the fact that the blind spot determination unit F20 in FIG. 8 is used as the blind spot determination unit F8 in FIG. 5 and for the following points in the second embodiment, the configurations and operations of the behavior evaluation device 10 and the system SYS are the same as those of the first embodiment. However, in the second embodiment, the blind spot determination value Q1 is used as the blind spot determination value Q, and the blind spot determination value Q1 at time t (i.e., Q1[t]) is expressed, for example, by the above formula (1A). The blind spot determination unit F20 includes a calculation unit F21, an instantaneous comparison unit F22, and a threshold correction unit F23.

[0077] An error ALoss and a confidence value CS are supplied to the calculation unit F21. The calculation unit F21 derives a dead angle determination value Q1 based on the error ALoss and the confidence value CS. At time t, the dead angle determination value Q1[t] is derived according to Equation (1A). In the controller 11, every time the image data of the latest input image IN based on the latest shooting result by the camera CM is supplied to the state detection unit F1 (see FIG. 5), the error ALoss and the confidence value CS are derived based on the image data of the latest input image IN. That is, the error ALoss and the confidence value CS are updated based on the image data of the latest input image IN. Every time the error ALoss and the confidence value CS are updated in the calculation unit F21, the dead angle determination value Q1 is updated based on the latest error ALoss and the confidence value CS. A signal having the dead angle determination value Q1 as a signal value is called a dead angle determination signal S Q1 as. FIG. 9(a) shows an example of the waveform of the dead angle determination signal S Q1 . The signal value of the dead angle determination signal S Q1 is the dead angle determination value Q1[t] at time t.

[0078] Every time the dead angle determination value Q1 is updated, the instantaneous comparison unit F22 compares the latest dead angle determination value Q1 with the determination threshold TH, and generates and outputs a comparison result signal H1 indicating the comparison result. The comparison result signal H1 is a binary signal having a value of "0" or "1". The instantaneous comparison unit F22 sets a value of "1" in the comparison result signal H1 when "Q1 < TH" holds, and sets a value of "0" in the comparison result signal H1 when "Q1 > TH" holds. Therefore, at time t, "H1 = 1" when Equation (1B) holds. When "Q1 = TH" holds, a value of "1" or "0" is set in the comparison result signal H1. In the second embodiment, the comparison result signal H1 functions as dead angle determination information BS and is sent to the behavior evaluation unit F4.

[0079] The threshold correction unit F23 has a periodicity determination unit F24. The periodicity determination unit F24 determines whether the dead angle determination signal S Q1 has periodicity by statistically processing the sequentially updated dead angle determination values Q1. The periodicity determination unit F23 obtains the autocorrelation of the dead angle determination signal S Q1 to obtain the dead angle determination signal S Q1can determine whether it has periodicity, and the blind spot determination signal S Q1 When it is determined to have periodicity, the blind spot determination signal S Q1 The period can be obtained. For any signal of interest, since the method of obtaining the autocorrelation of the signal of interest is well-known, a detailed description will be omitted.

[0080] The threshold correction unit F23 has a function of correcting the determination threshold TH. However, when the blind spot determination signal S Q1 is not determined to have periodicity, the correction of the determination threshold TH is not performed, and in this case, the operation of the blind spot determination unit F20 is the same as that of the blind spot determination unit F8 in the first embodiment.

[0081] The blind spot determination signal S Q1 is determined to have periodicity and the blind spot determination signal S Q1 The period of is the period T CYCL1 The case obtained as such is referred to as case CS2. Case CS2 is further subdivided into case CS2a and CS2b.

[0082] In case CS2a, as shown in Fig. 9(a), the periods when "Q1 > TH" holds and when "Q1 < TH" holds alternate, and the repetition period of the period when "Q1 < TH" holds is the period T CYCL1 This is the case. That is, in case CS2a, the blind spot determination value Q1 periodically falls below the determination threshold TH. Among cases CS2, the case that does not correspond to case CS2a is case CS2b. For example, the case where the blind spot determination signal S Q1 has periodicity but "Q1 > TH" always holds corresponds to case CS2b. Also, for example, the case where the periods when "Q1 > TH" holds and when "Q1 < TH" holds alternate but there is no periodicity in the occurrence of "Q1 < TH" also corresponds to case CS2b. In case CS2b, the correction of the determination threshold TH is not performed, and as a result, the operation of the blind spot determination unit F20 in case CS2b is the same as that of the blind spot determination unit F8 in the first embodiment.

[0083] The threshold correction unit F23 determines whether the blind spot determination value Q1 periodically falls below the determination threshold TH. When it is determined that the blind spot determination value Q1 periodically falls below the determination threshold TH, the threshold correction unit F23 increases the determination threshold TH. Referring to FIG. 9(b), now, at time t X The blind spot judgment value Q1 is not judged to periodically fall below the judgment threshold value TH until immediately before the reference value TH. REF The reference value TH REF is a value preset in the blind spot determination unit F20. X Assume that the threshold correction unit F23 determines that the blind spot determination value Q1 periodically falls below the determination threshold TH at time t X In this case, the threshold correction unit F23 corrects the judgment threshold TH to the reference value TH REF to correction value TH HIGH Here, the “TH HIGH >TH REF " is true. For example, "TH HIGH =TH REF +ΔTH” and “ΔTH>0”, or “TH HIGH =TH REF ×(1+k A )” and “k A >0". In this way, the threshold correction unit F23 X When it is determined that the blind spot determination value Q1 periodically falls below the determination threshold TH during the period immediately preceding the reference period PR1 (hereinafter referred to as the reference period PR1), the determination threshold TH after the reference period PR1 is increased to a value higher than that during the reference period PR1.

[0084] When a subject TG repeats the same work at regular intervals, blind spots may occur at regular intervals. For example, as shown in FIG. 10, assume that a series of steps are performed repeatedly in a factory, where the subject TG performs the first step at position 631, then the second step at position 632, and then the third step at position 633. In addition, in this case, due to the relationship between the camera CM, the obstacle 610, and the positions 631 to 633, it is assumed that a blind spot occurs only in the third step among the first to third steps. In this case, the blind spot judgment value Q1 is relatively high during the period when the subject TG performs the first and second steps at positions 631 and 632, and the blind spot judgment value Q1 is relatively low during the period when the subject TG performs the third step at position 633. In the situation shown in FIG. 10, every time the subject TG performs the third step during the reference period PR1, the blind spot judgment value Q1 is calculated. <TH REF " is established. Also, even after the reference period PR1, "Q1 <TH REF However, after the reference period PR1, depending on the way the third process is performed, the blind spot judgment value Q1 may become equal to the reference value TH REF However, if it is known that blind spots occur periodically during the reference period PR1, it is reasonable that blind spots will be detected at the same intervals after the reference period PR1. By increasing the determination threshold TH as shown in FIG. 9(b), it becomes easier to detect blind spots at the same intervals as during the reference period PR1, even after the reference period PR1, despite slight fluctuations in the blind spot determination value Q1. This is believed to contribute to improving the stability of blind spot detection.

[0085] In case CS2a, the determination threshold TH is set at time t X The reference value TH REF to correction value TH HIGH After correcting to period T CYCL1 Even if n times the time has passed, the judgment threshold TH (=TH HIGH ) is not obtained, the threshold correction unit F23 sets the judgment threshold TH to the reference value TH REF You can return it to n, where n has a value of 1 or greater.

[0086] When the threshold correction unit F23 determines that the blind spot determination value Q1 periodically falls below the determination threshold value TH during the reference period PR1, the threshold correction unit F23 calculates the period T during which the blind spot determination value Q1 falls below the determination threshold value TH during the reference period PR1. CYCL1 It may be estimated that a blind spot will occur after the reference period PR1 in the same cycle as the reference period PR1. A signal Sig_2a indicating this estimation result is output at time t X may be sent from the controller 11 to the evaluation utilization device 20 at signal Sig_2a. In response to receiving the signal Sig_2a, a corresponding notification may be made by the evaluation utilization device 20 to the target person TG or another person.

[0087] In addition, in case CS2a, when the threshold correction unit F23 determines that the blind spot determination value Q1 periodically falls below the determination threshold TH, the controller 11 determines that the blind spot periodically falls below the determination threshold TH. CYCL1 A signal Sig_2b indicating that the blind spot repeatedly occurs may be transmitted to the evaluation utilization device 20. In response to receiving the signal Sig_2b, a corresponding notification may be provided by the evaluation utilization device 20 to the subject TG or another person. The notification based on the signal Sig_2b may be useful for the subject TG or another person to consider the causes of the blind spot.

[0088] <<Third Example>> A third embodiment will now be described. In the third embodiment, the technique described in the second embodiment is applied to a blind spot determination value Q2 having characteristics opposite to those of the blind spot determination value Q1. FIG. 11 shows a blind spot determination unit F30 according to the third embodiment. In the third embodiment, the blind spot determination unit F30 of FIG. 11 is used as the blind spot determination unit F8 of FIG. 5. Except for the blind spot determination unit F30 of FIG. 11 being used as the blind spot determination unit F8 of FIG. 5 and the following points in the third embodiment, the configuration and operation of the behavior evaluation device 10 and the system SYS are the same as those of the first embodiment. However, in the third embodiment, the blind spot determination value Q2 is used as the blind spot determination value Q, and the blind spot determination value Q2 at time t (i.e., Q2[t]) is expressed, for example, by the above formula (2A). The blind spot determination unit F30 includes a calculation unit F31, an instantaneous comparison unit F32, and a threshold correction unit F33.

[0089] An error ALoss and a confidence value CS are supplied to the calculation unit F31. The calculation unit F31 derives a blind spot determination value Q2 based on the error ALoss and the confidence value CS. At time t, the blind spot determination value Q2[t] is derived according to equation (2A). In the controller 11, every time the image data of the latest input image IN based on the latest shooting result by the camera CM is supplied to the state detection unit F1 (see FIG. 5), the error ALoss and the confidence value CS are derived based on the image data of the latest input image IN. That is, the error ALoss and the confidence value CS are updated based on the image data of the latest input image IN. Every time the error ALoss and the confidence value CS are updated in the calculation unit F31, the blind spot determination value Q2 is updated based on the latest error ALoss and the confidence value CS. A signal having the blind spot determination value Q2 as a signal value is the blind spot determination signal S Q2 is referred to as. FIG. 12(a) shows an example of the waveform of the blind spot determination signal S Q2 . The signal value of the blind spot determination signal S Q2 is the blind spot determination value Q2[t] at time t.

[0090] Every time the blind spot determination value Q2 is updated, the instantaneous comparison unit F32 compares the latest blind spot determination value Q2 with the determination threshold TH, and generates and outputs a comparison result signal H2 indicating the comparison result. The comparison result signal H2 is a binary signal having a value of "0" or "1". The instantaneous comparison unit F32 sets a value of "1" to the comparison result signal H2 when "Q2>TH" holds, and sets a value of "0" to the comparison result signal H2 when "Q2<TH" holds. Therefore, at time t, "H2 = 1" when equation (2B) holds. When "Q2 = TH" holds, a value of "1" or "0" is set to the comparison result signal H2. In the third embodiment, the comparison result signal H2 functions as blind spot determination information BS and is sent to the behavior evaluation unit F4.

[0091] The threshold correction unit F33 includes a periodicity determination unit F34. The periodicity determination unit F34 determines whether the blind spot determination signal S Q2 has periodicity by statistically processing the sequentially updated blind spot determination values Q2. The periodicity determination unit F33 obtains the autocorrelation of the blind spot determination signal S Q2 to determine the blind spot determination signal S Q2can determine whether it has periodicity, and the blind spot determination signal S Q2 When it is determined that it has periodicity, the blind spot determination signal S Q2 The period of can be obtained. For any signal of interest, since the method of obtaining the autocorrelation of the signal of interest is well-known, a detailed explanation is omitted.

[0092] The threshold correction unit F33 has a function of correcting the determination threshold TH. However, when it is not determined that the blind spot determination signal S Q2 has periodicity, the correction of the determination threshold TH is not performed, and in this case, the operation of the blind spot determination unit F30 is the same as that of the blind spot determination unit F8 in the first embodiment.

[0093] The blind spot determination signal S Q2 is determined to have periodicity and the blind spot determination signal S Q2 The period of is determined to be the period T CYCL2 The case obtained in this way is referred to as case CS3. Case CS3 is further subdivided into case CS3a and CS3b.

[0094] In case CS3a, as shown in Fig. 12(a), the periods when "Q2 < TH" holds and when "Q2 > TH" holds visit alternately, and the repetition period of the period when "Q2 > TH" holds is the period T CYCL2 This is the case. That is, in case CS3a, the blind spot determination value Q2 periodically exceeds the determination threshold TH. Among cases CS3, the case that does not correspond to case CS3a is case CS3b. For example, the case where the blind spot determination signal S Q2 has periodicity but "Q2 < TH" always holds corresponds to case CS3b. Also, for example, the case where the periods when "Q2 < TH" holds and when "Q2 > TH" holds visit alternately but there is no periodicity in the period when "Q2 > TH" also corresponds to case CS3b. In case CS3b, the correction of the determination threshold TH is not performed, and as a result, the operation of the blind spot determination unit F30 in case CS3b is the same as that of the blind spot determination unit F8 in the first embodiment.

[0095] The threshold correction unit F33 determines whether the blind spot determination value Q2 periodically exceeds the determination threshold TH. When it is determined that the blind spot determination value Q2 periodically exceeds the determination threshold TH, the threshold correction unit F33 decreases the determination threshold TH. Referring to FIG. 12(b), now, at time t Y It is not judged that the blind spot judgment value Q2 periodically exceeds the judgment threshold value TH until just before the judgment threshold value TH REF The reference value TH REF is a value preset in the blind spot determination unit F30. Y Assume that the threshold correction unit F33 determines that the blind spot judgment value Q2 periodically exceeds the judgment threshold value TH at time t Y In this case, the threshold correction unit F33 corrects the determination threshold TH to the reference value TH REF to correction value TH LOW Here, the “TH LOW <TH REF " is true. For example, "TH LOW =TH REF -ΔTH" and "ΔTH>0", or "TH LOW =TH REF ×(1-k B )” and “1>k B >0". In this way, the threshold correction unit F33 Y When it is determined that the blind spot determination value Q2 periodically exceeds the determination threshold TH during the period immediately preceding the reference period PR2 (hereinafter referred to as the reference period PR2), the determination threshold TH after the reference period PR2 is reduced below the reference period PR2.

[0096] When the subject TG repeats the same task at regular intervals, blind spots may occur at regular intervals. As described in the second embodiment with reference to FIG. 10, if it is known that blind spots occur periodically during the reference period PR2, it is reasonable that blind spots will be detected at the same intervals after the reference period PR2. By decreasing the determination threshold TH as shown in FIG. 12(b), it becomes easier to detect blind spots occurring at the same intervals as during the reference period PR1, even after the reference period PR2, despite slight fluctuations in the blind spot determination value Q2. This is believed to contribute to improving the stability of blind spot detection.

[0097] In case CS3a, the determination threshold TH is set at time t Y The reference value TH REF to correction value TH LOW After correcting to period T CYCL2 Even if n times the time has passed, the judgment threshold TH (=TH LOW ), the threshold correction unit F33 corrects the determination threshold TH to the reference value TH REF You can return it to n, where n has a value of 1 or greater.

[0098] When it is determined that the blind spot determination value Q2 periodically exceeds the determination threshold value TH during the reference period PR2, the threshold value correction unit F33 calculates the period T during which the blind spot determination value Q2 periodically exceeds the determination threshold value TH during the reference period PR2. CYCL2 It may be estimated that a blind spot will occur after the reference period PR2 in the same cycle as the reference period PR1. A signal Sig_3a indicating this estimation result is output at time t Y may be sent from the controller 11 to the evaluation utilization device 20. In response to receiving the signal Sig_3a, a corresponding notification may be made by the evaluation utilization device 20 to the target person TG or another person.

[0099] In addition, in case CS3a, when the threshold correction unit F33 determines that the blind spot determination value Q2 periodically exceeds the determination threshold TH, the controller 11 determines that the blind spot periodically exceeds the determination threshold TH. CYCL2 A signal Sig_3b indicating that the blind spot repeatedly occurs may be transmitted to the evaluation utilization device 20. In response to receiving the signal Sig_3b, a corresponding notification may be provided by the evaluation utilization device 20 to the subject TG or another person. The notification based on the signal Sig_3b may be useful for the subject TG or another person to consider the causes of the blind spot.

[0100] <<Fourth Example>> A fourth embodiment will now be described. Up to this point, for the sake of specificity and clarity of explanation, it has been assumed that the target TG is a single person. However, if multiple people fit within the shooting area of the camera CM, each of the multiple people may be treated as a target TG. That is, if multiple people fit within the shooting area of the camera CM, the behavior evaluation device 10 may regard each person as a target TG and perform the above-described processes (such as processes for deriving behavior evaluation information EV) for each person. In this case, the controller 11 tracks and identifies each person based on the image data of multiple input images IN (i.e., moving images) arranged in time series.

[0101] <<Fifth Example>> A fifth embodiment will be described. The target person TG is not limited to a worker in a factory (production line). As cases where the target person TG is different from the worker, cases CS5a to CS5c will be shown below.

[0102] In case CS5a, the camera CM is installed in a vehicle such as an automobile and captures the vehicle's occupants as subjects TG. While the driver is primarily considered to be the vehicle occupant, the subjects captured by the camera CM may include other occupants (passengers) besides the driver. In case CS5a, the behavior evaluation device 10 may be mounted on the vehicle. The controller 11 in case CS5a can evaluate the driver's behavior, including the driver's driving operation (e.g., evaluate the safety or risk of the driving operation). In case CS5a, the evaluation utilization device 20 may be located inside or outside the vehicle. If the vehicle is a taxi or delivery truck, the evaluation utilization device 20 may include a server device installed outside the vehicle, and the behavior evaluation information EV stored in the server device can be used for driver performance evaluation, training, etc.

[0103] In case CS5b, the camera CM is installed in a vehicle such as an automobile and captures a nearby person as a target person TG. The nearby person here is a person (pedestrian, etc.) located near the vehicle. In case CS5b, the behavior evaluation device 10 may be mounted on the vehicle. The controller 11 related to case CS5b can evaluate the behavior of a nearby person, such as whether they will enter the direction of travel of the vehicle. In case CS5b, the evaluation utilization device 20 may be located inside or outside the vehicle. If the vehicle is a taxi or delivery truck, the evaluation utilization device 20 may include a server device installed outside the vehicle, and the notification device 21 may be located inside the vehicle.

[0104] A driving assistance system or an automated driving system may be installed in the vehicle according to case CS5b, and the driving assistance system or automated driving system may provide driving assistance or automated driving to the vehicle based on the behavior evaluation information EV. During driving assistance or automated driving, control is implemented to avoid a collision between the vehicle and a nearby person. In this case, if there is a blind spot in the camera CM capturing the nearby person, the reliability of the control may be reduced. If it is determined that there is a blind spot, the notification device 21 may notify the driver of the vehicle based on a blind spot notification signal (see step S20), thereby improving the safety of driving assistance or automated driving.

[0105] In case CS5c, the camera CM photographs a player as a target TG. The player here is a person playing or practicing a sport. Alternatively, the player is a person performing an exercise (such as fitness exercise). The controller 11 in case CS5c can, for example, evaluate the player's behavior (exercise) to derive the appropriateness of the form (for example, the appropriateness of the form in a golf or baseball swing) and include it in the behavior evaluation information EV.

[0106] Alternatively, the subject TG may be any person whose behavior is to be evaluated. Furthermore, the subject of behavior evaluation by the behavior evaluation device 10 may be something other than a person. That is, the behavior evaluation device 10 may be capable of evaluating the behavior of any object. The subject TG is an example of an object, and the object may be an animal other than a human, or an artificial object such as a robot or a vehicle. The movements of robots operating on production lines are often periodic, and the techniques shown in the second or third embodiment may be useful. Furthermore, there may be a case where the object is a racing vehicle traveling around a racing track, and the state of the racing vehicle traveling around the track is filmed by a camera commercial. Since racing vehicles travel around the racing track at roughly regular intervals, the techniques shown in the second or third embodiment may be useful.

[0107] The behavior evaluation device 10 includes a blind spot determination device. The blind spot determination device can be understood as equivalent to the behavior evaluation device 10 without the behavior evaluation unit F4. In this case, it can be considered that the behavior evaluation device 10 is formed by adding the behavior evaluation unit F4 to the blind spot determination device.

[0108] A program that causes a computer device to execute any of the methods described in the embodiments of the present invention, and a non-volatile recording medium on which the program is recorded, are included within the scope of the embodiments of the present invention. The program that causes a computer device to execute any of the methods described in the embodiments of the present invention may be a subprogram incorporated into any main program or called by any main program. The behavior evaluation device 10 is a type of computer device. Any processing in the embodiments of the present invention may be realized by hardware such as a semiconductor integrated circuit, software equivalent to the program, or a combination of hardware and software.

[0109] The embodiments of the present invention can be modified in various ways as appropriate within the scope of the technical ideas set forth in the claims. The above-described embodiments are merely examples of the present invention, and the meanings of the terms of the present invention and each constituent element are not limited to those described in the above-described embodiments. The specific numerical values shown in the above description are merely examples, and as a matter of course, they can be changed to various numerical values. [Explanation of symbols]

[0110] SYS System (Behavioral Assessment System) TG Target Audience Commercial camera 10 Behavioral assessment device 11 Controller 12 Memory 13 Communications Department 20 Evaluation and Use Device 21 Notification device 111 Behavioral Assessment Block 112 Blind Spot Detection Block F1 Status detection unit F2 Action recognition unit F3 Behavioral Prediction Department F4 Behavioral Assessment Department F5 Trust Identification Unit F6 Delay F7 Error calculation section F8, F20, F30 blind spot determination section F21, F31 calculation section F22, F32 Instantaneous comparison section F23, F33 Threshold correction section F24, F34 Periodicity judgment section

Claims

1. Deriving state detection information by detecting a state of the object based on an input image from a camera that photographs the object; deriving current behavior information by recognizing a current behavior of the object based on the state detection information, and deriving predicted behavior information by predicting a future behavior of the object based on the state detection information; deriving an error between the current behavior information at a target time and the predicted behavior information, which is information derived in a prediction before the target time and indicates a predicted result of the behavior of the object at the target time; The presence or absence of a blind spot of the camera when photographing the object is determined based on a reliability value indicating the accuracy of the state detection information at the target time and the error. , blind spot determination device.

2. The presence or absence of a blind spot is determined by comparing the reliability value and the blind spot determination value according to the error with a determination threshold. The blind spot determination device according to claim 1 .

3. When the blind spot determination value increases as the reliability value increases and decreases as the error increases, it is determined that the blind spot exists when the blind spot determination value is smaller than the determination threshold value. The blind spot determination device according to claim 2 .

4. If the blind spot determination value falls periodically below the determination threshold during a reference period, the determination threshold after the reference period is increased to a value higher than that during the reference period. The blind spot determination device according to claim 3 .

5. When the blind spot determination value decreases as the reliability value increases and increases as the error increases, it is determined that the blind spot exists when the blind spot determination value is greater than the determination threshold. The blind spot determination device according to claim 2 .

6. If the blind spot determination value periodically exceeds the determination threshold during a reference period, the determination threshold after the reference period is reduced to a value lower than that during the reference period. The blind spot determination device according to claim 5 .

7. A behavior evaluation device including the blind spot determination device according to any one of claims 1 to 6, The system is configured to be able to generate and output behavior evaluation information related to the behavior of the target object based on the current behavior information at the target time and the predicted behavior information obtained by prediction at the target time, and when it is determined that there is a blind spot, the system stops generating or outputting the behavior evaluation information. , behavioral assessment device.

8. A behavior evaluation device including the blind spot determination device according to any one of claims 1 to 6, a notification device capable of notifying information to the person or other person as the target is connected to the behavior evaluation device by wire or wireless; When it is determined that the blind spot exists, the behavior evaluation device outputs a blind spot detection signal to the notification device to cause the notification device to notify the person or the other person of information indicating the existence of the blind spot. , behavioral assessment device.

9. The behavior evaluation device according to claim 7 ; The camera. ,Behavioral Assessment System.

10. The behavior evaluation device according to claim 8 ; The camera. ,Behavioral Assessment System.

11. A blind spot determination method executed by a blind spot determination device, Deriving state detection information by detecting a state of the object based on an input image from a camera that photographs the object; deriving current behavior information by recognizing a current behavior of the object based on the state detection information, and deriving predicted behavior information by predicting a future behavior of the object based on the state detection information; deriving an error between the current behavior information at a target time and the predicted behavior information, which is information derived in a prediction before the target time and indicates a predicted result of the behavior of the object at the target time; The presence or absence of a blind spot of the camera when photographing the object is determined based on a reliability value indicating the accuracy of the state detection information at the target time and the error. , blind spot determination method.

12. Deriving state detection information by detecting a state of the object based on an input image from a camera that photographs the object; deriving current behavior information by recognizing a current behavior of the object based on the state detection information, and deriving predicted behavior information by predicting a future behavior of the object based on the state detection information; deriving an error between the current behavior information at a target time and the predicted behavior information, which is information derived in a prediction before the target time and indicates a predicted result of the behavior of the object at the target time; a computer device that executes a blind spot determination method for determining whether or not there is a blind spot for the camera when photographing the object, based on a reliability value indicating the accuracy of the state detection information at the target time and the error; , a blind spot detection program.

Images