Image recognition method and image recognition program

Abstract

Witness candidates of an incident are narrowed down from image data. An image recognition method includes detecting an incident and a plurality of persons, calculating, for each person, a first vector indicating a first direction that the person faces and a second vector indicating a second direction from the person to the incident, and assigning a score to each person using the first vector and the second vector, the score indicating a probability that the person has witnessed the incident.

Description

IMAGE RECOGNITION METHOD AND IMAGE RECOGNITION PROGRAM

[0001] The embodiments discussed herein relate to an image recognition method and an image recognition program.

[0002] Incidents may occur in places where people are active. Incidents may include accidents with unexpected and undesired consequences such as human damage and material damage. Incidents may also include hazardous situations that possibly lead to such accidents although no accidents have occurred.

[0003] When an incident is detected, an investigator may analyze the causes of the incident and develop measures to prevent recurrence. The investigator may identify the incident using image data captured by surveillance cameras. The investigator may also identify bystanders at the incident using the image data captured by the surveillance cameras. There may be a witness to the incident among the bystanders. The investigator may obtain important information on the incident by conducting interviews with the bystanders. An information processing system has room for improvement to support such incident investigations using image recognition techniques.

[0004] Examples of the image recognition techniques include a technique of detecting bystanders at an object from image data, a technique of creating, from image data, a skeleton graph with nodes representing person’s joints and edges representing the connections between the joints and predicting the person’s behavior using the skeleton graph, a technique of detecting violent behavior from image data, a technique of detecting a person falling from image data, and a technique of detecting a person’s gaze from image data.

[0005] Rakibul Hasan, David Crandall, Mario Fritz and Apu Kapadia, "Automatically Detecting Bystanders in Photos to Reduce Privacy Risks", Proc. of the 41st IEEE Symposium on Security and Privacy, pp. 318-335, May 2020Yuya Obinata and Takuma Yamamoto, "Temporal Extension Module for Skeleton-Based Action Recognition", Proc. of the 25th International Conference on Pattern Recognition (ICPR 2020), pp. 534-540, January 2021Khalid Raihan Talha, Koushik Bandapadya and Mohammad Monirujjaman Khan, "Violence Detection Using Computer Vision Approaches", Proc. of the 2022 IEEE World AI IoT Congress (AIIoT 2022), pp. 544-550, June 2022Ekram Alam, Abu Sufian, Paramartha Dutta and Marco Leo, "Real-Time Human Fall Detection Using a Lightweight Pose Estimation Technique", Proc. of the 5th International Conference on Computational Intelligence in Communications and Business Analytics (CICBA 2023), pp. 30-40, January 2023Matus Tanonwong, Naoya Chiba and Koichi Hashimoto, "Recognition of Human Relationships Using Interactions and Gazes through Video Analysis in Surveillance Footage", Proc. of the 2023 IEEE International Conference on Robotics and Biomimetics (ROBIO 2023), December 2023

[0006] By applying an image recognition technique to image data capturing an incident, an information processing system may be able to detect a plurality of people from the image data. However, not all bystanders at the incident have witnessed the incident or have important information on the incident. Therefore, an incident investigation involving interviews by an investigator with all people detected in the image data may be inefficient. In view of this, an object of the present disclosure is to narrow down witness candidates of the incident from the image data.

[0007] In one aspect, there is provided an image recognition method executed by a computer. The image recognition method includes: detecting an incident and a plurality of persons from image data; calculating, for each person of the plurality of persons, a first vector indicating a first direction that the person faces and a second vector indicating a second direction from the person to the incident; and assigning a score to each person using the first vector and the second vector, the score indicating a probability that the person has witnessed the incident.

[0008] In another aspect, there is provided an image recognition method executed by a computer. The image recognition method includes: detecting an incident, a plurality of persons, and one or more objects other than the plurality of persons from image data; calculating, for each person of the plurality of persons, a straight line between the person and the incident; searching for an object that the straight line crosses among the one or more objects; and assigning a score to each person using a result of the searching, the score indicating a probability that the person has witnessed the incident.

[0009] According to one aspect, witness candidates of an incident are narrowed down from image data.

[0010] FIG. 1 is a view for describing an information processing apparatus according to a first embodiment.FIG. 2 is a view for describing an information processing apparatus according to a second embodiment.FIG. 3 illustrates an example of an information processing system according to a third embodiment.FIG. 4 is a block diagram illustrating an example of the hardware configuration of an information processing apparatus.FIG. 5 illustrates an example of detecting an incident and persons from an image frame.FIG. 6 illustrates examples of vectors for calculating an attention score.FIG. 7 illustrates an example of searching for an occlusion for calculating an occlusion factor.FIG. 8 illustrates examples of a positional relationship between an incident and a person.FIG. 9 is a table for describing an example of calculating a total score.FIG. 10 is a table illustrating examples of comparing investigation durations.FIG. 11 is a block diagram illustrating examples of functions of the information processing apparatus.FIG. 12 is a flowchart illustrating an example of a procedure for image recognition.FIG. 13 is a continuation of the flowchart illustrating the example of the procedure for the image recognition.

[0011] Hereinafter, embodiments will be described with reference to the accompanying drawings.

[0012] (a) First Embodiment

[0013] FIG. 1 is a view for describing an information processing apparatus according to a first embodiment. The information processing apparatus 10 supports an incident investigation using image recognition techniques. The information processing apparatus 10 may be a client device or a server device. The information processing apparatus 10 may be called a computer, an image recognition apparatus, or a machine learning apparatus.

[0014] The information processing apparatus 10 includes a storage unit 11 and a processing unit 12. The storage unit 11 may be a volatile semiconductor memory, such as a random access memory (RAM). Alternatively, the storage unit 11 may be a non-volatile storage device, such as a hard disk drive (HDD) or a flash memory.

[0015] For example, the processing unit 12 is a processor such as a central processing unit (CPU), a graphics processing unit (GPU), or a digital signal processor (DSP). Note that the processing unit 12 may include an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or another electronic circuit. For example, the processor runs programs stored in a memory such as a RAM. A set of processors may be called a multiprocessor or simply a “processor.”

[0016] The storage unit 11 stores image data 13. For example, a surveillance camera generates the image data 13. The information processing apparatus 10 may receive the image data 13 from the surveillance camera. Alternatively, the information processing apparatus 10 may receive the image data 13 from another information processing apparatus. The surveillance camera captures images of a certain location. The certain location may be an indoor place such as a shop or a factory, or an outdoor place such as a street or a park. The image data 13 may include a plurality of image frames taken at different times. The processing unit 12 may process each image frame as follows.

[0017] The processing unit 12 detects an incident 14 from the image data 13. The incident 14 may be an accident with unexpected and undesired consequences. Alternatively, the incident 14 may be a hazardous situation that possibly leads to such an accident although no accident has occurred. One example of the incident 14 is a collision accident between a person and a machine. Another example of the incident 14 is a near-miss situation in which a person and a machine approach each other within less than a certain distance although they do not come into direct contact with each other. When the incident 14 is detected, an investigator may investigate the incident 14 to prevent recurrence.

[0018] In addition, the processing unit 12 detects a plurality of persons including persons 15a and 15b from the image data 13. The processing unit 12 detects the incident 14 and persons 15a and 15b from the image data 13 by using an image recognition technique. This image recognition technique may use pattern matching. In addition, the image recognition technique may use a machine learning model such as a convolutional neural network. The processing unit 12 may detect the incident 14 and persons 15a and 15b by using an image recognition technique described in the above-mentioned non-patent literature documents.

[0019] The processing unit 12 calculates a first vector and a second vector for each of the plurality of persons. The first vector indicates a first direction that the person faces on the image data 13. The second vector indicates a second direction from the person to the incident 14 on the image data 13. For example, each of the first vector and second vector is a two-dimensional numerical vector with X and Y components. The processing unit 12 calculates a vector 16a indicating the first direction and a vector 17a indicating the second direction for the person 15a. In addition, the processing unit 12 calculates a vector 16b indicating the first direction and a vector 17b indicating the second direction for the person 15b.

[0020] The vector 16a may indicate the direction in which the person 15a gazes. The processing unit 12 may detect the gaze from the image data 13 by using an image recognition technique described in the above-mentioned non-patent literature documents. This image recognition technique may use a machine learning model such as a convolutional neural network. Alternatively, the vector 16a may indicate the direction that the head of the person 15a faces. Yet alternatively, the vector 16a may indicate the direction that the body of the person 15a faces.

[0021] Here, the processing unit 12 may detect the skeleton of the person 15a from the image data 13 by using an image recognition technique described in the above-mentioned non-patent literature documents. This image recognition technique may use a machine learning model such as a convolutional neural network. The skeleton may be represented by a skeleton graph with nodes representing joints and edges representing the connections between the joints. The processing unit 12 may use the skeleton graph to detect the head-facing direction or the body-facing direction.

[0022] The starting point of the vector 17a may be set at the center of a bounding box containing the person 15a. A bounding box is the smallest rectangle that surrounds a detected entity. Alternatively, the starting point of the vector 17a may be set at a node representing the head of the person 15a. Yet alternatively, the starting point of the vector 17a may be set at a node representing the center of the body of the person 15a. The vector 17a may be directed toward the center of a bounding box containing the incident 14.

[0023] The processing unit 12 calculates the vector 16b for the person 15b in the same manner as with the vector 16a of the person 15a. The processing unit 12 also calculates the vector 17b for the person 15b in the same manner as with the vector 17a of the person 15a. The processing unit 12 may normalize the vectors 16a, 16b, 17a, and 17b. For example, the processing unit 12 normalizes each vector 16a, 16b, 17a, and 17b to a length of 1.

[0024] The processing unit 12 assigns a score to each of the plurality of persons using the first vector and the second vector. The score indicates the probability that the person has witnessed the incident 14. For example, the score is a scalar value in a certain numerical range of 0 to 1, inclusive. A higher score indicates that the person is more likely to have witnessed the incident 14, whereas a lower score indicates that the person is less likely to have witnessed the incident 14.

[0025] The processing unit 12 assigns a score 18a to the person 15a, and a score 18b to the person 15b. The processing unit 12 may calculate the score 18a using the inner product of the vector 16a and vector 17a. As the first direction indicated by the vector 16a and the second direction indicated by the vector 17a are closer to each other, the score 18a increases. The processing unit 12 calculates the score 18b for the person 15b in the same manner as with the score 18a of the person 15a.

[0026] For example, the first direction indicated by the vector 16a and the second direction indicated by the vector 17a are close to each other. Therefore, the score 18a is high, which indicates that the person 15a is highly likely to have witnessed the incident 14. On the other hand, the first direction indicated by the vector 16b and the second direction indicated by the vector 17b are far from each other. Therefore, the score 18b is low, which indicates that the person 15b is less likely to have witnessed the incident 14.

[0027] The processing unit 12 outputs the plurality of scores respectively corresponding to the plurality of persons. The processing unit 12 may store the plurality of scores in a non-volatile storage device. Alternatively, the processing unit 12 may display the plurality of scores on a display device connected to the information processing apparatus 10 or may send them to another information processing apparatus.

[0028] The processing unit 12 may sort the plurality of persons by score. For example, the processing unit 12 sorts the plurality of persons in descending order of scores. The processing unit 12 may output a witness candidate list listing the plurality of persons in the sorted order, or may output a witness candidate list listing persons whose scores exceed a threshold. The investigator may conduct interviews regarding the incident 14 with the persons, prioritizing those with higher scores.

[0029] As described above, the information processing apparatus 10 of the first embodiment detects the incident 14 and a plurality of persons from the image data 13. For each of the plurality of persons, the information processing apparatus 10 calculates a first vector indicating the first direction that the person faces and a second vector indicating the second direction from the person to the incident 14. Then, using the first vector and the second vector, the information processing apparatus 10 assigns each of the plurality of persons a score indicating the probability that the person has witnessed the incident 14.

[0030] By doing so, the information processing apparatus 10 is able to extract witness candidates who may have witnessed the incident 14 from the image data 13. Therefore, the information processing apparatus 10 is able to support the investigator in investigating the incident 14. In addition, the information processing apparatus 10 is able to narrow down the witness candidates using the scores. Therefore, the investigator is able to conduct interviews with only persons who are highly likely to have important information on the incident 14. This reduces the number of targets to be interviewed and thus improves the efficiency of the incident investigation.

[0031] Note that the investigator has a choice of conducting interviews with all persons detected in the image data 13. However, there is a possibility that some of them have not witnessed the incident 14 because of their having faced away from the incident 14. Such persons are likely not to have important information on the incident 14. To deal with this, the information processing apparatus 10 is able to narrow down to persons who are highly likely to have important information, on the basis of the facing direction of each person.

[0032] (b) Second Embodiment

[0033] FIG. 2 is a view for describing an information processing apparatus according to a second embodiment. The information processing apparatus 20 supports incident investigations using image recognition techniques. The information processing apparatus 20 may be a client device or a server device. The information processing apparatus 20 may be called a computer, an image recognition apparatus, or a machine learning apparatus. Note that the second embodiment may be implemented in combination with the first embodiment.

[0034] The information processing apparatus 20 includes a storage unit 21 and a processing unit 22. The storage unit 21 may be a volatile semiconductor memory such as a RAM. Alternatively, the storage unit 21 may be a non-volatile storage device such as an HDD or a flash memory. The storage unit 21 corresponds to the storage unit 11 of the first embodiment. The processing unit 22 is a processor such as a CPU, a GPU, or a DSP, for example. In this connection, the processing unit 22 may include an ASIC, an FPGA, or another electronic circuit. The processor runs programs stored in a memory such as a RAM, for example. The processing unit 22 corresponds to the processing unit 12 of the first embodiment.

[0035] The storage unit 21 stores image data 23. For example, a surveillance camera generates the image data 23. The image data 23 corresponds to the image data 13 of the first embodiment. The image data 23 may include a plurality of image frames taken at different times. The processing unit 22 may process each image frame as follows.

[0036] The processing unit 22 detects an incident 24 from the image data 23. The incident 24 corresponds to the incident 14 of the first embodiment. The processing unit 22 detects a plurality of persons including persons 25a and 25b from the image data 23. The persons 25a and 25b correspond to the persons 15a and 15b of the first embodiment. In addition, the processing unit 22 detects, other than the persons 25a and 25b, one or more objects including an object 26 from the image data 23.

[0037] The object 26 is an occlusion candidate that may obstruct the views of the persons 25a and 25b. One example of the object 26 is a machine placed in the space where the persons 25a and 25b are present. The processing unit 22 detects the object 26 by using an image recognition technique. This image recognition technique may use a machine learning model such as a convolutional neural network. The processing unit 22 may detect the object 26 by using an image recognition technique described in the above-mentioned non-patent literature documents.

[0038] For each of the plurality of persons, the processing unit 22 calculates a straight line between the person and the incident 24. The straight line may be a line segment that has one end point representing the location of the person and the other end point representing the location of the incident 24. The processing unit 22 calculates a straight line 27a between the person 25a and the incident 24. In addition, the processing unit 22 calculates a straight line 27b between the person 25b and the incident 24.

[0039] The end point representing the location of the incident 24 may be set at the center of a bounding box containing the incident 24. The end point representing the location of the person 25a may be set at the center of a bounding box containing the person 25a. Alternatively, the end point representing the location of the person 25a may be set at a node representing the head of the person 25a or at a node representing the center of the body of the person 25a. The processing unit 22 calculates the straight line 27b for the person 25b in the same manner as with the straight line 27a of the person 25a.

[0040] For each of the plurality of persons, the processing unit 22 searches for an object that the straight line crosses among the one or more objects. For example, the processing unit 22 searches for an object whose bounding box intersects the straight line. With respect to the person 25a, the straight line 27a does not cross any object. Therefore, the object 26 has not prevented the person 25a from witnessing the incident 24. With respect to the person 25b, on the other hand, the straight line 27b crosses the object 26. Therefore, there is a possibility that the object 26 has prevented the person 25b from witnessing the incident 24.

[0041] The processing unit 22 assigns a score to each of the plurality of persons using the above search results. The score indicates the probability that the person has witnessed the incident 24. For example, the score is a scalar value in a certain numerical range of 0 to 1, inclusive. The processing unit 22 assigns a score 28a to the person 25a, and a score 28b to the person 25b. The score 28a corresponds to the score 18a of the first embodiment, and the score 28b corresponds to the score 18b of the first embodiment.

[0042] When an object that the straight line crosses is found, the processing unit 22 may reduce the score. When such an object is not found, on the other hand, the processing unit 22 may increase the score. For example, the score 28a is higher than the score 28b. The processing unit 22 may update a base score on the basis of whether such an object exists or not. The base score may be a score that is calculated in the first embodiment.

[0043] The processing unit 22 may calculate the score 28b using the type of the object 26. The processing unit 22 increases the score 28b as the object 26 has a greater transparency. On the other hand, the processing unit 22 reduces the score 28b as the object 26 has a lower transparency. The transparency is the degree to which the view behind the object 26 is visible through the object 26. The transparency depends on the structure and material of the object 26.

[0044] The processing unit 22 may also calculate the score 28b on the basis of the relationship between the object 26 and the straight line 27b. For example, the processing unit 22 calculates the distance between the straight line 27b and the center of the bounding box of the object 26. The processing unit 22 may increase the score 28b as the distance is greater. The processing unit 22 may reduce the score 28b as the distance is less.

[0045] The processing unit 22 outputs the plurality of scores respectively corresponding to the plurality of persons. The processing unit 22 may store the plurality of scores in a non-volatile storage device. Alternatively, the processing unit 22 may display the plurality of scores on a display device connected to the information processing apparatus 20 or may send them to another information processing apparatus.

[0046] The processing unit 22 may sort the plurality of persons by score. For example, the processing unit 22 sorts the plurality of persons in descending order of scores. The processing unit 22 may output a witness candidate list listing the plurality of persons in the sorted order or may output a witness candidate list listing persons whose scores exceed a threshold. The investigator may conduct interviews regarding the incident 24 with the persons, prioritizing those with higher scores.

[0047] As described above, the information processing apparatus 20 of the second embodiment detects, from the image data 23, the incident 24, a plurality of persons, and one or more objects other than the plurality of persons. For each of the plurality of persons, the information processing apparatus 20 calculates a straight line between the person and the incident 24. The information processing apparatus 20 searches for an object that the straight line crosses among the one or more objects. The information processing apparatus 20 then assigns each of the plurality of persons a score indicating the probability that the person has witnessed the incident 24, using the search results.

[0048] By doing so, the information processing apparatus 20 is able to extract witness candidates who may have witnessed the incident 24 from the image data 23. Therefore, the information processing apparatus 20 is able to support the investigator in investigating the incident 24. In addition, the information processing apparatus 20 is able to narrow down the witness candidates using the scores. Therefore, the investigator is able to conduct interviews with only persons who are highly likely to have important information on the incident 24. This reduces the number of targets to be interviewed and thus improves the efficiency of the incident investigation.

[0049] Note that the investigator has a choice of conducting interviews with all persons detected in the image data 23. However, there is a possibility that some of them have not witnessed the incident 24 because of their views having been obstructed by an occlusion. Such persons are likely not to have important information on the incident 24. To deal with this, the information processing apparatus 20 is able to narrow down to persons who are highly likely to have important information, on the basis of whether an occlusion exists.

[0050] (c) Third Embodiment

[0051] FIG. 3 illustrates an example of an information processing system according to a third embodiment. The information processing system of the third embodiment includes a network 30, a warehouse 31, surveillance cameras 32 and 33, and an information processing apparatus 100. The network 30 may include a local area network (LAN). The network 30 may also include a wide area communication network, such as the Internet. The surveillance cameras 32 and 33 and the information processing apparatus 100 are connected to the network 30. The surveillance cameras 32 and 33 are installed in the warehouse 31.

[0052] A company owns the warehouse 31. Items such as crates are stored in the warehouse 31. In addition, a cargo-handling machine such as a forklift is located in the warehouse 31. Employees of the company work in the warehouse 31. Incidents may occur in the warehouse 31. One example of the incidents may be a collision accident between an employee and a forklift. Another example of the incidents is a rule violation such as a failure to wear a safety vest.

[0053] When an incident is detected, an investigator at the health and safety division of the company analyzes the causes of the incident and develops measures to prevent recurrence. However, the employees working in the warehouse 31 may tend to undervalue incidents and therefore may sometimes not report minor incidents to the company. To deal with this, the information processing system automatically detects incidents using the surveillance cameras 32 and 33 and automatically reports the incidents to the company.

[0054] The information processing apparatus 100 receives image data from the surveillance cameras 32 and 33 over the network 30. The information processing apparatus 100 analyzes the image data. The information processing apparatus 100 automatically detects an incident from the image data using a machine learning model. In addition, the information processing apparatus 100 automatically detects bystanders at the incident from the image data using the machine learning model. The information processing apparatus 100 outputs a list of the bystanders.

[0055] The bystanders at the incident may have witnessed the incident. The investigator may be able to obtain important information on the incident by conducting interviews with the bystanders. The information processing apparatus 100 performs the above-described image processing to support the investigator in the incident investigation. The information processing apparatus 100 corresponds to the information processing apparatus 10 of the first embodiment and the information processing apparatus 20 of the second embodiment.

[0056] FIG. 4 is a block diagram illustrating an example of the hardware configuration of the information processing apparatus. The information processing apparatus 100 includes a CPU 101, a RAM 102, an HDD 103, a GPU 104, an input interface 105, a media reader 106, and a communication interface 107, which are connected to a bus. The CPU 101 corresponds to the processing unit 12 of the first embodiment and the processing unit 22 of the second embodiment. The RAM 102 or HDD 103 corresponds to the storage unit 11 of the first embodiment and the storage unit 21 of the second embodiment.

[0057] The CPU 101 is a processor that executes program commands. The CPU 101 loads programs and data from the HDD 103 to the RAM 102 and executes the programs. The information processing apparatus 100 may be provided with a plurality of processors.

[0058] The RAM 102 is a volatile semiconductor memory that temporarily stores programs and data. The programs are executed by the CPU 101, and the data is used by the CPU 101 in processing. The information processing apparatus 100 may be provided with a different type of volatile memory than RAM.

[0059] The HDD 103 is a non-volatile storage device that stores software programs and other data. The software includes operating system (OS), middleware, and application software. The information processing apparatus 100 may be provided with another type of non-volatile storage device such as a solid state drive (SSD).

[0060] The GPU 104 performs image processing in collaboration with the CPU 101 and displays images on a display device 111 connected to the information processing apparatus 100. Examples of the display device 111 include a cathode ray tube (CRT) display, a liquid crystal display, an organic electro-luminescence (EL) display, and a projector.

[0061] The GPU 104 may be used as a general-purpose computing on graphics processing unit (GPGPU). The GPU 104 may run programs in accordance with commands from the CPU 101. The information processing apparatus 100 may be provided with a volatile semiconductor memory other than the RAM 102 as a GPU memory.

[0062] The input interface 105 receives an input signal from an input device 112 connected to the information processing apparatus 100. Examples of the input device 112 include a mouse, a touch panel, and a keyboard. A plurality of input devices may be connected to the information processing apparatus 100.

[0063] The media reader 106 is a reading device that reads programs and data from a storage medium 113. Examples of the storage medium 113 include a magnetic disk, an optical disc, and a semiconductor memory. Magnetic disks include flexible disks (FDs) and HDDs. Optical discs include compact discs (CDs) and digital versatile discs (DVDs). The media reader 106 copies the programs and data from the storage medium 113 to the RAM 102 or the HDD 103.

[0064] The read programs may be run by the CPU 101. The storage medium 113 may be a portable storage medium. The storage medium 113 may be used to distribute the programs and data. The storage medium 113 and HDD 103 may be referred to as computer-readable storage media.

[0065] The communication interface 107 communicates with the surveillance cameras 32 and 33 over the network 30. The communication interface 107 may be a wired communication interface that performs communication via a wired cable. The communication interface 107 may be a wireless communication interface that performs communication via a wireless link. The communication interface 107 receives image data from the surveillance cameras 32 and 33 and stores the image data in the RAM 102.

[0066] The following describes image recognition to identify witnesses of an incident. The image data captured by the surveillance cameras 32 and 33 is video data with a fixed frame rate of 5 frames per second (fps) or another. The information processing apparatus 100 detects an incident, persons, and other objects from each image frame included in the image data.

[0067] The information processing apparatus 100 detects the incident, persons, and other objects by using a machine learning model such as a convolutional neural network. For example, the other objects include a crate and a forklift. The information processing apparatus 100 or another information processing apparatus trains the machine learning model using training data in advance. The information processing apparatus 100 assigns each detected entity a rectangular bounding box defining the image region containing the detected entity and a class label indicating the type of the detected entity.

[0068] The information processing apparatus 100 compares bounding boxes across adjacent image frames. The information processing apparatus 100 determines the identity of detected entities on the basis of the proximity of the positions of their bounding boxes and the similarity in the features of the detected entities. For example, the information processing apparatus 100 determines that persons detected in adjacent image frames are the same person in the case where a shift in the coordinates between the centers of their bounding boxes is less than a threshold and the degree of similarity of their appearance features exceeds a threshold. The information processing apparatus 100 assigns the same identifier to the plurality of bounding boxes containing the same detected entity. The identification of an entity detected across a plurality of image frames may be called re-identification (ReID).

[0069] FIG. 5 illustrates an example of detecting an incident and persons from an image frame. The information processing apparatus 100 detects an incident 141 and persons 142, 143, 144, and 145 from the image frame 131. The information processing apparatus 100 may detect the incident 141 by using an image recognition technique described in the above-mentioned non-patent literature documents.

[0070] For example, the information processing apparatus 100 detects persons from the image frame 131 and then detects their skeletons. A skeleton is represented by a skeleton graph with nodes representing joints and edges representing the connections between the joints. The information processing apparatus 100 estimates the posture of each person from his / her skeleton. In addition, the information processing apparatus 100 detects a cargo-handling machine such as a forklift from the image frame 131. The information processing apparatus 100 detects an incident that is a collision between a person and the cargo-handling machine, on the basis of the proximity between the person and the cargo-handling machine and the posture of the person.

[0071] In addition, for example, the information processing apparatus 100 detects the persons from the image frame 131 and detects the features of their clothes. The information processing apparatus 100 detects an incident that is a rule violation such as a failure to wear a safety vest, on the basis of the features of the clothes.

[0072] It is not always true that all persons 142, 143, 144, and 145 captured together with the incident 141 in an image frame have witnessed the incident 141. For example, as seen in the image frame 131, the persons 142 and 144 face the opposite direction from the incident 141. The person 145 faces toward the incident 141, but there is a possibility that the view of the person 145 is obstructed by the person 144.

[0073] As described above, it is not always true that the bystanders at the incident 141 have witnessed the incident 141 and have important information on the incident 141. Therefore, interviews with all bystanders or randomly selected bystanders means that the investigator conducts interviews with bystanders who do not have any important information. This may result in an inefficient incident investigation.

[0074] To deal with this, the information processing apparatus 100 analyzes image data and assigns a score to each person detected in the image data. A score indicates the probability that a person has witnessed the incident 141. A person with a higher score is more likely to have witnessed the incident 141. The information processing apparatus 100 sorts the plurality of persons in descending order of scores. The investigator conducts interviews with the persons, prioritizing those with higher scores, which improves the efficiency of the incident investigation.

[0075] The information processing apparatus 100 does not need to identify the names of the persons detected in the image data. For example, the information processing apparatus 100 outputs a list associating appearance information such as a bounding box cut out from the image data with a corresponding calculated score.

[0076] The following describes how to calculate the score. The information processing apparatus 100 determines an incident duration between the start and the end of an incident from image data. The information processing apparatus 100 calculates an attention score and an occlusion factor for each of the plurality of persons from each image frame falling within the incident duration. The attention score indicates the probability of the person having looked toward the incident. The occlusion factor indicates the probability of the gaze of the person having been unobstructed. The information processing apparatus 100 updates the attention score using the occlusion factor. The information processing apparatus 100 then aggregates the updated scores over the incident duration. In this manner, the information processing apparatus 100 calculates a total score for each person.

[0077] FIG. 6 illustrates examples of vectors for calculating the attention score. The information processing apparatus 100 detects an incident 151 and a person 152 from an image frame. The information processing apparatus 100 calculates vectors 153, 154, 155, and 156 for the person 152. Each of the vectors 153, 154, 155, and 156 is a two-dimensional numerical vector indicating a direction on the image frame. The information processing apparatus 100 normalizes each vector 153, 154, 155, and 156 to a length of 1.

[0078] The vector 153 is an incident vector indicating the direction from the person 152 to the incident 151. ui,kdenotes an incident vector calculated from an image frame of time k. The vector 153 indicates the direction from the head of the person 152 to the center of the bounding box of the incident 151. The information processing apparatus 100 creates a skeleton graph for the person 152 by using an image recognition technique described in the above-mentioned non-patent literature documents. The information processing apparatus 100 calculates the vector 153 that is directed from a node representing the head to the center of the incident 151.

[0079] The vector 154 is a head vector indicating the direction that the head of the person 152 faces. uh,kdenotes a head vector calculated from the image frame of time k. The information processing apparatus 100 estimates the posture of the person 152 using the skeleton graph of the person 152 and calculates the vector 154. The starting point of the vector 154 is set at a node representing the head among the nodes included in the skeleton graph.

[0080] The vector 155 is a gaze vector indicating the direction in which the person 152 gazes. ug,kdenotes a gaze vector calculated from the image frame of time k. The information processing apparatus 100 estimates the gaze from the image frame by using an image recognition technique described in the above-mentioned non-patent literature documents.

[0081] The vector 156 is a body vector indicating the direction that the body of the person 152 faces. ub,kdenotes a body vector calculated from the image frame of time k. The information processing apparatus 100 estimates the posture of the person 152 using the skeleton graph of the person 152, and calculates the vector 156. The starting point of the vector 156 is set at a node representing the bottom of the neck among the nodes included in the skeleton graph.

[0082] The information processing apparatus 100 calculates the attention score for the person 152 using the vectors 153, 154, 155, and 156. Equation (1) defines an attention score Akat time k. Equation (2) defines the function f included in Equation (1).

[0083]

[0084]

[0085] The information processing apparatus 100 calculates the inner product ui,k・ub,kof the vectors 153 and 156. The inner product ui,k・ub,kis a numerical value ranging from -1 to 1, inclusive. The inner product ui,k・ub,kindicates the degree to which the body of the person 152 faces toward the incident 151. In addition, the information processing apparatus 100 calculates the value of the function f, which will be described below, using the vectors 153, 154, and 155. The value of the function f is a numerical value ranging from -1 to 1, inclusive. The value of the function f indicates the degree to which the view of the person 152 is directed toward the incident 151.

[0086] The information processing apparatus 100 calculates the sum of the inner product ui,k・ub,kand the value of the function f as the attention score Ak. In the case where the sum is less than 0, the attention score Akis set to 0. The attention score Akis a numerical value ranging from 0 to 2, inclusive. A higher attention score Akindicates a higher probability of the person 152 having witnessed the incident 151 at time k.

[0087] With respect to the function f, the information processing apparatus 100 determines a smaller arc out of the arcs formed by the vector 154 and 155. The information processing apparatus 100 then determines whether the vector 153 lies in the area of the smaller arc. More specifically, the information processing apparatus 100 determines whether the vector 153 is located between the vector 154 and the vector 155.

[0088] In the case where this condition is met, the value of the function f is 1. In the case where the condition is not met, on the other hand, the information processing apparatus 100 calculates the inner product ui,k・uh,kof the vectors 153 and 154 and the inner product ui,k・ug,kof the vectors 153 and 155. The information processing apparatus 100 takes the inner product ui,k・uh,kor the inner product ui,k・ug,k, whichever is greater, as the value of the function f.

[0089] FIG. 7 illustrates an example of searching for an occlusion for calculating the occlusion factor. An image frame 132 has a height h indicting the number of pixels in the vertical direction and a width w indicating the number of pixels in the horizontal direction. The information processing apparatus 100 detects an incident 161, persons 162 and 163, and a forklift 164 from the image frame 132. For the person 162, the information processing apparatus 100 calculates a straight line 165 between the incident 161 and the person 162. In this connection, the information processing apparatus 100 calculates the occlusion factor for the person 163 in the same manner as with the person 162.

[0090] One end of the straight line 165 is set at the center of a bounding box containing the incident 161. The other end of the straight line 165 is set at a node representing the head among the nodes included in the skeleton graph of the person 162. The information processing apparatus 100 calculates the length d of the straight line 165. The length d represents the distance between the incident 161 and the person 162 on the image frame 132.

[0091] The information processing apparatus 100 searches for an occlusion that the straight line 165 crosses among the other persons than the person 162 and the other objects. The occlusion is the one whose bounding box intersects the straight line 165. In the image frame 132, the straight line 165 crosses the person 163 and the forklift 164. Therefore, the person 163 and forklift 164 are occlusions for the person 162.

[0092] In the case where such an occlusion is found, the information processing apparatus 100 calculates the distance between the occlusion and the straight line 165. The information processing apparatus 100 calculates the perpendicular line from the center of the bounding box containing the occlusion to the straight line 165. The length of this perpendicular line is taken as the distance between the occlusion and the straight line 165. In addition, the information processing apparatus 100 determines the type of the occlusion, e.g., person or forklift.

[0093] The information processing apparatus 100 calculates the occlusion factor for the person 162 using the above information. Equation (3) defines an occlusion factor ωkat time k. The occlusion factor ωkis a numerical value ranging from 0 to 1, inclusive. The occlusion factor ωkindicates the degree to which the incident 161 is visible by the person 162. A higher occlusion factor ωkindicates a higher probability that the person 162 has witnessed the incident 161.

[0094]

[0095] In Equation (3), d denotes the length of the straight line 165, h denotes the height of the image frame 132, and w denotes the width of the image frame 132. As the incident 161 and the person 162 are closer to each other, the occlusion factor ωkincreases. In addition, γjdenotes the occlusion factor of a j-th occlusion. As an occlusion allows an easier passage of gaze, the occlusion factor γjincreases. In the case where two or more occlusions are arranged on the straight line 165, the occlusion factor ωkis proportional to the product of the occlusion factors γjof the two or more occlusions.

[0096] The information processing apparatus 100 calculates the occlusion factor γjusing the type of the occlusion and the distance between the occlusion and the straight line 165. For example, the information processing apparatus 100 calculates a type factor ranging from 0 to 1, inclusive, on the basis of the type of the occlusion. In the case where the occlusion has a structure or a material that allows an easy passage of gaze, the type factor is high. As one example, a crate has a shape and material that do not allow gaze to pass through easily, and therefore has a low type factor. As another example, the forklift has some holes and gaps therein, and therefore has a high type factor. The information processing apparatus 100 defines the correspondence between the type of an occlusion and the type factor in advance.

[0097] In addition, for example, the information processing apparatus 100 calculates a distance factor ranging from 0 to 1, inclusive, on the basis of the distance between the occlusion and the straight line 165. The information processing apparatus 100 may normalize the distance to a numerical value ranging from 0 to 1, inclusive, using the size of the bounding box of the occlusion or the size of the image frame 132. As the occlusion and the straight line 165 are further away from each other, the distance factor increases. The information processing apparatus 100 may calculate the product of the above type factor and distance factor as the occlusion factor γj. Note that the user may define the function of calculating the occlusion factor γjfrom the type and distance in advance.

[0098] The information processing apparatus 100 calculates the product of the attention score Akand occlusion factor ωkcalculated from the image frame of time k for each of the plurality of persons. The information processing apparatus 100 calculates the sum of the products ωkAkcalculated over the incident duration for the same person. This sum is taken as the total score. Equation (4) defines the total score Atotal. A person with a higher total score Atotalis more likely to have witnessed the incident. A person with a lower total score Atotalis less likely to have witnessed the incident.

[0099]

[0100] In Equation (4), kstartdenotes the time of the first image frame in which the incident is first detected, and kenddenotes the time of the last image frame in which the incident is last detected. The information processing apparatus 100 sets a margin Δk to cover the risk of incident detection being inaccurate. The margin Δk is set to about a few seconds. Therefore, the start time of the incident duration is the margin Δk before the time kstart. The end time of the incident duration is the margin Δk after the time kend.

[0101] The information processing apparatus 100 sorts the plurality of persons detected in the incident duration in descending order of total scores. The information processing apparatus 100 outputs a list listing the plurality of persons in the sorted order. The investigator conducts interviews with the persons, prioritizing those with higher total scores. In the case where the investigator is able to obtain sufficient information on the incident by the interviews with some persons, the investigator does not need to conduct further interviews with the other persons. The following describes an example of ranking a plurality of persons.

[0102] FIG. 8 illustrates examples of a positional relationship between an incident and a person. The information processing apparatus 100 detects an incident 171, persons 172, 173 and 174, and a forklift 175 from an image frame 133. Although the person 172 is close to the incident 171, the gaze of the person 172 is directed somewhat away from the incident 171. In addition, as an occlusion, the forklift 175 exists between the incident 171 and the person 172.

[0103] Although the person 173 is close to the incident 171, the gaze of the person 173 is directed somewhat away from the incident 171. However, there is no occlusion between the incident 171 and the person 173. The person 174 is further away from the incident 171 than the persons 172 and 173. However, there is no occlusion between the incident 171 and the person 174. In addition, the gaze of the person 174 is directed toward the incident 171.

[0104] FIG. 9 is a table for describing an example of calculating the total score. With respect to each person 172, 173, and 174, the information processing apparatus 100 calculates the attention score and the occlusion factor, and then calculates the total score from the attention score and the occlusion factor. The table 134 contains the attention score, occlusion factor, and total score of each person 172, 173, and 174. The table 134 may also contain the identifier of each person 172, 173, and 174 assigned by ReID. Assume now, for sake of simplicity, that the incident duration includes the image frame 133 with time k = 0 only.

[0105] The gaze of the person 172 is directed somewhat away from the incident 171. Therefore, the attention score of the person 172 is 1.20. The gaze of the person 173 is directed somewhat away from the incident 171. Therefore, the attention score of the person 173 is 1.20. The gaze of the person 174 is directed toward the incident 171. Therefore, the attention score of the person 174 is 2.00, higher than those of the persons 172 and 173.

[0106] In addition, the occlusion exists between the incident 171 and the person 172. Therefore, the occlusion factor of the person 172 is 0.10. There is no occlusion between the incident 171 and the person 173, and the person 173 is close to the incident 171. Therefore, the occlusion factor of the person 173 is 0.80, higher than that of the person 172. There is no occlusion between the incident 171 and the person 174, and the person 174 is far from the incident 171. Therefore, the occlusion factor of the person 174 is 0.50, higher than that of the person 172 and lower than that of the person 173.

[0107] The total score is the product of the attention score and the occlusion factor. Therefore, the total score of the person 172 is 0.12. The total score of the person 173 is 0.96, and the total score of the person 174 is 1.00. As seen in the table 134, the information processing apparatus 100 sorts the three persons in the order of the person 174, the person 173, and the person 172. With respect to the probability of having witnessed the incident 171, the person 174 has the highest, and the person 173 has the second highest. Therefore, the investigator conducts an interview with the person 174 first. In the case where the investigator does not obtain sufficient information, the investigator conducts an interview with the person 173 next.

[0108] FIG. 10 is a table illustrating examples of comparing investigation durations. The table 135 compares the investigation durations between two investigation methods. The first investigation method is to randomly select persons to be interviewed from a plurality of persons detected in image data. The second investigation method is to conduct interviews with the persons, prioritizing persons with higher total scores.

[0109] Here, n denotes the number of detected persons, and dt denotes an interview duration per person. The interview duration includes a personal identification time to identify the name of a person from image data. The table 135 indicates that the interview duration per person is one hour. Assume here that one of the n persons is a key witness with the most important information. The investigator ends the incident investigation when the interview with the key witness is complete. In this connection, the investigator does not know in advance who of the n persons is the key witness.

[0110] Equation (5) defines the expected value of the investigation duration T for the first investigation method. In Equation (5), wkdenotes the k-th person interviewed, widenotes the key witness, and p denotes the probability that the k-th person interviewed is the key witness.

[0111]

[0112] Equation (6) defines the expected value of the investigation duration T for the second investigation method. Assume here that the key witness is in the top third of persons with the highest total scores. This assumption is conservative. In actual, the key witness is highly likely to be in persons who rank higher in total scores. As is clear from Equations (5) and (6), the expected value of the investigation duration T of the second investigation method is less than that of the first investigation method.

[0113]

[0114] As seen in the table 135, in the case of n = 3, the first investigation method has an investigation duration of 2.0 hours, and the second investigation method has an investigation duration of 1.0 hour. Therefore, the time ratio in terms of percentage is 200%. In the case of n = 6, the first investigation method has an investigation duration of 3.5 hours, and the second investigation method has an investigation duration of 1.5 hours. Therefore, the time ratio is 233%. In the case of n = 15, the first investigation method has an investigation duration of 8.0 hours, and the second investigation method has an investigation duration of 3.0 hours. Therefore, the time ratio is 267%. The time ratio here refers to the investigation duration ratio of the first investigation method to the second investigation method. A higher time ratio indicates that the incident investigation of the second investigation method is more efficient than that of the first investigation method. The method of the third embodiment significantly reduces the investigation duration as the number of bystanders at the incident increases.

[0115] The following describes the functions of the information processing apparatus 100, and also describes a procedure for image recognition of ranking persons captured by the surveillance cameras 32 and 33.

[0116] FIG. 11 is a block diagram illustrating examples of functions of the information processing apparatus. The information processing apparatus 100 includes an image storage unit 121, a model storage unit 122, a detection unit 123, a tracking unit 124, and a ranking unit 125. The image storage unit 121 and model storage unit 122 are implemented by using the RAM 102 or HDD 103, for example. The detection unit 123, tracking unit 124, and ranking unit 125 are implemented by using the CPU 101, GPU 104, and programs, for example.

[0117] The image storage unit 121 stores image data received from the surveillance cameras 32 and 33. The image data includes a plurality of image frames taken at different times. The model storage unit 122 stores a trained machine learning model. The machine learning model may be a neural network such as a convolutional neural network. The model storage unit 122 may store a plurality of machine learning models for different detection targets.

[0118] The detection unit 123 feeds each image frame included in the image data to the machine learning model. Thereby, the detection unit 123 detects an incident, persons, and other objects from the image frame. The detection unit 123 assigns each detected entity a bounding box defining the image region containing the detected entity and a class label indicating the type of the detected entity. In addition, the detection unit 123 detects the skeleton of each person from the image frame and creates a skeleton graph. The detection unit 123 also detects the gaze of each person from the image frame.

[0119] The tracking unit 124 compares the bounding boxes included in image frames of different times in terms of their positions and class labels. The tracking unit 124 extracts a plurality of bounding boxes containing the same detected entity and assigns the same identifier to the plurality of bounding boxes. In this manner, the tracking unit 124 identifies the incident, persons and other objects in the image data as a whole. The tracking unit 124 may determine the identity of detected entities across image frames obtained by different surveillance cameras. By doing so, the tracking unit 124 is able to track the movements of persons and other objects across the different surveillance cameras.

[0120] The ranking unit 125 identifies surveillance cameras that have captured the incident. In addition, the ranking unit 125 identifies an incident duration between the start and the end of the incident. The ranking unit 125 calculates the attention score and the occlusion factor for each of the plurality of persons from each image frame received from the surveillance cameras having captured the incident. The ranking unit 125 calculates the sum of the products of the attention score and the occlusion factor over the incident duration as the total score for each of the plurality of persons.

[0121] The ranking unit 125 sorts the plurality of persons in descending order of total scores. The ranking unit 125 outputs a list listing the plurality of persons in the sorted order. The list associates identifiers assigned by the tracking unit 124 with their corresponding total score. The ranking unit 125 may store the list in a non-volatile storage device. Alternatively, the ranking unit 125 may display the list on the display device 111 or may send it to another information processing apparatus.

[0122] FIG. 12 is a flowchart illustrating an example of a procedure for image recognition. At step S10, the detection unit 123 detects the bounding box of each of persons and other objects from an image frame. In addition, the detection unit 123 infers the type of each of the other objects. At step S11, the detection unit 123 detects the skeleton and gaze of each person detected at step S10.

[0123] At step S12, the detection unit 123 detects the bounding box of the incident from the image frame. At step S13, the tracking unit 124 compares bounding boxes in consecutive image frames to track the bounding box containing the same detected entity. The tracking unit 124 assigns the same identifier to the bounding boxes of the same detected entity.

[0124] The ranking unit 125 executes the following steps S14 to S20 on each image frame. At step S14, for each of the plurality of persons, the ranking unit 125 calculates an incident vector indicating the direction from the head to the incident. At step S15, for each of the plurality of persons, the ranking unit 125 calculates a gaze vector indicating the direction in which the person gazes. In addition, for each of the plurality of persons, the ranking unit 125 calculates a head vector indicating the direction that the head faces and a body vector indicating the direction that the body faces. The ranking unit 125 calculates the head vector and body vector using the detected skeleton.

[0125] At step S16, for each of the plurality of persons, the ranking unit 125 calculates the attention score from the four vectors calculated at steps S14 and S15. At step S17, for each of the plurality of persons, the ranking unit 125 calculates a straight line between the head and the incident and calculates the length of the straight line.

[0126] At step S18, for each of the plurality of persons, the ranking unit 125 searches for an occlusion that the straight line obtained at step S17 crosses among the other persons and the other objects. The occlusion is the one whose bounding box intersects the straight line. At step S19, the ranking unit 125 draws a perpendicular line from the center of the bounding box of the occlusion to the straight line calculated at step S17. The ranking unit 125 calculates the length of the perpendicular line as the distance between the occlusion and the straight line. At step S20, for each of the plurality of persons, the ranking unit 125 calculates the occlusion factor on the basis of the type of the occlusion, the distance obtained at step S19, and the length of the straight line obtained at step S17.

[0127] FIG. 13 is a continuation of the flowchart illustrating the example of the procedure for the image recognition. At step S21, the ranking unit 125 determines an incident duration as follows. The ranking unit 125 obtains the time of the first bounding box and the time of the last bounding box among the bounding boxes of the incident assigned the same identifier. The ranking unit 125 then sets the start time of the incident duration to the time that is the margin Δk before the time of the first bounding box. The ranking unit 125 also sets the end time of the incident duration to the time that is the margin Δk after the time of the last bounding box.

[0128] At step S22, for each of the plurality of persons, the ranking unit 125 calculates the product of the attention score and the occlusion factor for each image frame falling within the incident duration. Then, for each of the plurality of persons, the ranking unit 125 calculates the sum of the products across the plurality of image frames falling within the incident duration as the total score. At step S23, the ranking unit 125 sorts the plurality of persons in descending order of total scores. At step S24, the ranking unit 125 outputs a list associating the identifier of each detected person with a corresponding total score. This list lists the plurality of persons in descending order of total scores.

[0129] As described above, the information processing apparatus 100 of the third embodiment automatically detects an incident from image data obtained by surveillance cameras. Thereby, the company is able to detect minor incidents that employees do not report, and is able to develop measures to prevent recurrences. In addition, the information processing apparatus 100 automatically detects bystanders at the incident from the image data obtained by the surveillance cameras. Thereby, the investigator is able to conduct interviews with the bystanders to obtain important information on the incident. In this manner, the information processing apparatus 100 is able to support the incident investigation.

[0130] In addition, the information processing apparatus 100 assigns a total score to each person detected in the image data, and ranks the plurality of persons by total score. The total score indicates the probability that the person has witnessed the incident. Therefore, the investigator is able to conduct interviews with the persons, prioritizing those who are more likely to have important information. This enables the investigator to reduce the number of persons to be interviewed and improves the efficiency of the incident investigation.

[0131] In addition, the information processing apparatus 100 calculates an attention score, and then calculates the total score using the attention score. The attention score indicates the probability of a person having looked toward the incident. The information processing apparatus 100 calculates an incident vector, a gaze vector, a head vector, and a body vector, and then calculates the attention score using inner products of the vectors. This improves the accuracy of estimating the probability of having witnessed the incident.

[0132] Furthermore, the information processing apparatus 100 calculates an occlusion factor, and then calculates the total score using the occlusion factor. The occlusion factor indicates the probability of the gaze of the person having been unobstructed. The information processing apparatus 100 calculates the occlusion factor using the distance between the person and the incident and whether another object exists on the straight line between the person and the incident. This improves the accuracy of estimating the probability of having witnessed the incident.

[0133] 10, 20: Information processing apparatus 11, 21: Storage unit 12, 22: Processing unit 13, 23: Image data 14, 24: Incident 15a, 15b, 25a, 25b: Person 16a, 16b, 17a, 17b: Vector 18a, 18b, 28a, 28b: Score 26: Object 27a, 27b: Straight line

Claims

1. An image recognition method executed by a computer, the image recognition method comprising: detecting an incident and a plurality of persons from image data; calculating, for each person of the plurality of persons, a first vector indicating a first direction that the each person faces and a second vector indicating a second direction from the each person to the incident; and assigning a score to the each person using the first vector and the second vector, the score indicating a probability that the each person has witnessed the incident.

2. The image recognition method according to claim 1, wherein the assigning includes calculating the score using an inner product of the first vector and the second vector.

3. The image recognition method according to claim 1, wherein the first vector includes a third vector indicating a third direction in which the each person gazes, a fourth vector indicating a fourth direction that a head of the each person faces, and a fifth vector indicating a fifth direction that a body of the each person faces.

4. The image recognition method according to claim 1, further comprising: detecting one or more objects other than the plurality of persons from the image data; calculating a straight line between the each person and the incident; and searching for an object that the straight line crosses among the one or more objects, wherein the assigning includes calculating the score using the first vector, the second vector, and a result of the searching.

5. The image recognition method according to claim 4, wherein the assigning includes reducing, in response to the object being found, the score depending on a type of the object and a distance between the straight line and a center of the object.

6. The image recognition method according to claim 1, further comprising sorting the plurality of persons by the score.

7. An image recognition method executed by a computer, the image recognition method comprising: detecting an incident, a plurality of persons, and one or more objects other than the plurality of persons from image data; calculating, for each person of the plurality of persons, a straight line between the each person and the incident; searching for an object that the straight line crosses among the one or more objects; and assigning a score to the each person using a result of the searching, the score indicating a probability that the each person has witnessed the incident.

8. An image recognition program that causes a computer to execute a process comprising: detecting an incident and a plurality of persons from image data; calculating, for each person of the plurality of persons, a first vector indicating a first direction that the each person faces and a second vector indicating a second direction from the each person to the incident; and assigning a score to the each person using the first vector and the second vector, the score indicating a probability that the each person has witnessed the incident.

Images