Information processing program, information processing method, and information processing device

Abstract

To solve the problem in which it is not easy to accurately determine whether or not an abnormality has occurred between persons from a video because a person at a dead angle of a monitoring camera cannot be detected although there is provided a technique for identifying a person where an abnormality such as a stray child and kidnapping has occurred, that is, a child, from monitoring videos in various facilities in a store, etc. by image recognition by a computer, and notifying an alert indicating an abnormality.SOLUTION: An information processing program causes a computer to perform processing of: acquiring a video photographed by one or more camera devices; specifying a relationship obtained by identifying an interaction between a plurality of persons included in the video by analyzing the acquired video; and determining whether or not an abnormality has occurred between the plurality of persons outside a photographing range of the camera device on the basis of the specified relationship.SELECTED DRAWING: Figure 21

Description

[Technical field]

[0001] The present invention relates to an information processing program, an information processing method, and an information processing device for identifying a person exhibiting abnormal behavior from within an image and that behavior. [Background technology]

[0002] For example, there is a technology that uses computer image recognition to identify people who have become lost or kidnapped, i.e., children, from surveillance footage of various facilities such as stores, and send out an alert to notify them of the abnormality. This can prevent accidents and incidents from occurring.

[0003] Such technology, for example, uses a machine learning model to extract a bounding box (Bbox) that surrounds an area containing a person from a video, and determines whether an abnormality has occurred between people, such as a parent and child, based on the positional relationship between the two Bboxes. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-165483 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, since it is not possible to detect people in the blind spots of a surveillance camera, it is not easy to accurately determine whether an abnormality has occurred between people from the video. Note that the blind spots of a surveillance camera may be, for example, blind spots that occur between the shooting ranges of multiple cameras when multiple cameras are shooting different areas, or blind spots that occur outside the shooting range of a single camera, such as a panning camera.

[0006] According to one aspect, an object of the present invention is to provide an information processing program, an information processing method, and an information processing device that can more accurately determine that an abnormality has occurred among a plurality of people from a video. [Means for solving the problem]

[0007] In one embodiment, the information processing program causes a computer to execute a process of acquiring video footage captured by one or more camera devices, analyzing the acquired video footage to identify relationships that identify interactions between multiple people contained in the video footage, and determining whether an abnormality has occurred between the multiple people outside the capture range of the camera devices based on the identified relationships. Effect of the Invention

[0008] On the one hand, it can more accurately determine when an abnormality has occurred between multiple people in a video feed. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of shooting by a surveillance camera. [Diagram 2] FIG. 2 is a diagram showing another example of photography by a surveillance camera. [Diagram 3] FIG. 3 is a diagram showing an example of the configuration of the information processing system 1 according to the present embodiment. [Figure 4] FIG. 4 is a diagram showing an example of the configuration of the information processing device 10 according to the present embodiment. [Diagram 5] FIG. 5 is a diagram showing an example of information stored in the camera installation DB 14 according to the present embodiment. [Figure 6] FIG. 6 is a diagram showing an example of information stored in the rule DB 16 according to the present embodiment. [Figure 7] FIG. 7 is a diagram showing an example of relationship specification using a scene graph according to the present embodiment. [Figure 8] FIG. 8 is a diagram for explaining a problem in generating a scene graph. [Figure 9]FIG. 9 is a diagram showing an example of attention map generation according to the present embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a NN configuration for estimating a relationship according to this embodiment. [Figure 11] FIG. 11 is a diagram showing an example of image feature extraction according to this embodiment. [Figure 12] FIG. 12 is a diagram showing an example of object detection according to the present embodiment. [Figure 13] FIG. 13 is a diagram showing an example of paired feature generation according to the present embodiment. [Figure 14] FIG. 14 is a diagram showing an example of relationship feature extraction according to the present embodiment. [Figure 15] FIG. 15 is a diagram showing an example of relationship estimation according to the present embodiment. [Figure 16] FIG. 16 is a diagram showing an example of skeleton information according to this embodiment. [Figure 17] FIG. 17 is a diagram showing an example of whole-body posture determination according to this embodiment. [Figure 18] FIG. 18 is a diagram showing an example of detection of a motion for each part according to the present embodiment. [Figure 19] FIG. 19 is a diagram showing an example of calculation of projective transformation coefficients according to this embodiment. [Figure 20] FIG. 20 is a diagram showing an example of conversion from the image coordinate system to the floor map coordinate system according to this embodiment. [Figure 21] FIG. 21 is a flowchart showing the flow of the abnormality occurrence determination process according to this embodiment. [Figure 22] FIG. 22 is a flowchart showing the flow of the relationship estimation process according to this embodiment. [Diagram 23] FIG. 23 is a diagram illustrating an example of a hardware configuration of the information processing device 10. As shown in FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Below, examples of the information processing program, the information processing method, and the information processing device according to the present embodiment will be described in detail with reference to the drawings. Note that the present embodiment is not limited to these examples. In addition, each example can be appropriately combined within a range that does not cause inconsistency.

[0011] First, an explanation will be given of the shooting by a surveillance camera and the blind spots of the shooting range. Fig. 1 is a diagram showing an example of shooting by a surveillance camera. Fig. 1 shows an example in which the inside of various facilities such as a store is shot by multiple surveillance cameras, camera devices 110-1 to 110-7 (hereinafter collectively referred to as "camera device 110"). In Fig. 1, camera device 110 is a fixed camera, and the approximate shooting range of camera device 110 is the part indicated by dots.

[0012] Although it depends on the number of camera devices 110 installed and the size of the facility to be photographed, there are some blind spots in the photographing range of the camera device 110. In FIG. 1, for example, the position where the person S and the person C are located is a blind spot between the photographing ranges that cannot be captured by any of the photographing ranges of the camera devices 110-1 to 110-7. For example, even if an abnormality occurs between the person S and the person C at such a position, it is difficult to determine whether an abnormality has occurred between the person S and the person C by a conventional method that detects a person from the image of the camera device 110 and performs image recognition. Here, an abnormality between the person S and the person C is, for example, a case where the person C is a child and the person S is an adult who is unrelated to the person C, and the person S is trying to kidnap the person C.

[0013] Fig. 2 is a diagram showing another example of shooting by a surveillance camera. Like Fig. 1, Fig. 2 shows camera device 120, which is a surveillance camera that shoots images inside various facilities such as stores, but camera device 120 is a so-called panning camera. Therefore, as shown in Fig. 2, the shooting range of camera device 120 moves within the range shown by the dashed line, for example.

[0014] In this way, since the shooting range of the camera device 120 moves, a position that was shot at one time may be outside the shooting range depending on the timing, becoming a blind spot. For example, on the left side of FIG. 2, person P and person C are captured in the shooting range of the camera device 120, but on the right side of FIG. 2, person C cannot be captured due to the movement of the shooting range. For example, even if an abnormality occurs between person P and person C at a timing such as the right side of FIG. 2, it is difficult to determine whether an abnormality has occurred between person P and person C in the conventional method, as in the case of the multiple camera devices 110 in FIG. 1. Here, an abnormality between person P and person C is, for example, a case where person C is a child, person P is the parent of person C, and person C is lost and separated from person P.

[0015] (Overall composition) Next, an information processing system for carrying out this embodiment will be described. Fig. 3 is a diagram showing an example of the configuration of information processing system 1 according to this embodiment. As shown in Fig. 3, information processing system 1 is a system in which information processing device 10 and camera devices 100-1 to 100-n (n is an arbitrary natural number) are connected to each other so as to be able to communicate with each other via network 50. Note that camera devices 100-1 to 100-n are hereinafter collectively referred to as "camera devices 100".

[0016] The network 50 may be any of a variety of communication networks, whether wired or wireless, such as an intranet or the Internet used within various facilities, such as within a store. The network 50 may not be a single network, but may be configured, for example, by connecting an intranet and the Internet via a network device such as a gateway or other device (not shown). The expression "within a facility" is not limited to indoors and may include outdoors.

[0017] The information processing device 10 is, for example, an information processing device such as a desktop PC (Personal Computer) or a notebook PC that is installed in various facilities such as a store and used by security guards, or a server computer. Alternatively, the information processing device 10 may be a cloud computer device managed by a service provider that provides cloud computing services.

[0018] The information processing device 10 receives, for example, video captured by the camera device 100 of a predetermined shooting range in various facilities such as a store from the camera device 100. Strictly speaking, the video is composed of a plurality of captured images captured by the camera device 100, that is, a series of frames of a video.

[0019] Furthermore, the information processing device 10 extracts people in various facilities such as a store from the video captured by the camera device 100, for example, using an existing object detection technology. Note that extracting a person from the video may be, for example, extracting a bounding box (Bbox) that encloses an area including the person from the video in a rectangular shape. Furthermore, the information processing device 10 specifies a relationship that identifies an interaction between people, such as people holding hands, walking together, talking, etc.

[0020] Furthermore, the information processing device 10 determines whether or not an abnormality has occurred between the multiple persons based on, for example, the relationships between the multiple identified persons. The abnormality between the multiple persons here refers to, for example, a child getting lost or being kidnapped, and in particular, the information processing device 10 determines whether or not an abnormality has occurred between the multiple persons outside the shooting range of the camera device 100 based on the relationships between the multiple identified persons.

[0021] Then, when the information processing device 10 determines that an abnormality has occurred among a plurality of people, for example, it notifies an alert. Note that the alert is merely a warning, and the occurrence of an abnormality may include, for example, the possibility that an abnormality has occurred. The alert may be, for example, a voice output or a message notification on a screen. The destination of the alert may be, for example, an output device included in the information processing device 10, an external device, or another output device communicably connected to the information processing device 10 via the network 50.

[0022] Furthermore, the information processing device 10 may specify the location where the abnormality occurred based on, for example, the installation location of the camera device 100 that captured the video in which the relationships between multiple people were specified, and limit the recipients of the alert. Note that limiting the recipients of the alert means, for example, limiting the notification of the alert to an information processing terminal carried by a security guard near the location where the abnormality occurred, or a PC installed near the location where the abnormality occurred.

[0023] Security guards in stores and other facilities can receive an alert and check the location and person where the abnormality occurred, and by speaking to or warning the person, they can prevent or resolve issues such as a child getting lost or being kidnapped.

[0024] Although the information processing device 10 is shown as one computer in FIG. 3, it may be a distributed computing system configured with a plurality of computers.

[0025] The camera device 100 is, for example, a surveillance camera installed in various facilities such as a store. The camera device 100 may be, for example, camera device 110 which is a plurality of surveillance cameras as described with reference to Figs. 1 and 2, or may be camera device 120 which is a panning camera. Note that, although Fig. 3 shows a plurality of camera devices 100, depending on the scale of the shooting range or when the camera device 100 is a panning camera, there may be only one camera device. The camera device 100 transmits, for example, the captured video to the information processing device 10.

[0026] (Functional configuration of information processing device 10) Next, a description will be given of the functional configuration of the information processing device 10. Fig. 4 is a diagram showing an example of the configuration of the information processing device 10 according to the present embodiment. As shown in Fig. 4, the information processing device 10 has a communication unit 11, a storage unit 12, and a control unit 20.

[0027] The communication unit 11 is a processing unit that controls communication with other devices such as the camera device 100, and is, for example, a communication interface such as a network interface card.

[0028] The storage unit 12 has a function of storing various data and programs executed by the control unit 20, and is realized by a storage device such as a memory or a hard disk. The storage unit 12 stores an imaging DB 13, a camera installation DB 14, a model DB 15, and a rule DB 16. DB is an abbreviation for database.

[0029] The imaging DB 13 stores a plurality of captured images which are a series of frames captured by the camera device 100. The plurality of captured images, i.e., videos captured by the camera device 100, are transmitted from the camera device 100 at any time, received by the information processing device 10, and stored in the imaging DB 13.

[0030] The camera installation DB 14 stores, for example, information for identifying the location where each of the camera devices 100 is installed. The information stored here may be set in advance by, for example, an administrator of the information processing system 1.

[0031] Fig. 5 is a diagram showing an example of information stored in the camera installation DB 14 according to the present embodiment. As shown in Fig. 5, the camera installation DB 14 stores information such as "camera ID, installation location" in association with each other. In the "camera ID" stored here, for example, information such as an identifier for uniquely identifying each of the camera devices 100 is set, and in the "installation location", for example, information for specifying the location where each of the camera devices 100 is installed is set. Note that in cases where only one camera device 100 is installed, the storage unit 12 does not need to include the camera installation DB 14.

[0032] The model DB15 stores, for example, information on a machine learning model for identifying an area including a person and a relationship between multiple people from a video captured by the camera device 100, and model parameters for constructing the machine learning model. The machine learning model is generated by machine learning using, for example, the video captured by the camera device 100, i.e., the captured image, as input data, and the area including a person and the type of relationship between multiple people as correct answer labels. Note that the type of relationship between multiple people may be, for example, people holding hands, walking together, talking, etc., but is not limited to these. In addition, the area including a person may be, for example, a bounding box (Bbox) that surrounds the area in a rectangular shape on the captured image.

[0033] The model DB 15 also stores, for example, information on a machine learning model for acquiring the types of objects including people and the relationships between objects from a video to generate a scene graph, and model parameters for constructing the model. Note that the types of objects for generating a scene graph are sometimes called "classes" and the relationships between objects are sometimes called "relation." The machine learning model is generated by machine learning using the video captured by the camera device 100, i.e., the captured image, as input data, and the locations (Bboxes) of objects, the types of objects, and the relationships between objects included in the captured image as correct answer labels.

[0034] The model DB 15 also stores, for example, information about a machine learning model for generating an Attention map described later and model parameters for constructing the model. The machine learning model is generated by training, for example, the feature amount of an object including a person detected from a captured image as input data and an important area on the image as a correct answer label. The training and generation of various machine learning models may be executed by the information processing device 10 or may be executed by another information processing device.

[0035] The rule DB 16 stores information about rules for determining that an abnormality has occurred among a plurality of people, for example. The information stored here may be set in advance by, for example, an administrator of the information processing system 1.

[0036] FIG. 6 is a diagram showing an example of information stored in the rule DB 16 according to the present embodiment. As shown in FIG. 6, the rule DB 16 stores information such as "rule ID, person, relationship" in association with each other. The "rule ID" stored here is set with information such as an identifier for uniquely identifying the rule. In addition, the "person" is set with a plurality of types of people, such as "adult and child". In addition, the "relationship" is set with a type of relationship between a plurality of people, such as holding hands, walking together, talking, etc.

[0037] For example, when multiple people are detected from a video and the multiple people set in "people" in the rule DB 16 show the relationship set in "relationship", the information processing device 10 can determine that an abnormality may occur between the multiple people. More specifically, for example, as shown in rule ID=1 in FIG. 6, it is assumed that the multiple people detected from the video are "adults and children" and the relationship between the multiple people is "holding hands". In this case, the information processing device 10 can determine that an abnormality may occur between, for example, "adults and children" who show the relationship of "holding hands".

[0038] Then, for example, assume that a child of "adult and child" who has shown the relationship of "holding hands" shows a relationship set in the "relationship" of the rule DB 16, such as "holding hands" or "walking together" with another adult in a video chronologically later than the video. In this case, the information processing device 10 can determine that an abnormality, such as abduction for kidnapping, has occurred between multiple people detected from the video, i.e., between an adult and a child, and notify an alert. In addition, in the case of such a determination, for example, when a mother and a child hold hands and the child then holds hands with a grandfather, it may be determined that an abnormality has occurred. However, the occurrence of an abnormality may include the possibility that an abnormality has occurred, and for example, a security guard who has received a notification of an alert can check the place where the abnormality has occurred and the person, thereby preventing the occurrence of an abnormality, etc.

[0039] Also, for example, assume that a child of an "adult and child" who shows a relationship of "holding hands" has been shown to be alone after the current video in which the relationship has been dissolved. In this case, the information processing device 10 can determine that an abnormality such as a child being lost has occurred between multiple people detected from the video, i.e., the adult and the child, and can issue an alert.

[0040] Moreover, the above information stored in the storage unit 12 is merely an example, and the storage unit 12 can store various information other than the above information.

[0041] The control unit 20 is a processing unit, such as a processor, that controls the entire information processing device 10. The control unit 20 includes an acquisition unit 21, a specification unit 22, a determination unit 23, and a notification unit 24. Each processing unit is an example of an electronic circuit included in the processor or an example of a process executed by the processor.

[0042] The acquisition unit 21 acquires, for example, images of various facilities such as the inside of a store captured by one or more camera devices 100 from the imaging DB 13. The images captured by the camera devices 100 are transmitted from the camera devices 100 to the information processing device 10 as needed, received by the information processing device 10, and stored in the imaging DB 13.

[0043] The identification unit 22, for example, analyzes the video acquired by the acquisition unit 21 to identify the relationship that identifies the interaction between multiple people included in the video. Each of the multiple people included in the video may be an area on the video that includes each person, such as a first area including person A and a second area including person B. The area may be, for example, a bounding box (Bbox). The relationship between the multiple people that is identified may include types of relationships such as people holding hands, walking together, talking, etc. The process of identifying the relationship that identifies the interaction between multiple people may include, for example, a process of inputting the video acquired by the acquisition unit 21 into a machine learning model to generate a scene graph that identifies the relationship for each person included in the video. The generation of the scene graph will be described more specifically with reference to FIG. 7.

[0044] Fig. 7 is a diagram showing an example of relationship specification by a scene graph according to the present embodiment. The captured image 150 in Fig. 7 is, for example, one frame of a video captured by the camera device 100. The specification unit 22 detects objects including a person from the captured image 150 using an existing detection algorithm, for example, estimates the relationship between each object, and generates a scene graph 159 expressing each object and the relationship therebetween, that is, a context. Here, the existing detection algorithm is, for example, YOLO (YOU Only Look Once), SSD (Single Shot Multibox Detector), RCNN (Region Based Convolutional Neural Networks), etc.

[0045] In the example of FIG. 7, at least two men indicated by Bbox 151 and 152, a woman indicated by Bbox 153, a box indicated by Bbox 154, and a shelf indicated by Bbox 155 are detected from the captured image 150. Then, the identification unit 22, for example, cuts out the Bbox region of each object including a person from the captured image 150, extracts the feature amount of each region, estimates the relationship of each object from the feature amount of the pair of objects (Subject, Object), and generates a scene graph 159. In FIG. 7, the scene graph 159 indicates, for example, the relationship that the man indicated by Bbox 151 is standing on the shelf indicated by Bbox 155. Also, for example, the relationship to the man indicated by Bbox 151 indicated by the scene graph 159 is not limited to one. 7, the scene graph 159 indicates all the estimated relationships, such as a shelf, being behind the man indicated by Bbox 152, and holding a box indicated by Bbox 154. In this way, the identification unit 22 can identify the relationships between multiple people included in the video by generating a scene graph.

[0046] However, there are problems with scene graphs, and by solving these problems, the identification unit 22 can more accurately identify the relationships between multiple people included in a video.

[0047] FIG. 8 is a diagram for explaining problems with scene graph generation. In scene graph generation, for example, as shown in FIG. 8, individual objects are extracted from a captured image 170, and therefore, important information in the context outside the areas of Bbox 171 of Subject and Bbox 172 of Object may be overlooked. More specifically, as shown on the right side of FIG. 8, the areas of Bbox 171 of a curtain and Bbox 172 of a cat are extracted, but information of areas other than these is not used when estimating the relationship between the cat and the curtain. Therefore, important information may be overlooked when estimating the relationship between the cat and the curtain. For example, by looking at the objects and the room structure around the cat and the curtain, it can be estimated that the curtain is behind the cat, but in general scene graph generation, surrounding information is not used when estimating the relationship between the cat and the curtain. Therefore, by using surrounding information when estimating the relationship between objects, it becomes possible to specify the relationship more accurately. For example, in the example of Figure 8, it is possible to infer from the captured image 170 that there is an apple next to the cat, and that there is a high possibility that a table is underneath the cat and the apple, and that the curtain is hanging on the door, and therefore the curtain is behind the cat.

[0048] Therefore, in this embodiment, contextually important regions are adaptively extracted from the entire image for each subject and object for which a relationship is to be estimated, and the relationship between the targets is recognized. The extraction of regions important for relationship recognition is realized, for example, by generating a map (hereinafter referred to as an "Attention map") that takes a value between 0 and 1 depending on the importance.

[0049] FIG. 9 is a diagram showing an example of Attention map generation according to the present embodiment. As shown in FIG. 9, when estimating the relationship between a curtain as a Subject and a cat as an Object, the identification unit 22 generates an Attention map 180 expressing an important area other than the areas of Bboxes 171 and 172, and uses the Attention map 180 to estimate the relationship. The Attention map 180 is generated using a machine learning model generated by training, for example, the feature amount of an object detected from a captured image as input data and an important area on the image as a correct answer label, so that the recognition rate of the relationship between the objects is high. In addition, as shown in FIG. 9, for example, the Attention map 180 expresses the importance of an important area in estimating the relationship between a Subject and an Object by color, shading, or the like. Note that since it is not known in advance where the contextually important area is located in the captured image, the identification unit 22 generates the Attention map 180, for example, for the entire image. This allows the identification unit 22 to extract the contextually important area more accurately, and thus to identify the relationship between the objects more accurately.

[0050] 10 to 15, the estimation of the relationship between the objects using the attention map 180 will be described in more detail. The estimation of the relationship between the objects using the attention map 180 may be performed using, for example, a neural network (NN).

[0051] FIG. 10 is a diagram showing an example of a NN configuration for relationship estimation according to the present embodiment. As shown in FIG. 10, the NN 40 for estimating the relationship between objects using the Attention map 180 has, for example, an image feature extraction unit 41, an object detection unit 42, a pair feature generation unit 43, a relationship feature extraction unit 44, and a relationship estimation unit 45. For example, when a video of a scene, that is, a captured image, is input, the NN 40 starts with processing by the image feature extraction unit 41, and outputs a Bbox of an object detected by the object detection unit 42 and a type (class) of the object. In addition, the NN 40 outputs the relationship (relation) of each object detected by the object detection unit 42 estimated by the relationship estimation unit 45. The processing by each processing unit included in such an NN 40 will be described for each processing.

[0052] First, feature extraction from a captured image performed by the image feature extraction unit 41 will be described. FIG. 11 is a diagram showing an example of image feature extraction according to this embodiment. As shown in FIG. 11, the image feature extraction unit 41 extracts, for example, an image feature amount 181 of a C channel from a captured image 170 of three RGB channels. The image feature extraction unit 41 may be configured, for example, by a convolutional neural network (CNN) or a transformer, which are existing technologies. In addition, the CNN or the transformer may be one that has previously learned a classification task or the like on a large-scale image data set such as ImageNet. In addition, the width W and height H of the image feature amount are generally smaller than the width and height of the input image, that is, the captured image 170 in the example of FIG. 11, due to the CNN or pooling process.

[0053] Next, object detection from image features executed by the object detection unit 42 will be described. FIG. 12 is a diagram showing an example of object detection according to the present embodiment. As shown in FIG. 12, the object detection unit 42 receives, for example, image features 181 extracted by the image feature extraction unit 41, and outputs Bboxes 171-175 of each object included in the captured image 170 and a class indicating the type of the object. In the example of FIG. 12, the Bboxes 171-175 are written as bbox. As an output example, the object detection unit 42 can draw and output the Bboxes 171-174 of each detected object superimposed on the captured image, as shown on the right side of FIG. 12. In addition, a class indicating the type of each object, such as a curtain or a cat, may be output near each Bbox.

[0054] The rectangle of the Bbox can be expressed by four real values, such as the upper left coordinate (x1, y2) and the lower right coordinate (x2, y2) of the rectangle. The class output from the object detection unit 42 is, for example, a probability value that the object detected in the Bbox is a predetermined object to be detected. More specifically, for example, if the object to be detected is {cat, table, car} (cat, table, car), in the example of FIG. 12, the class of the Bbox 172 corresponding to the cat is a probability value such as (0.9, 0.1, 0.2) that the detected object is each of the objects to be detected. That is, the class in this case indicates that the probability that the detected object is a cat, a table, and a car is 0.9, 0.1, and 0.2, respectively. The object detection unit 42 may be configured, for example, using Faster Region Based Convolutional Neural Networks (R-CNN), Mask R-CNN, DEtection TRansformer (DETR), or the like, which are NNs for object detection based on existing technology.

[0055] Next, the feature amount of each pair of detected objects executed by the pair feature amount generating unit 43 will be described. FIG. 13 is a diagram showing an example of pair feature amount generation according to the present embodiment. As shown in FIG. 13, the pair feature amount generating unit 43 extracts the feature amount of each detected object for each object based on the object detected by the object detection unit 42, that is, the output class and Bbox. At this time, the pair feature amount generating unit 43 can use RoIAlign in Mask R-CNN, etc. Also, when the object detection unit 42 is configured by DETR, the feature amount for each object output by the Transformer decoder can be used as it is. Also, the extracted feature amount can be, for example, a real vector of 256 dimensions for each object.

[0056] Then, the paired feature generation unit 43 pairs all combinations of detected objects with one object as a Subject and the other object as an Object. Paired feature 182 shown on the right side of Fig. 13 is an arrangement of features of Subject and Object for each pair. In order to adjust the number of dimensions of paired feature 182, the paired feature generation unit 43 may convert the features of the paired Subject and Object by MLP (Multi Layer Perceptron), which is an existing technology.

[0057] Next, extraction of features indicating the relationship between detected and paired objects, which is executed by the relationship feature extraction unit 44, will be described. Fig. 14 is a diagram showing an example of relationship feature extraction according to this embodiment. As shown in Fig. 14, the relationship feature extraction unit 44 has, for example, a conversion unit (1), a conversion unit (2), an attention map generation unit, and a synthesis unit.

[0058] First, as shown in Fig. 14, the relationship feature extraction unit 44 converts the image feature amount 181 extracted by the image feature extraction unit 41 by the conversion unit (1) and the conversion unit (2) for use in subsequent processing. The conversion unit (1) and the conversion unit (2) may be realized by, for example, an MLP (Multi Layer Perceptron). In the example of Fig. 14, the conversion unit (1) and the conversion unit (2) are distinguished as 1 and 2 because they are realized by MLPs with different weights.

[0059] Next, the relationship feature extraction unit 44 generates an Attention map 180 by correlating the paired features 182 generated by the paired feature generation unit 43, row by row, with the image features converted by the conversion unit (1) using the Attention map generation unit. Note that the paired features 182, row by row, means for each pair of Subject and Object. In addition, the relationship feature extraction unit 44 may convert the Attention map 180 by MLP or layer normalization after correlating the paired features 182 with the image features converted by the conversion unit (1).

[0060] Here, the correlation process between a certain pair feature 182 and the image feature converted by the conversion unit (1) will be described in more detail. It is assumed that the pair feature 182 has been adjusted to a C-dimensional vector by the previous process. It is also assumed that the image feature converted by the conversion unit (1) is a tensor of H×W dimension with a C-dimensional channel direction. It is also assumed that a pixel (x, y) having the image feature converted by the conversion unit (1) is focused on and set as a pixel of interest. Since the pixel of interest is 1×1×C, it can be regarded as a C-dimensional vector. Then, the attention map generation unit calculates a correlation value (scalar) between the C-dimensional vector of the pixel of interest and the pair feature 182 adjusted to the C-dimensional vector. This determines the correlation value at the pixel of interest (x, y). The attention map generation unit executes this process for all pixels to generate an attention map 180 of H×W×1.

[0061] Then, the relationship feature extraction unit 44 extracts features of important regions in the entire image corresponding to the Subject and Object pair by multiplying the generated Attention map 180 by the image features converted by the conversion unit (2) to obtain a weighted sum. Note that since the weighted sum is obtained for the entire image, the feature obtained by the weighted sum becomes a C-dimensional feature for one Subject and Object pair.

[0062] In addition, the weighted sum of the Attention map 180 and the image feature converted by the conversion unit (2) will be described more specifically. Note that the image feature converted by the conversion unit (2) is assumed to be a tensor of H×W×C. First, the relationship feature extraction unit 44 multiplies the image feature converted by the conversion unit (2) by the Attention map 180. At this time, since the Attention map 180 is H×W×1, the channel is copied to the C dimension. In addition, the relationship feature extraction unit 44 adds all the C-dimensional vectors of each pixel for the multiplication. As a result, one C-dimensional vector is generated. That is, one C-dimensional vector is generated for one Attention map 180. Furthermore, in reality, the Attention map 180 is generated in the number of pair features 182, so the number of C-dimensional vectors generated is the same as the number of pair features 182. Through the above processing, the relationship feature extraction unit 44 takes a weighted sum of the image feature amounts converted by the conversion unit (2) using the Attention map 180 as a weight.

[0063] Then, the relationship feature extraction unit 44 uses a synthesis unit to synthesize the feature of the important region extracted by the Attention map 180 and the pair feature 182 generated by the pair feature generation unit 43, and outputs the synthesized feature as the relationship feature 183. More specifically, the relationship feature extraction unit 44 can use the feature of the important region and the pair feature 182 concatenated in the dimensional direction. In addition, after concatenating the feature of the important region and the pair feature 182, the relationship feature extraction unit 44 may convert the concatenated feature by MLP or the like to adjust the number of dimensions.

[0064] Next, the estimation of the relationship of each pair of Subject and Object executed by the relationship estimation unit 45 will be described. FIG. 15 is a diagram showing an example of the relationship estimation according to this embodiment. The relationship estimation unit 45 receives the relationship feature amount 183 output by the relationship feature extraction unit 44, and outputs a relationship corresponding to each pair of Subject and Object. The relationship estimation unit 45 may be configured, for example, by MLP or Batch normalization, which are existing technologies. The output relationship is a probability value that the relationship is a predetermined relationship to be estimated. Specifically, for example, when the relationship to be estimated is {on, behind, hold}, in the example of FIG. 15, a probability value for each relationship such as (0.1, 0.9, 0.2) is output as the relationship between a curtain and a cat. In this case, the identification unit 22 can identify "behind" with the highest probability value as the relationship between the curtain and a cat.

[0065] The above-described processes for estimating the relationships between objects using the attention map 180 are summarized as a process for identifying relationships between multiple people, which is executed by the identification unit 22 using the NN 40.

[0066] First, the identification unit 22 extracts, for example, a first feature amount corresponding to a first region including an object included in the video or a second region including a person included in the video from the video. For example, the video may be a video captured by the camera device 100 inside various facilities such as a store, and the first region and the second region may be Bboxes. Moreover, such an extraction process corresponds to a process of extracting an image feature amount 181 from a captured image 170 by the image feature extraction unit 41 as described with reference to FIG. 11. That is, in the example of FIG. 11, the captured image 170 corresponds to the video, and the image feature amount 181 corresponds to the first feature amount.

[0067] Next, the identification unit 22 detects, for example, objects and people included in the video from the extracted first feature amount. Such object and person detection processing corresponds to the processing by the object detection unit 42 to detect the Bbox and class of the object and person from the image feature amount 181 corresponding to the first feature amount as described with reference to FIG.

[0068] Next, the identification unit 22 generates a second feature amount by combining, for example, the first feature amounts of the detected objects, the detected people, and the objects or people in at least one pair of the object and the person. Such a generation process corresponds to the process of generating paired feature amount 182 by the paired feature amount generation unit 43, in which the feature amounts of the detected objects and people corresponding to the first feature amounts are arranged by pair, as described with reference to Fig. 13. That is, in the example of Fig. 13, the paired feature amount 182 corresponds to the second feature amount.

[0069] Next, the identification unit 22 generates a first map indicating a first relationship that identifies a plurality of objects, a plurality of people, and at least one interaction between an object and a person, based on, for example, the first feature amount and the second feature amount. Such a generation process corresponds to a process of generating an Attention map 180 based on an image feature amount 181 corresponding to the first feature amount and a pair feature amount 182 corresponding to the second feature amount, performed by the relationship feature extraction unit 44, as described with reference to FIG. 14. That is, in the example of FIG. 14, the Attention map 180 corresponds to the first map.

[0070] Next, the identification unit 22 extracts a fourth feature based on, for example, a third feature obtained by converting the first feature and the first map. Such an extraction process corresponds to the process of extracting a relationship feature 183 by the relationship feature extraction unit 44 based on the feature converted by the conversion unit (2) and an Attention map 180 corresponding to the first map, as described with reference to FIG. 14. That is, in the example of FIG. 14, the feature converted by the conversion unit (2) is a feature obtained by converting an image feature 181 corresponding to the first feature by the conversion unit (2), and corresponds to the third feature, and the relationship feature 183 corresponds to the fourth feature.

[0071] Then, the identification unit 22 identifies a relationship that identifies an interaction between a plurality of people included in the video from, for example, the fourth feature amount. Such identification processing corresponds to the processing by the relationship estimation unit 45 of estimating and identifying a relationship between an object and a person from a relationship feature amount 183 corresponding to the fourth feature amount as described with reference to FIG.

[0072] In addition, based on the video acquired by the acquisition unit 21, the identification unit 22 identifies a first person from among multiple people included in the video, with whom the relationship between the identified multiple people transitions from a first relationship to a second relationship in time series.

[0073] Here, for example, the first relationship is a relationship between an adult and a child, such as holding hands, walking together, talking, etc., which is set in the rule DB 16. Also, for example, the second relationship may be a relationship similar to the first relationship, but is a relationship between an adult different from the first relationship. That is, since the adult showing the first relationship is different from the adult showing the second relationship, the relationship has not transitioned to the second relationship, and the first person who transitions from the first relationship to the second relationship in time series is a child. Therefore, the identification unit 22 identifies, for example, a child who has been shown to have a second relationship with an adult different from the adult who has been shown to have the first relationship as the first person. These explanations are examples of abduction for kidnapping.

[0074] On the other hand, in the case of a lost child, the first relationship is, for example, a relationship between an adult and a child, such as holding hands, walking together, talking, etc., as in the case of abduction for kidnapping. However, the second relationship is, for example, a relationship in which the first relationship is dissolved and the child is alone with the adult who was with the child. In this case, the identification unit 22 identifies, for example, the child who indicates the first relationship by holding hands with the adult and indicates the second relationship in which the first relationship is dissolved, that is, who transitions from the first relationship to the second relationship in time series, as the first person.

[0075] In addition, the process of identifying the first relationship and the second relationship may include, for example, a process of inputting the video acquired by the acquisition unit 21 into a machine learning model, and identifying areas including children and areas including adults, and the first relationship and the second relationship from the video.

[0076] For example, the identification unit 22 inputs the video into a machine learning model to identify from the video a first area including a child and a second area including an adult, and a first relationship that identifies an interaction between the child included in the first area and the adult included in the second area. Also, the identification unit 22 inputs the video into a machine learning model to identify from the video a third area including a child and a fourth area including an adult, and a second relationship that identifies an interaction between the child included in the third area and the adult included in the fourth area.

[0077] In addition, when identifying the second relationship, in the case of a lost child, the video may not include an adult. Therefore, in this case, the fourth area does not need to be identified, and the second relationship may be a relationship that identifies the interaction between the child included in the third area and the adult included in the second area. That is, the identification unit 22 inputs the video acquired by the acquisition unit 21 into a machine learning model, and identifies, from the video, the third area including the child and the second relationship that identifies the interaction between the child included in the third area and the adult included in the second area.

[0078] Furthermore, the identification unit 22 can also identify the first relationship and the second relationship and identify the first person by, for example, analyzing a scene graph.

[0079] Furthermore, the identification unit 22 identifies a first area where an abnormality has occurred among a plurality of people, for example, based on the captured camera device 100. More specifically, the identification unit 22 identifies a first area where an abnormality has occurred among a plurality of people, for example, from the installation location and capturing range of the camera device 100 that captured the video in which the relationship between the plurality of people is identified.

[0080] Furthermore, the identification unit 22 generates skeletal information of multiple people included in the video by, for example, analyzing the video acquired by the acquisition unit 21, and identifies a relationship that identifies an interaction between the multiple people included in the video based on the generated skeletal information. More specifically, the identification unit 22 extracts, for example, a bounding box (Bbox) that encloses an area including a person in a rectangular shape from the video acquired by the acquisition unit 21. Then, the identification unit 22 generates skeletal information by, for example, inputting image data of the extracted person's Bbox into a trained machine learning model constructed using an existing algorithm such as DeepPose or OpenPose.

[0081] FIG. 16 is a diagram showing an example of skeletal information according to this embodiment. The skeletal information can use 18 definition information (0 to 17) in which each joint specified by a known skeletal model is numbered. For example, the right shoulder joint (SHOULDER_RIGHT) is assigned the number 7, the left elbow joint (ELBOW_LEFT) is assigned the number 5, the left knee joint (KNEE_LEFT) is assigned the number 11, and the right hip joint (HIP_RIGHT) is assigned the number 14. Therefore, the coordinate information of the 18 skeletons shown in FIG. 16 can be acquired from the image data, and for example, "X coordinate=X7, Y coordinate=Y7, Z coordinate=Z7" is acquired as the position of the right shoulder joint number 7. For example, the Z axis can be defined as the distance direction from the imaging device toward the target, the Y axis as the height direction perpendicular to the Z axis, and the X axis as the horizontal direction.

[0082] The determination unit 22 can also determine the posture of the whole body of a person, such as standing, walking, crouching, sitting, lying down, etc., by using a machine learning model previously trained on a skeletal pattern. For example, the determination unit 22 can determine the closest posture of the whole body by using a machine learning model trained with a Multilayer Perceptron on some joints and angles between the joints, such as the skeletal information and the beauty technique diagram in FIG. 16.

[0083] Fig. 17 is a diagram showing an example of whole body posture determination according to this embodiment. As shown in Fig. 17, the determination unit 22 acquires, for example, the angle (a) of the joint between No. 10 "HIP_LEFT" and No. 11 "KNEE_LEFT", and the angle (b) of the joint between No. 14 "HIP_RIGHT" and No. 15 "KNEE_RIGHT". Furthermore, the determination unit 22 can detect the posture of the whole body of the person by acquiring, for example, the angle (c) of No. 11 "KNEE_LEFT" and the angle (d) of No. 15 "KNEE_RIGHT".

[0084] In addition, the determination unit 22 can detect the motion of each part by determining the posture of the part based on the 3D posture of the body joint. More specifically, the determination unit 22 can convert the 2D joint coordinates into the 3D joint coordinates by using an existing algorithm such as a 3D-baseline method.

[0085] Fig. 18 is a diagram showing an example of detection of a motion by part according to this embodiment. As shown in Fig. 18, for example, for the part "face", the identification unit 22 can detect whether the face is facing forward, left, right, up, or down (five types) depending on whether the angle between the face direction and each directional vector is equal to or less than a threshold. The identification unit 22 identifies the face direction, for example, by a vector defined by "start point: midpoint of both ears, end point: nose". Furthermore, the identification unit 22 can detect whether the face is facing backwards, for example, by "face facing right and waist twisted to the right" or "face facing left and waist twisted to the left".

[0086] Furthermore, for example, for the part "arm", the identification unit 22 can detect whether the orientation of the left and right arms is forward / backward, left / right, up / down (six types) depending on whether the angle between the orientation of the forearm and each directional vector is equal to or less than a threshold. Note that the identification unit 22 can detect the orientation of the arm, for example, by a vector defined by "start point: elbow, end point: wrist".

[0087] Furthermore, for example, for the part "legs", the specification unit 22 can detect whether the orientation of the left and right legs is forward / backward, left / right, up / down (six types) depending on whether the angle between the orientation of the lower legs and each directional vector is equal to or less than a threshold. Note that the specification unit 22 can detect the orientation of the lower legs, for example, from a vector defined by "start point: knee, end point: ankle".

[0088] Furthermore, for example, the identification unit 22 can detect (two types) that the angle of the elbow part is equal to or greater than a threshold and is straightened, and that the angle is less than the threshold and is bent. Note that the identification unit 22 can detect the angle of the elbow, for example, from the angle between vector A of "start point: elbow, end point: shoulder" and vector B of "start point: elbow, end point: wrist".

[0089] Furthermore, for example, the identification unit 22 can detect (two types) that the knee angle of the part "knee" is equal to or greater than a threshold and is straightened, or is less than the threshold and is bent. Note that the identification unit 22 can detect the knee angle, for example, from the angle between vector A of "start point: knee, end point: ankle" and vector B of "start point: knee, end point: hip."

[0090] Furthermore, for example, for the part "waist", the identification unit 22 can detect left and right twists (two types) depending on whether the angle between the waist and shoulders is equal to or greater than a threshold, and can detect the front if it is less than the threshold. Furthermore, the identification unit 22 can detect the angle between the waist and shoulders, for example, from the angle of rotation around an axis vector C "start point: midpoint of both waists: end point: midpoint of both shoulders". The angle between the waist and shoulders is detected, for example, for vector A of "start point: left shoulder, end point: right shoulder" and vector B of "start point: left hip (hip(L)), end point: right hip (hip(R))".

[0091] Further, the identification unit 22, for example, identifies the positions of the multiple people included in each of the images captured by each of the multiple camera devices 100, using a first index that is different for each of the multiple camera devices 100. The first index is, for example, an image coordinate system in which the coordinates of the upper left pixel of an image that is one frame of an image captured by the camera device 100 are set as the origin (0,0). Since the image coordinate system is different for each of the multiple camera devices 100, even if the coordinates are the same between the images captured by the multiple camera devices 100, they do not indicate the same position in real space. Therefore, the identification unit 22, for example, identifies the positions of the multiple people identified by the first index, using a second index that is common to the multiple camera devices 100. The second index is, for example, a coordinate system common to the multiple camera devices 100 that is obtained by transforming the image coordinate system that is the first index using a projective transformation (homography) coefficient, and is hereinafter referred to as a "floor map coordinate system" in contrast to the image coordinate system. The conversion from the image coordinate system to the floor map coordinate system will now be described in more detail.

[0092] First, calculation of the projection transformation coefficients used for transformation from the image coordinate system to the floor map coordinate system will be described. FIG. 19 is a diagram showing an example of calculation of the projection transformation coefficients according to this embodiment. As shown in FIG. 19, the identification unit 22 identifies, for example, any corresponding points (1) to (4) that correspond between the image coordinate system and the floor map coordinate system. For example, the identification unit 22 identifies points (x1, y1), (x2, y2), (x3, y3), and (x4, y4) from the image coordinate system. Similarly, for example, the identification unit 22 identifies points (X1, Y1), (X2, Y2), (X3, Y3), and (X4, Y4) from the floor map coordinate system. Then, the identification unit 22 calculates, for example, a projection transformation coefficient a from the image coordinate system (x, y) to the floor map coordinate system (X, Y). i (i=1 to 8) are calculated by solving the simultaneous equations shown in formula (1) in Fig. 19. Note that since the layout of the camera devices 100 and the floor layout of various facilities such as stores are generally fixed, it is sufficient to estimate the projective transformation coefficients only once. Also, the corresponding points may be designated by an administrator of the information processing system 1, or points at the same location may be identified as corresponding points by image analysis.

[0093] Then, the identification unit 22, for example, uses the calculated projective transformation coefficient to convert the positions of the multiple people identified in the image coordinate system into the floor map coordinate system and identifies them. FIG. 20 is a diagram showing an example of conversion from the image coordinate system to the floor map coordinate system according to this embodiment. As shown in FIG. 20, the identification unit 22 calculates floor map coordinates (X, Y) by taking the image coordinates (x, y) of the center of the bottom end of the Bbox of each of the multiple people as the position of each person identified in the image coordinate system, and converting the coordinates using the projective transformation coefficient. Note that the floor map coordinates (X, Y) are coordinates common to the multiple camera devices 100 and indicate the position of each person in the floor map coordinate system. In the example of FIG. 20, for example, a point (x a ,y a )~(x d ,y d ) is converted into a point (X a ,Ya )~(X d ,Y d )

[0094] Returning to the explanation of Figure 4, the judgment unit 23 judges whether or not an abnormality has occurred between multiple people outside the shooting range of the camera device 100, for example, based on the relationship that the identification unit 22 identifies by identifying the interaction between multiple people included in the video.

[0095] More specifically, for example, when the identification unit 22 identifies a first relationship between an adult and a child and then identifies a second relationship with the child, the determination unit 23 determines that an abnormality has occurred between the adult and the child. In this regard, even if a scene that determines the occurrence of an abnormality, such as an adult and a child being separated in the case of a lost child, or a child being forcibly pulled by an adult in the case of abduction, is not captured by the camera device 100, the determination unit 23 can thus determine that an abnormality has occurred even when an abnormality has occurred outside the capture range of the camera device 100, based on the relationships between multiple people identified from the images before and after the occurrence of the abnormality.

[0096] Furthermore, the determination unit 23 determines whether or not an abnormality has occurred for a first person among the multiple people outside the shooting range of the camera device 100, based on, for example, a relationship that identifies an interaction between the multiple people included in the video and that is identified by the identification unit 22. Note that the first person is a person who transitions from a first relationship to a second relationship in time series, as identified by the identification unit 22.

[0097] Furthermore, the determination unit 23 determines whether or not a first person among multiple people has become lost outside the shooting range of the camera device 100, based on, for example, a time-series transition from the first relationship to the second relationship. For example, if the first relationship is a relationship between an adult and a child holding hands, and the second relationship is a relationship in which the first relationship is dissolved and the child is alone with the adult who was with the child, it is determined that the child has become lost.

[0098] Furthermore, when the child included in the first area and the child included in the third area identified from the video by the identification unit 22 are the same, the determination unit 23 compares the first relationship and the second relationship identified by the identification unit 22 with a preset rule. Here, the preset rule may be, for example, a rule set in the rule DB 16. Then, the determination unit 23 determines whether or not an abnormality has occurred with respect to the same child among multiple people outside the shooting range of the camera device 100, based on, for example, a comparison result between the first relationship and the second relationship and the preset rule.

[0099] Furthermore, the determination unit 23 determines whether or not the multiple people included in each of the videos are the same person for each of the multiple people, for example, based on the positions of the multiple people using the second index identified by the identification unit 22. For example, the second index is a floor map coordinate system common to the multiple camera devices 100. Therefore, for example, when the floor map coordinate systems indicated by the positions of the people included in each video captured by the multiple camera devices 100 are the same or within a predetermined range and in close proximity, the determination unit 23 can determine that the people included in each video are the same person.

[0100] Returning to the description of FIG. 4, the notification unit 24 notifies, for example, an alert indicating that an abnormality has occurred between a plurality of people. The alert may be notified when the determination unit 23 determines that an abnormality has occurred between a plurality of people outside the shooting range of the camera device 100 based on the relationship that identifies the interaction between the plurality of people included in the video. The alert may be, for example, an audio output or a message notification on a screen. The destination of the alert may be, for example, an output device included in the information processing device 10, an external device, or another output device communicably connected to the information processing device 10 via the network 50.

[0101] The notification unit 24 also notifies an alert indicating that an abnormality has occurred in the first object, for example, in association with the first area. The first area is, for example, an area identified by the identification unit 22 as an area in which an abnormality may have occurred among a plurality of people. The alert may include, for example, information regarding the position of the first area.

[0102] (Processing flow) Next, a description will be given of the flow of the abnormality occurrence determination process executed by the information processing device 10. Fig. 21 is a flowchart showing the flow of the abnormality occurrence determination process according to this embodiment.

[0103] 21, the information processing device 10 acquires, for example, an image captured by the camera device 100 of a predetermined shooting range in various facilities such as a store from the imaging DB 13 (step S101). The image captured by the camera device 100 is transmitted from the camera device 100 to the information processing device 10 as needed and stored in the imaging DB 13.

[0104] Next, the information processing device 10 inputs the video acquired in step S101 into a machine learning model, for example, to identify an area including a person from the video and relationships between multiple people (step S102). Note that the area including a person may be, for example, a bounding box (Bbox) that surrounds a person on the video with a rectangle. Also, the relationships between multiple people may be, for example, relationships between an adult and a child, such as holding hands, walking together, talking, etc.

[0105] Next, the information processing device 10 determines whether or not an abnormality has occurred between the multiple persons based on, for example, the relationships between the multiple persons identified in step S102 (step S103). Note that the abnormal behavior referred to here is, for example, getting lost or being kidnapped, and may include getting lost or being kidnapped outside the shooting range of the camera device 100. If it is determined that no abnormality has occurred between the multiple persons (step S104: No), the abnormality occurrence determination process shown in FIG. 21 ends.

[0106] On the other hand, if it is determined that an abnormality has occurred among a plurality of people (step S104: Yes), the information processing device 10 issues, for example, an alert (step S105). After executing step S105, the abnormality occurrence determination process shown in FIG. 21 ends.

[0107] Next, a flow of a relationship estimation process executed by the information processing device 10 will be described. Fig. 22 is a flowchart showing the flow of the relationship estimation process according to this embodiment. The relationship estimation process shown in Fig. 22 is, for example, a process of estimating the relationship between an object and a person using the NN 40 described with reference to Figs. 10 to 15.

[0108] First, the information processing device 10 acquires, for example, a video captured by the camera device 100 of a predetermined shooting range in various facilities such as a store, i.e., an input image, from the imaging DB 13 (step S201). Note that the input image includes an image of one frame of the video, and if the input image is stored as a video in the imaging DB 13, one frame is acquired from the video as the input image.

[0109] Next, the information processing device 10 extracts, for example, an image feature amount 181 from the input image acquired in step S201 as an image feature of the input image (step S202).

[0110] Next, the information processing device 10 detects a Bbox indicating the location of each object included in the video and a class indicating the type of each object from the image feature 181 extracted in step S202, for example, by using an existing technique (step S203). Note that each object detected here may include a person, and each object in the following description may also include a person.

[0111] Next, the information processing device 10 generates, as the pair feature 182, for example, a second feature obtained by combining the first feature of each object in the set of objects detected in step S203 (step S204).

[0112] Next, the information processing device 10, for example, combines the feature of the region important for relationship estimation extracted by the attention map 180 with the pair feature 182 to extract the relationship feature 183 (step S205). Note that the attention map 180 is generated from the pair feature 182 extracted in step S204.

[0113] Then, the information processing device 10 estimates the relationship between the objects detected from the image based on the relationship feature amount 183 extracted in step S205 (step S206). Note that the estimation of the relationship may be, for example, calculating a probability value for each type of relationship. After step S206 is executed, the relationship estimation process shown in FIG. 22 ends.

[0114] (effect) As described above, the information processing device 10 acquires images captured by one or more camera devices 100, analyzes the acquired images, identifies relationships that identify interactions between multiple people included in the images, and determines whether or not an abnormality has occurred between the multiple people outside the capture range of the camera device 100 based on the identified relationships.

[0115] In this way, the information processing device 10 identifies the relationship between multiple people from the video, and based on the identified relationship, determines whether or not an abnormality has occurred between multiple people outside the shooting range of the camera device 100. This allows the information processing device 10 to more accurately determine that an abnormality has occurred between multiple people from the video.

[0116] In addition, the information processing device 10 identifies a first person from among the multiple people, whose identified relationship transitions over time from a first relationship to a second relationship, based on the acquired video, and the process of determining whether or not an abnormality has occurred, executed by the information processing device 10, includes a process of determining whether or not an abnormality has occurred with respect to the first person among the multiple people, outside the shooting range of the camera device 100, based on the identified relationship.

[0117] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0118] In addition, the process of identifying relationships performed by the information processing device 10 includes a process of inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child included in the first area and the adult included in the second area, and a process of inputting the acquired video into the machine learning model to identify from the video a third area including a child, a fourth area including an adult, and a second relationship that identifies an interaction between the child included in the third area and the adult included in the fourth area. The process of determining whether or not an abnormality has occurred performed by the information processing device 10 includes a process of determining whether or not an abnormality has occurred with respect to the same child among multiple people, outside the shooting range of the camera device 100, by comparing the identified first relationship and second relationship with a preset rule when the child included in the first area and the child included in the third area are the same.

[0119] This enables the information processing device 10 to more accurately notify that an abnormality has occurred among a plurality of people from the video.

[0120] In addition, the process of identifying relationships performed by the information processing device 10 includes a process of inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child included in the first area and the adult included in the second area, and a process of inputting the acquired video into a machine learning model to identify from the video a third area including a child, and a second relationship that identifies an interaction between the child included in the third area and the adult included in the second area. The process of determining whether or not an abnormality has occurred performed by the information processing device 10 includes a process of determining whether or not an abnormality has occurred with respect to the same child among multiple people, outside the shooting range of the camera device 100, by comparing the identified first relationship and second relationship with a preset rule when the child included in the first area and the child included in the third area are the same.

[0121] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0122] In addition, the information processing device 10 identifies a first area where an abnormality has occurred among the multiple people based on the image captured by the camera device 100, associates the first area with the first area, and notifies an alert indicating that an abnormality has occurred among the multiple people.

[0123] This enables the information processing device 10 to more accurately notify that an abnormality has occurred among a plurality of people from the video.

[0124] In addition, the information processing device 10 identifies a first person from among the multiple people, whose identified relationship transitions over time from a first relationship to a second relationship, based on the acquired video, and the process of determining whether or not an abnormality has occurred, executed by the information processing device 10, includes a process of determining whether or not the first person among the multiple people has become lost, outside the shooting range of the camera device 100, based on the transition over time from the first relationship to the second relationship.

[0125] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0126] In addition, the process of identifying the first person executed by the information processing device 10 includes a process of inputting the acquired video into a machine learning model to generate a scene graph that identifies the relationships for each person included in the video, and identifying the first person by analyzing the scene graph.

[0127] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0128] In addition, the process of identifying a relationship executed by the information processing device 10 includes processes of extracting from the video a first feature corresponding to an object or each of multiple persons included in the video, detecting the object and each person included in the video from the extracted first feature, generating a second feature combining first features possessed by the detected multiple objects, multiple persons, and at least one pair of an object and each person, generating a first map indicating a first relationship that identifies at least one interaction between the multiple objects, multiple persons, and an object and each person based on the first feature and the second feature, extracting a fourth feature based on a third feature converted from the first feature and the first map, and identifying the relationship from the fourth feature.

[0129] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0130] In addition, the process of identifying the relationship executed by the information processing device 10 includes a process of generating skeletal information of a plurality of persons by analyzing the acquired video, and identifying the relationship based on the generated skeletal information.

[0131] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0132] In addition, the information processing device 10 identifies the positions of multiple people included in each of the images captured by each of the multiple camera devices 100 using a first indicator that is different for each of the multiple camera devices 100, identifies the positions of the multiple people identified by the first indicator using a second indicator that is common to the multiple camera devices 100, and determines, for each of the multiple people, whether the multiple people included in each of the images are the same person based on the positions of the multiple people using the identified second indicator.

[0133] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0134] In addition, the information processing device 10 is equipped with a sensor or camera at a gate placed at a predetermined position in the facility, and acquires a person's biometric information based on the detection of the biometric information of a person passing through the gate by the sensor or camera.When authentication using the acquired biometric information of the person is successful, the information processing device 10 analyzes an image including the person passing through the gate to identify the person included in the image as a person who has checked in to the facility, and tracks multiple people while associating them with the identification information of the person specified from the biometric information.

[0135] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0136] Furthermore, the facility is a store, the gate is located at the entrance of the store, and when the acquired biometric information of the person is registered as a member of the store, the information processing device 10 determines that authentication using the person's biometric information has been successful.

[0137] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0138] In addition, the facility is either a railway facility or an airport, the gate is located at a ticket gate at a railway facility or a counter at an airport, and when the acquired biometric information of the person has been pre-registered as a train or airplane passenger, the information processing device 10 determines that authentication using the person's biometric information has been successful.

[0139] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0140] Furthermore, the information processing device 10 tracks a plurality of people within the facility while identifying the relationships between the plurality of people.

[0141] This enables the information processing device 10 to more accurately determine that an abnormality has occurred among a plurality of people from the video.

[0142] (Application example) Next, returning to Fig. 1, an application example will be described. The information processing device 10 can analyze the behavior of a person who has checked in by using images captured by multiple camera devices 100. The facility is a railway facility, an airport, a store, etc. Also, the gate placed at the facility is placed at the entrance of a store, a ticket gate at a railway facility, a counter at an airport, etc.

[0143] First, an example will be described in which the check-in target is a railway facility or an airport. In the case of a railway facility or an airport, the gate is located at the boarding gate of the railway facility or the airport. In this case, if the biometric information of the person is preregistered as a railroad or airplane passenger, the information processing device 10 determines that the authentication based on the biometric information of the person is successful.

[0144] Next, an example will be described in which the check-in target is a store. When the check-in target is a store, the gate is located at the entrance of the store. At this time, when the biometric information of the person is registered as a member of the store, the information processing device 10 determines that the authentication based on the biometric information of the person is successful.

[0145] Here, the details of check-in will be described. The information processing device 10 acquires, for example, a vein image acquired by a vein sensor from a biometric sensor and performs authentication. Thereby, the ID, name, etc. of the person to be checked in are identified. At this time, the information processing device 10 acquires an image of the person to be checked in using the information processing device 10. Next, the information processing device 10 detects a person from the image. The information processing device 10 tracks the person detected from the images captured by the multiple camera devices 100 between frames. At this time, the information processing device 10 associates the ID and name of the person to be checked in with the person to be tracked.

[0146] Returning to FIG. 2, an application example will now be described. When checking in, the information processing device 10 acquires biometric information of a person passing through a gate placed at a predetermined position in a store. Specifically, the information processing device 10 acquires a vein image or the like acquired by a biometric sensor, for example a vein sensor, mounted on a gate placed at the entrance of the store, from the biometric sensor, and performs authentication. At this time, the information processing device 10 identifies an ID and name associated with the biometric information. The information processing device 10 identifies the IDs and names of person P and person C from the biometric information. Then, the information processing device 10 identifies that person P and person C are parent and child using the identified IDs and names.

[0147] The biometric sensor is mounted on a gate placed at a predetermined position in the facility and detects biometric information of a person passing through the gate. The camera devices 100 are installed on the ceiling of the store. Instead of the biometric sensor, the information processing device 10 may obtain biometric information from a face image captured by a camera mounted on a gate placed at the entrance of the store to perform authentication.

[0148] Next, the information processing device 10 determines whether or not authentication based on the person's biometric information has been successful. If the authentication has been successful, the information processing device 10 analyzes an image including the person passing through the gate to identify the person included in the image as a person who has checked in to the facility. Then, the information processing device 10 stores in the storage unit the identification information of the person specified from the biometric information and the identified person in association with each other.

[0149] Thereafter, the information processing device 10 tracks multiple people while associating the identification information of the person specified from the biometric information. Specifically, the information processing device 10 stores the ID and name of the person who checks in with the identified person while associating them. In addition, the information processing device 10 analyzes the images captured by the multiple camera devices 100, and tracks the identified person while identifying person P and person C as parent and child. This makes it possible to accurately detect that person C has become lost after being separated from person P when person C and person P become separated.

[0150] (system) The information including the processing procedures, control procedures, specific names, various data and parameters shown in the above documents and drawings may be changed arbitrarily unless otherwise specified. In addition, the specific examples, distributions, numerical values, etc. described in the embodiments are merely examples and may be changed arbitrarily.

[0151] In addition, the specific form of distribution or integration of the components of each device is not limited to that shown in the figure. In other words, all or part of the components may be functionally or physically distributed or integrated in any unit depending on various loads, usage conditions, etc. Furthermore, all or any part of the processing functions of each device may be realized by a CPU and a program analyzed and executed by the CPU, or may be realized as hardware using wired logic.

[0152] (Hardware) Fig. 23 is a diagram illustrating an example of a hardware configuration of the information processing device 10. As shown in Fig. 23, the information processing device 10 includes a communication device 10a, a hard disk drive (HDD) 10b, a memory 10c, and a processor 10d. The components shown in Fig. 23 are connected to each other via a bus or the like.

[0153] The communication device 10a is a network interface card or the like, and communicates with other information processing devices. The HDD 10b stores programs and DBs that operate the functions shown in FIG.

[0154] The processor 10d is a hardware circuit that operates a process that executes each function described in FIG. 4 and the like by reading a program that executes the same processing as each processing unit shown in FIG. 4 from the HDD 10b or the like and expanding it in the memory 10c. That is, this process executes the same function as each processing unit of the information processing device 10. Specifically, the processor 10d reads a program having the same functions as the acquisition unit 21, the identification unit 22, the determination unit 23, the notification unit 24, and the like from the HDD 10b or the like. Then, the processor 10d executes a process that executes the same processing as the identification unit 22 and the like.

[0155] In this way, the information processing device 10 operates as an information processing device that executes operation control processing by reading and executing a program that executes the same processing as each processing unit shown in Fig. 4. The information processing device 10 can also realize functions similar to those of the above-mentioned embodiment by reading a program from a recording medium using a medium reading device and executing the read program. Note that the program in this other embodiment is not limited to being executed by the information processing device 10. For example, this embodiment may also be applied in the same way to a case where another information processing device executes a program, or a case where the information processing device 10 and the other information processing device cooperate to execute a program.

[0156] A program that executes the same processes as those of each processing unit shown in Fig. 4 can be distributed via a network such as the Internet. This program can be recorded on a computer-readable recording medium such as a hard disk, a flexible disk (FD), a CD-ROM, a magneto-optical disk (MO), or a digital versatile disk (DVD), and can be executed by being read out from the recording medium by a computer.

[0157] The following supplementary notes are further disclosed regarding the embodiments including the above examples.

[0158] (Appendix 1) Obtaining video footage captured by one or more camera devices; Analyzing the acquired video to identify relationships that identify interactions between a plurality of people included in the video; Based on the identified relationship, it is determined whether or not an abnormality has occurred among the plurality of persons outside the photographing range of the camera device. An information processing program that causes a computer to execute a process.

[0159] (Supplementary Note 2) Based on the acquired video, a first person whose identified relationship transitions from a first relationship to a second relationship in time series is identified from among the plurality of people. causing the computer to execute a process; The process of determining whether or not an abnormality has occurred includes: Based on the identified relationship, it is determined whether or not an abnormality has occurred with respect to the first person among the plurality of people outside the photographing range of the camera device. 2. The information processing program according to claim 1, further comprising:

[0160] (Additional Note 3) The process of identifying the relationship includes: inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child in the first area and the adult in the second area; The acquired video is input to a machine learning model to identify a third area including a child, a fourth area including an adult, and a second relationship that identifies an interaction between the child in the third area and the adult in the fourth area from the video. Processing The process of determining whether or not an abnormality has occurred includes: When the child included in the first area and the child included in the third area are the same, the specified first relationship and the specified second relationship are compared with a preset rule to determine whether or not an abnormality has occurred with respect to the same child among the plurality of persons outside the photographing range of the camera device. 3. The information processing program according to claim 1 or 2, comprising:

[0161] (Additional Note 4) The process of identifying the relationship includes: inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child in the first area and the adult in the second area; The acquired video is input to a machine learning model to identify a third area including a child from the video, and a second relationship that identifies an interaction between the child in the third area and an adult in the second area. Processing The process of determining whether or not an abnormality has occurred includes: When the child included in the first area and the child included in the third area are the same, the specified first relationship and the specified second relationship are compared with a preset rule to determine whether or not an abnormality has occurred with respect to the same child among the plurality of persons outside the photographing range of the camera device. 3. The information processing program according to claim 1 or 2, comprising:

[0162] (Appendix 5) Identifying a first area in which an abnormality occurs among the plurality of people based on the image captured by the camera device; Corresponding to the first area, an alert is issued indicating that an abnormality has occurred among the plurality of persons. 3. The information processing program according to claim 1 or 2, characterized in that the program causes the computer to execute a process.

[0163] (Supplementary Note 6) Based on the acquired video, a first person whose identified relationship transitions from a first relationship to a second relationship in time series is identified from among the plurality of people. causing the computer to execute a process; The process of determining whether or not an abnormality has occurred includes: Based on a time series transition from the first relationship to the second relationship, it is determined whether or not the first person among the plurality of people has become lost outside a photographing range of the camera device. 2. The information processing program according to claim 1, further comprising:

[0164] (Additional Note 7) The process of identifying the first person includes: The acquired video is input into a machine learning model to generate a scene graph that specifies the relationship for each person included in the video; Analyzing the scene graph to identify the first person 7. The information processing program according to claim 2 or 6, comprising a process for:

[0165] (Appendix 8) The process of identifying the relationship includes: extracting a first feature corresponding to an object or each of the plurality of persons included in the video from the video; Detecting the object and each of the people included in the video from the extracted first feature amount; generating a second feature amount obtained by combining the first feature amount of the detected plurality of objects, the detected plurality of persons, and at least one pair of the object and each of the persons; generating a first map indicating a first relationship that identifies a plurality of the objects, the plurality of the persons, and at least one interaction between the objects and each of the persons based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The relationship is identified from the fourth feature amount. 3. The information processing program according to claim 1 or 2, comprising:

[0166] (Additional Note 9) The process of identifying the relationship includes: generating skeletal information of the plurality of persons by analyzing the acquired images; The relationship is identified based on the generated skeleton information. 3. The information processing program according to claim 1 or 2, comprising:

[0167] (Appendix 10) The positions of the plurality of people included in each of the images captured by each of the plurality of camera devices are identified by a first index that is different for each of the plurality of camera devices; specifying the positions of the plurality of people specified by the first indicator using a second indicator common to the plurality of camera devices; Based on the positions of the plurality of people using the identified second indicator, it is determined for each of the plurality of people whether the plurality of people included in each of the videos are the same person. 3. The information processing program according to claim 1 or 2, characterized in that the program causes the computer to execute a process.

[0168] (Appendix 11) A sensor or a camera is mounted on a gate arranged at a predetermined position of a facility, and biometric information of a person passing through the gate is obtained based on detection of the biometric information of the person by the sensor or the camera; When authentication based on the acquired biometric information of the person is successful, an image including the person passing through the gate is analyzed to identify the person included in the image as a person who has checked in to the facility; Tracking the plurality of persons while associating the identification information of the persons specified from the biometric information. 2. The information processing program according to claim 1, characterized in that the information processing program causes the computer to execute a process.

[0169] (Appendix 12) The facility is a store, The gate is disposed at an entrance of the store, If the acquired biometric information of the person is registered as a member of the store, it is determined that authentication using the biometric information of the person has been successful. 12. The information processing program according to claim 11, which causes the computer to execute a process.

[0170] (Appendix 13) The facility is either a railway facility or an airport, The gate is disposed at a ticket gate of the railway facility or a counter at an airport, When the acquired biometric information of the person is preregistered as a passenger of a train or an airplane, it is determined that the authentication based on the biometric information of the person has been successful. 12. The information processing program according to claim 11, which causes the computer to execute a process.

[0171] (Appendix 14) Tracking the plurality of people within the facility while identifying the relationships between the plurality of people. 12. The information processing program according to claim 11, which causes the computer to execute a process.

[0172] (Appendix 15) Obtaining video footage captured by one or more camera devices; Analyzing the acquired video to identify relationships that identify interactions between a plurality of people included in the video; Based on the identified relationship, it is determined whether or not an abnormality has occurred among the plurality of persons outside the photographing range of the camera device. An information processing method characterized in that the processing is executed by a computer.

[0173] (Appendix 16) Based on the acquired video, a first person whose identified relationship transitions from a first relationship to a second relationship in time series is identified from among the plurality of people. The process is executed by the computer, The process of determining whether or not an abnormality has occurred includes: Based on the identified relationship, it is determined whether or not an abnormality has occurred with respect to the first person among the plurality of people outside the photographing range of the camera device. 16. The information processing method according to claim 15, further comprising the steps of:

[0174] (Appendix 17) The process of identifying the relationship includes: inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child in the first area and the adult in the second area; The acquired video is input to a machine learning model to identify a third area including a child, a fourth area including an adult, and a second relationship that identifies an interaction between the child in the third area and the adult in the fourth area from the video. Processing The process of determining whether or not an abnormality has occurred includes: When the child included in the first area and the child included in the third area are the same, the specified first relationship and the specified second relationship are compared with a preset rule to determine whether or not an abnormality has occurred with respect to the same child among the plurality of persons outside the photographing range of the camera device. 17. The information processing method according to claim 15 or 16, further comprising the steps of:

[0175] (Appendix 18) The process of identifying the relationship includes: inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child in the first area and the adult in the second area; The acquired video is input to a machine learning model to identify a third area including a child from the video, and a second relationship that identifies an interaction between the child in the third area and an adult in the second area. Processing The process of determining whether or not an abnormality has occurred includes: When the child included in the first area and the child included in the third area are the same, the specified first relationship and the specified second relationship are compared with a preset rule to determine whether or not an abnormality has occurred with respect to the same child among the plurality of persons outside the photographing range of the camera device. 17. The information processing method according to claim 15 or 16, further comprising the steps of:

[0176] (Appendix 19) Identifying a first area in which an abnormality occurs among the plurality of people based on the image captured by the camera device; Corresponding to the first area, an alert is issued indicating that an abnormality has occurred among the plurality of persons. 17. The information processing method according to claim 15 or 16, wherein the processing is executed by the computer.

[0177] (Supplementary Note 20) Based on the acquired video, a first person whose identified relationship transitions from a first relationship to a second relationship in time series is identified from among the plurality of people. The process is executed by the computer, The process of determining whether or not an abnormality has occurred includes: Based on a time series transition from the first relationship to the second relationship, it is determined whether or not the first person among the plurality of people has become lost outside a photographing range of the camera device. 16. The information processing method according to claim 15, further comprising the steps of:

[0178] (Appendix 21) The process of identifying the first person includes: The acquired video is input into a machine learning model to generate a scene graph that specifies the relationship for each person included in the video; Analyzing the scene graph to identify the first person 21. The information processing method according to claim 16 or 20, further comprising the steps of:

[0179] (Additional Note 22) The process of identifying the relationship includes: extracting a first feature corresponding to an object or each of the plurality of persons included in the video from the video; Detecting the object and each of the people included in the video from the extracted first feature amount; generating a second feature amount obtained by combining the first feature amount of the detected plurality of objects, the detected plurality of persons, and at least one pair of the object and each of the persons; generating a first map indicating a first relationship that identifies a plurality of the objects, the plurality of the persons, and at least one interaction between the objects and each of the persons based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The relationship is identified from the fourth feature amount. 17. The information processing method according to claim 15 or 16, further comprising the steps of:

[0180] (Additional Note 23) The process of identifying the relationship includes: generating skeletal information of the plurality of persons by analyzing the acquired images; The relationship is identified based on the generated skeleton information. 17. The information processing method according to claim 15 or 16, further comprising the steps of:

[0181] (Appendix 24) The positions of the plurality of people included in each of the images captured by each of the plurality of camera devices are identified by a first index that is different for each of the plurality of camera devices; specifying the positions of the plurality of people specified by the first indicator using a second indicator common to the plurality of camera devices; Based on the positions of the plurality of people using the identified second indicator, it is determined for each of the plurality of people whether the plurality of people included in each of the videos are the same person. 17. The information processing method according to claim 15 or 16, wherein the processing is executed by the computer.

[0182] (Appendix 25) A sensor or a camera is mounted on a gate arranged at a predetermined position of the facility, and biometric information of a person passing through the gate is obtained based on detection of the biometric information of the person by the sensor or the camera; When authentication based on the acquired biometric information of the person is successful, an image including the person passing through the gate is analyzed to identify the person included in the image as a person who has checked in to the facility; Tracking the plurality of persons while associating the identification information of the persons specified from the biometric information. 16. The information processing method according to claim 15, wherein the processing is executed by the computer.

[0183] (Appendix 26) The facility is a store, The gate is disposed at an entrance of the store, If the acquired biometric information of the person is registered as a member of the store, it is determined that authentication using the biometric information of the person has been successful. 26. The information processing method according to claim 25, wherein the processing is executed by the computer.

[0184] (Appendix 27) The facility is either a railway facility or an airport, The gate is disposed at a ticket gate of the railway facility or a counter at an airport, When the acquired biometric information of the person is preregistered as a passenger of a train or an airplane, it is determined that the authentication based on the biometric information of the person has been successful. 26. The information processing method according to claim 25, wherein the processing is executed by the computer.

[0185] (Appendix 28) Tracking the plurality of people within the facility while identifying the relationships between the plurality of people. 26. The information processing method according to claim 25, wherein the processing is executed by the computer.

[0186] (Appendix 29) Obtaining video footage captured by one or more camera devices; Analyzing the acquired video to identify relationships that identify interactions between a plurality of people included in the video; Based on the identified relationship, it is determined whether or not an abnormality has occurred among the plurality of persons outside the photographing range of the camera device. An information processing device comprising a control unit for executing processing.

[0187] (Appendix 30) Based on the acquired video, a first person whose identified relationship transitions from a first relationship to a second relationship in time series is identified from among the plurality of people. The control unit executes the process, The process of determining whether or not an abnormality has occurred includes: Based on the identified relationship, it is determined whether or not an abnormality has occurred with respect to the first person among the plurality of people outside the photographing range of the camera device. 30. The information processing device according to claim 29, further comprising:

[0188] (Additional Note 31) The process of identifying the relationship includes: inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child in the first area and the adult in the second area; The acquired video is input to a machine learning model to identify a third area including a child, a fourth area including an adult, and a second relationship that identifies an interaction between the child in the third area and the adult in the fourth area from the video. Processing The process of determining whether or not an abnormality has occurred includes: When the child included in the first area and the child included in the third area are the same, the specified first relationship and the specified second relationship are compared with a preset rule to determine whether or not an abnormality has occurred with respect to the same child among the plurality of persons outside the photographing range of the camera device. 31. The information processing device according to claim 29 or 30, further comprising a process.

[0189] (Additional Note 32) The process of identifying the relationship includes: inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child in the first area and the adult in the second area; The acquired video is input to a machine learning model to identify a third area including a child from the video, and a second relationship that identifies an interaction between the child in the third area and an adult in the second area. Processing The process of determining whether or not an abnormality has occurred includes: When the child included in the first area and the child included in the third area are the same, the specified first relationship and the specified second relationship are compared with a preset rule to determine whether or not an abnormality has occurred with respect to the same child among the plurality of persons outside the photographing range of the camera device. 31. The information processing device according to claim 29 or 30, further comprising a process.

[0190] (Appendix 33) Identifying a first area in which an abnormality occurs among the plurality of people based on the image captured by the camera device; Corresponding to the first area, an alert is issued indicating that an abnormality has occurred among the plurality of persons. 31. The information processing device according to claim 29 or 30, wherein the control unit executes the processing.

[0191] (Appendix 34) Based on the acquired video, a first person whose identified relationship transitions from a first relationship to a second relationship in time series is identified from among the plurality of people. The control unit executes the process, The process of determining whether or not an abnormality has occurred includes: Based on a time series transition from the first relationship to the second relationship, it is determined whether or not the first person among the plurality of people has become lost outside a photographing range of the camera device. 30. The information processing device according to claim 29, further comprising:

[0192] (Additional Note 35) The process of identifying the first person includes: The acquired video is input into a machine learning model to generate a scene graph that specifies the relationship for each person included in the video; Analyzing the scene graph to identify the first person 35. The information processing device according to claim 30 or 34, further comprising a process.

[0193] (Additional Note 36) The process of identifying the relationship includes: extracting a first feature corresponding to an object or each of the plurality of persons included in the video from the video; Detecting the object and each of the people included in the video from the extracted first feature amount; generating a second feature amount obtained by combining the first feature amount of the detected plurality of objects, the detected plurality of persons, and at least one pair of the object and each of the persons; generating a first map indicating a first relationship that identifies a plurality of the objects, the plurality of the persons, and at least one interaction between the objects and each of the persons based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The relationship is identified from the fourth feature amount. 31. The information processing device according to claim 29 or 30, further comprising a process.

[0194] (Additional Note 37) The process of identifying the relationship includes: generating skeletal information of the plurality of persons by analyzing the acquired images; The relationship is identified based on the generated skeleton information. 31. The information processing device according to claim 29 or 30, further comprising a process.

[0195] (Appendix 38) The positions of the people included in each of the images captured by each of the plurality of camera devices are identified by a first index that is different for each of the plurality of camera devices; specifying the positions of the plurality of people specified by the first indicator using a second indicator common to the plurality of camera devices; Based on the positions of the plurality of people using the identified second indicator, it is determined for each of the plurality of people whether the plurality of people included in each of the videos are the same person. 31. The information processing device according to claim 29 or 30, wherein the control unit executes the processing.

[0196] (Appendix 39) A sensor or a camera is mounted on a gate arranged at a predetermined position of the facility, and biometric information of a person passing through the gate is obtained based on detection of the biometric information of the person by the sensor or the camera; When authentication based on the acquired biometric information of the person is successful, an image including the person passing through the gate is analyzed to identify the person included in the image as a person who has checked in to the facility; Tracking the plurality of persons while associating the identification information of the persons specified from the biometric information. 30. The information processing device according to claim 29, wherein the control unit executes the processing.

[0197] (Appendix 40) The facility is a store, The gate is disposed at an entrance of the store, If the acquired biometric information of the person is registered as a member of the store, it is determined that authentication using the biometric information of the person has been successful. 40. The information processing device according to claim 39, wherein the control unit executes the processing.

[0198] (Appendix 41) The facility is either a railway facility or an airport, The gate is disposed at a ticket gate of the railway facility or a counter at an airport, When the acquired biometric information of the person is preregistered as a passenger of a train or an airplane, it is determined that the authentication based on the biometric information of the person has been successful. 40. The information processing device according to claim 39, wherein the control unit executes the processing.

[0199] (Appendix 42) Tracking the plurality of people within the facility while identifying the relationships between the plurality of people. 40. The information processing device according to claim 39, wherein the control unit executes the processing.

[0200] (Appendix 43) A processor; a memory operatively connected to the processor; An information processing device comprising: Obtaining video footage captured by one or more camera devices; Analyzing the acquired video to identify relationships that identify interactions between a plurality of people included in the video; Based on the identified relationship, it is determined whether or not an abnormality has occurred among the plurality of persons outside the photographing range of the camera device. 2. An information processing device that executes processing. [Explanation of symbols]

[0201] 1. Information Processing Systems 10. Information processing device 10a Communication equipment 10b HDD 10c Memory 10d Processor 11 Communications Department 12 Storage section 13 Imaging DB 14 Camera installation database 15 Model DB 16 Rules DB 20 Control section 21 Acquisition Department 22 Specific section 23 Judgment section 24 Notification Department 40 NN 41 Image feature extraction unit 42 Object detection unit 43 Pair feature generator 44 Relationship feature extraction unit 45 Relationship Estimation Department 50 Network 100 Camera Equipment 150 captured images 151~155 Bbox 159 Scene Graph 170 Captured Images 171~175 Bbox 180 Attention Map 181 Image Features 182 pair features 183 Relationship Features

Claims

1. Obtaining video footage captured by one or more camera devices; Analyzing the acquired video to identify relationships that identify interactions between a plurality of people included in the video; Based on the identified relationship, it is determined whether or not an abnormality has occurred among the plurality of people outside the photographing range of the camera device. An information processing program that causes a computer to execute a process.

2. Identifying a first person, whose identified relationship transitions from a first relationship to a second relationship in time series, from among the plurality of persons based on the acquired video. causing the computer to execute a process; The process of determining whether or not an abnormality has occurred includes: Based on the identified relationship, it is determined whether or not an abnormality has occurred with respect to the first person among the plurality of people outside the photographing range of the camera device.

2. The information processing program according to claim 1, further comprising a process for:

3. The process of identifying the relationship includes: inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child in the first area and the adult in the second area; The acquired video is input to a machine learning model to identify, from the video, a third area including a child, a fourth area including an adult, and a second relationship that identifies an interaction between the child in the third area and the adult in the fourth area. Processing The process of determining whether or not an abnormality has occurred includes: When the child included in the first area and the child included in the third area are the same, the specified first relationship and the specified second relationship are compared with a preset rule to determine whether or not an abnormality has occurred with respect to the same child among the plurality of persons outside the photographing range of the camera device.

3. The information processing program according to claim 1, further comprising a process for:

4. The process of identifying the relationship includes: inputting the acquired video into a machine learning model to identify from the video a first area including a child, a second area including an adult, and a first relationship that identifies an interaction between the child in the first area and the adult in the second area; The acquired video is input to a machine learning model to identify a third area including a child from the video, and a second relationship that identifies an interaction between the child in the third area and an adult in the second area. Processing The process of determining whether or not an abnormality has occurred includes: When the child included in the first area and the child included in the third area are the same, the specified first relationship and the specified second relationship are compared with a preset rule to determine whether or not an abnormality has occurred with respect to the same child among the plurality of persons outside the photographing range of the camera device.

3. The information processing program according to claim 1, further comprising a process for:

5. Identifying a first area where an abnormality has occurred among the plurality of people based on the image captured by the camera device; Corresponding to the first area, an alert is issued indicating that an abnormality has occurred among the plurality of persons.

3. The information processing program according to claim 1, which causes the computer to execute a process.

6. Identifying a first person, whose identified relationship transitions from a first relationship to a second relationship in time series, from among the plurality of persons based on the acquired video. causing the computer to execute a process; The process of determining whether or not an abnormality has occurred includes: Based on a time series transition from the first relationship to the second relationship, it is determined whether or not the first person among the plurality of people has become lost outside a photographing range of the camera device.

2. The information processing program according to claim 1, further comprising a process for:

7. The process of identifying the first person includes: The acquired video is input into a machine learning model to generate a scene graph that specifies the relationship for each person included in the video; Analyzing the scene graph to identify the first person 7. The information processing program according to claim 2, further comprising a process for:

8. The process of identifying the relationship includes: extracting a first feature amount corresponding to an object or each person included in the video from the video; Detecting the object and each of the people included in the video from the extracted first feature amount; generating a second feature amount obtained by combining the first feature amount of the detected plurality of objects, the detected plurality of persons, and the detected plurality of objects or the detected plurality of persons in at least one pair of the detected objects and the detected plurality of persons; generating a first map indicating a first relationship that identifies a plurality of the objects, the plurality of the persons, and at least one interaction between the objects and each of the persons based on the first feature amount and the second feature amount; extracting a fourth feature amount based on a third feature amount obtained by converting the first feature amount and the first map; The relationship is identified from the fourth feature amount.

3. The information processing program according to claim 1, further comprising a process for:

9. The process of identifying the relationship includes: generating skeletal information of the plurality of persons by analyzing the acquired images; The relationship is identified based on the generated skeleton information.

3. The information processing program according to claim 1, further comprising a process for:

10. Identifying positions of the plurality of people included in each of the images captured by each of the plurality of camera devices using a first index that is different for each of the plurality of camera devices; specifying the positions of the plurality of people specified by the first indicator using a second indicator common to the plurality of camera devices; Based on the positions of the plurality of people using the identified second indicator, it is determined for each of the plurality of people whether the plurality of people included in each of the videos are the same person.

3. The information processing program according to claim 1, which causes the computer to execute a process.

11. A sensor or a camera is mounted on a gate arranged at a predetermined position of a facility, and biometric information of a person passing through the gate is acquired based on detection of the biometric information of the person by the sensor or the camera; When authentication based on the acquired biometric information of the person is successful, an image including the person passing through the gate is analyzed to identify the person included in the image as a person who has checked in to the facility; Tracking the plurality of persons while associating the identification information of the persons specified from the biometric information.

2. The information processing program according to claim 1, which causes the computer to execute a process.

12. The facility is a store, The gate is disposed at an entrance of the store, If the acquired biometric information of the person is registered as a member of the store, it is determined that authentication using the biometric information of the person has been successful.

12. The information processing program according to claim 11, which causes the computer to execute a process.

13. The facility is either a railway facility or an airport, The gate is disposed at a ticket gate of the railway facility or a counter at an airport, When the acquired biometric information of the person is preregistered as a passenger of a train or an airplane, it is determined that the authentication based on the biometric information of the person has been successful.

12. The information processing program according to claim 11, which causes the computer to execute a process.

14. Tracking the plurality of people within the facility while identifying the relationships between the plurality of people.

12. The information processing program according to claim 11, which causes the computer to execute a process.

15. Acquiring video footage captured by one or more camera devices; Analyzing the acquired video to identify relationships that identify interactions between a plurality of people included in the video; Based on the identified relationship, it is determined whether or not an abnormality has occurred among the plurality of persons outside the photographing range of the camera device. An information processing method characterized in that the processing is executed by a computer.

16. Acquiring video footage captured by one or more camera devices; Analyzing the acquired video to identify relationships that identify interactions between a plurality of people included in the video; Based on the identified relationship, it is determined whether or not an abnormality has occurred among the plurality of persons outside the photographing range of the camera device. An information processing device comprising a control unit for executing processing.

Images