Information processing system and information processing method

Abstract

To enable effective detection of predetermined action.SOLUTION: An information processing system comprises a determination unit and a decision unit. The determination unit determines a question being a target of VQA (Visual Question Answering) corresponding to a predetermined detection processing result. The decision unit decides predetermined action by executing the VQA on the basis of the question determined by the determination unit.SELECTED DRAWING: Figure 5

Description

[Technical Field]

[0001] The present invention relates to an information processing system and an information processing method. [Background technology]

[0002] Conventionally, a technique for estimating predetermined behavior from video images using an AI technology called VQA (Visual Question Answering) has been known. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-071675 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional techniques, when the number of questions in VQA is large, the processing time can increase, so there is room for further improvement in order to effectively detect predetermined behaviors.

[0005] The present application has been made in view of the above, and aims to provide an information processing system and an information processing method that enable effective detection of predetermined behavior. [Means for solving the problem]

[0006] The information processing system according to the present application is characterized by having a determination unit that determines questions to be subjected to VQA (Visual Question Answering) in response to a predetermined detection processing result, and a determination unit that determines a predetermined behavior by performing VQA based on the questions determined by the determination unit. [Effects of the Invention]

[0007] According to one aspect of the embodiment, it is possible to effectively detect a predetermined behavior. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a diagram illustrating an example of the configuration of an information processing system according to an embodiment. [Figure 2A] FIG. 2A is an explanatory diagram for explaining types of detection processing according to the embodiment. [Figure 2B] FIG. 2B is an explanatory diagram illustrating the procedure of the detection process according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of information processing according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of the configuration of a terminal device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of the configuration of an information processing device according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of a detection processing procedure storage unit according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of a VQA model storage unit according to the embodiment. [Figure 8] FIG. 8 is a flowchart illustrating an example of information processing according to the embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, an information processing system and an information processing method according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing system and the information processing method according to the present application are not limited to these embodiments. Furthermore, the same components in the following embodiments will be denoted by the same reference numerals, and duplicated descriptions will be omitted.

[0010] There is a need to install surveillance cameras and the like at manufacturing sites and warehouses to detect unsafe behaviors (risky behaviors) and conditions that violate safety manuals. Conventionally, managers at manufacturing sites and warehouses have had to narrow down the video using person detection and other methods, and then a reviewer has to review the video. In addition, as in Patent Document 1 above, there are known technologies that use AI and other technologies to detect people engaging in unsafe behavior from video. For example, there is known a technology that uses AI technologies such as VQA to detect unsafe behavior in video by asking people captured in the video questions such as "Are you standing on a stepladder?" and "Are you not wearing a helmet?" and determining that the person is engaging in unsafe behavior if they answer "YES" to all questions.

[0011] However, processing time may increase if the number of questions is large. A wide variety of unsafe behaviors can occur in manufacturing sites and warehouses. For example, climbing a stepladder without wearing a helmet can be an unsafe behavior when multiple conditions overlap. Therefore, for example, detecting various unsafe behaviors using cameras installed in a factory can require enormous processing time. Furthermore, for example, a high-spec PC may be required. Therefore, there is room for further improvement to effectively detect predetermined behaviors. For example, by improving the information processing flow, such as asking all questions to people captured in a video and obtaining all answers before judging the next video, it is believed that detection of predetermined behaviors can be more effective. The present application has been made in consideration of the above, and aims to provide an information processing system and an information processing method that enable effective detection of predetermined behaviors.

[0012] VQA is a technology that integrates computer vision and natural language processing. It accepts questions in natural language about objects and scenes in a video and generates an answer to that question. VQA is a model that, when input with a video and a question, generates an answer to the question based on the video. For example, when VQA is input with a video of a worker wearing gloves and the question "Is he standing on a stepladder?", it will output "YES" based on the video. There is a need to use such VQA technology to detect violations of safety manuals or the occurrence of high-urgency situations. The video can be either a still image or a moving image.

[0013] In the following embodiment, a plurality of types of detection processes are efficiently combined to optimize the processing configuration so as to minimize the processing time, thereby enabling effective detection of a predetermined behavior.

[0014] (Embodiment) [Configuration of information processing system] An information processing system 1 shown in Fig. 1 will be described. As shown in Fig. 1, the information processing system 1 includes a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via wired or wireless communication. Fig. 1 is a diagram showing an example of the configuration of the information processing system 1 according to an embodiment.

[0015] The terminal device 10 is an information processing device used by a manager who desires to detect predetermined behaviors such as unsafe behaviors. The manager desires to detect unsafe behaviors (including unsafe conditions) (for example, whether a safety manual is violated) of workers (e.g., employees) working at a specific site such as a manufacturing site or a warehouse. The terminal device 10 may be any device that can implement the processing in the embodiment. The terminal device 10 may be, for example, a smartphone, a tablet terminal, a notebook PC, a desktop PC, a mobile phone, a PDA, or other device. FIG. 2 shows a case where the terminal device 10 is a smartphone.

[0016] The terminal device 10 is, for example, a smart device such as a smartphone or smart glasses, and is a portable terminal device capable of communicating with any server device via a wireless communication network such as 4G to 5G (Generations), or LTE (Long Term Evolution). The terminal device 10 may have a screen such as a liquid crystal display with a touch panel function, and may accept various operations on displayed data such as content, such as tapping, sliding, and scrolling, performed by an administrator using a finger or a stylus. In FIG. 2, the terminal device 10 is used by an administrator U1.

[0017] The information processing device 100 is an information processing device designed to enable effective detection of predetermined behaviors. Any device may be used as long as it can implement the processes described in the embodiments. The information processing device 100 is designed to enable effective detection of predetermined behaviors using VQA, for example, to meet administrator needs for effective detection of predetermined behaviors, such as unsafe behaviors. For example, the information processing device 100 efficiently combines multiple types of detection processes to minimize processing time, for example, to effectively detect near misses. Specifically, the information processing device 100 performs detection processes such as person detection, skeleton detection, and object detection, and then performs VQA only when necessary based on the detection process results. By asking only the necessary questions to the necessary people, the predetermined behavior is determined. This enables more effective detection of predetermined behaviors than, for example, conventional techniques that ask all questions to all people in all video. Furthermore, for example, it is possible to reduce AI processing costs (e.g., cloud usage fees and GPU consumption), enabling more video analysis to be performed on a single PC in on-premise systems.

[0018] Although FIG. 1 shows a case where the terminal device 10 and the information processing device 100 are separate devices, the terminal device 10 and the information processing device 100 may be integrated.

[0019] [Example of information processing] Fig. 2 is an explanatory diagram for explaining a detection process according to an embodiment. Fig. 2A is an explanatory diagram for explaining the type of detection process according to an embodiment, and Fig. 2B is an explanatory diagram for explaining the procedure of the detection process according to an embodiment. Fig. 2B is an explanatory diagram in which arrows indicating the procedure of the detection process according to an embodiment are added to the explanatory diagram of Fig. 2A.

[0020] In Figures 2A and 2B, the shaded (net) area indicates person detection, the shaded (grid) area indicates skeleton detection, the shaded (vertical line) area indicates object detection, and the shaded (horizontal line) area indicates questions targeted by VQA. For example, person detection in the shaded (net) area includes detection of whether a person is in an area where gloves are required or whether a person is in an area where gloves are prohibited. For example, skeleton detection in the shaded (grid) area includes detection of whether hands are visible or whether a person's head is visible. For example, object detection in the shaded (vertical line) area includes detection of whether a person is near a forklift or whether an object is under the body. For example, the set of questions targeted by VQA in the shaded (horizontal line) area includes questions asking whether gloves are required or not and questions asking whether a person has their hands in a press machine. 2A and 2B are merely examples and are not particularly limited. For example, the person detection shown in FIGS. 2A and 2B does not include detection of whether "a person is in an area where helmets are mandatory," but the person detection may include detection of whether "a person is in an area where helmets are mandatory."

[0021] Furthermore, in "first," person detection is performed to determine whether "second" should be confirmed (e.g., whether an object to be confirmed is included). For example, in person detection, a determination is made based on a comparison of the coordinate information of the person with the coordinate information of a preset area. For example, if the coordinate information of the area includes the coordinate information of a person, it is determined that a person is present, and if the coordinate information of the area does not include the coordinate information of a person, it is determined that a person is not present. In addition, in "second," skeleton detection or object detection is performed to determine whether "third" should be confirmed. For example, skeleton detection makes a determination based on the posture of the person, and object detection makes a determination based on objects around the person. Then, if it is determined that "third" should be confirmed, VQA is performed. Then, based on the result of the VQA, a determination is made as to whether the behavior is unsafe. In this way, a determination as to whether the behavior is unsafe is made based on a series of processes in the order of person detection, skeleton detection, and VQA, or a series of processes in the order of person detection, object detection, and VQA.

[0022] Priorities are assigned to each of the question groups targeted by person detection, skeletal detection, object detection, and VQA. The priorities are related to the execution of the detection process. For example, the more urgent the event, the higher the priority assigned to each detection process. For example, in person detection, the priority of person detection is assigned so that the highest priority is given to detecting whether a person is in an area where gloves are required, followed by whether a person is in an area where gloves are prohibited. For example, in skeletal detection, the priority is assigned so that the highest priority is given to detecting whether a person's hands are visible, followed by whether a person's head is visible. For example, in object detection, the priority is assigned so that the highest priority is given to detecting whether a person is near a forklift, followed by whether a person has an object under their body. For example, in the question group targeted by VQA, the priority is assigned so that the highest priority is given to a question asking whether gloves are required, followed by a question asking whether a person has their hands in a press machine. 2A and 2B is an example and is not particularly limited. For example, in the person detection shown in FIG. 2A and 2B, the highest priority is given to detecting whether or not "a person is in an area where wearing gloves is mandatory," but the highest priority may also be given to detecting whether or not "a person is in an area where wearing gloves is prohibited."

[0023] The information processing device 100 performs a determination of all human detections included in "first" and then performs a further determination only on "second" corresponding to a human detection determined as "YES" among the human detections. That is, the information processing device 100 performs a detection (skeleton detection or object detection) for "second" only for items that match the conditions in "first". The information processing device 100 also performs a further determination only on detections (skeleton detection or object detection) that are determined as "YES" among "second" corresponding to a human detection determined as "YES" in "first". That is, the information processing device 100 performs a determination of "third" only for items that match the conditions in "second". Then, the information processing device 100 detects an unsafe behavior when a "YES" is determined in "third". In this way, pruning for "second" is performed in "first", and pruning for "third" is performed in "second". Then, unsafe behavior is detected based on the result of "third". The information processing device 100 performs detection processing in the order of human detection, skeleton detection or object detection, and VQA, as indicated by the arrows in FIG. 2B. Note that the information processing device 100 may perform each determination for each detection processing in order of priority. For example, the information processing device 100 may perform each determination in order of priority for each detection process, and if the determination is "NO," may proceed to processing the next image without performing subsequent priority determinations.

[0024] The detection process in FIG. 2 may be adjustable by turning on / off for each priority number. For example, forklift detection may be turned off because it is not necessary in locations where forklifts are unlikely to enter. In this way, the detection process may be adjusted as needed depending on the installation environment of the camera. This makes it possible to reduce unnecessary analysis processing by turning on / off the function for each priority number.

[0025] 3 is a diagram showing an example of information processing of the information processing system 1 according to the embodiment. In the following embodiment, an example of video analysis will be described in which image analysis is performed, but the present invention is not limited to this example, and moving image analysis may also be performed.

[0026] The information processing device 100 acquires an image G1 (a captured image captured by the imaging device C) to be subjected to the detection process (step S101). After acquiring the image G1, the information processing device 100 performs detection processes for person detection, skeleton detection, and object detection on the image G1 in the order of the arrows shown in FIG. 2B (step S102), and performs VQA based on the results of these detection processes (step S103). Note that the image G1 is an image showing a worker A1 (for example, an employee of the manager U1) performing a predetermined task in a factory.

[0027] 2, the information processing device 100, based on the detection result of whether or not "a person is in an area where gloves are required," if the answer is "YES," performs detection of whether or not "hands are visible" (skeleton detection), which is detection (predetermined detection associated with) corresponding to the person detection of whether or not "a person is in an area where gloves are required," and, based on the detection result of whether or not "hands are visible," if the answer is "YES," asks a question asking whether or not "gloves are required," which is a question to be subjected to VQA (predetermined question to be subjected to VQA associated with) corresponding to the skeletal detection of whether or not "hands are visible," and, based on the determination result of the question asking whether or not "gloves are required," if the answer is "YES," determines the behavior as unsafe. In this case, for example, if the person detection of whether or not "a person is in an area where gloves are required" is "NO," the information processing device 100 does not perform the corresponding skeletal detection or VQA and determines the behavior as safe. Furthermore, for example, if the skeleton detection of whether "hands are visible" returns "NO," the information processing device 100 determines that the behavior is safe without performing the corresponding VQA. Furthermore, for example, if the question of whether "gloves are worn" returns "NO," the information processing device 100 determines that the behavior is safe.

[0028] Similarly, based on the detection result of whether or not "there is a person in the path of the forklift," if the answer is "YES," the information processing device 100 performs detection of whether or not "the person is near the forklift" (object detection), which is detection (predetermined detection linked to) corresponding to the person detection of whether or not "the person is in the path of the forklift," and, based on the detection result of whether or not "the person is near the forklift," if the answer is "YES," asks a question asking whether or not "the distance is closing," which is a question (predetermined question linked to) that is the target of VQA corresponding to the object detection of whether or not "the person is near the forklift," and, based on the determination result of the question asking whether or not "the distance is closing," if the answer is "YES," determines that the behavior is unsafe. In this case, for example, if the person detection of whether or not "there is a person in the path of the forklift" is "NO," the information processing device 100 does not perform the corresponding object detection or VQA and determines that the behavior is safe. Furthermore, for example, if the object detection result of whether or not the forklift is near is "NO," the information processing device 100 does not perform the corresponding VQA and determines that the behavior is safe. Furthermore, for example, if the question of whether or not the distance is getting closer is "NO," the information processing device 100 determines that the behavior is safe.

[0029] In this way, the information processing device 100 makes a judgment on all person detections (9 items in Figure 2) for the "first," makes a judgment on skeleton detections (5 items in Figure 2) or object detections (3 items in Figure 2) for all "second" corresponding to the "first" judged as "YES," and asks questions (15 items in Figure 2) that are the target of VQA for all "third" corresponding to the "second" judged as "YES," thereby judging unsafe behavior.

[0030] The information processing device 100 may then notify the administrator U1, if necessary, depending on the result of the determination of the unsafe behavior (step S104). For example, even if the information processing device 100 determines that the behavior is unsafe, if the urgency is low (e.g., if the urgency is low based on the items or number of items determined to be unsafe behaviors taking priority into consideration), the information processing device 100 may perform processing (which is an example and may not be particularly limited) such as recording (and distributing) an image G1 to the administrator U1. If the urgency is high (e.g., if the urgency is high based on the items or number of items determined to be unsafe behaviors taking priority into consideration), the information processing device 100 may notify the administrator U1 via email, SNS, or the like, and may also perform processing (which is an example and may not be particularly limited) to turn on a patrol lamp or the like at the scene. For example, if the information processing device 100 determines that the behavior is unsafe for an item requiring urgency, such as "falling down," the information processing device 100 may perform the processing for the high urgency case described above.

[0031] In the above embodiment, the information processing device 100 may perform "first" person detection using skeleton detection. Person detection has the advantage of requiring less processing time than skeleton detection, but since multiple AI models must be stored in the system, the capacity of the system itself may become large. Therefore, the information processing device 100 may perform person detection using skeleton detection when it is necessary to reduce the model weight.

[0032] In the above embodiment, the information processing device 100 performs pruning for the VQA of "third" using separate detection processes for "first" and "second." The information processing device 100 may acquire or generate a single condition that combines the conditions for "first" and "second," and perform pruning for the VQA of "third" by a single determination of whether or not such a condition is satisfied. That is, for example, instead of separately detecting whether "a person is in an area where gloves are required" and whether "hands are visible" for "first" and "second," the information processing device 100 may perform pruning for the VQA of "third" by a single determination of whether "a person is in an area where gloves are required" and whether "hands are visible."

[0033] [Configuration of terminal device] Next, the configuration of the terminal device 10 according to the embodiment will be described with reference to Fig. 4. Fig. 4 is a diagram showing an example of the configuration of the terminal device 10 according to the embodiment. As shown in Fig. 4, the terminal device 10 has a communication unit 11, an input unit 12, an output unit 13, and a control unit 14.

[0034] (Communications Department 11) The communication unit 11 is realized by, for example, a network interface card (NIC), etc. The communication unit 11 is connected to a predetermined network N by wire or wirelessly, and transmits and receives information to and from the information processing device 100, etc., via the predetermined network N.

[0035] (Input section 12) The input unit 12 accepts various operations from the administrator. In FIG. 4, the input unit 12 accepts various operations from the administrator U1. For example, the input unit 12 may accept various operations from the administrator via a display screen using a touch panel function. The input unit 12 may also accept various operations from buttons provided on the terminal device 10 or a keyboard or mouse connected to the terminal device 10.

[0036] (Output section 13) The output unit 13 is a display device for displaying various information, such as a display screen of a tablet terminal realized by a liquid crystal display, an organic EL (Electro-Luminescence) display, etc. The output unit 13 displays, for example, the content of a notification transmitted from the information processing device 100 in accordance with the determination result of the VQA (such as the content of a notification transmitted in the case of high urgency).

[0037] (Control unit 14) The control unit 14 is, for example, a controller, and is realized by a CPU (Central Processing Unit), an MPU (Micro Processing Unit), or the like executing various programs stored in a storage device inside the terminal device 10 using a RAM (Random Access Memory) as a work area. For example, these various programs include application programs installed in the terminal device 10. For example, these various programs include an application program that displays notification content transmitted from the information processing device 100 in accordance with the VQA determination result. The control unit 14 is also realized by an integrated circuit, such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0038] As shown in FIG. 4, the control unit 14 has a receiving unit 141 and a transmitting unit 142, and realizes or executes the information processing operations described below.

[0039] (Receiving unit 141) The receiving unit 141 receives, for example, information for displaying notification content transmitted from the information processing device 100 in accordance with the determination result of the VQA. For example, the receiving unit 141 receives information for displaying notification content that a patrol lamp or the like has been turned on at the scene due to high urgency.

[0040] (Transmitter 142) The transmission unit 142 transmits, for example, information about video designated by the administrator for image analysis, and also transmits information about on / off settings adjusted by the administrator for each priority number.

[0041] [Configuration of information processing device] Next, the configuration of the information processing device 100 according to the embodiment will be described with reference to Fig. 5. Fig. 5 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. As shown in Fig. 5, the information processing device 100 has a communication unit 110, a storage unit 120, and a control unit 130. Note that the information processing device 100 may also have an input unit (e.g., a keyboard or a mouse) that accepts various operations from an administrator of the information processing device 100, and a display unit (e.g., a liquid crystal display) that displays various information.

[0042] (Communication unit 110) The communication unit 110 is realized by, for example, a NIC etc. The communication unit 110 is connected to a network N by wire or wirelessly, and transmits and receives information to and from the terminal device 10 etc. via the network N.

[0043] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM or a flash memory, or a storage device such as a hard disk or an optical disk. As shown in FIG. 5 , the storage unit 120 includes a detection processing procedure storage unit 121 and a VQA model storage unit 122.

[0044] The detection processing procedure storage unit 121 stores information about a detection processing procedure for determining a predetermined behavior. An example of the detection processing procedure storage unit 121 according to the embodiment is shown in Fig. 6. As shown in Fig. 6, the detection processing procedure storage unit 121 has items such as "detection processing procedure ID," "priority," "first," "second," and "third."

[0045] "Detection processing procedure ID" indicates identification information for identifying the detection processing procedure. "Priority" indicates the priority of the detection processing. "First", "Second", and "Third" indicate detection processing for determining a predetermined behavior. Note that "Second" is a detection processing that is predetermined in association with "First", and "Third" is a detection processing that is predetermined in association with "Second".

[0046] The VQA model storage unit 122 stores information related to the VQA model. Fig. 7 shows an example of the VQA model storage unit 122 according to the embodiment. As shown in Fig. 7, the VQA model storage unit 122 has items such as "VQA model ID" and "VQA model."

[0047] "VQA model ID" indicates identification information for identifying a VQA model. "VQA model" indicates a VQA model. In the example shown in FIG. 7, conceptual information such as "VQA model #1" and "VQA model #2" is stored in "VQA model," but in reality, training data of the VQA model is stored.

[0048] (control unit 130) The control unit 130 is a controller, and is realized by, for example, a CPU or an MPU executing various programs stored in a storage device inside the information processing device 100 using RAM as a work area. The control unit 130 is also realized by, for example, an integrated circuit such as an ASIC or an FPGA.

[0049] 5, control unit 130 has an acquisition unit 131, a determination unit 132, a judgment unit 133, a notification unit 134, and a processing unit 135, and realizes or executes the information processing action described below. Note that the internal configuration of control unit 130 is not limited to the configuration shown in FIG. 5, and may be any other configuration as long as it performs the information processing described below.

[0050] (Acquisition part 131) The acquiring unit 131 acquires various pieces of information from the storage unit 120. The acquiring unit 131 also stores the acquired various pieces of information in the storage unit 120.

[0051] The acquisition unit 131 acquires various pieces of information from an external information processing device. The acquisition unit 131 acquires various pieces of information from other information processing devices such as the terminal device 10.

[0052] The acquisition unit 131 acquires, for example, a video (which may be a still image or a moving image) to be subjected to the detection process. For example, the acquisition unit 131 acquires a video captured by an imaging device (such as a surveillance camera). For example, the acquisition unit 131 acquires a video captured by an imaging device whose installation position is known.

[0053] (Decision unit 132) The determination unit 132 determines, for example, a question to be subjected to VQA. For example, the determination unit 132 determines a question to be subjected to VQA in response to a predetermined detection processing result. For example, the determination unit 132 determines a question to be subjected to VQA in response to a plurality of types of detection processing results. For example, the determination unit 132 determines a question to be subjected to VQA in response to a combination of a plurality of types of detection processing.

[0054] The determination unit 132 determines, for example, a skeleton detection or object detection process to be executed in response to the result of the human detection process. The determination unit 132 also determines, for example, a question to be subjected to VQA in response to the result of the skeleton detection or object detection process. The determination unit 132 also determines, for example, a question to be subjected to VQA in response to the result of the skeleton detection or object detection process to be executed in response to the result of the human detection process.

[0055] The determination unit 132 determines questions to be subjected to VQA in accordance with, for example, the detection processing results for a predetermined video. Furthermore, the determination unit 132 determines questions to be subjected to VQA for each video in accordance with, for example, the detection processing results for each video. Furthermore, the determination unit 132 determines questions to be subjected to VQA in accordance with, for example, the detection processing results for a predetermined object included in the predetermined video. Furthermore, the determination unit 132 determines questions to be subjected to VQA for each object included in the predetermined video in accordance with, for example, the detection processing results for each object included in the predetermined video.

[0056] (Judgment unit 133) The determination unit 133, for example, determines a predetermined behavior (such as an unsafe behavior). For example, the determination unit 133 determines the predetermined behavior by performing VQA based on the question determined by the determination unit 132. For example, the determination unit 133 determines whether information related to the video acquired by the acquisition unit 131 corresponds to the predetermined behavior.

[0057] The determination unit 133 determines whether to execute the detection process of skeleton detection or object detection based on the result of the detection process of person detection, for example. For example, the determination unit 133 determines to execute the detection process of skeleton detection or object detection when the result of the detection process of person detection is "YES" (such as when the answer result can affirm the predetermined behavior), and determines not to execute the detection process of skeleton detection or object detection when the result of the detection process of person detection is "NO".

[0058] The determination unit 133 determines whether to perform VQA based on, for example, the result of the detection process of skeleton detection or object detection. For example, the determination unit 133 determines to perform VQA when the result of the detection process of skeleton detection or object detection is "YES" (such as when the answer result is likely to affirm a predetermined behavior), and determines not to perform VQA when the result of the detection process of skeleton detection or object detection is "NO".

[0059] The determination unit 133 determines whether or not the behavior corresponds to the predetermined behavior based on, for example, the determination result of the VQA. For example, the determination unit 133 determines that the behavior corresponds to the predetermined behavior when the determination result of the VQA is "YES" (e.g., when the answer result is positive for the predetermined behavior), and determines that the behavior does not correspond to the predetermined behavior when the determination result of the VQA is "NO."

[0060] (Notification Department 134) The notification unit 134 issues a notification according to, for example, the determination result of the VQA by the determination unit 133. For example, the notification unit 134 issues a notification when it is determined that the VQA corresponds to a predetermined behavior according to the determination result of the VQA by the determination unit 133. For example, the notification unit 134 issues a notification when it is determined that an item requiring urgency corresponds to a predetermined behavior. For example, the notification unit 134 issues a notification that there is a possibility that the item corresponds to a predetermined behavior or that there is a high degree of urgency. Furthermore, for example, the notification unit 134 issues a notification that a patrol lamp or the like has been turned on at the scene.

[0061] (Processing section 135) The processing unit 135 performs processing according to, for example, the determination result of the VQA by the determination unit 133. For example, the processing unit 135 performs processing when it is determined that an action corresponds to a predetermined action according to the determination result of the VQA by the determination unit 133. For example, the processing unit 135 performs processing when it is determined that an action corresponds to a predetermined action for an item requiring urgency. For example, the processing unit 135 performs processing such as recording (and distributing) video for an administrator. Furthermore, for example, the processing unit 135 performs processing to turn on a patrol lamp or the like at the scene.

[0062] [Information processing flow] Next, the procedure of information processing by the information processing system 1 according to the embodiment will be described with reference to Fig. 8. Fig. 8 is a flowchart showing the procedure of information processing by the information processing system 1 according to the embodiment.

[0063] As shown in FIG. 8, the information processing device 100 acquires a video image to be subjected to the detection process (step S201).

[0064] The information processing device 100 determines questions to be subjected to VQA (step S202).

[0065] The information processing device 100 determines the predetermined behavior (step S203).

[0066] If it is determined that the behavior corresponds to the predetermined behavior (step S203; YES), the information processing device 100 notifies the user that the behavior may correspond to the predetermined behavior (step S204).

[0067] If it is determined that the behavior does not correspond to the predetermined behavior (step S203; NO), the information processing device 100 records the video for the administrator (step S205).

[0068] [Other system configuration examples] The above-described configuration of the information processing system 1 is merely an example, and the information processing system 1 can employ any device configuration as long as the desired processing is possible. For example, the information processing system 1 may be configured without the terminal device 10. Furthermore, the various processes performed by the information processing device 100 may be executed by any device included in the information processing system 1, not limited to the information processing device 100. For example, at least a part of the various processes performed by the information processing device 100 (e.g., the decision unit 132, the determination unit 133, etc.) may be executed by the terminal device 10. In this way, the various processes performed by the information processing device 100 may be distributed and processed by multiple devices included in the information processing system 1.

[0069] Although an embodiment of the present invention has been described, this embodiment is presented as an example and is not intended to limit the scope of the invention. This embodiment can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and their modifications are within the scope and spirit of the invention, as well as the invention and its equivalents as set forth in the claims. Furthermore, these embodiments and their modifications can be combined as appropriate within the scope of the processing content. [Explanation of symbols]

[0070] 1. Information Processing Systems 10 Terminal Equipment 11 Communications Department 12 Input section 13 Output section 14 Control Unit 100 Information processing device 110 Communications Department 120 Storage section 121 Detection processing procedure memory unit 122 VQA model memory 130 Control Unit 131 Acquisition Department 132 Decision Section 133 Judgment section 134 Notification Department 135 Processing section 141 Receiving unit 142 Transmitter N Network

Claims

1. a determination unit that determines questions to be subjected to VQA (Visual Question Answering) in response to a predetermined detection processing result; a determination unit that determines a predetermined behavior by performing VQA based on the question determined by the determination unit; An information processing system comprising:

2. The determination unit A question to be subjected to VQA is determined in response to a result of a detection process of skeleton detection or object detection that is executed in response to a result of a detection process of person detection.

2. The information processing system according to claim 1, wherein:

3. The determination unit Based on the result of the person detection process, it is determined whether to perform the skeleton detection process or the object detection process.

2. The information processing system according to claim 1, wherein:

4. The determination unit Determine whether to perform VQA based on the results of the skeleton detection or object detection process.

2. The information processing system according to claim 1, wherein:

5. The determination unit Based on the result of the VQA, it is determined whether the predetermined behavior is met.

2. The information processing system according to claim 1, wherein:

6. The determination unit A question to be subjected to VQA is determined for each video in response to the detection processing result for each video.

2. The information processing system according to claim 1, wherein:

7. The determination unit The predetermined behavior is determined to be an unsafe behavior.

2. The information processing system according to claim 1, wherein:

8. 1. A computer-implemented information processing method, comprising: a determination step of determining questions to be subjected to VQA (Visual Question Answering) in response to a predetermined detection processing result; a determination step of determining a predetermined behavior by performing VQA based on the questions determined in the determination step; An information processing method comprising:

Images