Information processing device, information processing method, and information processing program

Abstract

The accuracy of detecting a predetermined event in a system that detects an event from an image can be improved. [Solution] The information processing device of the present application comprises an acquisition unit that acquires information about a video, and a judgment unit that generates frame image information based on the information about the video, which is information that links time information about the time of each of a plurality of frame images that make up the video with object information about the type and position of an object included in each of the plurality of frame images, generates a prompt that includes the frame image information and instruction information that instructs the device to determine whether a specified event has occurred based on the frame image information, and inputs the prompt into a language model to generate judgment information indicating whether the specified event has occurred.

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Conventionally, there has been known a technique for detecting occurrences (hereinafter also referred to as events) belonging to a specific category by analyzing video footage from cameras such as security cameras, surveillance cameras, and drive recorders. For example, there is known a technique that includes a display control means for displaying a category setting screen for setting categories of events included in video together with the video data captured by the surveillance camera, a learning data storage means for storing category information set in response to an operator's operation on the category setting screen together with the video data as learning data, and a video analysis means for analyzing the video data to detect occurrences belonging to a specific category and outputting the detection results, wherein the video analysis means performs a learning process using the learning data stored in the learning data storage means. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Publication No. 2017-225122 Summary of the Invention [Means for solving the problem]

[0004] The information processing device of the present application comprises an acquisition unit that acquires information about a moving image, and a judgment unit that generates frame image information based on the information about the moving image, the frame image information being information that links time information about the time of each of a plurality of frame images that constitute the moving image with object information about the type of object included in each of the plurality of frame images and the position of the object, generates a prompt that includes the frame image information and instruction information that instructs the device to determine whether a specified event has occurred based on the frame image information, and inputs the prompt into a language model to generate judgment information indicating whether the specified event has occurred. [Brief explanation of the drawings]

[0005] [Figure 1] FIG. 1 is a diagram showing an example of a method for detecting a predetermined event from an image. [Figure 2] FIG. 2 is a diagram illustrating an example of the configuration of an information processing system according to the embodiment. [Figure 3] FIG. 3 is a diagram illustrating an example of the configuration of the information processing device according to the embodiment. [Figure 4] FIG. 4 is a diagram illustrating an example of information processing by the information processing device according to the embodiment. [Figure 5] FIG. 5 is a diagram illustrating an example of a prompt according to the embodiment. [Figure 6] FIG. 6 is a diagram illustrating an example of determination information according to the embodiment. [Figure 7] FIG. 7 is a diagram illustrating an example of information processing by an information processing device according to a modified example. [Figure 8] FIG. 8 is a diagram showing an example of a prompt according to the modified example. [Figure 9] FIG. 9 is a diagram illustrating an example of a hardware configuration. DETAILED DESCRIPTION OF THE INVENTION

[0006] Hereinafter, modes for implementing an information processing device, an information processing method, and an information processing program according to the present application (hereinafter referred to as "embodiments") will be described in detail with reference to the drawings. Note that the information processing device, the information processing method, and the information processing program according to the present application are not limited to these embodiments. Furthermore, the same components in the following embodiments will be given the same reference numerals, and duplicated descriptions will be omitted. Note that images according to the present embodiments may include still images and moving images. Furthermore, moving images may be referred to as moving images hereinafter. Furthermore, moving images include video.

[0007] (Embodiment) 1. Introduction Conventionally, a technology for detecting an event belonging to a specific category (hereinafter, a predetermined event) from an image is known. FIG. 1 is a diagram illustrating an example of a method for detecting a predetermined event from an image. FIG. 1 illustrates a case in which a terminal device 10 according to an embodiment determines whether a vehicle has violated a stop sign, as an example of a predetermined event. Note that the predetermined event is not limited to a vehicle's violation of a stop sign. For example, the predetermined event may be a vehicle ignoring a traffic signal. In this way, the predetermined event may be dangerous driving or driving violating traffic rules. Furthermore, the predetermined event is not limited to dangerous driving or driving violating traffic rules, but may also be a criminal act such as shoplifting or the appearance of a suspicious person. In this way, the predetermined event may be any event that is a target of monitoring for an administrator who performs image-based monitoring. For example, the terminal device 10 is an information processing device equipped with a drive recorder. The terminal device 10 acquires a video. For example, the terminal device 10 acquires a video from the drive recorder. When the terminal device 10 acquires a video, the terminal device 10 inputs each of a plurality of frame images constituting the video into an object detection model and acquires information output from the object detection model. For example, the terminal device 10 acquires, as information output from the object detection model, information linking time information regarding the time of each of a plurality of frame images constituting a video with object information regarding the type and position of an object included in each of the plurality of frame images. In FIG. 1, the terminal device 10 inputs each of the plurality of frame images constituting a video G1 into the object detection model to detect a stop line O1 and a stop road sign O2 included in each of the plurality of frame images constituting the video G1. The terminal device 10 also acquires, as the object information, the position coordinates of each of the stop line O1 and road sign O2 included in each of the plurality of frame images. For example, the terminal device 10 acquires, as the object information, the coordinates of the vertices of the bounding boxes surrounding each of the stop lines O1 and road signs O2 included in each of the plurality of frame images.

[0008] Here, the object detection model is a machine learning model that detects a predetermined object from an input image. Specifically, the object detection model is a machine learning model that is trained to output, when a video is input, information linking time information regarding the time of each of a plurality of frame images constituting the video with object information regarding the type and position of an object included in each of the plurality of frame images. For example, the object detection model may include a convolutional neural network (CNN). Furthermore, the object detection model may be a known object detection model such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Networks), or DETR (End-to-End Object Detection with Transformers).

[0009] Furthermore, the terminal device 10 determines whether a predetermined event has occurred based on information output from the object detection model and a predetermined event detection rule. Here, the event detection rule is a rule that defines conditions for determining whether a predetermined event has occurred. For example, the event detection rule is a rule that defines multiple conditions for determining whether a predetermined event has occurred. The terminal device 10 determines whether each of the multiple conditions defined in the event detection rule is satisfied based on the type of object captured in the image, the position of the object, the length of time the object is captured in the image, etc. For example, the terminal device 10 may determine that a predetermined event has occurred if it determines that a predetermined number of the multiple conditions defined in the event detection rule are satisfied. On the other hand, the terminal device 10 may determine that a predetermined event has not occurred if it determines that a predetermined number of the multiple conditions defined in the event detection rule are not satisfied. For example, an event detection rule for detecting whether a stop sign violation has occurred may be a rule that defines three conditions: (1) whether a stop road sign is detected; (2) whether a stop line is detected; and (3) whether the time from when the stop line appears in the image until when the stop line disappears is within three seconds. The terminal device 10 determines whether each of the conditions (1) to (3) is satisfied based on the positions of the stop line O1 and the road sign O2 in the image and the length of time the stop line O1 appears in the image. For example, if the terminal device 10 determines that all of the conditions (1) to (3) are satisfied, it determines that a stop sign violation has occurred. On the other hand, if the terminal device 10 determines that the condition (3) among the conditions (1) to (3) is not satisfied, it determines that a stop sign violation has not occurred. Note that if the terminal device 10 cannot determine whether each of the conditions defined in the event detection rule is satisfied, it may notify the administrator of the information processing system 1 that a visual inspection of the image is required.

[0010] Furthermore, among systems that detect predetermined events from images such as those described in Figure 1, there are systems that do not allow for false positives or false negatives. In such systems that do not allow for false positives or false negatives, human inspection (visual inspection) may be required in addition to mechanical event detection using machine learning models, etc. In this case, there is a problem in that personnel must be secured and deployed for the human inspection, resulting in high personnel costs for operation.

[0011] In response to this, an information processing device according to an embodiment acquires information about a moving image, and generates, based on the information about the moving image, a prompt including frame image information that links time information about the time of each of a plurality of frame images constituting the moving image with object information about the type and position of an object included in each of the plurality of frame images, and instruction information that instructs the device to determine whether a predetermined event has occurred based on the frame image information, and inputs the prompt into a language model to generate determination information indicating whether the predetermined event has occurred.

[0012] Here, the frame image information can be considered to be an extracted summary of information necessary to determine whether a predetermined event has occurred from a video. The frame image information can be considered to be a summary of information necessary to determine whether a predetermined event has occurred from a video. Therefore, by inputting the frame image information as a prompt into a language model, the information processing device can improve the accuracy of event detection using the language model. Therefore, the information processing device can improve the event detection accuracy in a system that detects predetermined events from images. In other words, the information processing device can improve the detection accuracy in detecting mechanical events. Furthermore, because the information processing device can improve the event detection accuracy in detecting mechanical events, it can reduce personnel costs required for visual inspections. Furthermore, because the information processing device can reduce personnel costs required for visual inspections, it can reduce the operating costs of a system that detects predetermined events from images.

[0013] The language model may be any machine learning model that is a generative AI capable of processing natural language, and may be a large-scale language model or a small-scale language model. The following describes the case where the language model is a large-scale language model. The large-scale language model may be, for example, a GPT (Generative Pre-trained Transformer) model.

[0014] [2. Information Processing System Configuration] Fig. 2 is a diagram showing an example of the configuration of an information processing system 1 according to an embodiment. As shown in Fig. 2, the information processing system 1 according to an embodiment includes a terminal device 10, an information processing device 100, and an administrator device 200. The terminal device 10, the information processing device 100, and the administrator device 200 are connected to each other via a predetermined communication network (network N) so as to be able to communicate with each other via wired or wireless communication. Note that the information processing system 1 may include a plurality of terminal devices 10, a plurality of information processing devices 100, and a plurality of administrator devices 200.

[0015] The terminal device 10 is an information processing device equipped with a camera such as a security camera, a surveillance camera, or a drive recorder. For example, the terminal device 10 may be a household electrical appliance (home appliance), a mobile device such as a smartphone owned by a person, or an automobile. The terminal device 10 may also be referred to as an edge device. The camera generates video (image) of the environment around the camera. The terminal device 10 acquires the video from the camera. When the terminal device 10 acquires the video, it inputs each of multiple frame images constituting the video into an object detection model and acquires information output from the object detection model. The terminal device 10 also determines whether a predetermined event has occurred based on the information output from the object detection model and predetermined event detection rules.

[0016] Furthermore, the terminal device 10 calculates a first reliability for the determination result of whether or not a predetermined event has occurred. Specifically, the terminal device 10 calculates the first reliability for the determination result of whether or not a predetermined event has occurred, determined using the object detection model of the terminal device 10. For example, the terminal device 10 calculates a higher first reliability as the number of conditions determined to be satisfied among the multiple conditions defined in the event detection rule increases. For example, the terminal device 10 calculates the first reliability to be the highest when all of the multiple conditions defined in the event detection rule are satisfied. On the other hand, the terminal device 10 calculates a lower first reliability as the number of conditions determined to be satisfied among the multiple conditions defined in the event detection rule decreases. When the terminal device 10 calculates the first reliability, it transmits the determination result of whether or not a predetermined event has occurred and the first reliability to the information processing device 100.

[0017] The information processing device 100 is an information processing device that performs information processing according to the embodiment. For example, the information processing device 100 may perform information processing according to the embodiment in accordance with an information processing method realized by an information processing program according to the embodiment. The information processing device 100 corresponds to a cloud computer (server device).

[0018] The administrator device 200 is, for example, an information processing device used by an administrator of the information processing system 1. For example, the administrator device 200 is an information processing device such as a desktop PC (Personal Computer) or a notebook PC. The administrator device 200 may also be a smart device such as a smartphone or a tablet. The administrator device 200 displays information received from the information processing device 100 or the like using a web browser or an application.

[0019] 3. Configuration of Information Processing Device 3 is a diagram showing an example of the configuration of the information processing device 100 according to the embodiment. The information processing device 100 according to the embodiment includes a communication unit 110, a storage unit 120, and a control unit .

[0020] (Communication unit 110) The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the terminal device 10 and the administrator device 200. For example, the communication unit 110 is realized by a NIC (Network Interface Card), an antenna, or the like.

[0021] (Storage unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a random access memory (RAM) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores an information processing program according to the embodiment. The storage unit 120 also stores information related to machine learning models. For example, the storage unit 120 stores information related to large language models (LLMs) as an example of a machine learning model. The storage unit 120 also stores information related to an object detection model as an example of a machine learning model.

[0022] (control unit 130) The control unit 130 is a controller, and is realized, for example, by a CPU (Central Processing Unit) or an MPU (Micro Processing Unit) executing various programs stored in a storage device inside the information processing device 100 using RAM as a work area. The control unit 130 is also a controller, and is realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0023] The control unit 130 has an acquisition unit 131, a determination unit 132, and a provision unit 133 as functional units, and may realize or execute the information processing actions described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in FIG. 3, and may have other configurations as long as they perform the information processing described below. Furthermore, each functional unit indicates a function of the control unit 130, and does not necessarily have to be physically distinct.

[0024] (Acquisition part 131) The acquisition unit 131 acquires various information. Specifically, the acquisition unit 131 acquires a first reliability regarding a determination result of whether or not a predetermined event has occurred by the terminal device 10 that generated the video, from the terminal device 10. More specifically, the acquisition unit 131 acquires the first reliability regarding a determination result of whether or not a predetermined event has occurred, determined by the terminal device 10 using an object detection model of the terminal device 10.

[0025] The acquisition unit 131 also acquires information related to the video. For example, the acquisition unit 131 acquires the video from the terminal device 10 as an example of the information related to the video. For example, the acquisition unit 131 may acquire the video from the terminal device 10 together with the first reliability. The acquisition unit 131 also inputs each of a plurality of frame images into an object detection model that detects a predetermined object from an input image, and acquires information output from the object detection model, as the information related to the video. For example, when acquiring a video from the terminal device 10, the acquisition unit 131 inputs each of a plurality of frame images constituting the video into the object detection model, and acquires information output from the object detection model. Note that the acquisition unit 131 may acquire, from the terminal device 10, information output from the object detection model of the terminal device 10.

[0026] (Judgment unit 132) The determination unit 132 determines whether a predetermined event has occurred from the video. FIG. 4 is a diagram illustrating an example of information processing by the information processing device 100 according to the embodiment. In FIG. 4, the acquisition unit 131 acquires a reliability (first reliability) of the event determination using the object detection model of the terminal device 10 (step S101). For example, the acquisition unit 131 acquires the first reliability along with the video from the terminal device 10. Furthermore, the determination unit 132 determines whether the first reliability acquired by the acquisition unit 131 exceeds a first threshold (step S102). When the determination unit 132 determines that the first reliability exceeds the first threshold (step S102; Yes), the determination unit 132 ends the processing. Note that when the determination unit 132 determines that the first reliability exceeds the first threshold, the provision unit 133 may transmit information regarding the determination result of the event determination using the object detection model of the terminal device 10 to the administrator device 200.

[0027] On the other hand, if the determination unit 132 determines that the first reliability does not exceed the first threshold (step S102; No), it performs event determination using the object detection model of the information processing device 100 (step S103). For example, if the determination unit 132 determines that the first reliability is equal to or less than the first threshold, it acquires information output from the object detection model from the acquisition unit 131. Next, the determination unit 132 determines whether a predetermined event has occurred based on the information output from the object detection model and a predetermined event detection rule. Here, the event detection rule is, for example, a rule that manually defines conditions for determining whether a predetermined event has occurred. For example, the event detection rule is a rule that defines multiple conditions for determining whether a predetermined event has occurred. The determination unit 132 determines whether each of multiple conditions defined by the event detection rule is satisfied based on the type of object captured in the image, the position of the object, the length of time the object is captured in the image, etc. For example, the determination unit 132 may determine that a predetermined event has occurred if it determines that a predetermined number of conditions among the multiple conditions defined by the event detection rule are satisfied. On the other hand, the determination unit 132 may determine that a predetermined event has not occurred if it determines that a predetermined number of conditions among the multiple conditions defined by the event detection rule are not satisfied.

[0028] Furthermore, the determination unit 132 calculates a second reliability for the determination result of whether or not a predetermined event has occurred. For example, the determination unit 132 calculates a higher second reliability as the number of conditions determined to be satisfied among the multiple conditions defined in the event detection rule increases. For example, the determination unit 132 calculates the second reliability to be the highest when all of the multiple conditions defined in the event detection rule are satisfied. On the other hand, the determination unit 132 calculates a lower second reliability as the number of conditions determined to be satisfied among the multiple conditions defined in the event detection rule decreases. Furthermore, the determination unit 132 determines whether or not the second reliability exceeds a second threshold (step S104). If the determination unit 132 determines that the second reliability exceeds the second threshold (step S104; Yes), the process ends. In addition, when the determination unit 132 determines that the second reliability exceeds the second threshold, the providing unit 133 may transmit information regarding the determination result of the event determination using the object detection model of the information processing device 100 to the administrator device 200.

[0029] On the other hand, when the determination unit 132 determines that the second reliability is equal to or lower than the second threshold (step S104; No), it performs event determination using the language model of the information processing device 100 (step S105). For example, the determination unit 132 performs event determination using the large-scale language model of the information processing device 100. For example, based on information about the moving image acquired by the acquisition unit 131, the determination unit 132 generates frame image information, which is information in which time information about the time of each of a plurality of frame images constituting the moving image is associated with object information about the type and position of an object included in each of the plurality of frame images. The determination unit 132 also generates a prompt including the frame image information and instruction information that instructs the determination unit 132 to determine whether a predetermined event has occurred based on the frame image information. In other words, the determination unit 132 generates a prompt including the frame image information, which is information in which time information and object information are paired. The determination unit 132 also inputs the generated prompt to the large-scale language model to generate determination information indicating whether a predetermined event has occurred.

[0030] FIG. 5 is a diagram illustrating an example of a prompt according to the embodiment. The determination unit 132 acquires instruction information by referring to the storage unit 120. For example, the instruction information may be a sentence prepared in advance by a system administrator. For example, in FIG. 5, the determination unit 132 acquires the following sentence as instruction information A1: "The video from the drive recorder is being analyzed to determine whether stop sign ignoring has occurred. Does the following situation indicate stop sign ignoring?" The determination unit 132 also generates a prompt P1 that includes the instruction information A1.

[0031] Furthermore, the determination unit 132 acquires condition information by referring to the storage unit 120. For example, the condition information may be a sentence prepared in advance by a system administrator. The determination unit 132 generates a prompt including condition information that defines the occurrence condition of a predetermined event, and inputs the prompt into a language model to generate determination information. For example, the determination unit 132 generates a prompt including the condition information, and inputs the prompt into a large-scale language model to generate determination information. In FIG. 5, the determination unit 132 acquires the sentence "Note that ignoring a stop sign means 'stopping at a stop line near a stop sign for a period of 3 seconds or less (including not stopping at all)'" as condition information B1 that defines the occurrence condition of ignoring a stop sign. Furthermore, the determination unit 132 generates a prompt P1 including the condition information B1.

[0032] The determination unit 132 also inputs information output from the object detection model into a language model to generate frame image information, which is information linking object information, which is a sentence describing each of a plurality of frame images, with time information. For example, the determination unit 132 inputs the information output from the object detection model into a large-scale language model to generate frame image information. The determination unit 132 also generates a prompt including the generated frame image information and instruction information, and inputs the generated prompt into the large-scale language model to generate determination information. In FIG. 5, the determination unit 132 generates a character string "1 second" as time information T11 indicating the time of the frame image one second from the start of the video. The determination unit 132 also generates a sentence, "A stop sign and a stop line are visible," as object information C11 regarding the type and position of an object included in the frame image one second from the start of the video. Furthermore, the determination unit 132 generates frame image information F11, which is a sentence saying, "1 second: A stop sign and a stop line are visible," as information linking the time information T11 and the object information C11. The determination unit 132 also generates a prompt P1 including the frame image information F11.

[0033] 5, the determination unit 132 generates a character string "2 seconds" as time information T12 indicating the time of the frame image 2 seconds from the start of the video. The determination unit 132 also generates a sentence, "The stop sign and stop line at 1 second are shown even larger, and the stop line is located at the bottom of the screen," as object information C12 regarding the type and position of an object included in the frame image 2 seconds from the start of the video. The determination unit 132 also generates frame image information F12, which is a sentence, "2 seconds: The stop sign and stop line at 1 second are shown even larger, and the stop line is located at the bottom of the screen," as information linking the time information T12 and the object information C12. The determination unit 132 also generates a prompt P1 including the frame image information F12.

[0034] 5, the determination unit 132 generates a character string "3 to 8 seconds" as time information T13 indicating the time of the frame image 3 to 8 seconds from the start of the video. The determination unit 132 also generates a sentence "There is no change in the image" as object information C13 regarding the type and position of an object included in the frame image 3 to 8 seconds from the start of the video. The determination unit 132 also generates frame image information F13, which is the sentence "3 to 8 seconds: There is no change in the image," as information linking the time information T13 and the object information C13. The determination unit 132 also generates a prompt P1 including the frame image information F13.

[0035] 5, the determination unit 132 generates a character string "9 seconds" as time information T14 indicating the time of the frame image 9 seconds from the start of the video. The determination unit 132 also generates a sentence "The stop sign and stop line are no longer visible" as object information C14 regarding the type and position of an object included in the frame image 9 seconds from the start of the video. The determination unit 132 also generates frame image information F14, which is the sentence "9 seconds: The stop sign and stop line are no longer visible," as information linking the time information T14 and the object information C14. The determination unit 132 also generates a prompt P1 including the frame image information F14.

[0036] In addition, in FIG. 5, the determination unit 132 generates a prompt P1 that displays a plurality of frame image information F11 to F14 arranged in the chronological order of the frame images.

[0037] Furthermore, the determination unit 132 calculates a third reliability regarding the determination result of whether or not a predetermined event has occurred. Specifically, the determination unit 132 inputs a prompt including an instruction sentence instructing the large-scale language model to calculate a third reliability regarding the determination result of whether or not a predetermined event has occurred, to cause the large-scale language model to generate the third reliability. Furthermore, the determination unit 132 determines whether or not the third reliability exceeds a third threshold (step S106). If the determination unit 132 determines that the third reliability exceeds the third threshold (step S106; Yes), the determination unit 132 terminates the processing. Note that, if the determination unit 132 determines that the third reliability exceeds the third threshold, the providing unit 133 may transmit information regarding the determination result of the event determination using the large-scale language model of the information processing device 100 to the administrator device 200.

[0038] (Provider 133) The providing unit 133 transmits information related to the determination result by the determining unit 132 to the administrator device 200. For example, if the determining unit 132 determines that the third reliability is equal to or less than the third threshold value (step S106; No), the providing unit 133 transmits a notification to the administrator device 200 that a visual inspection is required (step S107).

[0039] As described above, the determination unit 132 generates determination information when the first reliability is equal to or less than the first threshold. Specifically, when the first reliability is equal to or less than the first threshold, the determination unit 132 generates frame image information, generates a prompt including the frame image information and instruction information, and inputs the generated prompt to the large-scale language model to cause the large-scale language model to generate determination information. Furthermore, the determination unit 132 generates determination information when the second reliability is equal to or less than the second threshold. Specifically, when the second reliability is equal to or less than the second threshold, the determination unit 132 generates frame image information, generates a prompt including the frame image information and instruction information, and inputs the generated prompt to the large-scale language model to cause the large-scale language model to generate determination information.

[0040] FIG. 6 is a diagram illustrating an example of determination information according to an embodiment. In FIG. 6, the determination unit 132 generates determination information R1 based on a prompt P1. The determination information R1 includes information D11 indicating an analysis procedure for determining whether a predetermined event has occurred, information D12 indicating a reason for the determination, and information D13 indicating a determination result. In FIG. 6, the determination unit 132 generates determination information including the following sentence as information D11 indicating the analysis procedure: "By definition, ignoring a stop sign is defined as 'a stopping time at a stop line near a stop sign being within three seconds (including cases where the vehicle does not stop)'." Based on this definition, the situation is analyzed as follows: 1. The stop sign and stop line first appear in the video at 1 second. 2. The vehicle continues moving until the stop line reaches the bottom of the screen at 2 seconds. 3. From the description that there is no change in the video for 6 seconds, from 3 seconds to 8 seconds, it can be inferred that the vehicle was stopped. 4. Finally, from the description that the stop sign and stop line disappear from the video at 9 seconds, it can be inferred that the vehicle started moving again." The determination unit 132 also generates determination information including information D12 indicating a reason for determining whether a predetermined event has occurred. In FIG. 6, the determination unit 132 generates determination information including the sentence, "Based on this assumption, it is believed that the vehicle was stopped at the stop sign and stop line for approximately six seconds. In other words, this violates the definition of ignoring a stop sign, which is that 'the stopping time at a stop line near a stop sign is within three seconds,' and this driving cannot be said to be ignoring a stop sign." The determination unit 132 also generates determination information including information D13 indicating a determination result of whether a predetermined event has occurred. In FIG. 6, the determination unit 132 generates determination information including the sentence, "It is determined that ignoring a stop sign has not occurred." as information D13 indicating the determination result.

[0041] [4. Modifications] FIG. 7 is a diagram illustrating an example of information processing by the information processing device 100 according to a modified example. Note that in FIG. 7, description of processes similar to those in FIG. 4 will be omitted. In FIG. 7, the acquisition unit 131 acquires the reliability (first reliability) of the event determination using the object detection model of the terminal device 10 (step S201). Furthermore, the determination unit 132 determines whether the first reliability acquired by the acquisition unit 131 exceeds a fourth threshold (step S202). If the determination unit 132 determines that the first reliability exceeds the fourth threshold (step S202; Yes), the determination unit 132 ends the processing. Note that if the determination unit 132 determines that the first reliability exceeds the fourth threshold, the provision unit 133 may transmit information regarding the determination result of the event determination using the object detection model of the terminal device 10 to the administrator device 200.

[0042] On the other hand, if the determination unit 132 determines that the first reliability does not exceed the fourth threshold (step S202; No), it executes event determination using the large-scale language model of the information processing device 100 (step S203). The determination unit 132 also calculates a third reliability regarding the determination result of whether or not a predetermined event has occurred. The determination unit 132 also determines whether or not the third reliability exceeds a fifth threshold (step S204). If the determination unit 132 determines that the third reliability exceeds the fifth threshold (step S204; Yes), it terminates the processing. Note that, if the determination unit 132 determines that the third reliability exceeds the fifth threshold, the providing unit 133 may transmit information regarding the determination result of the event determination using the large-scale language model of the information processing device 100 to the administrator device 200.

[0043] On the other hand, when the determination unit 132 determines that the third reliability is equal to or less than the fifth threshold (step S204; No), it executes event determination using the object detection model of the information processing device 100 (step S205). Furthermore, the determination unit 132 calculates a second reliability regarding the determination result of whether or not a predetermined event has occurred. Furthermore, the determination unit 132 determines whether or not the second reliability exceeds a sixth threshold (step S206). When the determination unit 132 determines that the second reliability exceeds the sixth threshold (step S206; Yes), it ends the processing. Note that, when the determination unit 132 determines that the second reliability exceeds the sixth threshold, the providing unit 133 may transmit information regarding the determination result of the event determination using the object detection model of the information processing device 100 to the administrator device 200.

[0044] On the other hand, if the determining unit 132 determines that the second reliability is equal to or less than the sixth threshold (step S206; No), it transmits a notification to the administrator device 200 that a visual inspection is required (step S207).

[0045] FIG. 8 is a diagram illustrating an example of a prompt according to a modified example. The prompt P1 illustrated in FIG. 5 includes frame image information F11 to F14 generated by inputting information output from the object detection model into a large-scale language model. Thus, the prompt P1 illustrated in FIG. 5 includes data obtained by processing the information output from the object detection model using the large-scale language model. In contrast, the prompt P2 illustrated in FIG. 8 differs from the prompt illustrated in FIG. 5 in that it includes frame image information F21 to F23, which is information output from the object detection model. Thus, the prompt P2 illustrated in FIG. 8 includes information (unprocessed data) output from the object detection model. In FIG. 8, the determination unit 132 acquires instruction information by referring to the storage unit 120. For example, in FIG. 8, the determination unit 132 acquires, as instruction information A2, a sentence such as, "I am analyzing a drive recorder and would like to determine whether a certain video shows a driver running a red light." Furthermore, the determination unit 132 generates a prompt P2 including the instruction information A2.

[0046] Furthermore, the determination unit 132 acquires condition information by referring to the storage unit 120. For example, in Fig. 8, the determination unit 132 acquires the following sentence as condition information B2 defining the occurrence conditions of red light ignoring: "#Definition of red light ignoring: When the stop line is passed, the closest traffic light (the traffic light with the smallest y coordinate of the upper side of the bounding box) is red." Furthermore, the determination unit 132 generates a prompt P2 including the condition information B2.

[0047] The determination unit 132 also references the storage unit 120 to acquire structural information that defines the structure of the data to be analyzed by the large-scale language model. The structural information may be information that defines tags for the data. The determination unit 132 acquires structural information including information indicating the timestamp of the frame image, identification information for identifying an object detected by the object detection model, information indicating the type of object detected by the object detection model, and information indicating the position of the object detected by the object detection model (e.g., coordinate information within the image). For example, in FIG. 8, the determination unit 132 acquires the following sentence as definition information B3: "# Tag of log used for analysis · timestamp: timestamp of the frame where processing was performed · i: object recognition index · class: class number of the recognized object (class8 = traffic light, class10 = stop line) · bbox: upper left x coordinate, upper left y coordinate, lower right x coordinate, lower right y coordinate of the bounding box · speed: current vehicle speed (km / h) · trafficcolor: lighting color of the nearest traffic light (R = red, U = unknown)." The determination unit 132 also generates a prompt P2 that includes the definition information B3.

[0048] Furthermore, the determination unit 132 generates frame image information based on the information output from the object detection model, generates a prompt including the frame image information and instruction information, and inputs the prompt into a language model to generate determination information. Specifically, the determination unit 132 generates a prompt including the frame image information and instruction information, and inputs the prompt into a large-scale language model to generate determination information. For example, when a prompt including the frame image information and instruction information is input, the determination unit 132 may input the generated prompt into a large-scale language model trained to output determination information to generate determination information. Furthermore, the information output from the object detection model includes information indicating the timestamp of each of the multiple frame images input to the object detection model, identification information identifying an object included in each of the multiple frame images, information indicating the type of object, and information indicating the position of the object (e.g., coordinate information within the image). For example, the determination unit 132 generates frame image information including information indicating the timestamp of each of the multiple frame images input to the object detection model as time information regarding the time of each of the multiple frame images constituting the video. Furthermore, the determination unit 132 generates frame image information including information indicating the type of object included in each of the plurality of frame images and information indicating the position of the object as object information relating to the type of object included in each of the plurality of frame images and the position of the object. The determination unit 132 also generates frame image information in which time information and object information are linked.

[0049] 8, for example, determination unit 132 generates frame image information F21 including a timestamp "timestamp, 1649854366959" output from the object detection model as time information T21 indicating the time of each of the multiple frame images input to the object detection model. Furthermore, determination unit 132 generates frame image information F21 including a character string "class, 3" indicating the object class output from the object detection model as information C211 regarding the type of object included in each of the multiple frame images. Furthermore, determination unit 132 generates frame image information F21 including position coordinates of a bounding box surrounding an object, "bbox, 0.583060, 0.560044, 0.659778, 0.655942," output from the object detection model as information C212 regarding the position of an object included in each of the multiple frame images. The determination unit 132 also generates frame image information F21 in which time information T21, information C211 about the type of object, and information C212 about the position of the object are linked together. Similarly, the determination unit 132 generates frame image information F22 in which time information T22, information C221 about the type of object, and information C222 about the position of the object are linked together. The determination unit 132 also generates frame image information F23 in which time information T23, information C231 about the type of object, and information C232 about the position of the object are linked together.

[0050] In the above-described embodiment, the case where the information processing device 100 performs event determination using a large-scale language model has been described, but the terminal device 10 may perform event determination using a large-scale language model. For example, the terminal device 10 may perform event determination using a large-scale language model instead of performing event determination using an object detection model. Furthermore, the terminal device 10 may perform both event determination using an object detection model and event determination using a large-scale language model.

[0051] The information processing device 100 may calculate the determination accuracy of event determination using the object detection model based on the determination result of event determination using the object detection model. The information processing device 100 may calculate the determination accuracy of event determination using the large-scale language model based on the determination result of event determination using the large-scale language model. The information processing device 100 may compare the determination accuracy of event determination using the object detection model with the determination accuracy of event determination using the large-scale language model, and set the model with better determination accuracy as the model with higher priority. For example, the information processing device 100 may preferentially perform event determination using the model set as the model with higher priority. The information processing device 100 also acquires information regarding cloud resource usage. When the cloud resource usage is less than a predetermined threshold, the information processing device 100 preferentially performs event determination using the model with higher priority. On the other hand, when the cloud resource usage is equal to or greater than a predetermined threshold, the information processing device 100 preferentially performs event determination using the model with lower resource consumption.

[0052] In addition, the information processing device 100 may assign weights to the determination results of event determination using an object detection model and the determination results of event determination using a large-scale language model according to the respective accuracy of the determination accuracy of event determination using an object detection model and the determination accuracy of event determination using a large-scale language model, and obtain an overall determination result of both.

[0053] [5. Effects] As described above, the information processing device 100 according to the embodiment includes an acquisition unit 131 and a determination unit 132. The acquisition unit 131 acquires information related to a moving image. Based on the information related to the moving image, the determination unit 132 generates frame image information that associates time information related to the time of each of a plurality of frame images constituting the moving image with object information related to the type and position of an object included in each of the plurality of frame images, generates a prompt that includes the frame image information and instruction information that instructs the device to determine whether or not a predetermined event has occurred based on the frame image information, and inputs the prompt to a language model to generate determination information indicating whether or not the predetermined event has occurred.

[0054] As described above, the frame image information can be considered to be an extracted summary of information necessary to determine whether a predetermined event has occurred in a video. The frame image information can be considered to be a summary of information necessary to determine whether a predetermined event has occurred in a video. Therefore, the information processing device 100 can improve the accuracy of event detection by the language model by inputting the frame image information as a prompt into the language model. Therefore, the information processing device 100 can improve the event detection accuracy in a system that detects predetermined events from images. Furthermore, the information processing device 100 can improve the event detection accuracy in a system that detects predetermined events from images, thereby contributing to the achievement of Goal 9 of the Sustainable Development Goals (SDGs), "Build resilient infrastructure, promote inclusive and sustainable industrialization, and promote innovation and foster resilience." In other words, the information processing device 100 can improve the detection accuracy in mechanical event detection. Furthermore, because the information processing device 100 can improve the event detection accuracy in mechanical event detection, it can reduce personnel costs required for visual inspections. Furthermore, because the information processing device 100 can reduce personnel costs required for visual inspections, it can reduce the operating costs of a system that detects predetermined events from images.

[0055] Furthermore, the acquisition unit 131 inputs each of a plurality of frame images to an object detection model that detects a predetermined object from the input image as information related to the moving image, and acquires information output from the object detection model. The determination unit 132 generates frame image information based on the information output from the object detection model, generates a prompt including the frame image information and instruction information, and inputs the prompt to a language model to generate determination information.

[0056] This allows the information processing device 100 to efficiently generate frame image information based on the information output from the object detection model.

[0057] In addition, the judgment unit 132 inputs the information output from the object detection model into a language model to generate frame image information, which is information that links object information, which is sentences that explain each of the multiple frame images, with time information, generates a prompt that includes the frame image information and instruction information, and inputs the prompt into the language model to generate judgment information.

[0058] This allows the information processing device 100 to generate judgment information based on frame image information, which is in a sentence format that is easy for the language model to understand, thereby improving the accuracy of event detection in a system that detects specified events from images.

[0059] Furthermore, the determination unit 132 generates a prompt including condition information that defines the occurrence condition of a predetermined event, and inputs the prompt into a language model to generate determination information.

[0060] This allows the information processing device 100 to generate judgment information based on whether or not the conditions for the occurrence of a specified event are met, thereby improving the accuracy of event detection in a system that detects a specified event from an image.

[0061] Furthermore, the acquisition unit 131 acquires a first reliability regarding the determination result of whether or not a predetermined event has occurred by the terminal device 10 that generated the video. If the first reliability is equal to or less than a first threshold, the determination unit 132 generates determination information.

[0062] As a result, the information processing device 100 generates judgment information when the reliability of the judgment result by the terminal device 10 as to whether or not a specified event has occurred is low, thereby improving the accuracy of event detection in a system that detects specified events from images.

[0063] Furthermore, the acquisition unit 131 inputs each of a plurality of frame images to an object detection model that detects a predetermined object from the input image as information related to the moving image, and acquires information output from the object detection model. The determination unit 132 determines whether or not a predetermined event has occurred based on the information output from the object detection model and a predetermined event detection rule, calculates a second reliability regarding the determination result of whether or not the predetermined event has occurred, and generates determination information if the second reliability is equal to or less than a second threshold.

[0064] As a result, the information processing device 100 generates judgment information when the reliability of the judgment result as to whether or not a specified event has occurred is low based on the information output from the object detection model and predetermined event detection rules, thereby improving the event detection accuracy in a system that detects specified events from images.

[0065] [6. Hardware Configuration] The terminal device 10 and the information processing device 100 according to the above-described embodiments are realized by, for example, a computer 1000 configured as shown in Fig. 9. The following description will be given taking the information processing device 100 as an example. Fig. 9 is a diagram showing an example of a hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which a calculation device 1030, a primary storage device 1040, a secondary storage device 1050, an output I / F (Interface) 1060, an input I / F 1070, and a network I / F 1080 are connected via a bus 1090.

[0066] The arithmetic device 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, programs read from the input device 1020, and the like, and executes various processes. The arithmetic device 1030 is realized by, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or the like.

[0067] The primary storage device 1040 is a memory device such as a RAM (Random Access Memory) that temporarily stores data used by the arithmetic device 1030 for various calculations. The secondary storage device 1050 is a storage device in which data used by the arithmetic device 1030 for various calculations and various databases are registered, and is realized by a ROM (Read Only Memory), an HDD (Hard Disk Drive), an SSD (Solid State Drive), a flash memory, or the like. The secondary storage device 1050 may be an internal storage device or an external storage device. The secondary storage device 1050 may also be a removable storage medium such as a USB (Universal Serial Bus) memory or an SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), a NAS (Network Attached Storage), a file server, or the like.

[0068] The output I / F 1060 is an interface for transmitting information to be output to an output device 1010 that outputs various types of information, such as a display, a projector, a printer, etc., and is realized by a connector conforming to a standard such as USB (Universal Serial Bus), DVI (Digital Visual Interface), or HDMI (High Definition Multimedia Interface), etc. The input I / F 1070 is an interface for receiving information from various input devices 1020, such as a mouse, a keyboard, a keypad, a button, a scanner, etc., and is realized by a USB, etc.

[0069] Furthermore, the output I / F 1060 and the input I / F 1070 may be wirelessly connected to the output device 1010 and the input device 1020, respectively. That is, the output device 1010 and the input device 1020 may be wireless devices.

[0070] The output device 1010 and the input device 1020 may be integrated into one device, such as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated into one device as an input / output I / F.

[0071] The input device 1020 may be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), a DVD (Digital Versatile Disc), or a PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0072] The network I / F 1080 receives data from other devices via the network N and sends it to the arithmetic device 1030, and also transmits data generated by the arithmetic device 1030 to other devices via the network N.

[0073] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output I / F 1060 and the input I / F 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0074] For example, when the computer 1000 functions as the information processing device 100, the arithmetic unit 1030 of the computer 1000 executes a program loaded onto the primary storage device 1040 to realize the functions of the control unit 130. The arithmetic unit 1030 of the computer 1000 may also load a program acquired from another device via the network I / F 1080 onto the primary storage device 1040 and execute the loaded program. The arithmetic unit 1030 of the computer 1000 may also cooperate with the other device via the network I / F 1080 to call and use the functions and data of a program from another program of the other device.

[0075] [7. Other] Although the embodiments of the present application have been described above, the present invention is not limited to the contents of these embodiments. Furthermore, the above-described components include those that can be easily imagined by a person skilled in the art, those that are substantially the same, and those that are within the scope of so-called equivalents. Furthermore, the above-described components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the spirit of the above-described embodiments.

[0076] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically using a known method. In addition, the information including the processing procedures, specific names, various data, and parameters shown in the above documents and drawings can be changed as desired unless otherwise specified. For example, the various information shown in each drawing is not limited to the information shown in the drawings.

[0077] Furthermore, the components of each device shown in the figure are conceptual functional components and do not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit depending on various loads, usage conditions, etc.

[0078] For example, the information processing device 100 described above may be realized by multiple server computers, and depending on the function, the configuration can be flexibly changed, such as by calling an external platform using an API (Application Programming Interface) or network computing.

[0079] Furthermore, the above-described embodiments and modifications can be combined as appropriate within the scope of not causing any contradiction in the processing content. [Explanation of symbols]

[0080] 1. Information Processing Systems 10 Terminal Equipment 100 Information processing device 110 Communications Department 120 Storage section 130 control section 131 Acquisition Department 132 Judgment section 133 Provision Department

Claims

1. an acquisition unit that acquires information about the video; a determination unit that generates frame image information based on information about the video, the frame image information being information linking time information about the time of each of a plurality of frame images constituting the video with object information about the type of object included in each of the plurality of frame images and the position of the object; generates a prompt including the frame image information and instruction information that instructs the system to determine whether or not a predetermined event has occurred based on the frame image information; and inputs the prompt into a language model to generate determination information indicating whether or not the predetermined event has occurred; An information processing device comprising:

2. The acquisition unit inputting each of the plurality of frame images into an object detection model that detects a predetermined object from an input image as information about the moving image, and acquiring information output from the object detection model; The determination unit generating the frame image information based on the information output from the object detection model, generating the prompt including the frame image information and the instruction information, and inputting the prompt into a language model to generate the determination information; The information processing device according to claim 1 .

3. The determination unit inputting information output from the object detection model into a language model to generate frame image information, which is information in which the object information, which is a sentence describing each of the plurality of frame images, is linked with the time information; generating the prompt, which includes the frame image information and the instruction information; and inputting the prompt into the language model to generate the determination information. The information processing device according to claim 2 .

4. The determination unit generating the prompt including condition information defining a condition for the occurrence of the predetermined event, and inputting the prompt into a language model to generate the determination information; The information processing device according to claim 1 .

5. The acquisition unit acquiring a first reliability regarding a determination result of whether or not the predetermined event has occurred by the terminal device that generated the video; The determination unit generating the determination information when the first reliability is equal to or less than a first threshold; The information processing device according to claim 1 .

6. The acquisition unit inputting each of the plurality of frame images into an object detection model that detects a predetermined object from an input image as information about the moving image, and acquiring information output from the object detection model; The determination unit determining whether the predetermined event has occurred based on information output from the object detection model and a predetermined event detection rule, calculating a second reliability regarding the determination result of whether the predetermined event has occurred, and generating the determination information if the second reliability is equal to or less than a second threshold value; The information processing device according to claim 1 .

7. An information processing method realized by a program executed by an information processing device, an acquisition step of acquiring information about the video; a determination step of generating frame image information based on information about the video, the frame image information being information linking time information about the time of each of a plurality of frame images constituting the video with object information about the type of object included in each of the plurality of frame images and the position of the object; generating a prompt including the frame image information and instruction information instructing the user to determine whether or not a predetermined event has occurred based on the frame image information; and inputting the prompt into a language model to generate determination information indicating whether or not the predetermined event has occurred. An information processing method including:

8. a retrieval procedure for retrieving information about the video; a determination step of generating frame image information based on information about the video, the frame image information being information linking time information about the time of each of a plurality of frame images constituting the video with object information about the type of object included in each of the plurality of frame images and the position of the object; generating a prompt including the frame image information and instruction information instructing the user to determine whether or not a predetermined event has occurred based on the frame image information; and inputting the prompt into a language model to generate determination information indicating whether or not the predetermined event has occurred. An information processing program that causes a computer to execute the above.

Images