Information processing device, information processing method, and information processing program

Abstract

This can improve the accuracy of event detection in a system that detects specific events from images. [Solution] The information processing device according to the present invention comprises: an acquisition unit that acquires information related to a video; a determination unit that generates frame image information, which is information that links time information relating to the time of each of a plurality of frame images constituting the video and object information relating to the type of object and the position of the object contained in each of the plurality of frame images, and generates a prompt that includes the frame image information and instruction information that instructs the determination of whether or not a predetermined event has occurred based on the frame image information, and inputs the prompt to a language model to generate determination information indicating whether or not a predetermined event has occurred.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, there is known a technique for analyzing images of cameras such as security cameras, surveillance cameras, or drive recorders to detect events (hereinafter also referred to as events) belonging to a specific category. For example, display control means for displaying a category setting screen for setting the category of an event included in the video together with the video of the video data captured by the surveillance camera, and category information set according to an operator's operation on the category setting screen are used as learning data together with the video data. Learning data storage means for storing, and video analysis means for analyzing video data to detect an event belonging to a specific category and outputting the detection result. The video analysis means is known to perform learning processing using the learning data stored in the learning data storage means.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Means for Solving the Problems

[0004] The information processing device according to the present invention includes: an acquisition unit that acquires information related to a video; a determination unit that generates frame image information based on the video information, which is information that links time information relating to the time of each of a plurality of frame images constituting the video and object information relating to the type of object contained in each of the plurality of frame images and the position of the object; a prompt that includes the frame image information and instruction information that instructs the determination unit to determine whether or not a predetermined event has occurred based on the frame image information; and inputs the prompt to a language model to generate determination information indicating whether or not the predetermined event has occurred. [Brief explanation of the drawing]

[0005] [Figure 1] Figure 1 shows an example of a method for detecting a predetermined event from an image. [Figure 2] Figure 2 shows an example of the configuration of an information processing system according to the embodiment. [Figure 3] Figure 3 shows an example of the configuration of an information processing device according to the embodiment. [Figure 4] Figure 4 shows an example of information processing by the information processing device according to the embodiment. [Figure 5] Figure 5 shows an example of a prompt according to the embodiment. [Figure 6] Figure 6 shows an example of determination information according to the present invention. [Figure 7] Figure 7 shows an example of information processing by an information processing device related to a modified example. [Figure 8] Figure 8 shows an example of a prompt related to a modified example. [Figure 9] Figure 9 shows an example of a hardware configuration. [Modes for carrying out the invention]

[0006] The following describes in detail, with reference to the drawings, embodiments for implementing the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing apparatus, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted. The images in these embodiments may include both still images and moving images. In the following, moving images may be referred to as moving images. Moving images include video footage.

[0007] (Embodiment) [1. Introduction] Conventionally, techniques for detecting events belonging to a specific category (hereinafter referred to as "predetermined events") from images are known. Figure 1 is a diagram illustrating an example of a method for detecting predetermined events from images. In Figure 1, the case in which the terminal device 10 according to the embodiment determines whether or not a vehicle has violated a stop sign as an example of a predetermined event is described. Note that the predetermined event is not limited to a vehicle violating a stop sign. For example, the predetermined event may be running a red light. Thus, the predetermined event may be dangerous driving or driving that violates traffic rules. Furthermore, the predetermined event is not limited to dangerous driving or driving that violates traffic rules, but may also be criminal acts such as shoplifting or the appearance of a suspicious person. Thus, the predetermined event may be any event that is subject to monitoring by an administrator performing image-based monitoring. For example, the terminal device 10 is an information processing device equipped with a drive recorder. The terminal device 10 acquires video. For example, the terminal device 10 acquires video from the drive recorder. When the terminal device 10 acquires video, it inputs each of the multiple frame images constituting the video into an object detection model and acquires the information output from the object detection model. For example, terminal device 10 obtains information output from the object detection model that links time information for each of the multiple frame images constituting the video with object information regarding the type of object and the position of the object contained in each of the multiple frame images. In Figure 1, terminal device 10 inputs each of the multiple frame images constituting the video G1 into the object detection model to detect the stop line O1 and the stop road sign O2 contained in each of the multiple frame images constituting the video G1. Terminal device 10 also obtains the position coordinates of each of the stop line O1 and the road sign O2 contained in each of the multiple frame images as object information. For example, terminal device 10 obtains the coordinates of the vertices of the bounding boxes surrounding each of the stop line O1 and the road sign O2 contained in each of the multiple frame images as object information.

[0008] Here, the object detection model is a machine learning model that detects a predetermined object from an input image. Specifically, the object detection model is a machine learning model that, when a video is input, outputs information that links time information for each of the multiple frame images constituting the video with object information regarding the type of object contained in each of the multiple frame images and the position of the object. For example, the object detection model may include a convolutional neural network (CNN). Alternatively, the object detection model may be a known object detection model such as YOLO (You Only Look Once), SSD (Single Shot MultiBox Detector), R-CNN (Region-based Convolutional Neural Networks), or DETR (nd-to-End Object Detection with Transformers).

[0009] Furthermore, the terminal device 10 determines whether a predetermined event has occurred based on the information output from the object detection model and predetermined event detection rules. Here, the event detection rules are rules that define conditions for determining whether a predetermined event has occurred. For example, an event detection rule is a rule that defines multiple conditions for determining whether a predetermined event has occurred. The terminal device 10 determines whether each of the multiple conditions defined by the event detection rule is met, based on the type of object in the image, the position of the object, and the length of time the object is in the image. For example, if the terminal device 10 determines that a predetermined number of the multiple conditions defined by the event detection rule are met, it may determine that a predetermined event has occurred. On the other hand, if the terminal device 10 determines that a predetermined number of the multiple conditions defined by the event detection rule are not met, it may determine that a predetermined event has not occurred. For example, an event detection rule for detecting whether or not a stop sign violation has occurred may be a rule that defines three conditions: (1) whether a stop sign is detected, (2) whether a stop line is detected, and (3) whether or not the time from when the stop line appears in the image until it disappears from view is within 3 seconds. The terminal device 10 determines whether or not each of conditions (1) to (3) is met based on the positions of the stop line O1 and the road sign O2 in the image and the length of time that the stop line O1 is visible in the image. For example, if the terminal device 10 determines that all of conditions (1) to (3) are met, it determines that a stop sign violation has occurred. On the other hand, if the terminal device 10 determines that condition (3) of (1) to (3) is not met, it determines that no stop sign violation has occurred. If the terminal device 10 is unable to determine whether or not each of the multiple conditions defined by the event detection rule is met, it may notify the administrator of the information processing system 1 that a visual inspection of the image is necessary.

[0010] Furthermore, among systems that detect predetermined events from images, as explained in Figure 1, there are systems where false detections or false detections are unacceptable. In such systems where false detections or false detections are unacceptable, it may be necessary to combine mechanical event detection using machine learning models with visual inspection by humans. In this case, it is necessary to secure and deploy personnel for visual inspection, which presents the challenge of increased personnel costs in operation.

[0011] In response to this, the information processing device according to the embodiment acquires information related to a video. The information processing device also generates a prompt that includes, based on the information related to the video, frame image information which is information that links time information related to the time of each of the multiple frame images constituting the video with object information related to the type of object and the position of the object contained in each of the multiple frame images, and instruction information which instructs the device to determine whether or not a predetermined event has occurred based on the frame image information. The prompt is input to a language model to generate determination information indicating whether or not a predetermined event has occurred.

[0012] Here, frame image information can be considered as an extraction of an overview of the information necessary to determine whether or not a predetermined event has occurred from a video. Frame image information can be considered as a summary of the information necessary to determine whether or not a predetermined event has occurred from a video. Therefore, the information processing device can improve the accuracy of event detection by the language model by inputting frame image information as a prompt to the language model. Consequently, the information processing device can improve the accuracy of event detection in a system that detects predetermined events from images. In other words, the information processing device can improve the detection accuracy in mechanical event detection. Furthermore, since the information processing device can improve the accuracy of event detection in mechanical event detection, it can reduce the personnel costs associated with visual inspection. Furthermore, since the information processing device can reduce the personnel costs associated with visual inspection, it can reduce the operating costs of a system that detects predetermined events from images.

[0013] The language model can be any machine learning model capable of processing natural language, and it can be either a large-scale or small-scale language model. Below, we will explain the case where the language model is a large-scale language model. A large-scale language model may be, for example, a GPT (Generative Pre-trained Transformer) model.

[0014] [2. Configuration of the Information Processing System] Figure 2 shows an example configuration of an information processing system 1 according to an embodiment. As shown in Figure 2, the information processing system 1 according to an embodiment includes a terminal device 10, an information processing device 100, and an administrator device 200. The terminal device 10, the information processing device 100, and the administrator device 200 are connected to each other via a predetermined communication network (network N) by wired or wireless means. The information processing system 1 may include multiple terminal devices 10, multiple information processing devices 100, and multiple administrator devices 200.

[0015] The terminal device 10 is an information processing device equipped with a camera such as a security camera, a surveillance camera, or a drive recorder. For example, the terminal device 10 may be a household electrical appliance (home appliance), a mobile device such as a smartphone owned by a person, an automobile, or the like. Note that the terminal device 10 may be referred to as an edge device. The camera generates a video (image) regarding the environment around the camera. The terminal device 10 acquires the video from the camera. When the terminal device 10 acquires the video, the terminal device 10 inputs each of a plurality of frame images constituting the video into an object detection model and acquires the information output from the object detection model. Further, the terminal device 10 determines whether or not a predetermined event has occurred based on the information output from the object detection model and a predetermined event detection rule.

[0016] Further, the terminal device 10 calculates a first reliability regarding the determination result as to whether or not a predetermined event has occurred. Specifically, the terminal device 10 calculates a first reliability regarding the determination result as to whether or not a predetermined event determined using the object detection model of the terminal device 10 has occurred. For example, the terminal device 10 calculates the first reliability higher as the number of conditions determined to satisfy the conditions among a plurality of conditions defined by the event detection rule is larger. For example, when all of a plurality of conditions defined by the event detection rule are satisfied, the terminal device 10 calculates the first reliability to be the highest. On the other hand, the terminal device 10 calculates the first reliability lower as the number of conditions determined to satisfy the conditions among a plurality of conditions defined by the event detection rule is smaller. When the terminal device 10 calculates the first reliability, the terminal device 10 transmits the determination result as to whether or not a predetermined event has occurred and the first reliability to the information processing device 100.

[0017] The information processing device 100 is an information processing device that performs the information processing according to the embodiment. For example, the information processing device 100 may perform the information processing according to the information processing method realized by the information processing program according to the embodiment. The information processing device 100 corresponds to a cloud computer (server device).

[0018] The management device 200 is an information processing device used by, for example, the administrator of the information processing system 1. For example, the management device 200 is an information processing device such as a desktop PC (Personal Computer) or a notebook PC. Also, the management device 200 may be a smart device such as a smartphone or a tablet. Further, the management device 200 displays the information received from the information processing device 100 etc. on a web browser or an application.

[0019] [3. Configuration of Information Processing Device] FIG. 3 is a diagram showing a configuration example of the information processing device 100 according to the embodiment. The information processing device 100 according to the embodiment includes a communication unit 110, a storage unit 120, and a control unit 130.

[0020] (Communication Unit 110) The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the terminal device 10 and the management device 200. For example, the communication unit 110 is realized by a NIC (Network Interface Card), an antenna, or the like.

[0021] (Storage Unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores the information processing program according to the embodiment. Also, the storage unit 120 stores information regarding the machine learning model. For example, the storage unit 120 stores information regarding a large language model (LLM: Large Language Models) as an example of the machine learning model. Also, the storage unit 120 stores information regarding an object detection model as an example of the machine learning model.

[0022] (Control Unit 130) The control unit 130 is a controller, and is realized, for example, by executing various programs stored in the memory device inside the information processing device 100 using RAM as the working area, using a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc. Alternatively, the control unit 130 is a controller and can be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0023] The control unit 130 has an acquisition unit 131, a determination unit 132, and a provision unit 133 as functional units, and may realize or execute the information processing operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 3, and other configurations are also acceptable as long as they perform the information processing described later. Furthermore, each functional unit represents the function of the control unit 130 and does not necessarily have to be physically separated.

[0024] (Acquisition part 131) The acquisition unit 131 acquires various types of information. Specifically, the acquisition unit 131 acquires a first confidence level from the terminal device 10 regarding the determination result of whether or not a predetermined event occurred by the terminal device 10 that generated the video. More specifically, the acquisition unit 131 acquires a first confidence level regarding the determination result of whether or not a predetermined event occurred, which was determined by the terminal device 10 using its object detection model.

[0025] Furthermore, the acquisition unit 131 acquires information related to the video. For example, as an example of information related to the video, the acquisition unit 131 acquires the video from the terminal device 10. For example, the acquisition unit 131 may acquire the video from the terminal device 10 along with the first confidence level. Also, as information related to the video, the acquisition unit 131 inputs each of the multiple frame images into an object detection model that detects predetermined objects from the input image, and acquires the information output from the object detection model. For example, when the acquisition unit 131 acquires the video from the terminal device 10, it inputs each of the multiple frame images that make up the video into the object detection model and acquires the information output from the object detection model. The acquisition unit 131 may also acquire the information output from the object detection model of the terminal device 10 from the terminal device 10.

[0026] (Judgment unit 132) The determination unit 132 determines whether a predetermined event has occurred from the video. Figure 4 is a diagram showing an example of information processing by the information processing device 100 according to the embodiment. In Figure 4, the acquisition unit 131 acquires the confidence level (first confidence level) of the event determination using the object detection model of the terminal device 10 (step S101). For example, the acquisition unit 131 acquires the first confidence level along with the video from the terminal device 10. The determination unit 132 also determines whether the first confidence level acquired by the acquisition unit 131 exceeds a first threshold (step S102). If the determination unit 132 determines that the first confidence level exceeds the first threshold (step S102; Yes), the process ends. If the determination unit 132 determines that the first confidence level exceeds the first threshold, the provision unit 133 may transmit information regarding the determination result of the event determination using the object detection model of the terminal device 10 to the administrator device 200.

[0027] On the other hand, if the determination unit 132 determines that the first confidence level does not exceed the first threshold (step S102; No), it performs event determination using the object detection model of the information processing device 100 (step S103). For example, if the determination unit 132 determines that the first confidence level is less than or equal to the first threshold, it acquires the information output from the object detection model from the acquisition unit 131. Subsequently, the determination unit 132 determines whether a predetermined event has occurred based on the information output from the object detection model and predetermined event detection rules. Here, the event detection rules are rules that define conditions for determining whether a predetermined event has occurred, for example, manually. For example, the event detection rules are rules that define multiple conditions for determining whether a predetermined event has occurred. The determination unit 132 determines whether each of the multiple conditions defined by the event detection rules is met based on the type of object in the image, the position of the object, and the length of time the object is in the image. For example, the determination unit 132 may determine that a predetermined event has occurred if it determines that a predetermined number of conditions among the multiple conditions defined by the event detection rule are met. On the other hand, the determination unit 132 may determine that a predetermined event has not occurred if it determines that a predetermined number of conditions among the multiple conditions defined by the event detection rule are not met.

[0028] Furthermore, the determination unit 132 calculates a second confidence level regarding the determination result of whether or not a predetermined event has occurred. For example, the determination unit 132 calculates a higher second confidence level the more conditions among the multiple conditions defined by the event detection rule are determined to be met. For example, the determination unit 132 calculates the highest second confidence level when all of the multiple conditions defined by the event detection rule are met. On the other hand, the determination unit 132 calculates a lower second confidence level the fewer conditions among the multiple conditions defined by the event detection rule are determined to be met. The determination unit 132 also determines whether or not the second confidence level exceeds a second threshold (step S104). If the determination unit 132 determines that the second confidence level exceeds a second threshold (step S104; Yes), it terminates the process. Furthermore, if the determination unit 132 determines that the second confidence level exceeds the second threshold, the provision unit 133 may transmit information regarding the determination result of the event determination using the object detection model of the information processing device 100 to the administrator device 200.

[0029] On the other hand, if the determination unit 132 determines that the second confidence level is less than or equal to the second threshold (step S104; No), it performs event determination using the language model of the information processing device 100 (step S105). For example, the determination unit 132 performs event determination using the large-scale language model of the information processing device 100. For example, the determination unit 132 generates frame image information, which is information that links time information for each of the multiple frame images constituting the video with object information, which is information that links the type of object and the position of the object contained in each of the multiple frame images, based on the video information acquired by the acquisition unit 131. The determination unit 132 also generates a prompt that includes the frame image information and instruction information that instructs the system to determine whether or not a predetermined event has occurred based on the frame image information. In other words, the determination unit 132 generates a prompt that includes frame image information, which is information that is a pair of time information and object information, and instruction information. The determination unit 132 also inputs the generated prompt to the large-scale language model to generate determination information indicating whether or not a predetermined event has occurred.

[0030] Figure 5 shows an example of a prompt according to the embodiment. The determination unit 132 obtains instruction information by referring to the storage unit 120. For example, the instruction information may be a sentence prepared in advance by the system administrator. For example, in Figure 5, the determination unit 132 obtains the sentence "We are analyzing the drive recorder footage to determine whether a stop sign violation has occurred. Does the following situation constitute a stop sign violation?" as instruction information A1. The determination unit 132 also generates a prompt P1 that includes instruction information A1.

[0031] Furthermore, the determination unit 132 obtains condition information by referring to the storage unit 120. For example, the condition information may be a sentence prepared in advance by the system administrator. The determination unit 132 generates a prompt containing condition information that defines the conditions for the occurrence of a predetermined event, inputs the prompt into the language model, and generates determination information. For example, the determination unit 132 generates a prompt containing condition information, inputs the prompt into the large-scale language model, and generates determination information. In Figure 5, the determination unit 132 obtains the sentence "Furthermore, ignoring a stop sign means that the stopping time at the stop line near the stop sign is 3 seconds or less (including cases where no stop was made)" as condition information B1 that defines the conditions for the occurrence of ignoring a stop sign. The determination unit 132 also generates a prompt P1 containing condition information B1.

[0032] Furthermore, the determination unit 132 inputs the information output from the object detection model into the language model to generate frame image information, which is information that links object information (a sentence describing each of multiple frame images) with time information. For example, the determination unit 132 inputs the information output from the object detection model into the large-scale language model to generate frame image information. The determination unit 132 also generates a prompt that includes the generated frame image information and instruction information, and inputs the generated prompt into the large-scale language model to generate determination information. In Figure 5, the determination unit 132 generates the string "1 second" as time information T11 indicating the time of the frame image at 1 second from the start of the video. The determination unit 132 also generates the sentence "A stop sign and stop line are visible" as object information C11 concerning the type of object and the position of the object included in the frame image at 1 second from the start of the video. Furthermore, the determination unit 132 generates frame image information F11, which is the text "1 second: Stop sign and stop line are visible," as information that links time information T11 and object information C11. The determination unit 132 also generates prompt P1, which includes frame image information F11.

[0033] In Figure 5, the determination unit 132 generates the string "2 seconds" as time information T12, which indicates the time of the frame image at 2 seconds from the start of the video. The determination unit 132 also generates the sentence "The stop sign and stop line from 1 second are shown larger, and the stop line is located at the bottom of the screen" as object information C12, which concerns the type and position of the object included in the frame image at 2 seconds from the start of the video. The determination unit 132 also generates frame image information F12, which is the sentence "2 seconds: The stop sign and stop line from 1 second are shown larger, and the stop line is located at the bottom of the screen" as information linking the time information T12 and object information C12. The determination unit 132 also generates a prompt P1 that includes the frame image information F12.

[0034] In Figure 5, the determination unit 132 generates the string "3-8 seconds" as time information T13, which indicates the time of the frame image from 3 to 8 seconds from the start of the video. The determination unit 132 also generates the sentence "No changes in the image" as object information C13, which concerns the type and position of the object included in the frame image from 3 to 8 seconds from the start of the video. The determination unit 132 also generates frame image information F13, which is the sentence "3-8 seconds: No changes in the image," as information linking the time information T13 and the object information C13. The determination unit 132 also generates a prompt P1 that includes the frame image information F13.

[0035] In Figure 5, the determination unit 132 generates the string "9 seconds" as time information T14, which indicates the time of the frame image at 9 seconds from the start of the video. The determination unit 132 also generates the sentence "The stop sign and stop line are no longer visible" as object information C14, which concerns the type and position of the object contained in the frame image at 9 seconds from the start of the video. The determination unit 132 also generates frame image information F14, which is the sentence "9 seconds: The stop sign and stop line are no longer visible," as information linking the time information T14 and the object information C14. The determination unit 132 also generates a prompt P1 that includes the frame image information F14.

[0036] Furthermore, in Figure 5, the determination unit 132 generates a prompt P1 that displays multiple frame image information F11 to F14 arranged in the chronological order of the frame images.

[0037] Furthermore, the determination unit 132 calculates a third confidence level regarding the determination result of whether or not a predetermined event has occurred. Specifically, the determination unit 132 inputs a prompt to the large-scale language model that includes an instruction to calculate a third confidence level regarding the determination result of whether or not a predetermined event has occurred, causing the large-scale language model to generate the third confidence level. The determination unit 132 also determines whether or not the third confidence level exceeds a third threshold (step S106). If the determination unit 132 determines that the third confidence level exceeds the third threshold (step S106; Yes), it terminates the process. If the determination unit 132 determines that the third confidence level exceeds the third threshold, the provision unit 133 may transmit information regarding the determination result of the event determination using the large-scale language model of the information processing device 100 to the administrator device 200.

[0038] (Provider 133) The supply unit 133 transmits information regarding the determination result by the determination unit 132 to the administrator device 200. For example, if the determination unit 132 determines that the third confidence level is below the third threshold (step S106; No), the supply unit 133 sends a notification to the administrator device 200 indicating that a visual inspection is required (step S107).

[0039] As described above, the determination unit 132 generates determination information if the first confidence level is below the first threshold. Specifically, if the first confidence level is below the first threshold, the determination unit 132 generates frame image information, generates a prompt including the frame image information and instruction information, inputs the generated prompt to the large-scale language model, and causes the large-scale language model to generate determination information. Furthermore, the determination unit 132 generates determination information if the second confidence level is below the second threshold. Specifically, if the second confidence level is below the second threshold, the determination unit 132 generates frame image information, generates a prompt including the frame image information and instruction information, inputs the generated prompt to the large-scale language model, and causes the large-scale language model to generate determination information.

[0040] Figure 6 is a diagram showing an example of judgment information according to the embodiment. In Figure 6, the judgment unit 132 generates judgment information R1, which includes information D11 indicating the analysis procedure for determining whether a predetermined event has occurred, information D12 indicating the reason for the judgment, and information D13 indicating the judgment result, based on prompt P1. In Figure 6, the judgment unit 132 generates judgment information including the following text as information D11 indicating the analysis procedure: "According to the definition, ignoring a stop sign means that the stopping time at the stop line near the stop sign is 3 seconds or less (including cases where the vehicle did not stop). In accordance with this definition, the situation is analyzed as follows: 1. The stop sign and stop line first appeared in the video at 1 second. 2. The car continued to move until the stop line was at the bottom of the screen at 2 seconds. 3. From the description that there was no change in the video for 6 seconds from 3 seconds to 8 seconds, it can be inferred that the car was stopped. 4. Finally, at 9 seconds, the stop sign and stop line disappeared from the video, so it can be inferred that the car started moving again." Furthermore, the determination unit 132 generates determination information including information D12 indicating the reason for determining whether or not a predetermined event has occurred. In Figure 6, the determination unit 132 generates determination information including the following sentence as information D12 indicating the reason for the determination: "Based on this estimation, it is thought that the vehicle was stopped at the stop sign and stop line for approximately 6 seconds. In other words, this contradicts the definition of ignoring a stop sign, which is that the stopping time at the stop line near the stop sign is 3 seconds or less, and therefore this driving cannot be said to be ignoring a stop sign." Furthermore, the determination unit 132 generates determination information including information D13 indicating the result of determining whether or not a predetermined event has occurred. In Figure 6, the determination unit 132 generates determination information including the following sentence as information D13 indicating the result of the determination: "It is determined that no ignoring of a stop sign occurred."

[0041] [4. Variations] Figure 7 shows an example of information processing by the information processing device 100 according to a modified example. Note that in Figure 7, the same processing as in Figure 4 is omitted from the explanation. In Figure 7, the acquisition unit 131 acquires the confidence level (first confidence level) of the event determination using the object detection model of the terminal device 10 (step S201). The determination unit 132 then determines whether the first confidence level acquired by the acquisition unit 131 exceeds the fourth threshold (step S202). If the determination unit 132 determines that the first confidence level exceeds the fourth threshold (step S202; Yes), the process ends. Note that if the determination unit 132 determines that the first confidence level exceeds the fourth threshold, the provision unit 133 may transmit information regarding the determination result of the event determination using the object detection model of the terminal device 10 to the administrator device 200.

[0042] On the other hand, if the determination unit 132 determines that the first confidence level does not exceed the fourth threshold (step S202; No), it performs event determination using the large-scale language model of the information processing device 100 (step S203). The determination unit 132 also calculates a third confidence level regarding the determination result of whether or not a predetermined event has occurred. The determination unit 132 also determines whether or not the third confidence level exceeds the fifth threshold (step S204). If the determination unit 132 determines that the third confidence level exceeds the fifth threshold (step S204; Yes), it terminates the process. If the determination unit 132 determines that the third confidence level exceeds the fifth threshold, the provision unit 133 may transmit information regarding the determination result of the event determination using the large-scale language model of the information processing device 100 to the administrator device 200.

[0043] On the other hand, if the determination unit 132 determines that the third confidence level is less than or equal to the fifth threshold (step S204; No), it performs event determination using the object detection model of the information processing device 100 (step S205). The determination unit 132 also calculates a second confidence level regarding the determination result of whether or not a predetermined event has occurred. The determination unit 132 also determines whether or not the second confidence level exceeds the sixth threshold (step S206). If the determination unit 132 determines that the second confidence level exceeds the sixth threshold (step S206; Yes), it terminates the process. If the determination unit 132 determines that the second confidence level exceeds the sixth threshold, the providing unit 133 may transmit information regarding the determination result of the event determination using the object detection model of the information processing device 100 to the administrator device 200.

[0044] On the other hand, if the determination unit 132 determines that the second confidence level is below the sixth threshold (step S206; No), it sends a notification to the administrator device 200 indicating that a visual inspection is required (step S207).

[0045] Figure 8 shows an example of a prompt related to a modified example. The prompt P1 shown in Figure 5 includes frame image information F11~F14 generated by inputting information output from the object detection model into a large-scale language model. Thus, the prompt P1 shown in Figure 5 includes data processed by the large-scale language model from information output from the object detection model. In contrast, the prompt P2 shown in Figure 8 differs from Figure 5 in that it includes frame image information F21~F23, which is information output from the object detection model. Thus, the prompt P2 shown in Figure 8 includes information output from the object detection model (unprocessed data). In Figure 8, the determination unit 132 obtains instruction information by referring to the storage unit 120. For example, in Figure 8, the determination unit 132 obtains the sentence "We are analyzing a drive recorder and want to determine if a certain video shows a 'running a red light'" as instruction information A2. The determination unit 132 also generates prompt P2 which includes instruction information A2.

[0046] Furthermore, the determination unit 132 obtains condition information by referring to the storage unit 120. For example, in Figure 8, the determination unit 132 obtains the following sentence as condition information B2, which defines the conditions for a traffic light to run a red light: "#Definition of running a red light: When passing the stop line, the nearest traffic light (the one with the smallest y-coordinate of the top edge of the bounding box) is red." The determination unit 132 also generates a prompt P2 that includes the condition information B2.

[0047] Furthermore, the determination unit 132 refers to the storage unit 120 and obtains structural information that defines the structure of the data to be parsed by the large-scale language model. The structural information may be information that defines the tags of the data. The determination unit 132 obtains structural information that includes information indicating the timestamp of the frame image, identification information that identifies the object detected by the object detection model, information indicating the type of object detected by the object detection model, and information indicating the position of the object detected by the object detection model (for example, coordinate information in the image). For example, in Figure 8, the determination unit 132 obtains the following text as definition information B3: "#Log tags to be used for analysis ·timestamp: timestamp of the frame that was processed ·i: index of object recognition ·class: class number of the recognized object (class8=traffic light, class10=stop line) ·bbox: x coordinate of the top left, y coordinate of the top left, x coordinate of the bottom right, y coordinate of the bottom right of the bounding box ·speed: current vehicle speed (km / h) ·traffcolor: color of the nearest traffic light (R=red, U=unknown)". The determination unit 132 also generates a prompt P2 that includes the definition information B3.

[0048] Furthermore, the determination unit 132 generates frame image information based on the information output from the object detection model, generates a prompt including the frame image information and instruction information, and inputs the prompt to the language model to generate determination information. Specifically, the determination unit 132 generates a prompt including the frame image information and instruction information, and inputs the prompt to the large-scale language model to generate determination information. For example, the determination unit 132 may input the generated prompt to a large-scale language model that has been trained to output determination information when a prompt including the frame image information and instruction information is input, in order to generate determination information. The information output from the object detection model includes information indicating the timestamp of each of the multiple frame images input to the object detection model, identification information that identifies the object contained in each of the multiple frame images, information indicating the type of object, and information indicating the position of the object (e.g., coordinate information in the image). For example, the determination unit 132 generates frame image information that includes information indicating the timestamp of each of the multiple frame images input to the object detection model as time information relating to the time of each of the multiple frame images constituting the video. Furthermore, the determination unit 132 generates frame image information that includes information indicating the type of object and information indicating the position of the object contained in each of the multiple frame images, as object information relating to the type of object and the position of the object contained in each of the multiple frame images. In addition, the determination unit 132 generates frame image information in which time information and object information are linked.

[0049] For example, in Figure 8, the determination unit 132 generates frame image information F21 containing a timestamp "timestamp, 1649854366959" output from the object detection model, as time information T21 indicating the time of each of the multiple frame images input to the object detection model. The determination unit 132 also generates frame image information F21 containing a string indicating the object class "class, 3" output from the object detection model, as information C211 regarding the type of object contained in each of the multiple frame images. The determination unit 132 also generates frame image information F21 containing the position coordinates of the bounding box surrounding the object "bbox, 0.583060, 0.560044, 0.659778, 0.655942" output from the object detection model, as information C212 regarding the position of the object contained in each of the multiple frame images. Furthermore, the determination unit 132 generates frame image information F21, which is linked to time information T21, information about the type of object C211, and information about the position of the object C212. Similarly, the determination unit 132 generates frame image information F22, which is linked to time information T22, information about the type of object C221, and information about the position of the object C222. Furthermore, the determination unit 132 generates frame image information F23, which is linked to time information T23, information about the type of object C231, and information about the position of the object C232.

[0050] In the embodiments described above, the case in which the information processing device 100 performs event determination using a large-scale language model was explained, but the terminal device 10 may also perform event determination using a large-scale language model. For example, the terminal device 10 may perform event determination using a large-scale language model instead of using an object detection model. Alternatively, the terminal device 10 may perform event determination using both an object detection model and a large-scale language model.

[0051] Furthermore, the information processing device 100 may calculate the accuracy of event determination using the object detection model based on the determination result of event determination using the object detection model. Furthermore, the information processing device 100 may calculate the accuracy of event determination using the large-scale language model based on the determination result of event determination using the large-scale language model. Furthermore, the information processing device 100 may compare the accuracy of event determination using the object detection model and the accuracy of event determination using the large-scale language model and set the model with better accuracy as the higher-priority model. For example, the information processing device 100 may prioritize the execution of event determination using the model set as the higher-priority model. Furthermore, the information processing device 100 acquires information regarding the usage of cloud resources. Furthermore, if the usage of cloud resources is below a predetermined threshold, the information processing device 100 prioritizes the execution of event determination using the higher-priority model. On the other hand, if the usage of cloud resources is above a predetermined threshold, the information processing device 100 prioritizes the execution of event determination using the model with lower resource consumption.

[0052] Furthermore, the information processing device 100 may assign weights to the event determination results using the object detection model and the event determination results using the large-scale language model, respectively, according to the accuracy of the event determination using the object detection model and the event determination accuracy using the large-scale language model, and obtain an overall determination result from both.

[0053] [5. Effects] As described above, the information processing device 100 according to the embodiment includes an acquisition unit 131 and a determination unit 132. The acquisition unit 131 acquires information related to a video. Based on the information related to the video, the determination unit 132 generates frame image information, which is information that links time information related to the time of each of the multiple frame images constituting the video with object information related to the type of object and the position of the object contained in each of the multiple frame images. The determination unit 132 generates a prompt that includes the frame image information and instruction information that instructs the system to determine whether or not a predetermined event has occurred based on the frame image information. The determination unit 132 inputs the prompt to a language model and generates determination information indicating whether or not a predetermined event has occurred.

[0054] As described above, frame image information can be considered as an extraction of an overview of the information necessary to determine whether or not a predetermined event has occurred from a video. Frame image information can be considered as a summary of the information necessary to determine whether or not a predetermined event has occurred from a video. Therefore, the information processing device 100 can improve the accuracy of event detection by the language model by inputting frame image information as a prompt to the language model. Thus, the information processing device 100 can improve the accuracy of event detection in a system that detects predetermined events from images. Furthermore, since the information processing device 100 can improve the accuracy of event detection in a system that detects predetermined events from images, it can contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation." In other words, the information processing device 100 can improve the accuracy of detection in mechanical event detection. Furthermore, since the information processing device 100 can improve the accuracy of event detection in mechanical event detection, it can reduce the personnel costs associated with visual inspection. Furthermore, since the information processing device 100 can reduce the personnel costs associated with visual inspection, it can reduce the operating costs of a system that detects predetermined events from images.

[0055] Furthermore, the acquisition unit 131 inputs each of the multiple frame images into an object detection model that detects predetermined objects from the input image as information related to the video, and acquires the information output from the object detection model. The determination unit 132 generates frame image information based on the information output from the object detection model, generates a prompt that includes the frame image information and instruction information, inputs the prompt into a language model, and generates determination information.

[0056] As a result, the information processing device 100 can efficiently generate frame image information based on the information output from the object detection model.

[0057] Furthermore, the determination unit 132 inputs the information output from the object detection model into the language model to generate frame image information, which is information that links object information (a sentence describing each of the multiple frame images) with time information. It then generates a prompt that includes the frame image information and instruction information, inputs the prompt into the language model, and generates determination information.

[0058] As a result, the information processing device 100 can generate judgment information based on frame image information, which is in a text format that is easily understood by the language model, thereby improving the accuracy of event detection in a system that detects predetermined events from images.

[0059] Furthermore, the determination unit 132 generates a prompt that includes condition information defining the conditions for the occurrence of a predetermined event, inputs the prompt to the language model, and generates determination information.

[0060] As a result, the information processing device 100 can generate determination information based on whether or not the conditions for the occurrence of a predetermined event are met, thereby improving the event detection accuracy in a system that detects predetermined events from images.

[0061] Furthermore, the acquisition unit 131 acquires a first confidence level regarding the determination result of whether or not a predetermined event occurred by the terminal device 10 that generated the video. The determination unit 132 generates determination information if the first confidence level is less than or equal to a first threshold.

[0062] As a result, the information processing device 100 generates determination information when the reliability of the determination result of whether or not a predetermined event has occurred by the terminal device 10 is low, thereby improving the event detection accuracy in a system that detects predetermined events from images.

[0063] Furthermore, the acquisition unit 131 inputs each of the multiple frame images into an object detection model that detects predetermined objects from the input image as information related to the video, and acquires the information output from the object detection model. The determination unit 132 determines whether a predetermined event has occurred based on the information output from the object detection model and predetermined event detection rules, calculates a second confidence level regarding the determination result of whether a predetermined event has occurred, and generates determination information if the second confidence level is less than or equal to a second threshold.

[0064] As a result, the information processing device 100 generates determination information when the confidence level regarding the determination result of whether or not a predetermined event has occurred is low, based on the information output from the object detection model and predetermined event detection rules. This improves the event detection accuracy in a system that detects predetermined events from images.

[0065] [6. Hardware Configuration] Furthermore, the terminal device 10 and information processing device 100 according to the above-described embodiment are realized by a computer 1000 having a configuration such as that shown in Figure 9. The following explanation will use the information processing device 100 as an example. Figure 9 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0066] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0067] The primary storage device 1040 is a memory device, such as RAM (Random Access Memory), that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as USB (Universal Serial Bus) memory or SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), file server, etc.

[0068] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010, such as displays, projectors, and printers, and is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners, and is implemented using, for example, USB.

[0069] Furthermore, the output interface 1060 and input interface 1070 may be wirelessly connected to the output device 1010 and input device 1020, respectively. In other words, the output device 1010 and input device 1020 may be wireless devices.

[0070] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0071] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0072] The network interface 1080 receives data from other devices via network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via network N.

[0073] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0074] For example, when computer 1000 functions as an information processing device 100, the arithmetic unit 1030 of computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Alternatively, the arithmetic unit 1030 of computer 1000 may load a program obtained from another device via the network interface 1080 onto the primary storage device 1040 and execute the loaded program. Furthermore, the arithmetic unit 1030 of computer 1000 may cooperate with other devices via the network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0075] [7. Other] Although embodiments of the present invention have been described above, the present invention is not limited by the content of these embodiments. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above.

[0076] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0077] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0078] For example, the information processing device 100 described above may be implemented using multiple server computers, and depending on the function, it may be implemented by calling external platforms, etc., via APIs (Application Programming Interfaces) or network computing, allowing for flexible configuration changes.

[0079] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent. [Explanation of symbols]

[0080] 1. Information Processing System 10 Terminal devices 100 Information Processing Devices 110 Communications Department 120 Storage section 130 Control Unit 131 Acquisition Department 132 Judgment section 133 Provision Department

Claims

1. An acquisition unit that acquires information about the video, Based on information about the video, a determination unit generates frame image information which is information that links time information for each of the multiple frame images constituting the video with object information which is the type of object contained in each of the multiple frame images and the position of the object; generates a prompt which includes the frame image information and instruction information which instructs the determination unit to determine whether or not a predetermined event has occurred based on the frame image information; inputs the prompt to a language model to generate determination information which indicates whether or not the predetermined event has occurred; An information processing device equipped with the following features.

2. The acquisition unit is, As information related to the aforementioned video, each of the multiple frame images is input to an object detection model that detects a predetermined object from the input image, and the information output from the object detection model is obtained. The determination unit, Based on the information output from the object detection model, the frame image information is generated, the prompt including the frame image information and the instruction information is generated, and the prompt is input to the language model to generate the determination information. The information processing apparatus according to claim 1.

3. The determination unit, The information output from the object detection model is input to the language model to generate frame image information, which is information linking the object information, which is a sentence describing each of the multiple frame images, with the time information; the prompt, which includes the frame image information and the instruction information, is generated; and the prompt is input to the language model to generate the judgment information. The information processing apparatus according to claim 2.

4. The determination unit, The system generates a prompt that includes conditional information defining the conditions for the occurrence of the predetermined event, inputs the prompt into a language model, and generates the determination information. The information processing apparatus according to claim 1.

5. The acquisition unit is, A first confidence level is obtained regarding the determination result of whether or not the predetermined event occurred by the terminal device that generated the video. The determination unit, If the first confidence level is less than or equal to the first threshold, the determination information is generated. The information processing apparatus according to claim 1.

6. The acquisition unit is, As information related to the aforementioned video, each of the multiple frame images is input to an object detection model that detects a predetermined object from the input image, and the information output from the object detection model is obtained. The determination unit, Based on the information output from the object detection model and predetermined event detection rules, it is determined whether or not the predetermined event has occurred, a second confidence level is calculated for the determination result of whether or not the predetermined event has occurred, and if the second confidence level is less than or equal to a second threshold, the determination information is generated. The information processing apparatus according to claim 1.

7. An information processing method implemented by a program executed by an information processing device, The acquisition process for obtaining information about the video, Based on the video information, generate frame image information which is information that links time information for each of the multiple frame images constituting the video with object information which is the type of object contained in each of the multiple frame images and the position of the object; generate a prompt which includes the frame image information and instruction information which instructs to determine whether or not a predetermined event has occurred based on the frame image information; input the prompt to a language model to generate determination information which indicates whether or not the predetermined event has occurred; Information processing methods including

8. Procedure for obtaining information about the video, A determination procedure that generates frame image information, which is information that links time information for each of the multiple frame images constituting the video with object information, which is information that includes the type of object and the position of the object contained in each of the multiple frame images, based on information about the video; generates a prompt that includes the frame image information and instruction information that instructs the system to determine whether or not a predetermined event has occurred based on the frame image information; inputs the prompt to a language model to generate determination information indicating whether or not the predetermined event has occurred; An information processing program that causes a computer to execute something.

Images