Information processing device, method, program and system

Abstract

According to the present disclosure, a summary of a video can be easily understood. [Solution] A program for operating a computer having a processor and a memory, the program causing the processor to execute the following steps: acquiring a video consisting of multiple frames; inputting the video or specific frames that make up the video and a prompt for explaining the content of the frames that make up the video into a large-scale language model, and causing the large-scale language model to output an explanatory text for the frame; inputting multiple explanatory texts and a prompt for outputting a summary of the video into the large-scale language model, and causing the large-scale language model to output the summary of the video; and presenting the summary output by the large-scale language model to a user.

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, a method, a program, and a system. [Background technology]

[0002] Patent Document 1 describes a technology for providing an image that allows easy understanding of the movement and sound of an object depicted in the image. For example, in Patent Document 1, an information processing device receives video data composed of multiple frames and depicting one or more objects (people or things), and selects a characteristic frame from the video data based on the movement of the one or more objects or the sound from the one or more objects. The information processing device selects a characteristic object from one or more objects based on the movement of the one or more objects or the sound from the one or more objects depicted in the video data. The information processing device associates text information indicating at least one of the movement of the characteristic object and the sound from the characteristic object with the characteristic object and displays it on the image of the characteristic frame.

[0003] [Patent Document 1] JP 2015-073198 A Summary of the Invention [Problem to be solved by the invention]

[0004] In Patent Document 1, text information indicating at least one of the movement of the characteristic object and the sound from the characteristic object is associated with the characteristic object and displayed in the image of the characteristic frame. In Patent Document 1, it is possible to understand the movement and sound of the object depicted in the image by checking the image, but associating information with the image causes problems such as a large capacity and difficulty in searching for desired information.

[0005] An object of the present disclosure is to enable easy comprehension of a summary of a video. [Means for solving the problem]

[0006] A program for operating a computer having a processor and a memory, the program causing the processor to execute the following steps: acquiring a video consisting of multiple frames; inputting the video or specific frames that make up the video and a prompt for explaining the content of the frames that make up the video into a large-scale language model, and causing the large-scale language model to output an explanatory text for the frame; inputting multiple explanatory texts and a prompt for outputting a summary of the video into the large-scale language model, and causing the large-scale language model to output the summary of the video; and presenting the summary output by the large-scale language model to a user. Effect of the Invention

[0007] You can easily understand the summary of the video. [Brief description of the drawings]

[0008] [Figure 1] 1 is a block diagram showing the overall configuration of a system 1. FIG. [Diagram 2] 2 is a block diagram showing an example of a functional configuration of a terminal device 10. FIG. [Diagram 3] 2 is a block diagram showing an example of the functional configuration of a server 20. FIG. [Figure 4] FIG. 2 is a diagram illustrating a data structure of a table. [Diagram 5] FIG. 2 is a diagram illustrating a data structure of a table. [Figure 6] FIG. 2 is a diagram showing an example of a processing flow in the system 1. [Figure 7] FIG. 13 is a diagram showing an example screen of the present disclosure. [Figure 8] FIG. 9 is a block diagram showing the basic hardware configuration of a computer 90. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0009] Hereinafter, an embodiment of the present disclosure will be described with reference to the drawings. In all the drawings explaining the embodiment, the same reference numerals are given to common components, and repeated explanations are omitted. Note that the following embodiment does not unduly limit the contents of the present disclosure described in the claims. In addition, not all of the components shown in the embodiment are essential components of the present disclosure. In addition, each figure is a schematic diagram and is not necessarily illustrated strictly.

[0010] In the following description, a "processor" refers to one or more processors. The at least one processor is typically a microprocessor such as a CPU (Central Processing Unit), but may be another type of processor such as a GPU (Graphics Processing Unit). The at least one processor may be a single-core or multi-core.

[0011] Furthermore, the at least one processor may be a processor in the broad sense, such as a hardware circuit (for example, a field-programmable gate array (FPGA) or an application specific integrated circuit (ASIC)) that performs part or all of the processing.

[0012] In the following explanation, information that gives an output for an input may be described using an expression such as "xxx table", but this information may be data of any structure or a learning model such as a neural network that generates an output for an input. Therefore, the "xxx table" may be called "xxx information".

[0013] Furthermore, in the following description, the configuration of each table is an example, and one table may be divided into two or more tables, or all or part of two or more tables may be one table.

[0014] In addition, in the following explanation, the processing may be described with the "program" as the subject, but since the program is executed by a processor to perform a specified processing step by appropriately using a memory unit and / or an interface unit, etc., the subject of the processing may be the processor (or a device such as a controller having the processor).

[0015] The program may be installed in a device such as a computer, or may be, for example, in a program distribution server or a computer-readable (e.g., non-transitory) recording medium. In the following description, two or more programs may be realized as one program, or one program may be realized as two or more programs.

[0016] In the following description, identification numbers are used as identification information for various objects, but other types of identification information (for example, identifiers including alphabetic characters or symbols) may be used instead.

[0017] In addition, in the following description, when describing elements of the same type without distinguishing between them, reference signs (or common signs among the reference signs) may be used, and when describing elements of the same type with distinction between them, the identification numbers (or reference signs) of the elements may be used.

[0018] In the following description, the control lines and information lines are those that are considered necessary for the description, and not all control lines and information lines in the product are necessarily shown. All components may be connected to each other.

[0019] Each information processing device is configured by a computer equipped with a calculation device and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For the server 20 and the terminal device 10, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0020] <Summary> The system of this embodiment generates explanatory text for multiple chronologically consecutive frames (e.g., a video) in a large language model (LLM) according to predetermined rules, and has the LLM generate a summary of the multiple frames (e.g., a summary of the video) based on the generated explanatory texts.

[0021] <1. Overall system configuration> Fig. 1 is a block diagram showing an example of the overall configuration of a system 1. The system 1 shown in Fig. 1 includes, for example, a terminal device 10, a server 20, and an LLM system 30. The terminal device 10, the server 20, and the LLM system 30 are communicatively connected via, for example, a network 80.

[0022] 1 shows an example in which the system 1 includes two terminal devices 10, but the number of terminal devices 10 included in the system 1 is not limited to two. The number of terminal devices 10 included in the system 1 may be one, or three or more.

[0023] 1 illustrates an example in which the system 1 includes one LLM system 30, but the number of LLM systems 30 included in the system 1 is not limited to one. The number of LLM systems 30 included in the system 1 may be two or more.

[0024] 1 shows an example in which the server 20 is independent of the LLM system 30, the server 20 may include the functionality of the LLM system 30. In other words, the server 20 may store the LLM.

[0025] In this embodiment, a collection of multiple devices may be one server. The method of allocating multiple functions required to realize the server 20 according to this embodiment to one or multiple pieces of hardware can be appropriately determined in consideration of the processing capacity of each piece of hardware and / or the specifications required for the server 20.

[0026] 1 is an information processing device used by a user who utilizes a service provided by a server 20. For example, the terminal device 10 is an information processing device operated by a user who utilizes a document output service provided by the server 20 and creates a document. The terminal device 10 is realized by, for example, a stationary PC (Personal Computer), a laptop PC, a head mounted display, or the like. The terminal device 10 may also be a portable computer such as a smartphone or a tablet terminal.

[0027] The terminal device 10 includes a communication IF (Interface) 12, an input device 13, an output device 14, a memory 15, a storage 16, and a processor 19. The input device 13 is a device for receiving an input operation from a user (for example, a touch panel, a touch pad, a pointing device such as a mouse, a keyboard, etc.). The output device 14 is a device for presenting information to a user (a display, a speaker, etc.).

[0028] The server 20 is an information processing device that provides a service of generating a summary of a video. The server 20 is an information processing device realized by, for example, a computer connected to a network 80. As shown in Fig. 1, the server 20 includes a communication IF 22, an input / output IF 23, a memory 25, a storage 26, and a processor 29. The input / output IF 23 functions as an interface with an input device for receiving an input operation from a user and an output device for outputting information to the user.

[0029] The LLM system 30 is a system in which a large-scale artificial intelligence model (LLM) used in the field of natural language processing (NLP), for example, is constructed. By learning a large amount of text data (web pages, books, articles, etc.), the LLM can understand patterns of language used by humans and effectively perform natural language generation (NLG) tasks.

[0030] LLMs are used in many NLP tasks, such as generating answers to specific questions, automatic sentence generation, text summarization, translation, and sentiment analysis. They can also be used in a variety of applications, including education, entertainment, customer service, and product development. An example of an LLM is a multimodal LLM. Examples of LLMs include the following: GPT-4 (registered trademark) (OpenAI, Inc.) Gemini (registered trademark) (Google Inc.) ·StableLM (StableAI) Llama 3.2 (Meta)

[0031] The LLM system 30 inputs the text data and prompts received from the server 20 to the LLM, and causes the LLM to output answers based on the input text data and prompts. The LLM system 30 transmits the answers output from the LLM to the server 20.

[0032] The photographing device 31 is a device for receiving light by a light receiving element and outputting it as an image signal. The photographing device 31 is installed so as to be able to photograph the situation of the space (hereinafter referred to as the site) where the user wishes to grasp the situation. Specifically, the photographing device 31 is installed in a position where the entire site can be viewed without any obstructions. For example, the photographing device 31 is installed at a high place in a corner of the site. When one photographing device 31 cannot capture the entire situation of the site, multiple photographing devices 31 are installed. The photographing device 31 is, for example, a surveillance camera installed in a downtown area, a park, a station, an office, a school, etc. The photographing device 31 is, for example, a camera that photographs a surgery in a hospital.

[0033] Each information processing device is configured by a computer equipped with a calculation device and a storage device. The basic hardware configuration of the computer and the basic functional configuration of the computer realized by the hardware configuration will be described later. For each of the terminal device 10 and the server 20, descriptions that overlap with the basic hardware configuration and basic functional configuration of the computer described later will be omitted.

[0034] <2. Terminal device configuration> Fig. 2 is a block diagram showing an example of a functional configuration of the terminal device 10. As shown in Fig. 2, the terminal device 10 includes a communication unit 120, an input device 13, an output device 14, an audio processing unit 17, a microphone 171, a speaker 172, a camera 160, a position information sensor 150, a storage unit 180, and a control unit 190. The blocks included in the terminal device 10 are electrically connected to each other, for example, by a bus or the like.

[0035] The communication unit 120 performs processing such as modulation and demodulation processing for the terminal device 10 to communicate with other devices. The communication unit 120 performs transmission processing on a signal generated by the control unit 190 and transmits the signal to the outside (for example, the server 20). The communication unit 120 performs reception processing on a signal received from the outside and outputs the signal to the control unit 190.

[0036] The input device 13 is a device for inputting instructions or information by a user who operates the terminal device 10. The input device 13 is realized, for example, by a touch-sensitive device 131 or the like in which an instruction is input by touching an operation surface. In the case where the terminal device 10 is a PC or the like, the input device 13 may be realized by a reader, a keyboard, a mouse, or the like. The input device 13 converts an instruction input by a user into an electrical signal, and outputs the electrical signal to the control unit 190. Note that the input device 13 may include, for example, a receiving port that receives an electrical signal input from an external input device.

[0037] The output device 14 is a device for presenting information to a user who operates the terminal device 10. The output device 14 is realized, for example, by a display 141 or the like. The display 141 displays data according to the control of the control unit 190. The display 141 is realized, for example, by an LCD (Liquid Crystal Display) or an organic EL (Electro-Luminescence) display or the like.

[0038] The audio processing unit 17 performs, for example, digital-analog conversion processing of an audio signal. The audio processing unit 17 converts a signal provided from the microphone 171 into a digital signal and provides the converted signal to the control unit 190. The audio processing unit 17 also provides the audio signal to the speaker 172. The audio processing unit 17 is realized, for example, by a processor for audio processing. The microphone 171 accepts audio input and provides an audio signal corresponding to the audio input to the audio processing unit 17. The speaker 172 converts the audio signal provided from the audio processing unit 17 into audio and outputs the audio to the outside of the terminal device 10.

[0039] The camera 160 is a device for receiving light with a light receiving element and outputting the received light as an image capturing signal.

[0040] The position information sensor 150 is a sensor that detects the position of the terminal device 10, and is, for example, a GPS (Global Positioning System) module. The GPS module is a receiving device used in a satellite positioning system. In the satellite positioning system, signals are received from at least three or four satellites, and the current position of the terminal device 10 equipped with the GPS module is detected based on the received signals. The position information sensor 150 may detect the current position of the terminal device 10 from the position of the wireless base station to which the terminal device 10 is connected.

[0041] The storage unit 180 is realized by, for example, the memory 15, the storage 16, etc., and stores data and programs used by the terminal device 10. The storage unit 180 stores, for example, user information 181.

[0042] The user information 181 includes, for example, information about the user who uses the terminal device 10. The information about the user includes, for example, the user's name, age, address, date of birth, contact information, and the like.

[0043] The control unit 190 is realized by the processor 19 reading a program stored in the storage unit 180 and executing instructions included in the program. The control unit 190 controls the operation of the terminal device 10. The control unit 190 performs the functions of an operation reception unit 191, a transmission / reception unit 192, and a presentation control unit 193 by operating according to the program.

[0044] The operation reception unit 191 performs processing for receiving instructions or information input from the input device 13. Specifically, for example, the operation reception unit 191 receives instructions or information input from the touch-sensitive device 131 or the like.

[0045] Furthermore, the operation reception unit 191 receives voice instructions input from the microphone 171. Specifically, for example, the operation reception unit 191 receives a voice signal input from the microphone 171 and converted into a digital signal by the voice processing unit 17. For example, the operation reception unit 191 analyzes the received voice signal and extracts a predetermined noun, thereby acquiring an instruction from the user.

[0046] The transmitting / receiving unit 192 performs processing for the terminal device 10 to transmit and receive data to and from an external device such as the server 20 in accordance with a communication protocol. Specifically, for example, the transmitting / receiving unit 192 transmits information input by a user or an instruction from the user to the server 20. In addition, the transmitting / receiving unit 192 receives information provided by the server 20.

[0047] The presentation control unit 193 controls the output device 14 to present information provided by the server 20 to the user. Specifically, for example, the presentation control unit 193 causes the display 141 to display information related to a document transmitted from the server 20. In addition, the presentation control unit 193 causes the speaker 172 to output the information transmitted from the server 20.

[0048] <3. Functional configuration of the server> 3 is a diagram showing an example of the functional configuration of the server 20. As shown in FIG.

[0049] The communication unit 201 performs processing for the server 20 to communicate with external devices.

[0050] The storage unit 202 includes, for example, a user information table 2021 and a video log table 2022. The tables stored in the storage unit 202 are not limited to these.

[0051] The user information table 2021 is a table for storing information about users, which will be described in detail later.

[0052] The video log table 2022 is a table for storing information about frames that make up a video, as will be described in detail later.

[0053] The control unit 203 is realized by the processor 29 reading a program stored in the storage unit 202 and executing instructions included in the program. The program includes an application such as a web browser application. The program includes a programming language such as JavaScript (registered trademark) that is executed on the web browser application stored in the terminal device 10. The control unit 203 operates according to the program to fulfill functions shown as a reception control module 2031, a transmission control module 2032, a service processing module 2033, and a presentation control module 2034.

[0054] The reception control module 2031 controls the process in which the server 20 receives a signal from an external device in accordance with a communication protocol.

[0055] The transmission control module 2032 controls the process in which the server 20 transmits signals to external devices in accordance with a communication protocol.

[0056] The service processing module 2033 controls the process of inputting prompts to the LLM and generating video summaries.

[0057] The presentation control module 2034 controls the process of presenting information to the user.

[0058] The image analysis module 2035 analyzes the video to detect a specific object, a specific action, or a combination thereof.

[0059] <4. Data Structure> The data structure of the table stored in the server 20 will be described. Note that the data structure described is an example, and does not exclude data that is not described. In addition, even if data is described in the same table, it may be stored in separate storage areas in the storage unit 202.

[0060] Fig. 4 is a diagram showing the data structure of the user information table 2021. The user information table 2021 shown in Fig. 4 is a table having columns of name, age, sex, date of birth, and contact information, with a user ID as a key.

[0061] The user ID is an item that stores an identifier for uniquely identifying a user. The name is an item that stores the name of the user. The age is an item that stores the age of the user. The gender is an item that stores the gender of the user. The date of birth is an item that stores the date of birth of the user. The contact information is an item that stores the contact information (e.g., a telephone number, an email address, etc.) of the terminal device 10 owned by the user.

[0062] Fig. 5 is a diagram showing the data structure of the video log table 2022. As shown in Fig. 5, the video log table 2022 has items such as video ID, frame ID, time, and description. The video log table 2022 lists frame IDs below video IDs. The video log table 2022 links the time and description with one frame ID as a key.

[0063] The moving image ID indicates identification information for identifying a moving image captured by the imaging device 31. For example, a different moving image ID is assigned to a moving image for each imaging device 31 that captures the moving image. Even if a moving image is captured by the same imaging device 31, the moving images may be recognized as different moving images for each time period (for example, each day, every three hours, etc.) and assigned different moving image IDs.

[0064] The frame ID indicates identification information for identifying a frame that constitutes a moving image. The frame ID is assigned to a specific frame among the frames that constitute a moving image. Details will be described later. The time indicates the time when the frame was extracted.

[0065] The description indicates a sentence that explains the contents of the frame. Details will be described later.

[0066] <5. Operation> An example of the process flow in the system 1 will be described.

[0067] FIG. 6 is a flowchart showing an example of the operation of the server 20 when causing the LLM to generate a summary of a video.

[0068] In step S1001, the server 20 receives a video from the image capturing device 31. Specifically, the reception control module 2031 receives a video of the site captured by the image capturing device 31 from the image capturing device 31. The video is assigned a video ID based on, for example, the identification information of the image capturing device 31. The server 20 stores the received video in, for example, the storage unit 202. The image capturing device 31 may transmit the video to the server 20 all at once as one file, or may transmit the video to the server 20 in packet units.

[0069] The reception control module 2031 may receive, for example, a predetermined frame out of a plurality of frames constituting a moving image from the photographing device 31 instead of a moving image. In this case, the photographing device 31 transmits a frame to the server 20 at a predetermined cycle (for example, every 30 seconds). The photographing device 31 may also transmit a frame at a random timing to the server 20.

[0070] In step S1002, the server 20 selects a frame. Specifically, the service processing module 2033 selects a predetermined frame from among a plurality of frames that make up the received video, in accordance with a predetermined rule.

[0071] For example, the service processing module 2033 selects frames from a video at predetermined intervals (for example, every 30 seconds).

[0072] Also, for example, the service processing module 2033 may select a frame at a random time in a video.

[0073] Also, for example, the service processing module 2033 may select a frame in which the image analysis module 2035 detects a predetermined object or a predetermined action. In this case, for example, the image analysis module 2035 analyzes the video received in step S1001. The predetermined object is, for example, a vehicle, an animal, a weapon, etc. The predetermined action is, for example, entering or leaving a room, falling, a collision, etc. The image analysis module 2035 detects the predetermined object, the predetermined action, or a combination thereof by analyzing the video.

[0074] The service processing module 2033 assigns a frame ID to the selected frame.

[0075] In step S1003, the server 20 transmits the frame and prompt A to the LLM system 30. Specifically, the service processing module 2033 transmits the selected frame and prompt A for explaining the content of the frame to the LLM system 30. The prompt A is set as a default by, for example, a video summary service provider. The text of the prompt A is, for example, "Please output a sentence explaining the content of the input frame." The LLM system 30 inputs the transmitted frame and prompt A to the LLM.

[0076] The service processing module 2033 may transmit a video to the LLM system 30 instead of a frame. In this case, the text of the prompt A may be, for example, "Please output a sentence explaining the content of the frame to which the frame ID is assigned." The text of the prompt A may be one that causes the content of a frame at a predetermined timing to be explained. Specifically, for example, the text of the prompt A may be, "Please output a sentence explaining the content of the frame to which the tag is assigned." In addition, the text of the prompt A may be, for example, "Please output a sentence explaining the content of the frame at a predetermined cycle."

[0077] In step S1004, the server 20 causes the LLM system 30 to output an explanatory text. Specifically, the LLM outputs an explanatory text in response to the input prompt A. The reception control module 2031 receives the output explanatory text from the LLM system 30. The server 20 stores the received explanatory text, as well as the video ID, frame ID, and time corresponding to the explanatory text, in the video log table 2022. In other words, the service processing module 2033 causes the LLM to output the explanatory text for the frame and stores it in the video log table 2022, even without an instruction from the user.

[0078] In step S1005, the server 20 presents information related to the explanatory text to the terminal device 10. Specifically, the transmission control module 2032 transmits the video data, the video ID, the frame ID, the time, and the explanatory text to the terminal device 10. The presentation control module 2034 transmits, to the terminal device 10, layout information for displaying the video data, the video ID, the frame ID, the time, and the explanatory text on the terminal device 10.

[0079] In step S1006, the terminal device 10 presents information related to the explanatory text to the user. Specifically, the presentation control unit 193 displays the video data, the video ID, the frame ID, the time, and the explanatory text on the display 141 according to the arrangement information transmitted from the server 20. At this time, the presentation control unit 193 may display the explanatory text of the frame in which the predetermined object or the predetermined action appears in a manner that can be distinguished from other explanatory texts. For example, the presentation control unit 193 may attach a mark that attracts the user's attention to the explanatory text of the frame in which the predetermined object or the predetermined action appears. The mark that attracts the user's attention is, for example, a star mark. The predetermined object is, for example, a vehicle, an animal, a weapon, etc. The predetermined action is, for example, entering or leaving a room, falling, a collision, etc. Note that the presentation control unit 193 does not need to display the information related to the explanatory text on the display 141.

[0080] In step S1007, the terminal device 10 accepts an instruction to output a summary from the user. Specifically, the operation receiving unit 191 accepts a prompt B for outputting a summary of the video. For example, the user refers to a screen on which a description of a plurality of frames is displayed, and inputs the prompt B on the display 141. The user may input the prompt B on a screen on which no description is displayed. The text of the prompt B is, for example, "If a dangerous situation occurs in the video, please tell us the situation and the time." The text of the prompt B may also be, for example, "Please tell us what happened from HH:MM1 to MM2." The server 20 receives the prompt B from the terminal device 10. The user may or may not specify the time span (from the start time to the end time) of the video to be summarized in the prompt B. If the user does not specify the time span of the video to be summarized in the prompt B, the service processing module 2033 considers that, for example, the entire video, that is, the entire time from the start time to the end time of the video, is specified as the time span.

[0081] In step S1008, the server 20 transmits prompt B and an explanatory text to the LLM system 30. Specifically, the service processing module 2033 transmits prompt B transmitted from the terminal device 10 and an explanatory text for a time width required to create a summary to the LLM system 30. If the time width of the target video is specified in prompt B, the service processing module 2033 transmits an explanatory text for the time width specified in prompt B to the LLM system 30.

[0082] In step S1009, the server 20 causes the LLM system 30 to output a video summary. Specifically, the LLM outputs the video summary based on the prompt B and the explanatory text transmitted in step S1008. The video summary output from the LLM is based on the contents of the multiple explanatory texts, for example. The reception control module 2031 receives the video summary output from the LLM from the LLM system 30.

[0083] In step S1010, the server 20 presents information related to the video summary to the terminal device 10. Specifically, the presentation control module 2034 presents information related to the video summary to the terminal device 10. The transmission control module 2032 transmits, to the terminal device 10, layout information for displaying the video summary on the terminal device 10.

[0084] In step S1011, the terminal device 10 displays information related to the video summary to the user. Specifically, the presentation control unit 193 displays the video summary on the display 141 based on the arrangement information transmitted from the server 20. The presentation control unit 193 may display words, phrases, and sentences indicating noteworthy events in the video summary so as to be distinguishable from other words, phrases, and sentences. Specifically, for example, the presentation control unit 193 may apply decoration that attracts the user's attention. Decoration that attracts the user's attention is, for example, bold text, colored text, and the like.

[0085] The LLM system 30 that outputs the explanatory text in step S1004 and the LLM system 30 that outputs the video summary in step S1009 may be separate. For example, the LLM system 30A may output an explanatory text in response to prompt A, and the LLM system 30B may output a video summary in response to prompt B.

[0086] <6. Screen Examples> An example of a screen of the display 141 of the terminal device 10 in the present disclosure will be described.

[0087] Figure 7 shows an example of a screen where the description and summary are displayed and prompt B is entered.

[0088] Area 501 is an area in which the video is displayed in step 1006. In Fig. 7, a frame of a person lying on the floor is displayed in area 501. Information about the video is displayed in a corner of area 501. The information about the video is, for example, a video ID, the time of the displayed frame, etc.

[0089] A bar 502 indicates a playback position of a video on a time axis. A user can adjust the playback position by adjusting a dot on the bar 502.

[0090] The icon group 503 is an icon for operating the video. The icon group 503 is not limited to the three icons shown in FIG.

[0091] Table 504 is a table for displaying the frame ID, time, and description in step 1006. When the user clicks on any of the frame ID, time, and description in the table, the corresponding frame is displayed in table region 501.

[0092] Box 505 is a box for inputting prompt B from the user in step 1007. Prompt B input by the user may be kept displayed in box 505 until the user ends the video summary service (for example, until the screen in FIG. 7 is closed by the user).

[0093] Box 506 is a box in which the summary is displayed in step 1011. The summary output as the answer to prompt B may be maintained on display in box 506, for example, until the user ends the video summary service (for example, until the screen in FIG. 7 is closed by the user). Note that, although FIG. 7 illustrates a case in which prompt B and the answer as a summary are displayed in different boxes, the display format is not limited to this. Prompt B and the answer as a summary may be displayed in a chat format.

[0094] <7.Summary> As described above, in the above embodiment, the server 20 acquires a video consisting of a plurality of frames. The server 20 inputs the video or a predetermined frame constituting the video and a prompt for explaining the contents of the frame constituting the video to the large-scale language model, and causes the large-scale language model to output an explanatory text for the frame. The server 20 inputs a plurality of explanatory texts and a prompt for outputting a summary of the video to the large-scale language model, and causes the large-scale language model to output the summary of the video. The server 20 presents the summary output by the large-scale language model to the user. This allows the user to easily grasp the outline of events occurring in the video through the summary of the video, even if the user does not watch the entire video.

[0095] In the above embodiment, in the step of outputting the explanatory text, the server 20 inputs frames of the multiple frames at a predetermined interval into the large-scale language model. This makes it possible to select frames based on time. Also, the frames are automatically input into the large-scale language model.

[0096] In the above embodiment, in the step of outputting the explanatory text, the server 20 inputs a frame at a random timing among the multiple frames to the large-scale language model. This allows random frame selection. Also, the frame is automatically input to the large-scale language model.

[0097] In the above embodiment, in the step of outputting the explanatory text, the server 20 inputs the frame in which the predetermined object or the predetermined action is detected to the large-scale language model. This makes it possible to select a frame based on the object or action that the user focuses on. Also, the frame is automatically input to the large-scale language model.

[0098] In the above embodiment, the server 20 presents a list of explanatory texts to the user and receives a prompt from the user to output a summary of the video. This improves the user's ability to view explanatory texts, making it easier for the user to understand the information needed to instruct the user to summarize the video.

[0099] In the above embodiment, in the receiving step, the server 20 presents the user with a description of the frame in which the predetermined object or the predetermined action appears, in a manner that allows the user to instantly know the frame in which the object or action that the user is focusing on appears.

[0100] In the above embodiment, in the step of outputting the summary, the server 20 inputs the description of the frames included in the predetermined time span and a prompt for outputting the summary of the video of the predetermined time span to the large-scale language model, and causes the large-scale language model to output the summary of the video of the predetermined time span. In this way, the user can obtain a summary of the video of the desired time span.

[0101] In the above embodiment, the video is a video captured by a surveillance camera. The video may be a video captured during surgery. This allows for summarization of videos in various locations or situations.

[0102] <8. Variations> A modification of the above embodiment will now be described. <8.1. Variation 1> In the above embodiment, the user specifies the time span to be summarized in the video in step S1007 at prompt B, i.e., in a linguistic manner. However, the user may specify the time span to be summarized in the video in a non-linguistic manner. For example, the terminal device 10 may accept the user's specification of the time span by dragging the bar 502 in FIG. 7 from the start time point to the end time point. The terminal device 10 may also accept the user's specification of the time span by dragging the table 504 in FIG. 7 from the start time row to the end time row.

[0103] In these cases, the user does not specify a time range in prompt B. The text of prompt B is, for example, "If a dangerous situation occurs in the video, please tell us the situation and the time."

[0104] <8.2. Variation 2> In the above embodiment, the server 20 causes the LLM system 30 to output the description of the entire region in the selected frame. However, the server 20 may cause the LLM system 30 to output only the description of a part of the region in the selected frame. For example, a part of the region 501 in which the moving image in FIG. 7 is displayed on the display 141 is dragged and specified by the user, and a prompt is input for an explanation of the image represented by the region. The prompt at this time is, for example, "How many XX people are there in this?" or "Please tell me the situation in this area." The server 20 receives information identifying the specified region and the prompt from the terminal device 10. The server 20 extracts an image from the frame based on the specified region. The server 20 inputs the extracted image and the prompt to the LLM system 30, and causes the LLM system 30 to output the description of the specified part of the region in the selected frame.

[0105] If there is a scene to be processed in the specified partial area, the server 20 may process the scene. The scene to be processed is, for example, a scene related to privacy. The processing is, for example, a process to reduce the discernibility of the object, such as a masking process or a mosaic process. For example, when the server 20 detects a scene that requires consideration of privacy (such as a person's face) in the specified area, it applies a mosaic to the object.

[0106] <8.3. Variation 3> The server 20 may refer to the description of the frames and create a video in which important frames are connected together, that is, a highlight video. For example, an important frame is a frame whose description is long (five or more sentences, fifty or more characters, etc.). Also, for example, an important frame is a frame whose description refers to a specific object or a specific action. Examples of specific objects include vehicles, animals, weapons, etc. Examples of specific actions include entering or leaving a room, falling, collision, etc.

[0107] The server 20 receives, for example, an instruction from a user to instruct the creation of a highlight video. When the instruction is received, the server 20 analyzes, for example, a stored explanatory text and extracts important frames. The analysis may be, for example, natural language analysis, may be performed using a trained model trained to extract important frames, or may be performed using an LLM. The server 20 extracts frames that may constitute a highlight video based on the important frames. The server 20 may treat an important frame as a frame that may constitute a highlight video, or may treat a plurality of frames based on an important frame as frames that may constitute a highlight video. The server 20 connects the extracted frames to create a highlight video. This allows the user to easily obtain a highlight video, and reduces the burden of checking the video.

[0108] <8.4. Variation 4> In the above embodiment, in step S1007, prompt B is a prompt for outputting a summary of the video, and the text of prompt B is, for example, "If a dangerous situation occurs in the video, please tell me the situation and the time." However, prompt B may be a prompt for analyzing an event in the video. That is, the text of prompt B may be, for example, "Please tell me the distribution of the ages of customers in the video." In this case, in step S1009, the LLM may output the result in an intuitive format. That is, for example, the text of prompt B may be "Please tell me the distribution of the ages of customers in the video in a graph." For example, the LLM outputs a graph of the age distribution of customers in step S1009 based on the explanatory text referring to the ages of customers in each frame output in step S1004.

[0109] <Basic computer hardware configuration> 8 is a block diagram showing the basic hardware configuration of a computer 90. The computer 90 includes at least a processor 901, a main storage device 902, an auxiliary storage device 903, and a communication IF 991 (interface). These are electrically connected to each other by a communication bus 921.

[0110] The processor 901 is hardware for executing an instruction set described in a program, and is composed of an arithmetic unit, a register, a peripheral circuit, and the like.

[0111] The main memory device 902 is for temporarily storing programs, data to be processed by the programs, etc. For example, it is a volatile memory such as a DRAM (Dynamic Random Access Memory).

[0112] The auxiliary storage device 903 is a storage device for saving data and programs, such as a flash memory, a hard disk drive (HDD), a magneto-optical disk, a CD-ROM, a DVD-ROM, or a semiconductor memory.

[0113] The communication IF 991 is an interface for inputting and outputting signals for communicating with other computers via a network using a wired or wireless communication standard. The network is composed of the Internet, a LAN, various mobile communication systems constructed by wireless base stations, etc. For example, the network includes 3G, 4G, 5G mobile communication systems, LTE (Long Term Evolution), wireless networks that can connect to the Internet via a specified access point (e.g., Wi-Fi (registered trademark)), etc. In the case of wireless connection, communication protocols include, for example, Z-Wave (registered trademark), ZigBee (registered trademark), Bluetooth (registered trademark), etc. In the case of wired connection, the network also includes a network that is directly connected by a USB (Universal Serial Bus) cable or the like.

[0114] It should be noted that the computer 90 can be virtually realized by distributing all or part of each hardware configuration among multiple computers 90 and connecting them together via a network. In this way, the computer 90 is a concept that includes not only a computer 90 housed in a single housing or case, but also a virtualized computer system.

[0115] <Basic functional configuration of computer 90> A description will now be given of the functional configuration of a computer realized by the basic hardware configuration (FIG. 8) of a computer 90. The computer comprises at least the functional units of a control unit, a storage unit, and a communication unit.

[0116] The functional units of the computer 90 can also be realized by distributing all or part of the functional units among multiple computers 90 connected to each other via a network. The computer 90 is a concept that includes not only a single computer 90 but also a virtualized computer system.

[0117] The control unit is realized by the processor 901 reading out various programs stored in the auxiliary storage device 903, expanding the programs in the main storage device 902, and executing processes according to the programs. The control unit can realize functional units that perform various information processing depending on the type of program. In this way, the computer is realized as an information processing device that performs information processing.

[0118] The storage unit is realized by a main storage device 902 and an auxiliary storage device 903. The storage unit stores data, various programs, and various databases. Furthermore, the processor 901 can secure a storage area corresponding to the storage unit in the main storage device 902 or the auxiliary storage device 903 in accordance with a program. Furthermore, the control unit can cause the processor 901 to execute processes of adding, updating, and deleting data stored in the storage unit in accordance with the various programs.

[0119] The term database refers to a relational database, which is used to manage sets of data called masters and tables in a tabular format structurally defined by rows and columns, by associating them with each other. In a database, a table is called a table or master, a column in a table is called a column, and a row in a table is called a record. In a relational database, relationships between tables and masters can be set and associated. Usually, a column that serves as a primary key for uniquely identifying a record is set in each table and each master, but setting a primary key in a column is not essential. The control unit can cause the processor 901 to add, delete, or update records in a specific table or master stored in the storage unit according to various programs. Furthermore, by storing data, various programs, and various databases in the storage unit, it can be considered that the information processing device and information processing system according to the present disclosure have been manufactured.

[0120] In addition, the database and master in this disclosure may include any data structure (such as a list, a dictionary, an associative array, or an object) in which information is structurally defined. The data structure also includes data that can be considered as a data structure by combining data with a function, class, method, or the like written in any programming language.

[0121] The communication unit is realized by the communication IF 991. The communication unit realizes a function of communicating with other computers 90 via a network. The communication unit can receive information transmitted from other computers 90 and input the information to the control unit. The control unit can cause the processor 901 to execute information processing on the received information in accordance with various programs. In addition, the communication unit can transmit information output from the control unit to other computers 90.

[0122] In addition, the above-mentioned configurations, functions, processing units, processing means, etc. may be realized in part or in whole by hardware, for example, by designing them as integrated circuits. The present invention can also be realized by software program code that realizes the functions of the embodiments. In this case, a storage medium on which the program code is recorded is provided to a computer, and a processor included in the computer reads the program code stored in the storage medium. In this case, the program code itself read from the storage medium realizes the functions of the above-mentioned embodiments, and the program code itself and the storage medium storing it constitute the present invention. Examples of storage media for supplying such program code include flexible disks, CD-ROMs, DVD-ROMs, hard disks, SSDs, optical disks, magneto-optical disks, CD-Rs, magnetic tapes, non-volatile memory cards, ROMs, etc.

[0123] Furthermore, the program code for realizing the functions described in this embodiment can be implemented in a wide range of program or script languages, such as assembler, C / C++, perl, Shell, PHP, Java (registered trademark), and the like.

[0124] Furthermore, the program code of the software that realizes the functions of the embodiments may be distributed over a network and stored in a storage means such as a computer's hard disk or memory, or in a storage medium such as a CD-RW or CD-R, and the processor of the computer may read out and execute the program code stored in the storage means or storage medium.

[0125] The functions performed by the components described herein may be implemented in circuitry or processing circuitry, including general purpose processors, application specific processors, integrated circuits, ASICs (Application Specific Integrated Circuits), a CPU (a Central Processing Unit), conventional circuits, and / or combinations thereof, programmed to perform the functions described. Processors include transistors and other circuits and are considered to be circuitry or processing circuitry. A processor may be a programmed processor that executes a program stored in a memory. In this specification, a circuitry, unit, or means is hardware that is programmed to realize or performs the described functions, which may be any hardware disclosed in this specification or any hardware known to be programmed to realize or perform the described functions. If the hardware is a processor considered to be a type of circuitry, the circuitry, means, or unit is a combination of the hardware and software used to configure the hardware and / or processor.

[0126] Although several embodiments of the present disclosure have been described above, these embodiments can be implemented in various other forms, and various omissions, substitutions, and modifications can be made without departing from the spirit of the invention. These embodiments and modifications are within the scope of the invention and its equivalents as described in the claims, as well as the scope and spirit of the invention.

[0127] (Additional Note) The matters described in the above embodiments will be supplemented below.

[0128] (Appendix 1) A program for operating a computer having a processor and a memory, the program causing the processor to execute the following steps: acquiring a video consisting of multiple frames; inputting the video or specific frames that make up the video and a prompt for explaining the content of the frames that make up the video into a large-scale language model, and causing the large-scale language model to output an explanatory text for the frame; inputting multiple explanatory texts and a prompt for outputting a summary of the video into the large-scale language model, and causing the large-scale language model to output the summary of the video; and presenting the summary output by the large-scale language model to a user. (Appendix 2) The program according to (Appendix 1), wherein in the step of outputting an explanatory text, frames at a predetermined interval among the multiple frames are input to a large-scale language model. (Appendix 3) The program according to (Appendix 1) or (Appendix 2), wherein in the step of outputting an explanatory text, frames at random times among the multiple frames are input to a large-scale language model. (Appendix 4) A program described in any one of (Appendix 1) to (Appendix 3), wherein, in a step of outputting an explanatory text, a frame in which a specified object or a specified action is detected among multiple frames is input into a large-scale language model. (Appendix 5) A program described in any one of (Appendix 1) to (Appendix 4), which causes a processor to execute a step of presenting a list of explanatory texts to a user and receiving a prompt from the user to output a summary of the video. (Appendix 6) A program as described in (Appendix 5), wherein in the receiving step, a description of a frame in which a specified object or a specified action is displayed is presented to the user in a manner that makes it distinguishable from other descriptions. (Appendix 7) A program described in any one of (Appendix 1) to (Appendix 6), in which, in a step of outputting a summary, a description of a plurality of frames included in a predetermined time width and a prompt for outputting a summary of a video of the predetermined time width are input to a large-scale language model, and the summary of the video of the predetermined time width is output from the large-scale language model. (Appendix 8) The program according to any one of (Supplementary Note 1) to (Supplementary Note 7), wherein in the acquiring step, the video is a video captured by a surveillance camera. (Appendix 9) A program described in any one of (Appendix 1) to (Appendix 8), wherein in the acquiring step, the video is a video taken during surgery. (Appendix 10) A method implemented by a computer having a processor and a memory, wherein the processor executes all of the steps performed in any of the inventions according to (Appendix 1) to (Appendix 9). (Appendix 11) An information processing device comprising a control unit and a memory unit, wherein the control unit executes all of the steps executed in the invention according to any one of (Appendix 1) to (Appendix 9). (Appendix 12) A system comprising means for executing all the steps performed in any of the inventions according to (Appendix 1) to (Appendix 9). [Explanation of symbols]

[0129] 1. System 10...Terminal device 12…Communication Interface 13...Input device 14...Output device 15…Memory 16…Storage 19…Processor 20…Server 22…Communication IF 23... Input / Output IF 25…Memory 26…Storage 29…Processor 30…LLM System 31...Photographic equipment 80…Network

Claims

1. A program for operating a computer having a processor and a memory, the program causing the processor to: acquiring a video of a number of frames; inputting the video or a predetermined frame constituting the video and a prompt for explaining the content of the frame constituting the video into a large-scale language model, and causing the large-scale language model to output an explanatory text for the frame; presenting the list of descriptions to a user and receiving a prompt from the user to output a summary of the video; inputting the plurality of description sentences and the received prompt into the large-scale language model, and causing the large-scale language model to output a summary of the video; presenting a summary output by the large-scale language model to the user; A program that executes the following.

2. 2. The program according to claim 1, wherein in the step of outputting the explanatory text, frames at a predetermined interval among the plurality of frames are input to the large-scale language model.

3. 2. The program according to claim 1, wherein in the step of outputting the explanatory text, frames at random times among the plurality of frames are input to the large-scale language model.

4. 2. The program according to claim 1, wherein in the step of outputting the explanatory text, a frame in which a predetermined object or a predetermined action is detected among the multiple frames is input to the large-scale language model.

5. 2. The program according to claim 1, wherein in the receiving step, the explanatory text of the frame in which the predetermined object or the predetermined action appears is presented to the user in a manner that makes it distinguishable from other explanatory texts.

6. The program of claim 1, wherein in the step of outputting the summary, a description of a plurality of frames included in a predetermined time range and a prompt for outputting a summary of the video of the predetermined time range are input to the large-scale language model, and the summary of the video of the predetermined time range is output from the large-scale language model.

7. The program according to claim 1 , wherein in the acquiring step, the video is a video captured by a surveillance camera.

8. The program according to claim 1 , wherein in the acquiring step, the video is a video taken during surgery.

9. A method implemented on a computer having a processor and a memory, the processor performing all of the steps performed in any one of claims 1 to 8.

10. 9. An information processing apparatus comprising a control unit and a storage unit, the control unit executing all steps executed in the invention according to any one of claims 1 to 8.

11. A system comprising means for executing all the steps performed in any one of claims 1 to 8.

Images