Information processing device, information processing method, and information processing program

Abstract

To improve the user experience when searching for videos using natural language. [Solution] The information processing device according to the present invention comprises: a first generation unit that generates multiple caption vectors, which are embedding vectors corresponding to each of the multiple captions corresponding to each of the multiple videos; a reception unit that receives a search statement from a user; a second generation unit that generates a search statement vector, which is an embedding vector corresponding to the search statement received by the reception unit; and a search unit that obtains videos from among the multiple videos as search results videos corresponding to caption vectors whose similarity between the caption vector and the search statement vector exceeds a first threshold.

Description

Technical Field

[0001] The present invention relates to an information processing apparatus, an information processing method, and an information processing program.

Background Art

[0002] Conventionally, techniques for managing videos such as video data have been known. For example, text data of video elements included in video data is generated, a sentence is constructed using the text data of the video elements, metadata of the video content is generated, and the video data and the metadata are associated and stored in a storage unit. Further, a technique is known in which metadata corresponding to a user's request is specified from the metadata in the storage unit, and the video data corresponding to the specified metadata is extracted and output from the storage unit.

Prior Art Documents

Patent Documents

[0003]

Patent Document 1

Summary of the Invention

Means for Solving the Problems

[0004] The information processing apparatus according to the present application includes a first generation unit that generates each of a plurality of caption vectors, which are embedding vectors corresponding to each of a plurality of captions corresponding to each of a plurality of videos; a reception unit that receives a search sentence from a user; a second generation unit that generates a search sentence vector, which is an embedding vector corresponding to the search sentence received by the reception unit; and a search unit that acquires, as a search result, a video corresponding to the caption vector whose similarity with the search sentence vector exceeds a first threshold value from among the plurality of videos.

Brief Description of the Drawings

[0005] [Figure 1]Figure 1 shows an example of the configuration of an information processing system according to an embodiment. [Figure 2] Figure 2 shows an example of the configuration of an information processing device according to the embodiment. [Figure 3] Figure 3 shows an example of information stored in the video information storage unit according to this embodiment. [Figure 4] Figure 4 shows an example of information processing by the information processing device according to the embodiment. [Figure 5] Figure 5 shows an example of input data according to the present invention. [Figure 6] Figure 6 shows an example of a prompt according to the present invention. [Figure 7] Figure 7 shows an example of information processing by an information processing device related to a modified example. [Figure 8] Figure 8 shows an example of a hardware configuration. [Modes for carrying out the invention]

[0006] The following describes in detail, with reference to the drawings, embodiments for implementing the information processing apparatus, information processing method, and information processing program according to the present application (hereinafter referred to as "embodiments"). Note that these embodiments do not limit the information processing apparatus, information processing method, and information processing program according to the present application. Furthermore, the same parts are denoted by the same reference numerals in each of the following embodiments, and redundant descriptions are omitted. The images in these embodiments may include both still images and moving images. In the following, moving images may be referred to as moving images. Moving images include video footage.

[0007] (Embodiment) [1. Introduction] Conventionally, there are known technologies that allow users to search for videos they desire from a pre-stored collection of videos. For example, a user may want to search for videos that include events belonging to a specific category (hereinafter referred to as "specified events"). Here, a specified event may be, for example, dangerous driving or driving that violates traffic rules, such as failing to stop at a stop sign or running a red light. Furthermore, a specified event is not limited to dangerous driving or driving that violates traffic rules; it may also include criminal acts such as shoplifting or the appearance of a suspicious person. Thus, a specified event is, for example, an event that the user is monitoring. However, a specified event is not limited to events that the user is monitoring. For example, a specified event may be an event that is of interest or concern to the user.

[0008] For example, to make it possible for users to search for desired videos, there is a method of recording all video footage acquired from an imaging device such as a camera at predetermined intervals (e.g., every 5 minutes) (hereinafter sometimes referred to as "full recording"). In the case of full recording, the storage capacity of the memory unit that stores the video is consumed heavily, and it cannot be said that storage can be used efficiently. Also, in order to find the part of the video that contains the desired event, the user needs to watch the entire video.

[0009] There is also a method of recording only videos that contain predetermined events defined by the user (hereinafter sometimes referred to as "spot recording"). In the case of spot recording, the user does not have to spend time checking videos compared to full recording, but it has the disadvantage that the user cannot search for videos that contain events that have not been defined in advance, as it only stores videos that contain predetermined events defined by the user.

[0010] In contrast, the information processing device according to this embodiment generates multiple caption vectors, which are embedding vectors corresponding to each of the multiple captions corresponding to each of the multiple videos. The information processing device also receives a search query from the user and generates a search query vector, which is an embedding vector corresponding to the received search query. Furthermore, the information processing device retrieves videos as search results from among the multiple videos for which the similarity between the caption vector and the search query vector exceeds a first threshold. In this way, by generating captions corresponding to videos in advance, the information processing device can improve the convenience for users when searching for videos using natural language.

[0011] [2. Configuration of the Information Processing System] Figure 1 is a diagram showing an example configuration of an information processing system 1 according to an embodiment. As shown in Figure 1, the information processing system 1 according to the embodiment includes a terminal device 10 and an information processing device 100. The terminal device 10 and the information processing device 100 are connected to each other via a predetermined communication network (network N) by wired or wireless means. Note that the information processing system 1 may include multiple terminal devices 10 and multiple information processing devices 100.

[0012] Terminal device 10 is an information processing device equipped with a camera. Terminal device 10 may be a household electrical appliance (home appliance), a mobile device such as a smartphone owned by a person, or an automobile. The camera may be a camera mounted on an information device such as a smartphone, a camera mounted on an industrial robot, an in-vehicle camera (dashcam), a security camera, or a surveillance camera. Terminal device 10 may also be referred to as an edge device. The camera generates video (images) of the environment around the camera. Terminal device 10 acquires video from the camera.

[0013] The information processing apparatus 100 is an information processing apparatus that performs information processing according to the embodiment. For example, the information processing apparatus 100 may perform the information processing according to the embodiment in accordance with the information processing method realized by the information processing program according to the embodiment. For example, the information processing apparatus 100 corresponds to a cloud computer (server apparatus).

[0014] [3. Configuration of Information Processing Apparatus] FIG. 2 is a diagram showing a configuration example of the information processing apparatus 100 according to the embodiment. The information processing apparatus 100 according to the embodiment includes a communication unit 110, a storage unit 120, and a control unit 130.

[0015] (Communication Unit 110) The communication unit 110 is connected to the network N by wire or wirelessly, and transmits and receives information to and from the terminal device 10. For example, the communication unit 110 is realized by a NIC (Network Interface Card), an antenna, or the like.

[0016] (Storage Unit 120) The storage unit 120 is realized by, for example, a semiconductor memory element such as a RAM (Random Access Memory) or a flash memory, or a storage device such as a hard disk or an optical disk. Specifically, the storage unit 120 stores the information processing program according to the embodiment. The storage unit 120 also includes a video information storage unit 121.

[0017] (Video Information Storage Unit 121) The video information storage unit 121 stores various information related to videos. FIG. 3 is a diagram showing an example of the information stored in the video information storage unit 121 according to the embodiment. As shown in FIG. 3, the video information storage unit 121 stores information of items such as "video", "timestamp", "caption vector", and "caption" in association with each other.

[0018] The "Video" field stores information that associates video data with identification information that identifies the video data. Note that video data (hereinafter sometimes simply referred to as "video") may be a group of multiple frames that make up a video, or it may be individual frames that make up a video. In Figure 3, the video data stored in the first row of the "Video" field is shown as "MV1". The video data stored in the second row of the "Video" field is shown as "MV2". The video data stored in the third row of the "Video" field is shown as "MV3".

[0019] The "Timestamp" field stores information about the time of the video data. For example, if the video data stored in the "Video" field is a group of multiple frames, the "Timestamp" field will store information indicating the time of the earliest frame or the time of the last frame included in the group of multiple frames. Alternatively, the "Timestamp" field may store the average time of the frames included in the group of multiple frames. Furthermore, if the video data stored in the "Video" field is individual frames, the "Timestamp" field will store information indicating the time of each individual frame. In Figure 3, the data stored in the first row of the "Timestamp" field is shown as "T1". The data stored in the second row of the "Timestamp" field is shown as "T2". The data stored in the third row of the "Timestamp" field is shown as "T3".

[0020] The "Caption" field stores the captions corresponding to the videos stored in the "Video" field. For example, if the video data stored in the "Video" field is a group of multiple frames, the "Caption" field stores the captions corresponding to the multiple frames (for example, descriptive text explaining the content of the multiple frames). Also, if the video data stored in the "Video" field is individual frames, the "Caption" field stores the captions corresponding to each individual frame (for example, descriptive text explaining the content of each individual frame). In Figure 3, the data stored in the first row of the "Caption" field is shown as "CA1". The data stored in the second row of the "Caption" field is shown as "CA2". The data stored in the third row of the "Caption" field is shown as "CA3".

[0021] The "Caption Vector" field stores embedded vectors corresponding to the captions associated with the videos stored in the "Video" field. In other words, the "Caption Vector" field stores embedded vectors corresponding to the captions stored in the "Caption" field. For example, if the "Caption" field contains descriptive text describing the content of multiple frames, the "Caption Vector" field stores embedded vectors corresponding to the descriptive text describing the content of multiple frames. Also, if the "Caption" field contains descriptive text describing the content of individual frames, the "Caption Vector" field stores embedded vectors corresponding to the descriptive text describing the content of individual frames. In Figure 3, the data stored in the first row of the "Caption Vector" field is shown as "CV1". The data stored in the second row of the "Caption Vector" field is shown as "CV2". The data stored in the third row of the "Caption Vector" field is shown as "CV3".

[0022] (Control unit 130) The control unit 130 is a controller, and is realized, for example, by executing various programs stored in the memory device inside the information processing device 100 using RAM as the working area, using a CPU (Central Processing Unit) or MPU (Micro Processing Unit), etc. Alternatively, the control unit 130 is a controller and can be realized, for example, by an integrated circuit such as an ASIC (Application Specific Integrated Circuit) or FPGA (Field Programmable Gate Array).

[0023] The control unit 130 has an acquisition unit 131, a first generation unit 132, a reception unit 133, a second generation unit 134, a search unit 135, a third generation unit 136, a storage unit 137, and a notification unit 138 as functional units, and may realize or execute the information processing operations described below. Note that the internal configuration of the control unit 130 is not limited to the configuration shown in Figure 2, and other configurations are also acceptable as long as they perform the information processing described later. In addition, each functional unit indicates the function of the control unit 130 and does not necessarily have to be physically separated.

[0024] (Acquisition part 131) The acquisition unit 131 acquires video from the terminal device 10. The acquisition unit 131 acquires multiple video. Figure 4 is a diagram showing an example of information processing by the information processing device according to the embodiment. In Figure 4, the acquisition unit 131 acquires three video MV1 to MV3 from the terminal device 10. Note that in Figure 4, for simplicity, the case in which the acquisition unit 131 acquires three video from the terminal device 10 is described, but the acquisition unit 131 may acquire four or more video from the terminal device 10.

[0025] When the acquisition unit 131 acquires a video, it stores information related to the video in the storage unit 120. For example, as an example of information related to a video, the acquisition unit 131 stores the video and identification information that identifies the video in the video information storage unit 121.

[0026] (first generation unit 132) The first generation unit 132 inputs each of multiple videos into a caption generation model, which is a machine learning model that generates captions corresponding to videos from videos, and generates each of multiple captions corresponding to each of the multiple videos. For example, the first generation unit 132 obtains a caption generation model that has been trained to output a caption corresponding to a video when a video is input, based on a dataset of videos and captions corresponding to those videos. For example, the first generation unit 132 obtains a caption generation model that has been trained to generate multiple captions for one video at a time. For example, when a video is input, the first generation unit 132 obtains a caption generation model that has been trained to extract multiple frames from the video at predetermined time intervals (e.g., 5 seconds, etc.) and output a caption corresponding to each of the extracted frames. Alternatively, the first generation unit 132 may call the caption generation model multiple times for one video and generate multiple captions for one video. Furthermore, the caption generation model may be a multimodal large-scale language model (multimodal LLM). For example, the caption generation model may be Gemini-1.5-Pro ​​or MV-GPT, etc. The first generation unit 132 inputs the video acquired by the acquisition unit 131 into the caption generation model and generates captions corresponding to the video. In Figure 4, the first generation unit 132 inputs each of the three videos MV1 to MV3 acquired by the acquisition unit 131 into the caption generation model M1 and generates three captions CA1 to CA3 corresponding to each of the three videos MV1 to MV3.

[0027] Furthermore, the first generation unit 132 inputs each of the multiple videos into the caption generation model and generates each of the multiple timestamps corresponding to each of the multiple videos. For example, when a video is input, the first generation unit 132 acquires a caption generation model that has been trained to output a caption and a timestamp corresponding to the video. In Figure 4, the first generation unit 132 inputs each of the three videos MV1 to MV3 into the caption generation model M1 and generates each of the three captions CA1 to CA3 corresponding to each of the three videos MV1 to MV3, and each of the three timestamps T1 to T3 corresponding to each of the three videos MV1 to MV3.

[0028] Furthermore, when the first generation unit 132 generates each of the multiple captions and each of the multiple timestamps, it associates each of the multiple videos, each of the multiple captions, and each of the multiple timestamps and stores them in the video information storage unit 121. In Figure 4, the first generation unit 132 associates each of the three videos MV1 to MV3, each of the three captions CA1 to CA3, and each of the three timestamps T1 to T3 and stores them in the video information storage unit 121.

[0029] Furthermore, the first generation unit 132 inputs each of the multiple captions into an embedding model, which is a machine learning model that generates embedding vectors corresponding to text from text, and generates each of the multiple caption vectors. For example, the first generation unit 132 obtains an embedding model that has been trained to output an embedding vector corresponding to text when text is input, based on a dataset of text and embedding vectors corresponding to text. For example, the embedding model may be a text encoder. For example, the text encoder may include a recurrent neural network (RNN). The text encoder may use the RNN to extract text features (corresponding to embedding vectors) from text. Alternatively, the text encoder may include a GRU (Gated Recurrent Unit) or an LSTM (Long Short Term Memory). The text encoder may use the GRU or LSTM to extract text features from text. Furthermore, the text encoder may include, for example, a Transformer (Ashish Vaswani et al., 2017), a Transformer-based BERT (Bidirectional Encoder Representations from Transformers), GPT-3 (Generative Pre-Training 3), or T5 (Text-to-Text Transfer Transformer). The text encoder may then extract text features from the text using the Transformer, BERT, GPT-3, or T5. Additionally, the text encoder may include the CLIP text encoder, and it may then extract text features from the text using the CLIP text encoder. The embedding model may be Word2Vec, GloVe (Global Vectors for Word Representation), or Multimodal Embeddings.The embedding model may be an encoder for a multimodal large-scale language model. For example, the embedding model may be an encoder for text-davinci-003 or GPT-4o. The first generation unit 132 inputs the captions generated using the caption generation model into the embedding model to generate caption vectors, which are embedding vectors corresponding to the captions generated using the caption generation model. In Figure 4, the first generation unit 132 inputs each of the three captions CA1 to CA3 into the embedding model M2 to generate three caption vectors CV1 to CV3, each corresponding to each of the three captions CA1 to CA3.

[0030] Furthermore, when the first generation unit 132 generates each of the multiple caption vectors, it associates each of the multiple videos, each of the multiple captions, each of the multiple timestamps, and each of the multiple caption vectors and stores them in the video information storage unit 121. In Figure 4, the first generation unit 132 associates each of the three videos MV1 to MV3, each of the three captions CA1 to CA3, each of the three timestamps T1 to T3, and each of the three caption vectors CV1 to CV3 and stores them in the video information storage unit 121.

[0031] As described above, the first generation unit 132 generates each of the multiple caption vectors, which are embedding vectors corresponding to each of the multiple captions corresponding to each of the multiple videos. In Figure 4, the first generation unit 132 generates each of the three caption vectors CV1 to CV3, which are embedding vectors corresponding to each of the three captions CA1 to CA3, corresponding to each of the three videos MV1 to MV3 acquired by the acquisition unit 131.

[0032] (Reception desk 133) The reception unit 133 receives search queries from users. For example, the reception unit 133 receives search queries from information processing devices used by users. The search query may be any text, such as words, phrases, or sentences.

[0033] (Second generation unit 134) The second generation unit 134 generates a search statement vector, which is an embedded vector corresponding to the search statement received by the reception unit 133. Specifically, the second generation unit 134 inputs the search statement into the embedded model to generate the search statement vector. For example, when the second generation unit 134 receives a search statement from the reception unit 133, it inputs the search statement received by the reception unit 133 into the embedded model M2 explained in Figure 4 to generate the search statement vector.

[0034] (Search section 135) The search unit 135 retrieves videos as search results from among multiple videos in which the similarity between the caption vector and the search text vector exceeds a first threshold. Specifically, when a search text vector is generated by the second generation unit 134, the search unit 135 refers to the video information storage unit 121 and retrieves multiple caption vectors corresponding to each of the multiple videos generated by the first generation unit 132. Subsequently, the search unit 135 calculates the similarity between each of the multiple caption vectors generated by the first generation unit 132 and the search text vector generated by the second generation unit 134. Subsequently, the search unit 135 retrieves videos as search results in which the similarity between the caption vector and the search text vector exceeds a first threshold. In this specification, the similarity between vectors corresponds to the distance between vectors. Furthermore, the distance used as an indicator of the similarity between vectors can be any distance that is applicable as a distance between vectors. For example, various distances such as L2 distance (Euclidean distance), Mahalanobis distance, and cosine distance may be used.

[0035] (Third generation unit 136) The third generation unit 136 determines whether a predetermined event has occurred based on the similarity of multiple caption vectors, and if it determines that a predetermined event has occurred, it generates event information indicating the predetermined event based on multiple captions corresponding to each of the multiple caption vectors. For example, when a new set of data is stored in the video information storage unit 121 by the first generation unit 132, the third generation unit 136 refers to the video information storage unit 121 and obtains multiple caption vectors that are consecutive in time. For example, the third generation unit 136 obtains multiple caption vectors arranged in chronological order as multiple caption vectors that are consecutive in time. For example, the third generation unit 136 refers to the video information storage unit 121 and obtains multiple caption vectors V1, V2, V3, V4, ..., V N-1 , V N (N is a natural number) is obtained. Next, the third generation unit 136 determines whether the similarity of multiple time-sequential caption vectors exceeds the second threshold. For example, the third generation unit 136 determines whether the similarity of multiple caption vectors arranged in time series exceeds the second threshold. For example, the third generation unit 136 obtains the cosine similarity of V1 and V2, the cosine similarity of V2 and V3, the cosine similarity of V3 and V4, ..., V N-1 and V NThe third generation unit 136 calculates the cosine similarity of adjacent caption vectors in a time series, similar to the cosine similarity of the previous example, and arranges them in chronological order. The third generation unit 136 also determines whether there are any consecutive sequences of cosine similarity values ​​that exceed a second threshold (e.g., 0.9). For example, the third generation unit 136 determines whether there are any sequences of a predetermined number or more consecutive sequences of cosine similarity values ​​that exceed the second threshold. If the third generation unit 136 determines that there are sequences of a predetermined number or more consecutive sequences of cosine similarity values ​​that exceed the second threshold, it determines that a predetermined event has occurred. In this way, the third generation unit 136 determines that a predetermined event has occurred if it determines that the similarity of multiple temporally consecutive caption vectors exceeds the second threshold. Furthermore, when the third generation unit 136 determines that a predetermined event has occurred, it generates event information indicating the predetermined event based on multiple captions corresponding to each of a plurality of temporally consecutive caption vectors.

[0036] Furthermore, the third generation unit 136 determines whether the similarity of multiple time-sequential caption vectors is less than or equal to the third threshold. For example, the third generation unit 136 determines whether the similarity of multiple caption vectors arranged in time-series order is less than or equal to the third threshold. For example, the third generation unit 136 determines the cosine similarity of V1 and V2, the cosine similarity of V2 and V3, the cosine similarity of V3 and V4, ..., V N-1 and V NSimilar to the cosine similarity of the previous caption vector, the third generation unit 136 calculates the cosine similarity of adjacent caption vectors in the time series and arranges them in the time series order. The third generation unit 136 also determines whether there is a portion where a predetermined number of consecutive values ​​of the cosine similarity of the time-series-ordered caption vectors are below the third threshold (e.g., -0.9). If the third generation unit 136 determines that there is a portion where a predetermined number of consecutive values ​​of the cosine similarity of the time-series-ordered caption vectors are below the third threshold, it determines that a predetermined event has occurred. In this way, the third generation unit 136 determines that a predetermined event has occurred if it determines that the similarity of multiple time-sequentially consecutive caption vectors is below the third threshold. Furthermore, if the third generation unit 136 determines that a predetermined event has occurred, it generates event information indicating the predetermined event based on multiple captions corresponding to each of the multiple time-sequentially consecutive caption vectors. Furthermore, the third generator 136 is not limited to using cosine similarity as an index to indicate the similarity between caption vectors. For example, the third generator 136 may calculate a similarity corresponding to the L2 distance (Euclidean distance), a similarity corresponding to the Mahalanobis distance, or the dot product as an index to indicate the similarity between caption vectors.

[0037] More specifically, the third generation unit 136 determines whether a predetermined event has occurred based on the similarity of multiple temporally consecutive caption vectors. If it determines that a predetermined event has occurred, it prompts the unit to generate event information indicating the predetermined event based on multiple captions corresponding to each of the multiple temporally consecutive caption vectors, and inputs the multiple temporally consecutive caption vectors into the language model to generate the event information. The language model can be any machine learning model capable of processing natural language, and can be a large-scale or small-scale language model. The following describes the case where the language model is a large-scale language model. A large-scale language model may be, for example, a GPT (Generative Pre-trained Transformer) model.

[0038] Figure 5 shows an example of input data according to the embodiment. The third generation unit 136 inputs the input data shown in Figure 5 into a large-scale language model to generate event information. In Figure 5, the third generation unit 136 refers to the video information storage unit 121 and obtains data IN1, which is a set of timestamp T11, caption vector CV11, and caption CA11. In this way, the third generation unit 136 obtains data that is a set of timestamp, caption vector, and caption. The third generation unit 136 also obtains data that is a set of timestamps whose times are consecutive in time. The third generation unit 136 obtains data that is a set of timestamps, caption vectors, and captions whose times are consecutive in time. In this way, the third generation unit 136 obtains multiple caption vectors that are consecutive in time. The third generation unit 136 inputs data that is a set of timestamps, caption vectors, and captions whose times are consecutive in time into a large-scale language model to generate event information. In this way, the third generation unit 136 inputs multiple temporally consecutive caption vectors into a large-scale language model to generate event information.

[0039] Figure 6 shows an example of a prompt according to the embodiment. The third generation unit 136 inputs a prompt P1 as shown in Figure 6 to the large-scale language model and generates event information. For example, the third generation unit 136 inputs a prompt to the large-scale language model as an input prompt, which includes an instruction that tells the system to generate a new event by referring to the input data as described in Figure 5, and generates event information. In Figure 6, the third generation unit 136 inputs a prompt P1 including the sentence P11 "Input prompt: Refer to the input data and create a new event" to the large-scale language model and generates event information.

[0040] Furthermore, the third generation unit 136 generates event information by inputting a prompt to the large-scale language model containing an instruction to determine whether a predetermined event has occurred based on the similarity of a plurality of temporally consecutive caption vectors. More specifically, the third generation unit 136 determines whether the similarity of a plurality of temporally consecutive caption vectors exceeds a second threshold, and if it determines that the similarity of a plurality of temporally consecutive caption vectors exceeds the second threshold, it inputs a prompt to the large-scale language model containing an instruction to determine that a predetermined event has occurred, thereby generating event information. In addition, the third generation unit 136 determines whether the similarity of a plurality of temporally consecutive caption vectors is less than or equal to a third threshold, and if it determines that the similarity of a plurality of temporally consecutive caption vectors is less than or equal to the third threshold, it inputs a prompt to the large-scale language model containing an instruction to determine that a predetermined event has occurred, thereby generating event information. Furthermore, if the third generation unit 136 determines that a predetermined event has occurred, it inputs a prompt to the large-scale language model instructing it to generate event information indicating the predetermined event based on multiple captions corresponding to each of multiple temporally consecutive caption vectors, thereby generating event information. In Figure 6, the third generation unit 136 inputs a prompt P1 containing the text P12, "Event determination method: Data that shows a high degree of similarity from {timestamp,[embedding],…} consecutively or changes abruptly is particularly likely to be an event. If it is determined that there is a high probability that it is an event, generate the event using captions," to the large-scale language model, thereby generating event information.

[0041] Furthermore, the third generation unit 136 generates event information by inputting a prompt containing an instruction sentence that tells the large-scale language model to output event information in text format as output data. In Figure 6, the third generation unit 136 generates event information by inputting a prompt P1 containing the sentence P13 "Output: {event}" into the large-scale language model. Furthermore, the third generation unit 136 generates event information by inputting a prompt containing information showing an example of event determination into the large-scale language model. In Figure 6, the third generation unit 136 generates event information by inputting a prompt P1 containing the sentence P14 "Event determination example: The similarity of the embedding vectors from 12:00:01 to 12:02:00 is continuously high. This state is "overturned". Output: {"overturned"}" into the large-scale language model.

[0042] (Preservation Department 137) The storage unit 137 stores a video corresponding to a predetermined event in the storage unit based on the event information. Specifically, when event information is generated by the third generation unit 136, the storage unit 137 stores a video corresponding to a plurality of temporally consecutive caption vectors whose similarity is determined by the third generation unit 136 to exceed the second threshold in the storage unit. For example, the storage unit 137 may refer to the video information storage unit 121 to obtain a video corresponding to a plurality of temporally consecutive caption vectors whose similarity is determined by the third generation unit 136 to exceed the second threshold. The storage unit 137 also generates identification information to identify the video corresponding to the plurality of temporally consecutive caption vectors whose similarity is determined by the third generation unit 136 to exceed the second threshold, and stores it in the video information storage unit 121 in association with the video corresponding to the plurality of temporally consecutive caption vectors whose similarity is determined by the third generation unit 136 to exceed the second threshold. In the following, a video corresponding to multiple temporally consecutive caption vectors whose similarity is determined by the third generation unit 136 to exceed the second threshold may be referred to as a "video corresponding to a predetermined event."

[0043] Furthermore, when event information is generated by the third generation unit 136, the storage unit 137 stores in the storage unit the video corresponding to a plurality of temporally consecutive caption vectors whose similarity is determined to be below the third threshold by the third generation unit 136. For example, the storage unit 137 may refer to the video information storage unit 121 to obtain a video corresponding to a plurality of temporally consecutive caption vectors whose similarity is determined to be below the third threshold by the third generation unit 136. The storage unit 137 also generates identification information to identify the video corresponding to a plurality of temporally consecutive caption vectors whose similarity is determined to be below the third threshold by the third generation unit 136, and stores it in the video information storage unit 121 in association with the video corresponding to a plurality of temporally consecutive caption vectors whose similarity is determined to be below the third threshold by the third generation unit 136. In the following, the video corresponding to a plurality of temporally consecutive caption vectors whose similarity is determined to be below the third threshold by the third generation unit 136 may be referred to as "a video corresponding to a predetermined event."

[0044] Furthermore, the storage unit 137 may delete videos other than those corresponding to a predetermined event. For example, when event information is generated by the third generation unit 136, the storage unit 137 may refer to the video information storage unit 121 and delete videos from time periods other than those corresponding to the predetermined event.

[0045] Furthermore, the first generation unit 132 generates multiple caption vectors from each of the multiple captions corresponding to each of the multiple videos, including the video corresponding to a predetermined event. For example, the first generation unit 132 generates multiple caption vectors from each of the multiple captions corresponding to each of the multiple videos, including the video corresponding to multiple temporally consecutive caption vectors whose similarity has been determined by the third generation unit 136 to exceed the second threshold. Also, the first generation unit 132 generates multiple caption vectors from each of the multiple captions corresponding to each of the multiple videos, including the video corresponding to multiple temporally consecutive caption vectors whose similarity has been determined by the third generation unit 136 to be below the third threshold.

[0046] (Notification Department 138) The notification unit 138 notifies the user that a video corresponding to a predetermined event has been saved, based on the event information. Specifically, the notification unit 138 transmits notification information regarding the saving of a video corresponding to a predetermined event by the saving unit 137 to the information processing device used by the user.

[0047] [4. Variations] Figure 7 shows an example of information processing by an information processing device according to a modified example. In Figure 7, the first generation unit 132 does not input each of the multiple videos into the caption generation model and generate each of the multiple captions corresponding to each of the multiple videos, but rather inputs each of the multiple videos into the embedding model and generates each of the multiple video vectors. In Figure 7, the first generation unit 132 inputs each of the three videos MV1 to MV3 into the embedding model M2 and generates each of the video vectors EV1 to EV3, which are embedding vectors corresponding to each of the three videos MV1 to MV3. The first generation unit 132 also inputs each of the three videos MV1 to MV3 into the embedding model M2 and generates each of the three timestamps T1 to T3, which correspond to each of the three videos MV1 to MV3. In Figure 7, the first generation unit 132 associates each of the three videos MV1 to MV3, each of the three timestamps T1 to T3, and each of the three video vectors EV1 to EV3 and stores them in the video information storage unit 121.

[0048] [5. Effects] As described above, the information processing device 100 according to the embodiment comprises a first generation unit 132, a reception unit 133, a second generation unit 134, and a search unit 135. The first generation unit 132 generates multiple caption vectors, which are embedding vectors corresponding to each of the multiple captions corresponding to each of the multiple videos. The reception unit 133 receives a search statement from the user. The second generation unit 134 generates a search statement vector, which is an embedding vector corresponding to the search statement received by the reception unit 133. The search unit 135 obtains videos from the multiple videos as search results that correspond to caption vectors whose similarity between the caption vector and the search statement vector exceeds a first threshold.

[0049] This allows the information processing device 100 to improve the convenience of users when searching for videos using natural language by pre-generating captions corresponding to the videos. Furthermore, because the information processing device 100 can improve the convenience of users when searching for videos using natural language, it can contribute to achieving Sustainable Development Goal (SDG) 9, "Build resilient infrastructure, promote inclusive and sustainable industrialization and foster innovation."

[0050] Furthermore, the first generation unit 132 inputs each of the multiple videos into a caption generation model, which is a machine learning model that generates captions corresponding to videos from videos, and generates each of the multiple captions. It then inputs each of the multiple captions into an embedding model, which is a machine learning model that generates embedding vectors corresponding to text from text, and generates each of the multiple caption vectors. The second generation unit 134 inputs a search sentence into the embedding model and generates a search sentence vector.

[0051] In this way, the information processing device 100 can appropriately calculate the similarity between the caption corresponding to the video and the search text by vectorizing each of the captions and search texts corresponding to the video.

[0052] The information processing device 100 also includes a third generation unit 136 and a storage unit 137. The third generation unit 136 determines whether a predetermined event has occurred based on the similarity of a plurality of caption vectors, and if it determines that a predetermined event has occurred, it generates event information indicating the predetermined event based on a plurality of captions corresponding to each of the plurality of caption vectors. The storage unit 137 stores the video corresponding to the predetermined event in the storage unit based on the event information. The first generation unit 132 generates each of a plurality of caption vectors from each of a plurality of captions corresponding to each of a plurality of videos, including the video corresponding to the predetermined event.

[0053] As a result, the information processing device 100 can store only the portion of the video that contains the event determined to be an event, and does not need to store the entire video. In this way, the information processing device 100 does not need to store the portion of the video that does not contain the event, thus reducing the consumption of the memory capacity of the memory unit. Furthermore, since the information processing device 100 can store videos that contain events other than those predetermined by the user, it is possible for the user to search for videos that contain events that have not been predetermined.

[0054] Furthermore, the third generation unit 136 determines whether the similarity of multiple time-sequential caption vectors exceeds a second threshold. If it determines that the similarity of multiple time-sequential caption vectors exceeds the second threshold, it determines that a predetermined event has occurred and generates event information indicating the predetermined event based on multiple captions corresponding to each of the multiple time-sequential caption vectors.

[0055] This allows the information processing device 100 to determine, for example, that a predetermined event has occurred when similar scenes occur consecutively.

[0056] Furthermore, the third generation unit 136 determines whether the similarity of multiple time-sequential caption vectors is below a third threshold. If it determines that the similarity of multiple time-sequential caption vectors is below the third threshold, it determines that a predetermined event has occurred and generates event information indicating the predetermined event based on multiple captions corresponding to each of the multiple time-sequential caption vectors.

[0057] This allows the information processing device 100 to determine, for example, that a predetermined event has occurred when the scene changes.

[0058] Furthermore, the third generation unit 136 determines whether a predetermined event has occurred based on the similarity of multiple temporally consecutive caption vectors. If it determines that a predetermined event has occurred, it issues a prompt instructing the generation of event information indicating the predetermined event based on multiple captions corresponding to each of the multiple temporally consecutive caption vectors, and inputs the multiple temporally consecutive caption vectors into the language model to generate the event information.

[0059] This allows the information processing device 100 to appropriately determine the occurrence of an event using a language model.

[0060] Furthermore, the information processing device 100 includes a notification unit 138. The notification unit 138 notifies the user that a video corresponding to a predetermined event has been saved, based on event information.

[0061] This allows the information processing device 100 to notify the user, for example, when a new event occurs that is different from an event predefined by the user.

[0062] [6. Hardware Configuration] Furthermore, the information processing device 100 according to the above-described embodiment is realized by a computer 1000 having a configuration such as that shown in Figure 8. The following explanation will use the information processing device 100 as an example. Figure 8 is a diagram showing an example of the hardware configuration. The computer 1000 is connected to an output device 1010 and an input device 1020, and has a configuration in which an arithmetic unit 1030, a primary storage device 1040, a secondary storage device 1050, an output interface 1060, an input interface 1070, and a network interface 1080 are connected by a bus 1090.

[0063] The arithmetic unit 1030 operates based on programs stored in the primary storage device 1040 and the secondary storage device 1050, as well as programs read from the input device 1020, and executes various processes. The arithmetic unit 1030 can be implemented using, for example, a CPU (Central Processing Unit), an MPU (Micro Processing Unit), a GPU (Graphics Processing Unit), an ASIC (Application Specific Integrated Circuit), or an FPGA (Field Programmable Gate Array).

[0064] The primary storage device 1040 is a memory device, such as RAM (Random Access Memory), that temporarily stores data used by the arithmetic unit 1030 for various calculations. The secondary storage device 1050 is a storage device where data used by the arithmetic unit 1030 for various calculations and various databases are registered, and can be implemented using ROM (Read Only Memory), HDD (Hard Disk Drive), SSD (Solid State Drive), flash memory, etc. The secondary storage device 1050 may be internal storage or external storage. The secondary storage device 1050 may also be a removable storage medium such as USB (Universal Serial Bus) memory or SD (Secure Digital) memory card. The secondary storage device 1050 may also be cloud storage (online storage), NAS (Network Attached Storage), file server, etc.

[0065] The output I / F 1060 is an interface for transmitting information to be output to output devices 1010, such as displays, projectors, and printers, and is implemented using connectors of standards such as USB (Universal Serial Bus), DVI (Digital Visual Interface), and HDMI (High Definition Multimedia Interface). The input I / F 1070 is an interface for receiving information from various input devices 1020, such as mice, keyboards, keypads, buttons, and scanners, and is implemented using, for example, USB.

[0066] Furthermore, the output interface 1060 and input interface 1070 may be wirelessly connected to the output device 1010 and input device 1020, respectively. In other words, the output device 1010 and input device 1020 may be wireless devices.

[0067] Furthermore, the output device 1010 and the input device 1020 may be integrated as a touch panel. In this case, the output I / F 1060 and the input I / F 1070 may also be integrated as an input / output I / F.

[0068] The input device 1020 may also be a device that reads information from, for example, an optical recording medium such as a CD (Compact Disc), DVD (Digital Versatile Disc), or PD (Phase Change Rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0069] The network interface 1080 receives data from other devices via network N and sends it to the computing unit 1030, and also transmits data generated by the computing unit 1030 to other devices via network N.

[0070] The arithmetic unit 1030 controls the output device 1010 and the input device 1020 via the output interface 1060 and the input interface 1070. For example, the arithmetic unit 1030 loads a program from the input device 1020 or the secondary storage device 1050 onto the primary storage device 1040 and executes the loaded program.

[0071] For example, when computer 1000 functions as an information processing device 100, the arithmetic unit 1030 of computer 1000 realizes the functions of the control unit 130 by executing a program loaded onto the primary storage device 1040. Alternatively, the arithmetic unit 1030 of computer 1000 may load a program obtained from another device via the network interface 1080 onto the primary storage device 1040 and execute the loaded program. Furthermore, the arithmetic unit 1030 of computer 1000 may cooperate with other devices via the network interface 1080 and call and use program functions, data, etc., from other programs on other devices.

[0072] [7. Other] Although embodiments of the present invention have been described above, the present invention is not limited by the content of these embodiments. Furthermore, the aforementioned components include those that can be easily conceived by those skilled in the art, those that are substantially the same, and those that fall within the so-called equivalent range. Moreover, the aforementioned components can be combined as appropriate. Furthermore, various omissions, substitutions, or modifications of the components can be made without departing from the gist of the embodiments described above.

[0073] Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by known methods. In addition, the processing procedures, specific names, and information including various data and parameters shown in the above document and drawings can be arbitrarily changed unless otherwise specified. For example, the various information shown in each figure is not limited to the information shown.

[0074] Furthermore, the components of each illustrated device are functionally conceptual and do not necessarily need to be physically configured as shown. In other words, the specific forms of distribution and integration of each device are not limited to those shown, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads and usage conditions.

[0075] For example, the information processing device 100 described above may be implemented using multiple server computers, and depending on the function, it may be implemented by calling external platforms, etc., via APIs (Application Programming Interfaces) or network computing, allowing for flexible configuration changes.

[0076] Furthermore, the embodiments and modifications described above can be combined as appropriate, provided that the processing content is not inconsistent. [Explanation of Symbols]

[0077] 1. Information Processing System 10 Terminal devices 100 Information Processing Devices 110 Communications Department 120 Storage section 130 Control Unit 131 Acquisition Department 132 1st generation part 133 Reception Department 134 Second generation part 135 Search Section 136 Third generation part 137 Preservation Department 138 Notification Department

Claims

1. A first generation unit generates multiple caption vectors, which are embedding vectors corresponding to each of the multiple captions, from each of the multiple captions corresponding to each of the multiple videos, A reception desk that receives search queries from users, A second generation unit generates a search sentence vector, which is an embedding vector corresponding to the search sentence received by the reception unit, A search unit that retrieves videos from among the aforementioned multiple videos as search results videos corresponding to the caption vector whose similarity between the caption vector and the search text vector exceeds a first threshold, An information processing device equipped with the following features.

2. The first generation unit is, Each of the multiple videos is input into a caption generation model, which is a machine learning model that generates captions corresponding to the videos from the videos, and each of the multiple captions is generated; each of the multiple captions is input into an embedding model, which is a machine learning model that generates embedding vectors corresponding to the text from the text, and each of the multiple caption vectors is generated; The second generation unit is, The search statement is input to the embedded model to generate the search statement vector. The information processing apparatus according to claim 1.

3. A third generation unit determines whether a predetermined event has occurred based on the similarity of the plurality of caption vectors, and if it is determined that the predetermined event has occurred, generates event information indicating the predetermined event based on the plurality of captions corresponding to each of the plurality of caption vectors. A storage unit that stores a video corresponding to the predetermined event in a storage unit based on the event information, Equipped with, The first generation unit is, Each of the multiple caption vectors is generated from each of the multiple captions corresponding to each of the multiple videos, each of which includes a video corresponding to the predetermined event. The information processing apparatus according to claim 1.

4. The third generation unit is, The system determines whether the similarity of multiple time-consecutive caption vectors exceeds a second threshold, and if it determines that the similarity of the multiple time-consecutive caption vectors exceeds the second threshold, it determines that the predetermined event has occurred, and generates event information indicating the predetermined event based on the multiple captions corresponding to each of the multiple time-consecutive caption vectors. The information processing apparatus according to claim 3.

5. The third generation unit is, The system determines whether the similarity of multiple time-consecutive caption vectors is below a third threshold, and if it determines that the similarity of the multiple time-consecutive caption vectors is below the third threshold, it determines that the predetermined event has occurred, and generates event information indicating the predetermined event based on the multiple captions corresponding to each of the multiple time-consecutive caption vectors. The information processing apparatus according to claim 3.

6. The third generation unit is, Based on the similarity of multiple time-sequential caption vectors, a prompt is issued to determine whether a predetermined event has occurred, and if it is determined that the predetermined event has occurred, the prompt instructs the system to generate event information indicating the predetermined event based on multiple captions corresponding to each of the multiple time-sequential caption vectors, and the system inputs the multiple time-sequential caption vectors into a language model to generate the event information. The information processing apparatus according to claim 3.

7. A notification unit that notifies the user that a video corresponding to the predetermined event has been saved based on the event information. The information processing apparatus according to claim 3, comprising:

8. An information processing method implemented by a program executed by an information processing device, A first generation step of generating multiple caption vectors, which are embedding vectors corresponding to each of the multiple captions, from each of the multiple captions corresponding to each of the multiple videos, The process of receiving search queries from users, A second generation step generates a search sentence vector, which is an embedding vector corresponding to the search sentence received in the above-mentioned reception step, A search step of obtaining, from among the aforementioned multiple videos, a video corresponding to the caption vector whose similarity between the caption vector and the search text vector exceeds a first threshold as a search result; Information processing methods including

9. A first generation procedure for generating multiple caption vectors, which are embedding vectors corresponding to each of the multiple captions corresponding to each of the multiple videos, The procedure for receiving search queries from users, A second generation procedure generates a search text vector, which is an embedded vector corresponding to the search text received by the above-mentioned reception procedure, A search procedure for obtaining, from among the aforementioned multiple videos, a video corresponding to the caption vector whose similarity to the search text vector exceeds a first threshold, as a search result, An information processing program that causes a computer to execute something.

Images