Video scene search system and video scene search method

Abstract

When using artificial intelligence for video searches, the system suppresses hallucination to search for the desired video scene. [Solution] The video scene search system 1 acquires video data and audio data from a video targeted for scene search, detects a subject from the scene image of the video data corresponding to each scene in the video, creates a context based on the subject in the scene image, creates scene information including the context and a timestamp of the scene section containing the scene image, acquires the sound content of the scene section of the audio data corresponding to each scene in the video, creates a context based on the sound content of the scene section, creates scene information based on the context and a timestamp of the scene section, creates structured data of the scene information of the scene section based on the video data or audio data, vectorizes the structured data and stores it as a RAG scene section search model used in LLM.

Description

Technical Field

[0001] The present invention relates to a video scene search system and a video scene search method that enable video scene search.

Background Art

[0002] Conventionally, for a terminal device that browses videos, an information processing device such as a server has a video search function that enables searching for videos, and provides the searched videos to the terminal device.

[0003] For example, in the information processing method disclosed in Patent Document 1, a receiver that outputs a broadcast program acquires browsing information of web information from a mobile terminal carried by a viewer, the receiver transmits the browsing information to a management server, the management server searches for videos provided by a video distribution service according to the browsing information, the management server transmits video information regarding the searched videos to the receiver, and the receiver outputs the video information.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] The video search function in an information processing device works by, for example, allowing the user to input search criteria for a video, searching for videos that match those criteria, and providing the candidate videos to the terminal device. However, with conventional video search functions, there are cases where the user cannot input the desired search criteria or find the desired video. Furthermore, with conventional video search functions, when the artificial intelligence that has learned the search criteria and videos searches for videos based on the search criteria, hallucination may occur, making it impossible to find the desired video. In particular, when searching for a specific scene in a video, hallucination is more likely to occur, and there is a problem in that the desired scene cannot be found.

[0006] Therefore, taking the above circumstances into consideration, the present invention aims to provide a video scene search system and a video scene search method that can search for desired video scenes while suppressing the occurrence of hallucination when artificial intelligence is used for video searching. [Means for solving the problem]

[0007] To solve the above problems, the video scene search system of the present invention is characterized by inputting a video to be searched for, acquiring video data and audio data from the video, detecting a subject from a scene image of the video data corresponding to each scene of the video in order to assist in the scene search of the video, creating a context based on the subject of the scene image, creating scene information including the context and a timestamp of the scene section including the scene image, acquiring the sound content of the scene section of the audio data corresponding to each scene of the video, creating a context based on the sound content of the scene section, creating scene information based on the context and the timestamp of the scene section, creating structured data of the scene information of the scene section based on the video data or the audio data, and vectorizing the structured data and storing it as a search model of the scene section for search extension generation used in a large-scale language model.

[0008] Furthermore, in order to solve the above problems, the present invention provides a video scene search method that inputs a video to be searched for, obtains video data and audio data from the video, detects a subject from the scene image of the video data corresponding to each scene of the video in order to assist in the scene search of the video, creates a context based on the subject of the scene image, creates scene information including the context and a timestamp of the scene section including the scene image, obtains the pronunciation content of the scene section of the audio data corresponding to each scene of the video, creates a context based on the pronunciation content of the scene section, creates scene information based on the context and the timestamp of the scene section, creates structured data of the scene information of the scene section based on the video data or the audio data, vectorizes the structured data and stores it as a search model of the scene section for search extension generation used in a large-scale language model. [Effects of the Invention]

[0009] According to the present invention, it is possible to provide a video scene search system and a video scene search method that can search for desired video scenes while suppressing the occurrence of hallucination in video search using artificial intelligence. [Brief explanation of the drawing]

[0010] [Figure 1] This is a schematic block diagram showing a video scene search system according to one embodiment of the present invention. [Figure 2] This flowchart shows an example of the operation for creating a search model in a video scene search system according to one embodiment of the present invention. [Figure 3] This flowchart shows an example of how video scene search works in a video scene search system according to one embodiment of the present invention. [Figure 4] This flowchart shows an example of the operation when selecting a video candidate in a video list in a video scene search system according to one embodiment of the present invention. [Figure 5]This flowchart shows an example of an operation involving behavioral analysis when creating a list of candidate videos in a video scene search system according to one embodiment of the present invention. [Modes for carrying out the invention]

[0011] First, with reference to Figure 1, the overall configuration of the video scene search system 1 according to an embodiment of the present invention will be described. As shown in Figure 1, the video scene search system 1 comprises a search server 2 that performs scene searches of various videos including video data and audio data, and a terminal device 3 for viewing videos, and the search server 2 and the terminal device 3 are connected to each other via a network 5 such as the Internet.

[0012] The video scene search system 1 is a system in which, in response to a scene search request from a terminal device 3 that specifies search conditions, the search server 2 uses artificial intelligence to search for video scenes that are candidates for viewing and provides them to the terminal device 3.

[0013] Here, the video scene search system 1 uses Retrieval Augmented Generation (RAG) to assist in video scene searching by creating and storing search models for each scene in the target video that is the subject of the scene search. The video scene search system 1 then creates and stores a learning model (generative model) that has been machine-learned using Large Language Model (LLM) artificial intelligence based on scene search requests and search models. The video scene search system 1 uses the search model via RAG and the LLM-applied artificial intelligence that utilizes this generative model to search for viewing candidates for video scenes in response to scene search requests from terminal device 3, and generates a response that provides viewing candidates for video scenes as an answer to the scene search request.

[0014] Although Figure 1 illustrates an example where the video scene search system 1 has one search server 2 and one terminal device 3, it is possible to have multiple terminal devices 3 connected to one search server 2.

[0015] The search server 2 is configured, for example, to include a control unit 10, a storage unit 11, a communication unit 12, a display unit 13, an operation unit 14, and a database 15.

[0016] The control unit 10 provides overall control over all parts and functions of the search server 2. It is composed of a computer such as a CPU (Central Processing Unit) and is connected to the storage unit 11 and the communication unit 12. The storage unit 11 is composed of memory such as ROM (Read Only Memory) and RAM (Random Access Memory), and recording media such as a hard disk, and stores programs and data for controlling all parts and functions of the search server 2. The communication unit 12 is the interface for the search server 2 to connect to the network 5, that is, it connects the search server 2 to the terminal device 3 via the network 5.

[0017] The control unit 10 controls various components and functions of the search server 2 by executing calculations based on programs and data stored in the storage unit 11. For example, the control unit 10 operates as the video input unit 20, video data processing unit 21, sound data processing unit 22, search model creation unit 23, scene search unit 24, search model editing unit 25, behavior analysis unit 26, restructuring unit 27, and tagging unit 28 by executing programs stored in the storage unit 11. The video input unit 20, video data processing unit 21, sound data processing unit 22, search model creation unit 23, scene search unit 24, search model editing unit 25, behavior analysis unit 26, restructuring unit 27, and tagging unit 28 realize the video input process, video data processing process, sound data processing process, search model creation process, scene search process, search model editing process, behavior analysis process, restructuring process, and tagging process of the video scene search method according to the present invention.

[0018] The video input unit 20 inputs the target video to be searched for scenes for creating a search model that aids in the search for each scene. At this time, the video input unit 20 separates and acquires video data and audio data from the target video and stores them in the storage unit 11.

[0019] For example, the video input unit 20 inputs the video transmitted or transferred (uploaded) from the terminal device 3 as the target video. Alternatively, the video input unit 20 inputs the video stored in the storage unit 11 or the database 15 as the target video. Or, the video input unit 20 inputs the video published via the network 5 such as the Internet as the target video.

[0020] The video data processing unit 21 performs image processing on the video data of the target video acquired by the video input unit 20, extracts scene images corresponding to each scene from the video data, and detects the subjects appearing in the target video from the scene images. At this time, the video data processing unit 2 may detect the subject from the scene image by artificial intelligence based on a learning model that has machine-learned images of various subjects. For example, the video data processing unit 21 detects people, animals, buildings, objects, characters, scenery, etc. as subjects from the scene image and discriminates the names of the detected subjects. Further, when the video data processing unit 21 detects a person as a subject from the scene image, it further detects and discriminates the face, hands and feet, clothing, etc. of the person, and also detects and discriminates the expression of the face, the pose and movement of the hands and feet and the body.

[0021] Based on the discrimination result of the subject detected from the scene image, the video data processing unit 21 creates a context indicating the subject as a character string by artificial intelligence applying an LLM via RAG, and stores the scene information including this context in the database 15. The video data processing unit 21 may create a context indicated by a character string for all the subjects appearing in the scene image. ​​For example, the video data processing unit 21 identifies the names of subjects such as people, objects (table, birthday cake, etc.), and animals (dogs, cats, etc.), and creates a string context indicating those names. The video data processing unit 21 may identify individual names, or it may distinguish between adults and children, or between men and women. Furthermore, the video data processing unit 21 creates a string context indicating the facial expressions (smiling, crying, etc.), actions (making a peace sign with hands, running, etc.), and state of the subjects.

[0023] The video data processing unit 21 extracts the timestamp from the target video of the scene image in which the subject was detected, and stores the scene information, which associates the timestamp of the scene image with the context created as described above, in the database 15.

[0024] Furthermore, the video data processing unit 21 performs structuring processing on this scene information to create structured data, and then vectorizes this structured data to create vector data, which is stored in the database 15. The video data processing unit 21 compares the previous vector data created from the previous scene information (scene information of the previous scene) with the current vector data created from the current scene information. If the cosine similarity is within a predetermined range and it can be determined that the detected subject content of both scene information is similar, it merges the previous structured data and the current structured data to create the current vector data. In this way, the video data processing unit 21 creates one structured data for a section in which scenes with similar detected subject content are consecutive, treating that section as one scene section.

[0025] The video data processing unit 21 sets the timestamps of the start and end points of the scene information that constitute a single scene section as the start and end points of the timestamp for that scene section. The video data processing unit 21 also associates the timestamps (start and end points) of that scene section with the context corresponding to the structured data of that scene section and stores them in the database 15 as scene information for that scene section.

[0026] The sound data processing unit 22 analyzes the sound data of the target video acquired by the video input unit 20 to detect scene sections in which similar sounds are consecutive, and detects the content of the sounds in each scene section. At this time, the sound data processing unit 22 may use artificial intelligence to detect scene sections from the sound data and detect the content of the sounds in each scene section, based on a learning model that has been trained on various sounds such as human voices (speech) and other sounds.

[0027] The sound data processing unit 22 creates a context that represents the pronunciation content as a string based on the determination result of the pronunciation content detected from the scene section, and stores the scene information including this context in the database 15. The sound data processing unit 22 should create a context that represents the pronunciation content as a string for all pronunciation content that appears in the scene section.

[0028] Specifically, the sound data processing unit 22 distinguishes and extracts human voices (speech) and other sounds from the sound data. The sound data processing unit 22 may, for example, distinguish between speech and other sounds by performing frequency analysis on the sound data, or it may distinguish between speech and other sounds using artificial intelligence based on a machine learning model that distinguishes between speech and other sounds.

[0029] Regarding human voices (speech), the sound data processing unit 22 extracts the speech from the sound data for consecutive sections where the same person's voice is heard, as the sound for the scene section related to that person. If the sound data processing unit 22 detects the voices of multiple people in the sound data, it extracts the sound for the scene section related to each person. The sound data processing unit 22 may, for example, identify each person's voice by performing frequency analysis on the speech, or it may identify each person's voice using artificial intelligence based on a machine learning model that has acquired various people's voices through machine learning. The sound data processing unit 22 extracts the speech for each person's scene section from the sound data and creates a context by transcribing that speech into text strings as the content of each person's pronunciation.

[0030] For sounds other than human voices (speech), the sound data processing unit 22 identifies non-speech-like sound types, such as animal noises, thunder, applause, and music, and extracts sounds from the sound data in sections where sounds of the same sound type are consecutive, as sounds of the scene section related to that sound type. The sound data processing unit 22 may, for example, identify the sound type by frequency analysis of the sound of the sound type, or it may identify the sound type using artificial intelligence based on a learning model that has been trained on sounds of various sound types. The sound data processing unit 22 extracts sounds from each scene section of each sound type from the sound data and creates a context as the sound content of each sound type.

[0031] The audio data processing unit 22 extracts the timestamps (start and end points) of the target video for each scene section, associates the timestamps of each scene section with the context created for each scene section as described above, and stores them in the database 15 as scene information for each scene section.

[0032] The search model creation unit 23 creates a search model based on the target video acquired by the video input unit 20, using scene information including the context and timestamp of each subject processed by the video data processing unit 21, and scene information including the context and timestamp of each sound content processed by the sound data processing unit 22.

[0033] Specifically, the search model creation unit 23 creates structured data by structuring the scene information of a scene section in the target video, regardless of whether the scene information for that scene section was created from video data or audio data. The search model creation unit 23 also vectorizes the structured data based on each scene information of the target video and stores it in the database 15 as a search model for the scene sections of the target video that will be used by RAG in the LLM's artificial intelligence.

[0034] At this time, the search model creation unit 23 creates a search model by adding metadata and indexes to the vector data, which is a vectorized version of the structured data of each scene information of the target video, and saves it as a vector group file (vector store) related to the target video. The metadata records the viewing position (timestamp) and the number of views for each scene section of the target video. The metadata also records the number of times the target video has been enumerated in the video list by the scene search unit 24, and the number of times it has not been selected in the video list (enumeration count - view count).

[0035] The scene search unit 24 receives a scene search request from the terminal device 3, and in response to this scene search request, uses artificial intelligence to search for viewable scenes in the video and provides the viewable scenes to the terminal device 3 as a search result. In this embodiment in particular, the scene search unit 24 uses artificial intelligence that applies LLM, which utilizes a search model created by the search model creation unit 23 via RAG and a generation model for answering with viewable scenes in the video, to search for scenes in the video in response to a scene search request from the terminal device 3 and generate viewable scenes in the video as an answer to the scene search request.

[0036] Specifically, the scene search unit 24 searches for a search model corresponding to the subject and spoken content according to the search conditions indicated by the scene search request, and provides the scene section of the video corresponding to the searched search model as a viewing candidate. At this time, the scene search unit 24 vectorizes the scene search request via RAG to create vector data. The scene search unit 24 compares the vector data of the scene search request with the vector group files of each target video stored in the database 15, and lists a predetermined number of target videos that are determined to be close because their cosine similarity is within a predetermined range as video candidates.

[0037] The scene search unit 24 creates a video list of the enumerated video candidates, sorted in ascending order of cosine similarity (closer to 0), and provides, for example, a video search screen displaying the video list.

[0038] Furthermore, for each video candidate in the video list, the scene search unit 24 obtains a search model for each scene information from the vector group file, and uses RAG to determine whether each scene information satisfies the search conditions of the scene search request based on the search model for each scene information. For example, the scene search unit 24 determines that scene information (scene interval) satisfies the search conditions if the cosine similarity between the vector data of each search model (each scene information) and the vector data of the scene search request is within a predetermined range.

[0039] If the scene search unit 24 finds one or more scene pieces (scene segments) that satisfy the search criteria, it retrieves the timestamp (viewing position) of each scene segment from the metadata of one or more corresponding search models as the viewing start position and adds it to the metadata of the video candidate. On the other hand, if the scene search unit 24 finds no scene pieces (scene segments) that satisfy the search criteria, it retrieves the starting point of the video candidate as the viewing start position and adds it to the metadata of the video candidate.

[0040] The scene search unit 24 then displays the video search screen with the video list, along with the viewing start position (seek position) added to the metadata, as a viewing candidate.

[0041] Furthermore, the scene search unit 24 obtains the number of views for the scene information (scene section) of the viewing start position added to the video candidates from the metadata of the corresponding search model. The scene search unit 24 then adjusts the placement priority points of the video candidates according to the number of views for the scene section. For example, the more views a video receives, the higher its placement priority points are set, and the fewer views a video receives, the lower its placement priority points are set. The scene search unit 24 also sets higher placement priority points the closer the cosine similarity is to the scene search request, and lower placement priority points the further the cosine similarity is.

[0042] Then, the scene search unit 24 adjusts the ranking of each video candidate in the video list based on the ranking priority points.

[0043] The scene search unit 24 allows users to select any video candidate from the video list on the video search screen, and also allows users to select any starting position (viewing candidate) from the list of starting positions for each video candidate.

[0044] When a video candidate and a viewing start position are selected from the video list on the video search screen provided to the terminal device 3, the scene search unit 24 plays the video candidate from the beginning of the scene section at the viewing start position and provides it to the terminal device 3.

[0045] At this time, the scene search unit 24 stores the selection results of video candidates and viewing start positions (scene segments) in the video list in the database 15 as a user viewing log. For example, the scene search unit 24 obtains metadata of the search model of scene information corresponding to the selected viewing start position (scene segment) from the vector group file of target videos corresponding to the selected video candidates, and counts up the number of views of said metadata.

[0046] In this specification, when a video candidate and a starting position for viewing are selected in the video list in response to a scene search request, that scene search request will be referred to as a selected request.

[0047] Furthermore, in the video list corresponding to the scene search request, the scene search unit 24 adds the vector data of the scene search request (selected request) to the positive processing vector group file (vector store) in order to improve the suitability of the scene search request to the selected video candidate and viewing start position (scene section), and adds metadata related to the video candidate and viewing start position (viewing log) to the vector data.

[0048] On the other hand, the scene search unit 24, in the video list corresponding to the scene search request, adds vector data of the scene search request (selected request) to a negative processing vector group file (vector store) for video candidates and viewing start positions (scene sections) that were not selected, in order to reduce the suitability of the scene search request to the video candidates and viewing start positions. It also adds metadata related to the video candidates and viewing start positions (viewing logs) to the vector data.

[0049] The search model editing unit 25 accepts manual editing by the administrator for the search models of each scene section of the target video stored in the storage unit 11 by the search model creation unit 23. For example, the search model editing unit 25 accepts edits by the administrator regarding the context corresponding to the structured data of the search model. In addition, the search model editing unit 25 accepts additions and edits by the administrator regarding the metadata attached to the vector store of the search model.

[0050] The behavior analysis unit 26 analyzes the user's behavior based on the user's video scene search request to the scene search unit 24 and the user's viewing log of the video scene section provided in response to the scene search request. Based on the results of this behavior analysis, it restructures the structured data of the search model corresponding to the scene section, vectorizes it, and stores it again as a search model.

[0051] The behavioral analysis unit 26 compares the vector data of the scene search request input to the scene search unit 24 with the positive processing vector group file of scene search requests (selected requests) stored in the database 15, and enumerates a predetermined number of selected requests from the positive processing vector group file whose cosine similarity is within a predetermined range and can be determined to be close. The behavioral analysis unit 26 obtains video candidates and viewing start positions that are paired with the selected requests from the metadata attached to the vector data of the enumerated selected requests, and if the video candidates and viewing start positions are not present in the video list enumerated by the scene search unit 24, it adds the video candidates and viewing start positions to the video list.

[0052] Furthermore, the behavioral analysis unit 26 adjusts the inclusion priority points in the video list for the video candidate and the viewing start position based on the cosine similarity with the scene search request input to the scene search unit 24. For example, the closer the cosine similarity, the higher the inclusion priority points are set. Alternatively, the behavioral analysis unit 26 may obtain a search model for scene information corresponding to the video candidate and the viewing start position from a vector group file related to the target video, obtain the number of views from the metadata of the search model, and set higher inclusion priority points based on the weighted average value of the number of views.

[0053] Furthermore, the behavioral analysis unit 26 compares the vector data of the scene search request input to the scene search unit 24 with the negative processing vector group file of scene search requests (selected requests) stored in the database 15, and enumerates a predetermined number of selected requests from the negative processing vector group file that are determined to be close because their cosine similarity is within a predetermined range.

[0054] The behavioral analysis unit 26 obtains video candidates and viewing start positions that correspond to the selected requests from the metadata attached to the vector data of the enumerated selected requests. If the video candidates and viewing start positions exist in the video list enumerated by the scene search unit 24, it adjusts the listing priority points based on the cosine similarity with the scene search request input to the scene search unit 24. For example, the closer the cosine similarity, the lower the listing priority points are set.

[0055] Furthermore, the behavioral analysis unit 26 may obtain a search model for scene information corresponding to the video candidate and the viewing start position from a vector group file related to the target video, obtain the number of times it was not selected in the video list from the metadata of the search model, and set a lower listing priority point based on the weighted average value of the number of times it was not selected.

[0056] Then, after the behavioral analysis by the behavioral analysis unit 26, the scene search unit 24 adjusts the ranking of each video candidate based on the listing priority points, stores the resulting video list as a viewing log in the database 15, and provides it to the terminal device 3.

[0057] Based on the user behavior analysis results from the behavior analysis unit 26, the restructuring unit 27 restructures the structured data of the search model, which includes video candidates with adjusted placement priority points and scene information (scene intervals) corresponding to the viewing start position, vectorizes it, and re-stores it in the database 15 as a search model.

[0058] For example, in the video list corresponding to the scene search request, the restructuring unit 27 modifies (refuses) the vector data of the structured data so that the search model for the selected video candidate and scene information (scene interval) corresponding to the viewing start position becomes closer to the vector data of the scene search request.

[0059] On the other hand, the restructuring unit 27 modifies (refuses) the vector data of the structured data so that the cosine similarity with the vector data of the scene search request becomes greater for the search model of the video candidates that were not selected and the scene information (scene interval) corresponding to the viewing start position in the video list in response to the scene search request.

[0060] The tagging unit 28 extracts tag elements such as genre, category, participant, and video content from the context corresponding to the structured data of the search model stored in the database 15, tags the extracted results of each tag element, associates them with the search model, and stores them in the database 15. For example, the tagging unit 28 pre-determines the tag elements to be tagged and extracts the strings corresponding to the tag elements from the context.

[0061] The scene search unit 24 described above not only compares the vector data of the scene search request with the vector files of each target video stored in the database 15, but also compares the tag elements extracted from the context of the scene search request with the tag elements associated with each search model in the vector files, thereby listing more appropriate target videos as video candidates for the scene search request.

[0062] Terminal device 3 is a device for the user to perform search operations on video scenes, and is composed of, for example, a personal computer, smartphone, or tablet terminal. Terminal device 3 is composed of, for example, a control unit 30, a storage unit 31, a communication unit 32, a display unit 33, and an operation unit 34.

[0063] The control unit 30 provides overall control over all parts and functions of the terminal device 3. It is composed of a computer such as a CPU and is connected to the storage unit 31 and the communication unit 32. The storage unit 31 is composed of memory such as ROM or RAM and recording media such as a hard disk, and stores programs and data for controlling all parts and functions of the terminal device 3. The communication unit 32 is the interface for the terminal device 3 to connect to the network 5, that is, it connects the terminal device 3 to the search server 2 via the network 5.

[0064] In this embodiment, the terminal device 3 is equipped with a browser and displays a video search screen provided by the search server 2, which is connected via the network 5, on the display unit 33 using the browser. The terminal device 3 inputs search conditions for a desired video scene via the video search screen displayed on the display unit 33 by the browser, in response to the user's operation of the operation unit 34, and sends a scene search request indicating the search conditions to the search server 2.

[0065] Furthermore, the terminal device 3 displays a video search screen on the display unit 33, which includes a list of videos provided by the search server 2 in response to a scene search request. In addition, the terminal device 3 selects a video candidate and a viewing start position from the video list in response to the user's operation on the operation unit 34, and plays the video candidate from the start of the scene section of the viewing start position provided by the search server 2 according to the selection result.

[0066] Next, we will explain an example of how to create a search model using the video scene search system 1, referring to the flowchart in Figure 2.

[0067] First, the video input unit 20 in the search server 2 inputs the target video to be searched for (step S1). The video input unit 20 also separates and obtains the video data and audio data from the target video (step S2).

[0068] Then, the video data processing unit 21 extracts scene images corresponding to each scene from the video data (step S3). The video data processing unit 21 also detects and identifies the subject from each scene image and creates a context indicating the subject (step S4). Furthermore, the video data processing unit 21 stores the subject context in each scene image and the timestamp of each scene image as scene information in the database 15 (step S5).

[0069] Then, the sound data processing unit 22 detects scene sections corresponding to each scene from the sound data (step S6). The sound data processing unit 22 also detects the pronunciation content of each scene section and creates a context indicating the pronunciation content (step S7). Furthermore, the sound data processing unit 22 stores the context of the pronunciation content in each scene section and the timestamp of each scene image as scene information in the database 15 (step S8).

[0070] Then, the search model creation unit 23 performs structuring processing on the above scene information based on the video data and / or sound data to create structured data, vectorizes the structured data, and stores the vector data as a search model in the database 15 (step S9). At this time, the search model creation unit 23 saves the vector data of each scene information (each scene section) as a vector group file (vector store) related to the target video (step S10).

[0071] Next, we will explain an example of how the video scene search system 1 works, referring to the flowchart in Figure 3.

[0072] First, when search conditions are entered in terminal device 3, a scene search request indicating the search conditions is sent to search server 2 (step S11).

[0073] Then, in the search server 2, the scene search unit 24 vectorizes the scene search request via RAG to create vector data (step S12). The scene search unit 24 also compares the input scene search request vector data with the vector group file (vector store) of target videos stored in the database 15 and lists a predetermined number of target videos that are determined to be close because their cosine similarity is within a predetermined range as video candidates (step S13).

[0074] Furthermore, the scene search unit 24 processes each of the enumerated video candidates. At this time, for each video candidate, the scene search unit 24 obtains a search model for each scene information (each scene interval) from the vector group file and uses RAG to determine whether each scene information is close to the scene search request (step S14).

[0075] If each video candidate has a scene section that is close to the scene search request (Step S14: Yes), a timestamp is obtained from the search model metadata of that scene section and added to the video candidate's metadata as the viewing start position (viewing candidate) (Step S15).

[0076] On the other hand, if there is no scene section close to the scene search request for each video candidate (step S14: No), the starting point of the video candidate is added to the video candidate's metadata as the viewing start position (viewing candidate) (step S16).

[0077] Then, the scene search unit 24 displays the video list on the video search screen along with the viewing start position (viewing candidate) added to the metadata (step S17).

[0078] Next, we will explain an example of the operation when selecting a video candidate in the video list using the video scene search system 1, referring to the flowchart in Figure 4.

[0079] First, when a video candidate and viewing start position are selected from the video list displayed on the video search screen in terminal device 3, the selection results are sent to search server 2 (step S21).

[0080] Then, in the search server 2, the scene search unit 24 plays the selected video candidates from the starting point of the scene section of the selected viewing start position and provides them to the terminal device 3 (step S22).

[0081] At this time, the scene search unit 24 stores the video candidates and the selection results of the viewing start position (viewing candidate) in the video list in response to the scene search request in the database 15 as a viewing log associated with the scene search request (step S23).

[0082] Furthermore, the scene search unit 24 adds the vector data of the corresponding scene search request (selected request) to the positive processing vector group file for the selected video candidate and viewing start position (viewing candidate), saves it, and adds metadata of the video candidate and viewing start position to the vector data (step S24).

[0083] Meanwhile, the scene search unit 24 adds the vector data of the corresponding scene search request (selected request) to the negative processing vector group file for video candidates that were not selected (other video candidates and viewing start position), saves it, and adds metadata of the video candidate and the viewing start position to the vector data (step S25).

[0084] Next, referring to the flowchart in Figure 3, we will explain an example of the video scene search operation described above, in step S17 where a list of candidate videos is created, with the operation involving behavioral analysis, referring to the flowchart in Figure 5.

[0085] In this example, first, as described in steps S11 to S17 above, the search server 2 creates a list of candidate videos in response to the scene search request input from the terminal device 3.

[0086] Then, in the search server 2, the behavioral analysis unit 26 compares the input scene search request vector data with the positive processing vector file of selected requests stored in the database 15 and enumerates the selected requests that are similar to the input scene search request (step S31).

[0087] The behavioral analysis unit 26 retrieves video candidates that correspond to the selected request from the metadata of the vector data of the selected request, and determines whether the video candidate exists in the video list (step S32). If the video candidate corresponding to the selected request does not exist in the video list, the unit adds the video candidate to the video list (step S33).

[0088] Furthermore, the behavioral analysis unit 26 adjusts the placement priority points for the video candidates paired with the selected requests based on the comparison result between the vector data of the input scene search request and the vector data of the selected request (step S34).

[0089] Furthermore, the behavioral analysis unit 26 compares the input scene search request vector data with the negative processing vector file of selected requests stored in the database 15 to enumerate selected requests that are similar to the input scene search request (step S35).

[0090] The behavioral analysis unit 26 retrieves video candidates that correspond to the selected requests from the metadata of the vector data of the selected requests, and determines whether or not the video candidates exist in the video list (step S36).

[0091] If a video candidate that matches a selected request is present in the video list, the listing priority points for the video candidate that matches the selected request are adjusted based on the comparison result between the vector data of the entered scene search request and the vector data of the selected request (step S37).

[0092] Then, the scene search unit 24 provides the terminal device 3 with a list of videos whose ranking order has been adjusted based on the ranking priority points (step S38).

[0093] In this embodiment, as described above, the video scene search system 1 inputs a video to be searched, obtains video data and audio data from the video, detects a subject from the scene image of the video data corresponding to each scene in the video to assist in the scene search of the video, creates a context based on the subject of the scene image, creates scene information including the context and a timestamp of the scene section containing the scene image, obtains the sound content of the scene section of the audio data corresponding to each scene in the video, creates a context based on the sound content of the scene section, creates scene information based on the context and a timestamp of the scene section, creates structured data of the scene information of the scene section based on the video data or audio data, vectorizes the structured data and stores it as a scene section search model for Search Extension Generation (RAG) used in Large-Scale Language Models (LLMs).

[0094] In other words, the video scene search method of the present invention takes a video to be searched as input, obtains video data and audio data from the video, detects a subject from the scene image of the video data corresponding to each scene in the video in order to assist in the scene search of the video, creates a context based on the subject of the scene image, creates scene information including the context and a timestamp of the scene section containing the scene image, obtains the sound content of the scene section of the audio data corresponding to each scene in the video, creates a context based on the sound content of the scene section, creates scene information based on the context and a timestamp of the scene section, creates structured data of the scene information of the scene section based on the video data or audio data, vectorizes the structured data and stores it as a scene section search model for Search Extension Generation (RAG) used in Large-Scale Language Models (LLMs).

[0095] With this configuration, according to this embodiment, by analyzing the video and audio data of a video and creating a search model for each scene segment usable in Search Augmentation Generator (RAG), when performing scene searches on a video using artificial intelligence of a Large-Scale Language Model (LLM), hallucination can be avoided and more appropriate scenes can be found.

[0096] Furthermore, the video scene search system 1 of this embodiment transcribes the audio of the scene section of the sound data into a string of characters and obtains the string of characters as the content of the pronunciation.

[0097] With this configuration, according to this embodiment, it is possible to understand the content of a video not only from video data but also from audio data, and to create a search model for each scene section.

[0098] Furthermore, the video scene search system 1 of this embodiment edits the search model and / or adds and / or edits metadata to the search model in response to manual operation by the administrator.

[0099] With this configuration, according to this embodiment, a search model can be created for each scene segment of a video in accordance with the intentions of the administrator managing the video search.

[0100] Furthermore, the video scene search system 1 of this embodiment performs user behavior analysis based on a video scene search request from the user and the user's viewing log of the video scene section provided in response to the scene search request. Based on the results of this behavior analysis, it restructures the structured data of the search model corresponding to the scene section, vectorizes it, and re-stores it as a search model. At this time, the video scene search system 1 adjusts the posting priority of videos based on the results of the behavior analysis.

[0101] With this configuration, according to this embodiment, not only can video candidates and viewing start positions be searched and provided in response to a scene search request, but the search model can also be reconstructed according to the user's viewing status (operation status) for the provided video candidates and viewing start positions.

[0102] Furthermore, the video scene search system 1 of this embodiment extracts tag elements from structured data and performs tagging.

[0103] With this configuration, according to this embodiment, by classifying the search models for each scene segment of a video using tag elements, the search model can be retrieved more quickly and appropriately by identifying the scene search request using the tag elements.

[0104] The present invention may be modified as appropriate, provided that it does not contradict the gist or idea of ​​the invention as can be read from the claims and the specification as a whole, and video scene search systems and video scene search methods that involve such modifications are also included in the technical concept of the present invention. [Explanation of Symbols]

[0105] 1. Video Scene Search System 2 Search Server 3 Terminal devices 5 Network 10 Control Unit 11 Storage section 12 Communications Department 13 Display section 14 Control section 15 Databases 20 Video Input Section 21 Video Data Processing Unit 22 Audio Data Processing Unit 23 Search Model Creation Department 24 Scene Search Section 25 Search Model Editorial Department 26 Behavior Analysis Department 27 Restructuring part 28 Tagging Section 30 Control Unit 31 Storage section 32 Communications Department 33 Display section 34 Control section

Claims

1. Enter the video you want to search for scenes, From the aforementioned video, video data and audio data are obtained, To assist in scene searching of the aforementioned video, The system detects a subject from the scene image of the video data corresponding to each scene of the aforementioned video, creates a context based on the subject in the scene image, and creates scene information including the context and a timestamp of the scene section containing the scene image. The audio content of the scene section of the sound data corresponding to each scene of the aforementioned video is obtained, a context is created based on the audio content of the scene section, and scene information is created based on the context and the timestamp of the scene section. A video scene search system characterized by creating structured data of the scene information of the scene section based on the video data or the sound data, vectorizing the structured data, and storing it as a search model of the scene section for search extension generation used in a large-scale language model.

2. The video scene search system according to claim 1, characterized in that it transcribes the audio of the scene section of the sound data into a string of characters and obtains the string of characters as the content of the sound.

3. The video scene search system according to claim 1, characterized in that, with respect to the search model stored for the scene section of the video, the system performs editing of the search model and / or adds and / or edits metadata to the search model in response to manual operation by the administrator.

4. The video scene search system according to claim 1, characterized in that it performs user behavior analysis based on a user's request for a scene search of the video and the user's viewing log of the scene section of the video provided in response to the scene search request, and based on the results of the behavior analysis, restructures the structured data of the search model corresponding to the scene section, vectorizes it, and stores it again as the search model.

5. The video scene search system according to claim 4, characterized in that it adjusts the priority of posting the videos based on the results of the behavioral analysis.

6. The video scene search system according to claim 1, characterized by extracting tag elements from the aforementioned structured data and tagging them.

7. Enter the video you want to search for scenes, From the aforementioned video, video data and audio data are obtained, To assist in scene searching of the aforementioned video, The system detects a subject from the scene image of the video data corresponding to each scene of the aforementioned video, creates a context based on the subject in the scene image, and creates scene information including the context and a timestamp of the scene section containing the scene image. The audio content of the scene section of the sound data corresponding to each scene of the aforementioned video is obtained, a context is created based on the audio content of the scene section, and scene information is created based on the context and the timestamp of the scene section. A video scene search method characterized by creating structured data of the scene information of the scene section based on the video data or the sound data, vectorizing the structured data, and storing it as a search model of the scene section for search extension generation used in a large-scale language model.

Images