Video retrieval method and electronic equipment

Abstract

The invention provides a video retrieval method, electronic equipment, a computer program product and a computer readable storage medium. The video retrieval method comprises the following steps: identifying content feature information of each video in a video set; constructing a knowledge base of the video set based on the content feature information of each video, wherein the knowledge base comprises association between the content feature information of each video in the video set; in response to a received user query, searching in the knowledge base to identify at least one video or video clip matched with the user query in the video set; and generating a natural language answer fusing the identified content feature information of the at least one video or video segment.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to the technical field of video processing, and more particularly, to a video retrieval method, an electronic device, a computer readable storage medium and a computer program product. BACKGROUND

[0002] The existing video retrieval method generally has two defects of content island and semantic discontinuity. The content island refers to that the existing video retrieval method processes the video content as isolated information units from each other, only extracts the label of a single video independently without establishing the association between videos, resulting in that the user cannot obtain complete information through one retrieval. The semantic discontinuity refers to that the existing video retrieval method only realizes the basic video classification, cannot meet the natural interaction demand of the user to directly obtain the answer, and often needs the user to manually filter the video content. SUMMARY

[0003] An aspect of the present disclosure provides a video retrieval method, comprising: identifying content feature information of each video in a video set; constructing a knowledge base of the video set based on the content feature information of each video, the knowledge base comprising associations between the content feature information of each video in the video set; in response to receiving a user query, searching in the knowledge base to identify at least one video or video segment in the video set matching the user query; and generating a natural language answer fusing the content feature information of the identified at least one video or video segment.

[0004] Optionally, the knowledge base comprises at least one of a knowledge graph, a vector database, and a hybrid vector retrieval database.

[0005] Optionally, the searching in the knowledge base in response to receiving the user query to identify at least one video or video segment in the video set matching the user query comprises: in response to receiving the user query, extracting at least one key semantic information in the user query; and searching in the knowledge base according to the at least one key semantic information to identify at least one video or video segment in the video set matching the at least one key semantic information.

[0006] Optionally, the key semantic information comprises a keyword and / or a semantic vector.

[0007] Optionally, the generating a natural language answer fusing the content feature information of the identified at least one video or video segment comprises: utilizing a trained machine learning model, generating the natural language answer based on the content feature information of the identified at least one video or video segment and the at least one key semantic information.

[0008] Optionally, the natural language answer is generated based on a portion of the knowledge base corresponding to the at least one video or video clip.

[0009] Optionally, the natural language answer is generated based on an association between the content feature information of the at least one video or video clip and the content feature information of each video in the video set in the knowledge base.

[0010] Optionally, the knowledge base comprises a knowledge graph, and constructing the knowledge graph comprises: separately analyzing the content feature information of each video in the video set to separately identify entities, attributes of the entities, and relationships between the entities in the each video; performing fusion analysis on the content feature information of each video in the video set to identify an association between the entities, the attributes, and the relationships in the each video; and constructing a knowledge graph of the video set based on the identified entities, attributes, and relationships and the association between the entities, the attributes, and the relationships in the each video.

[0011] Optionally, identifying the association between the entities, the attributes, and the relationships in the each video comprises identifying at least one of: a common entity of two or more videos in the video set; a common attribute of two or more videos in the video set; a common relationship of two or more videos in the video set; and a relationship between event-type entities in entities of two or more videos in the video set.

[0012] Optionally, the relationship between the event-type entities comprises at least one of: a temporal relationship, a causal relationship, a spatial relationship, and a classification relationship.

[0013] Optionally, extracting the at least one key semantic information in the user query comprises: parsing the user query into user query data comprising at least one modality, the at least one modality comprising at least one of a text modality, an audio modality, or a visual modality; and extracting key semantic information for each modality of the user query data respectively to obtain the at least one key semantic information.

[0014] Optionally, extracting key semantic information for the text modality of the user query data comprises: normalizing the text modality of the user query data; removing redundant text in the text modality of the user query data; and extracting key semantic information in the text modality of the user query data.

[0015] Optionally, the at least one key semantic information comprises action key semantic information, object key semantic information, and constraint condition key semantic information.

[0016] Optionally, the knowledge base comprises a knowledge graph, and identifying the at least one video or video clip matching the at least one key semantic information comprises: traversing nodes and edges of the knowledge graph to query nodes and / or edges matching all of the at least one key semantic information by starting from nodes or edges in the knowledge graph matching any of the at least one key semantic information, to identify the at least one video or video clip.

[0017] Optionally, the video retrieval method further comprises: outputting the natural language answer; or outputting the natural language answer with a link of the identified at least one video or video clip embedded.

[0018] Optionally, the video retrieval method further comprises: updating the knowledge base of the video set in response to an update of the video set.

[0019] Optionally, the content feature information comprises at least one of: information indicating an object in a video; information indicating an event in a video; information indicating a time in a video; information indicating a label in a video; information indicating a semantic in a video; and multi-modal information of a video.

[0020] Another aspect of the present disclosure provides an electronic device comprising: at least one memory having instructions stored thereon; and at least one processor coupled to the at least one memory, which, when executing the instructions, causes the video retrieval method as described above to be performed.

[0021] Still another aspect of the present disclosure provides a computer program product having instructions stored thereon, which, when executed by a processor, causes the video retrieval method as described above to be performed.

[0022] Yet another aspect of the present disclosure provides a computer-readable storage medium having instructions stored thereon, which, when executed by a processor, causes the video retrieval method as described above to be performed.

[0023] The video retrieval method provided by the present disclosure is advantageous in improving the speed and accuracy of retrieval by constructing a knowledge base of a video set and searching in the knowledge base for retrieval of videos or video clips. Moreover, since the knowledge base defines the association between the content feature information of each video in the video set, it is advantageous to query videos or video clips that cannot be directly matched by traditional retrieval. Furthermore, the video retrieval method can also generate a natural language answer that integrates the retrieved videos or video clips, which is advantageous in meeting the natural interaction needs of users and improving user experience. BRIEF DESCRIPTION OF DRAWINGS

[0024] In order to more clearly illustrate the technical solutions of the embodiments of the present disclosure, the drawings of the embodiments will be briefly introduced below. Obviously, the drawings described below only relate to some of the embodiments of the present disclosure and not limit the present disclosure.

[0025] Figure 1 An exemplary application scenario of the video retrieval method according to the embodiments of the present disclosure is shown.

[0026] Figure 2 A schematic flowchart of the video retrieval method according to the embodiments of the present disclosure is shown.

[0027] Figure 3 An exemplary knowledge graph according to the embodiments of the present disclosure is shown; and

[0028] Figure 4 A schematic block diagram of an electronic device according to the embodiments of the present disclosure is shown. DETAILED DESCRIPTION

[0029] The embodiments of the present disclosure will be described in more detail below with reference to the drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure should not be construed as being limited to the embodiments set forth herein, but rather, the embodiments are provided to make the present disclosure more thorough and complete. It should be understood that the drawings and embodiments of the present disclosure are only for exemplary purposes and are not intended to limit the scope of protection of the present disclosure.

[0030] Figure 1 An exemplary application scenario of the video retrieval method according to the embodiments of the present disclosure is shown.

[0031] Reference Figure 1 , the exemplary application scenario 100 includes video capture devices 110, 120 and 130, a network 140 and a computing device 150. The video capture devices 110, 120 and 130 can be, for example, a camera, a video camera, a mobile phone, a tablet computer or any other device with video capture function. The network 140 can provide a communication connection between the video capture devices 110, 120 and 130 and the computing device 150 through a wired communication link, a wireless communication link or an optical fiber cable. The computing device 150 can include any device implemented in the form of hardware, software or a combination thereof, which can process and output the processing result of the video captured by the video capture devices 110, 120 and 130. The video retrieval method according to the embodiments of the present disclosure can be implemented in the video capture devices 110, 120 or 130, or in the computing device 150. It should be understood that, Figure 1 The number of video capture devices, networks and computing devices in the exemplary application scenario 100 is only exemplary. According to the implementation needs, there can be any number of video capture devices, networks and computing devices.

[0032] The video retrieval method according to embodiments of this disclosure, described below, can be implemented in video capture devices 110, 120, 130 and / or computing device 150. For example, when program code, computer-readable storage media, or computer program products corresponding to the video retrieval method are deployed in video capture devices 110, 120, 130, the video capture devices 110, 120, 130 can implement the video retrieval method. When no program code, computer-readable storage media, or computer program products corresponding to the video retrieval method are deployed in video capture devices 110, 120, 130 but are deployed in computing device 150, the video capture devices 110, 120, 130 can transmit the captured video to computing device 150 via network 140, and computing device 150 can implement the video retrieval method.

[0033] Figure 2 A schematic flowchart of a video retrieval method according to an embodiment of the present disclosure is shown.

[0034] refer to Figure 2 The video retrieval method 200 according to embodiments of this disclosure may include steps 210 to 250, and, as shown by the dashed box in the figure, may optionally include step 260. Before starting to execute the video retrieval method 200, the implementer of the method may obtain videos from its internal, external, or internal and external sources. For example, in Figure 1 In the example, the processors in video capture devices 110, 120, and 130 can acquire a video set from their internal capture components. Computing device 150 can acquire a video set from video capture devices 110, 120, and / or 130.

[0035] In step 210, the content feature information of each video in the video set is identified. The content feature information may include structured and unstructured data obtained from the videos, where unstructured data may include, but is not limited to, vector data, tag data (obtained through techniques such as semantic generalization), etc. The content feature information may include, but is not limited to, information indicating various elements present in the video, such as information indicating objects in the video (e.g., people, animals, objects involved in the video), information indicating events in the video (e.g., changes in the actions and states of people or animals involved in the video, changes in the states of objects, etc.), information indicating time in the video (e.g., video duration, frame rate, keyframes, start and end times of each event in the video, text contained in the video, times mentioned in audio, etc.), and information indicating identifiers in the video (e.g., video title, ID, event ID, information about the video recording device, video recording...). Author information, video copyright information, etc.), semantic information in the video (e.g., action category, state category, emotion, etc.), multimodal information of the video (audio modality (e.g., speech content, speaker information, background music, ambient sound, sound effects, volume, pitch, rhythm, emotional tendency, etc.), text modality (e.g., subtitle text, OCR-recognized on-screen text, text descriptions inferred from speech / visual, etc.), visual modality (e.g., resolution, frame rate, brightness, color, shot segmentation, keyframes, scenes such as indoor and outdoor, detection and classification of animals, people and objects, action recognition, motion trajectory, pose estimation, facial expressions and emotions, etc.)).

[0036] In step 220, a knowledge base for the video set is constructed based on the content feature information of each identified video. This knowledge base includes the relationships between the content feature information of each video in the video set. Constructing this knowledge base can include building a knowledge graph, vector database, hybrid vector retrieval database, or other forms of knowledge representation for the video set. A knowledge graph is a semantic network describing the objective world using a graph structure composed of nodes and edges. Essentially, it describes entities, entity attributes, and relationships between entities. Nodes in a knowledge graph represent entities, edges between nodes represent relationships between entities, and each node or edge can include information representing its attributes. A vector database describes the objective world using a vector space. The essence of a vector database for a video set is to transform video data into vectors. The metadata attached to each vector represents the original information of the entity, and the similarity between entities can be measured by calculating the distance between vectors. A hybrid vector retrieval database describes the objective world using both vector space and symbolic indexes. The essence of a hybrid vector retrieval database for video collections is to transform video data into composite index records. Each record contains a semantic vector component and a textual feature component. The metadata associated with each component collectively represents the complete original information of the entity. By calculating the weighted fusion of semantic similarity between vectors and matching scores between texts, the comprehensive relevance between entities can be measured. Taking the construction of a knowledge graph for a video collection as an example, the content feature information of each video identified in step 210 can be analyzed separately to identify the entities, relationships, and attributes in each video. Then, the associations between the entities, attributes, and relationships in each video can be identified through fusion analysis of the content feature information of each video. Finally, a knowledge graph can be formed based on these identified entities, relationships, and attributes, as well as the associations between the entities, attributes, and relationships in each video. Identifying the relationships between entities, attributes, and connections in various videos can include, for example, identifying common entities in two or more videos within a video collection; common attributes in two or more videos within a video collection; common relationships in two or more videos within a video collection; and relationships between event-type entities in two or more videos within a video collection, such as temporal relationships (the chronological order of events), causal relationships (e.g., whether events occur in the same space), and classification relationships (whether events belong to the same category). For ease of understanding, the following will combine... Figure 3 An example knowledge graph will be used to illustrate this.

[0037] Figure 3 An exemplary knowledge graph is shown according to an embodiment of this disclosure.

[0038] For example, Figure 1The video capture devices 110, 120, and 130 are deployed in the living room, kitchen, and balcony of the user's residence, respectively, and capture videos V1, V2, and V3. These videos are transmitted to the computing device 150 via the network 140, and the computing device 150 stores these videos in its memory, forming a video set V={V1,V2,V3}.

[0039] For example, video V1, captured by video capture device 110 deployed in the living room, includes the following scenes: from 10:00:00 to 10:01:59, the orange cat plays with a cat toy on the sofa; from 10:02:00 to 10:03:59, the orange cat sits next to a bookshelf licking its fur, next to a potted plant. Video V2, captured by video capture device 120 deployed in the kitchen, includes the following scenes: from 10:04:00 to 10:04:59, the orange cat plays with a ball next to the refrigerator in the kitchen; from 10:05:00 to 10:05:59, the owner pets the orange cat's head. Video V3, captured by video capture device 130 deployed on the balcony, includes the following scene: from 10:06:00 to 10:09:59, the orange cat eats cat food from a food bowl on the balcony.

[0040] For example, based on the content features of video V1, such as its identifier information, the entity "Video V1" and its time attribute "10:00:00~10:03:59" and location attribute "living room" can be extracted, thus forming the node "Video V1" and its associated attribute information. Based on the content features of video V1, such as the information of the indicated objects and / or visual modal information, entities in video V1, including an orange cat, a cat stick, a sofa, a bookshelf, and a potted plant, can be extracted, thus forming... Figure 3 Nodes N1 to N5 in the video. Based on the content feature information of video V1, such as visual modality information, audio modality information, event information, and time information, the relationship between the orange cat and the cat stick, including the action relationship "orange cat", can also be extracted. play The text mentions a "cat stick" and the corresponding time is 10:00:00~10:01:59. It also mentions the relationship between the orange cat and the sofa, including their positional relationship. lie in The text mentions a "sofa" and a time period of 10:00:00~10:01:59, and the relationship between the orange cat and the bookshelf, including their positional relationship. The phrase "orange cat" appears to be unrelated and likely refers to a separate topic. Located in / squat The bookshelf is mentioned, and the corresponding time is 10:02:00~10:03:59. The relationship between the bookshelf and the pothos plant includes their positional relationship. Located next to The text describes a series of data points related to a bookshelf, including: edge E1 with its associated action attribute "playing" and time attribute "10:00:00~10:01:59"; edge E2 with its associated position attribute "located at" and time attribute "10:00:00~10:01:59"; edge E3 with its associated position attribute "located at / crouching" and time attribute "10:02:00~10:03:59"; and edge E4 with its associated position attribute "located next to". This information is based on the content features of video V1, such as events within it. Information (including but not limited to the participants in the event, the spatial location of the event, the time of the event, etc.) can also form node N11 "orange cat playing with a cat toy" representing the event entity and its time attribute "10:00:00~10:01:59" and location attribute "living room sofa", node N12 "orange cat licking its fur" representing the event entity and its time attribute "10:02:00~10:03:59" and location attribute "living room bookshelf", as well as edges E18 and E19 indicating that the orange cat is a participant in these events.

[0041] Similarly, based on the content feature information of video V2, the entity "Video V2" and its time attribute "10:04:00~10:05:59" and location attribute "kitchen" can be extracted, thus forming the node "Video V2" and its attached attributes. Entities extracted from video V2 also include the orange cat, the ball, the refrigerator, and the owner, thus forming nodes N6 to N8. The relationship between the orange cat and the ball can also be extracted, including the action relationship "orange cat...". play The text appears to be a mix of unrelated phrases and sentences, making it difficult to translate coherently. It includes fragments about a "ball" (referring to a time period between 10:04:00 and 10:04:59), the relationship between an orange cat and a refrigerator (including their location), and a reference to an orange cat. lie in The refrigerator is mentioned, and the corresponding time is 10:04:00~10:04:59. The relationship between the orange cat and its owner includes their interaction. patting head The orange cat and the corresponding time is 10:05:00~10:05:59, from which a pattern can be formed. Figure 3 Edges E8 to E13 and their associated attributes are defined in the video. Based on the content feature information of video V2, such as the event information therein, nodes N13 "Orange cat playing with a ball" representing event-type entities, along with their time attribute "10:04:00~10:04:59" and location attribute "kitchen refrigerator", nodes N14 "Orange cat being petted on the head by its owner" representing event-type entities, along with their time attribute "10:05:00~10:05:59" and location attribute "kitchen refrigerator", and edges E20 and E21 indicating that the orange cat is a participant in these events.

[0042] Similarly, based on the content feature information of video V3, the entity "Video V3" and its time attribute "10:06:00~10:09:59" and location attribute "balcony" can be extracted, thus forming the node "Video V3" and its associated attributes. Entities in video V3 can also be extracted, including the orange cat, cat food, and food bowl, thus forming nodes N9 to N10. The relationship between the orange cat and cat food can also be extracted, including the action relationship "orange cat eats cat food" with a corresponding time of "10:06:00 to 10:09:59," and the relationship between cat food and the food bowl includes the functional relationship "the food bowl holds cat food," thus forming edges E14 and E15 and their associated attributes. Based on the content feature information of video V3, such as the event information within it, the node N15 "orange cat eats cat food," representing an event-type entity, along with its time attribute "10:06:00~10:09:59" and location attribute "balcony," and the edge E22 indicating that the orange cat is a participant in these events, can also be formed.

[0043] In addition, edges E5 to E7, E11 to E13, and E16 and E17 can be formed based on the attribution relationship between entities and videos (for the sake of simplicity, the edges representing the attribution relationship between event nodes N11 to N12 and video V1, the attribution relationship between N13 to N14 and video V2, and the attribution relationship between N15 and video V3 are not shown in the figure).

[0044] Furthermore, based on the fusion analysis of the content feature information of videos V1, V2, and V3, the relationships between entities, attributes, and relationships in videos V1, V2, and V3 can be further identified. For example, the common entity among these three can be identified as "orange cat," so node N1 can be used as the central node; the temporal attributes of nodes N11 to N15 can be used to identify that the events represented by nodes N11 to N15 occur sequentially in time, thus forming edges E23 to E26 representing this temporal relationship; the spatial relationship between nodes N11 and N12 can be identified as both occurring in the living room based on the shared spatial attributes; and the classification relationship between nodes N11 and N13 can be identified as both belonging to the "orange cat playing" event category based on the fact that nodes N11 and N13 both belong to the play category, and so on.

[0045] It should be understood that the relationships between entities, attributes, and connections among the individual videos in a video collection depend on the specific content of the video. Figure 3Examples are not exhaustive. For instance, in another example, a camera located on a road in a certain district of a city captured video V4, which includes the following scene: At 08:42:00 on June 18, 2025, in heavy traffic and on a slippery road, a white sedan with license plate No.1 rear-ended a heavy truck, and the white sedan emitted thick smoke and flames; at 09:10:00, a rescue vehicle and an ambulance arrived, and subsequently, the rescue vehicle extinguished the fire in the white sedan, and paramedics carried the injured onto the ambulance; at 09:25:00, traffic police dispersed the crowd and directed the blocking of vehicles; at 09:30:00, the road returned to normal. A camera located at a rescue station in the same district captured video V5, which includes the following scene: From 08:45:00 to 09:00:00 on June 18, 2025, rescue personnel prepared fire extinguishing equipment in a rescue vehicle with license plate No.2, and then the rescue vehicle drove away from the rescue station. A camera at a hospital in the district captured video V6, which includes the following scene: from 08:45:00 to 08:48:00 on June 18, 2025, medical staff are loading medical equipment into an ambulance with license plate number No.3, after which the ambulance leaves the hospital. For this example, a knowledge graph of the video set V={V4, V5, V6} can be constructed using a method similar to that used in the aforementioned "orange cat" example. In this process, based on the separate analysis of the content feature information (such as event information and time information) of videos V4, V5, and V6, we can construct nodes representing event entities from video V4: M1 "car rear-ends collision with heavy truck", M2 "rescue personnel extinguish fire on car", M3 "medical personnel lift injured onto ambulance", and M4 "traffic police evacuate crowd and direct blocked vehicles"; nodes representing event entities from video V5: M5 "rescue personnel prepare fire extinguishing equipment" and M6 "rescue vehicle leaves rescue station"; and nodes representing event entities from video V6: M7 "medical personnel prepare rescue equipment" and M8 "ambulance leaves hospital". These nodes also include their time, location, and other attributes. Based on the time attributes of these nodes, the temporal relationship between them can be identified as M1, M5, M6, M7, M2, M3, and M4 occurring sequentially in time. Based on the fusion analysis of the content feature information (such as event information, time information and semantic information) of videos V4, V5 and V6, it can be identified that node M1 is classified as the cause of the accident, nodes M5, M6, M7, M2, M3 and M8 are classified as the rescue process, node M4 is classified as the follow-up processing, and it can also be identified that node M1 is the reason for nodes M2 to M8.

[0046] It should be understood that the specific content of the knowledge graph varies as the content of each video in the video collection changes, and this disclosure cannot exhaustively list them all. The foregoing examples are merely illustrative and not intended to limit this disclosure. (See references...) Figure 2In step 230, a user query can be received. In step 240, a search can be performed in the knowledge base to identify at least one video or video segment in the video collection that matches the user query.

[0047] For example, in response to receiving a user query, in step 240, at least one key semantic information can be extracted from the user query. This key semantic information may include keywords and semantic vectors. Keywords focus on the original characters or words appearing in the user query. Semantic vectors focus on understanding the user's deeper intent and are not limited to the original characters or words appearing in the user query. For example, for the user query "I want to find a high-end restaurant with a sea view and a romantic atmosphere," the extracted keywords may include, for example, "sea," "romantic," "high-end," and "restaurant," while the extracted semantic vectors may include, for example, "beach," "waterside," "sea view," "candlelight," "expensive," "Michelin," and "exquisite."

[0048] User queries can be in text, audio, image, video, or a combination thereof. To extract at least one key semantic information from a user query, multimodal techniques can be used to parse the user query into user query data including at least one modality. Then, key semantic information can be extracted from the user query data for each modality. At least one modality can include text, audio, visual, or a combination thereof.

[0049] For user query data in text form, such as text directly entered by the user, text transcribed from user-input speech, or text identified from user-input images or videos, the natural language text can first be normalized using techniques such as trie, double-array trie, distance dictionary, and thesaurus. For example, texts such as "orange cat," "kitten," and "cat" in the user query can be normalized to "orange cat." Then, techniques such as regular expression filtering and pre-built stop word lists can be used to remove redundant text from the natural language text. For example, for the user query "Please tell me about the cat's breakfast," the word "please" can be removed because it is merely a polite interjection with no practical value. Afterward, part-of-speech tagging, syntactic analysis, NER (Named Entity Recognition), semantic vector retrieval techniques, and / or other natural language processing techniques can be used to extract keywords or semantic vectors representing actions, objects, constraints, etc. For example, when a user queries "Please tell me about the cat's breakfast this morning", the system can identify keywords representing actions such as "eating" and semantic vectors such as "eating" and "dining", key semantic words representing objects such as "cat" and semantic vectors such as "pet" and "meow", and key semantic words representing constraints such as "morning" and semantic vectors such as "breakfast" and "this morning".

[0050] After extracting key semantic information from the user query, it can be searched in the knowledge base to identify at least one video or video segment in the video collection that matches at least one extracted key semantic information.

[0051] Taking searching in a knowledge graph as an example, the at least one video or video segment can be identified by starting from a node or edge in the knowledge graph that matches any of the key semantic information in at least one key semantic information, traversing the nodes and edges of the knowledge graph, and querying nodes and / or edges that match all the key semantic information in the at least one key semantic information.

[0052] For example, in one example, one could start by searching for nodes matching keywords from the entity layer of the knowledge graph, and then expand outwards along the edges representing the relationships from the found matching nodes. For example, using... Figure 3 Taking the knowledge graph shown as an example, for a user query "Please tell me about the cat's morning meal situation," we can first query the entity layer to find node N1 that matches the object keyword "cat." Then, by traversing all edges E1~E3, E8~E10, E14, and E18~E22 of node N1, we can query the edges that match the action keyword "eat" and the attributes that match the constraint keyword "morning." Thus, we can find edge E14, which represents the action relationship "eat" and has the time attribute "10:06:00 to 10:09:59." Next, we can also query the entity "cat food" (i.e., node N9) that has this action relationship "eat" with the entity "orange cat." Subsequently, by continuing to traverse all edges E15 and E16 connected to node N9, representing the node source relationship, we can find the "video V3" node. After that, after traversing the knowledge graph, we can determine that the target video that matches all keywords "orange cat," "eat," and "morning" includes video V3.

[0053] For example, in another example, one could start by searching for attributes matching keywords from the attribute layer of the knowledge graph, and then expand outwards along the edges of nodes possessing those attributes. For example, using... Figure 3Taking the knowledge graph shown as an example, for a user query "Please tell me about the cat's morning meal schedule", we can first query the attribute layer to find the time attributes "10:00:00 to 10:03:59", "10:04:00 to 10:05:59", and "10:06:00 to 10:09:59" that match the constraint keyword "morning". Then, starting from the nodes "Video V1", "Video V2", and "Video V3" with the corresponding attributes, we expand outward along their edges to find the node N1 that matches the object keyword "cat" and the edge "E14" that matches the action keyword. After traversing the knowledge graph, we can also determine that the target video that matches all keywords "orange cat", "eating", and "morning" includes video V3.

[0054] For example, in yet another example, one could start by querying edges matching keywords from the relational layer of the knowledge graph, then query the nodes connected to those edges, and expand outwards along other edges connecting those nodes. For example, with Figure 3 Taking the knowledge graph shown as an example, for a user query "Please tell me about the cat's breakfast," we can first find edge E14 matching "bread" in the relation layer. This edge also contains the time attribute "10:06:00 to 10:09:59," matching the constraint keyword "morning." Then, we find that node N1, "orange cat," connected to edge E14, matches the object keyword. After traversing the knowledge graph, we can also determine that the target video matching all keywords "orange cat," "bread," and "morning" includes video V3. Therefore, by searching the knowledge graph within the video set V={V1, V2, V4}, we can retrieve video V3, which matches the user query "Please tell me about the cat's breakfast."

[0055] In another example, the search results that match a user's query could also be video clips. For example, still using... Figure 3Taking the knowledge graph shown as an example, for the user query "Please tell me about the cat playing with toys this morning", key semantic information such as "cat", "meow", "pet", "play", "toy", "morning", "early morning", and "this morning" can be extracted. Then, in the entity layer, node N1 matching the object keyword "cat" can be found, and by traversing all edges of node N1, edges E1 and E8 matching the action keyword "play" can be found. Next, nodes N2 and N6, connected by edges E1 and E8, matching the object keyword "toy", can be found. The knowledge graph can be further traversed to find event-type entities N11 and N13 matching the keyword "play with toys". Thus, the search results matching this user query "Please tell me about the cat playing with toys this morning" can be identified as including video segments from 10:00:00 to 10:01:59 in video V1 and video segments from 10:04:00 to 10:04:59 in video V2.

[0056] In step 250, a natural language response that incorporates content feature information from at least one identified video or video segment can be generated.

[0057] In one example, a trained machine learning model can be used to generate a natural language answer based on content features of at least one identified video or video segment and at least one key semantic information extracted from the user query. For example, in the aforementioned example, after identifying search results matching the user query "Please tell me about the cat playing with toys this morning," including video segments V1 from 10:00:00 to 10:01:59 and video V2 from 10:04:00 to 10:04:59, the content features of these two video segments and the key semantic information extracted from the user query—"cat," "meow," "pet," "play," "toy," "morning," "early morning," "this morning"—can be input into the trained machine learning model. This machine learning model can, for example, have a feature fusion function, establishing semantic associations between the content features of the two video segments and these key semantic information through an attention mechanism, forming fused features. Then, the machine learning model can, for example, use an autoregressive generation method to generate a natural language answer word-by-word based on the fused features according to a predetermined probability model. An example of the natural language response is: "The orange cat played twice in the morning. It started at 10:00 AM, playing with a cat toy on the sofa in the living room for 1 minute. At 10:04 AM, it came to the kitchen and played with a ball next to the refrigerator for 1 minute. It was in good play condition and was healthy and energetic."

[0058] It should be understood that in step 240, the entire knowledge base, such as a collaborative knowledge graph, a vector database, and a hybrid vector retrieval database, can be used to find videos or video clips that match the user. For example, key semantic information extracted from the user query is matched with the knowledge graph to obtain structured retrieval results. The user query is converted into semantic vectors, and a similarity retrieval is performed in the vector database to obtain vector retrieval results. The vector retrieval results and structured retrieval results are input into the hybrid vector retrieval database, which then fuses and sorts the videos or video clips in the video collection to obtain the final retrieval results.

[0059] Reference Figure 2 Optionally, step 260 can also be performed to output the search results. For example, in Figure 1 The output is displayed in the display unit of the computing device 150 or on a terminal device networked with the video capture devices 110, 120, and 130. In this step, a natural language response may be output, or a natural language response with an embedded access link to at least one identified video or video segment may be output. Alternatively, a natural language response and other attributes of the identified at least one video or video segment, such as the videographer and video title, may be output. Alternatively, a natural language response and the identified at least one video or video segment itself may be output.

[0060] Thus, by constructing a knowledge base for the video collection, instead of extracting tags from individual videos, searching for videos or video frequency bands that match the user's query no longer involves searching the information of each video separately, avoiding content silos and improving retrieval speed and accuracy. Furthermore, because the knowledge base includes the correlation between the content feature information of each video in the video collection, inference and indirect matching are possible, allowing for the retrieval of videos that traditional retrieval methods cannot directly match. It also reduces ambiguous search results (for example, if a user wants to search for videos related to smartphones with the brand "Apple," simply matching the keyword "Apple" with the tags of individual videos without using video retrieval method 200 might result in videos related to the fruit "apple" but unrelated to smartphones). In addition, as another implementation of step 250, a natural language answer can be generated based on the portion of the knowledge base corresponding to at least one identified video or video segment. Taking a knowledge graph as an example, entities, relationships, and attributes can be searched in the portion of the knowledge graph of the video collection corresponding to at least one identified video or video segment, and then a natural language answer can be generated based on all or part of the retrieved entities, relationships, and attributes.

[0061] For example, in the aforementioned example, after identifying video segments from video V1 (times 10:00:00~10:01:59) and video V2 (times 10:04:00~10:04:59) that match the user query "Please tell me about the cat's playtime with toys this morning", the knowledge graph can be searched for entities, relations, and attributes corresponding to these video segments. Then, a natural language answer can be generated based on all or part of these entities, relations, and attributes.

[0062] As another implementation of step 250, a natural language response can also be generated based on the correlation between the content feature information of at least one identified video and the content feature information of each video in the video set defined in the knowledge base.

[0063] Since the knowledge base is generated based on content feature information, which contains information equivalent to or greater than that in the knowledge base itself, and the knowledge base contains the relationships between the content feature information of various videos—such as the temporal, causal, spatial, and classification relationships of event entities in a knowledge graph—it is possible to generate natural language answers that meet user needs using both of these elements. Using the previously described example of traffic accident-related videos, an example of a natural language response to a user query for "Please inform me of the traffic accident situation on a certain road section in a certain district of a certain city on June 18, 2025" would be: "At 8:42 AM on June 18, 2025, a white sedan with license plate number No.1 was rear-ended by a heavy truck in front of it. The white sedan emitted thick smoke and flames. A hospital in the district quickly dispatched an ambulance to the scene at 8:48 AM, and a rescue station in the district also quickly dispatched a rescue vehicle to the scene at 9:00 AM. The ambulance and rescue vehicle arrived at the scene simultaneously at 9:10 AM and quickly began extinguishing the fire on the vehicle and loading the injured into the ambulance. After the ambulance and rescue vehicle left the scene, traffic police began dispersing the crowd and directing traffic around the area at 9:15 AM, and the road traffic situation returned to normal by 9:30 AM."

[0064] Furthermore, the video retrieval method 200 may also include a step of updating the knowledge base of the video set in response to updates to the video set. For example, the knowledge base may be updated based on the content feature information of each video in the updated video set. For instance, when videos are added, deleted, or changed (e.g., videos are edited manually) in the video set, the content feature information of the newly added videos may be extracted, the content features of the deleted videos may be deleted, and the content feature information of the changed videos may be modified accordingly.

[0065] In this way, the video retrieval method 200 can not only retrieve the videos or video clips that match the user's query, but also generate natural language answers that integrate the content feature information of these videos and video clips, so that users can obtain the information they want more directly and avoid semantic gaps and manual video selection by users. Figure 4 A schematic block diagram of an electronic device according to an embodiment of the present disclosure is shown.

[0066] refer to Figure 4 An electronic device 400 according to an embodiment of this disclosure may include at least one processor 410 and at least one memory 420. Instructions are stored in the at least one memory 410. The at least one processor 410 is coupled to the at least one memory 420, and when the instructions are executed by the at least one processor 410, the electronic device 400 causes the electronic device 400 to perform the aforementioned video retrieval method 200. The electronic device 400 may be, for example, Figure 1 The video capture device 110, 120, 130 or computing device 150 shown are illustrated.

[0067] Examples of at least one processor 410 may include a microcontroller, a digital signal processor (DSP), a field-programmable gate array (FPGA), a programmable logic device (PLD), a state machine, gated logic, discrete hardware circuits, and other hardware circuits capable of performing instruction-level arithmetic, signal processing, or control functions. At least one processor 410 can execute software. Software should be broadly understood as instructions, instruction sets, code, code segments, program code, programs, subroutines, software modules, application programs, software applications, software packages, routines, subroutines, objects, executable files, threads of execution, procedures, functions, etc., whether referred to as software, firmware, middleware, microcode, hardware description languages, or other forms. The software may be stored in at least one memory 420.

[0068] At least one memory 420 may be a non-volatile computer-readable medium. For example, non-volatile computer-readable media include magnetic storage devices (such as hard disks, floppy disks, magnetic stripes), optical disks (such as optical discs (CDs) or digital versatile optical discs (DVDs)), smart cards, flash memory devices (such as cards, flash memory sticks, or USB flash drives), random access memory (RAM), read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, removable disks, and any other suitable medium that can be used to store software and / or instructions and can be accessed and read by a computer.

[0069] At least one processor 410 and at least one memory 420 can be connected via a bus for communicating information. The bus can consist of a single bus or multiple different buses.

[0070] Furthermore, an embodiment of this disclosure also provides a computer program product. This computer program product stores instructions that, when executed by a processor, cause one or more steps of the video retrieval method 200 described above to be performed. As an example, the computer program product includes a non-volatile computer-readable storage medium having instructions executable by a processor. For example, the computer program product can be deployed on... Figure 1 The video capture device 110, 120, 130 or the computing device 150 shown.

[0071] Furthermore, an embodiment of this disclosure also provides a computer-readable medium. This computer-readable medium stores instructions that, when executed by a processor, cause one or more steps of the video retrieval method 200 as described above to be performed. For example, this computer-readable medium can be deployed on... Figure 1 The video capture device 110, 120, 130 or the computing device 150 shown.

[0072] The embodiments of this disclosure have been described above with reference to the accompanying drawings. It should be understood that the above embodiments are merely illustrative, and those skilled in the art should understand that the combination of constituent elements and processes of this embodiment can be modified in various ways, and such modifications also fall within the scope of this disclosure.

Claims

1. A video retrieval method, comprising: Identify the content features of each video in the video collection; A knowledge base for the video set is constructed based on the content feature information of each video, and the knowledge base includes the association between the content feature information of each video in the video set; In response to receiving a user query, the system searches the knowledge base to identify at least one video or video segment in the video collection that matches the user query. as well as Generate a natural language response that integrates content feature information from at least one identified video or video segment.

2. The video retrieval method according to claim 1, wherein, The knowledge base includes at least one of the following: knowledge graph, vector database, and hybrid vector retrieval database.

3. The video retrieval method according to claim 1, wherein, In response to receiving a user query, searching in the knowledge base to identify at least one video or video segment in the video collection that matches the user query includes: In response to receiving a user query, at least one key semantic information is extracted from the user query; The knowledge base is searched based on the at least one key semantic information to identify at least one video or video segment in the video set that matches the at least one key semantic information.

4. The video retrieval method according to claim 3, wherein, The key semantic information includes keywords and / or semantic vectors.

5. The video retrieval method according to claim 3, wherein, Generating a natural language response that integrates content feature information from at least one identified video or video segment includes: The natural language response is generated using a trained machine learning model based on the content feature information and the key semantic information of at least one identified video or video segment.

6. The video retrieval method according to claim 1, wherein, The natural language response is generated based on the knowledge base and the portion corresponding to the at least one video or video segment.

7. The video retrieval method according to claim 1, wherein, The natural language response is generated based on the association between the content feature information of at least one identified video or video segment and the content feature information of each video in the video set in the knowledge base.

8. The video retrieval method according to claim 2, wherein, The knowledge base includes a knowledge graph, and constructing the knowledge graph includes: The content feature information of each video in the video collection is analyzed separately to identify entities in each video, the attributes of the entities, and the relationships between the entities. The content feature information of each video in the video collection is fused and analyzed to identify the associations between the entities, attributes, and relationships in each video; A knowledge graph of the video set is constructed based on the identified entities, attributes, and relationships, as well as the associations between the entities, attributes, and relationships in each video.

9. The video retrieval method according to claim 8, wherein, Identifying the relationships between the entities, attributes, and relations in the various videos includes identifying at least one of the following: Identify a common entity in two or more videos within the video set; Identify common attributes of two or more videos in the video set; Identify common relationships between two or more videos in the video set; Identify the relationships between event-class entities in the entities of two or more videos in the video collection.

10. The video retrieval method according to claim 9, wherein, The relationships between the event-type entities include at least one of the following: temporal relationship, causal relationship, spatial relationship, and classification relationship.

11. The video retrieval method according to claim 3, wherein, Extracting at least one key semantic information from the user query includes: The user query is parsed into user query data including at least one modality, wherein the at least one modality includes at least one of a text modality, an audio modality, or a visual modality; Extract key semantic information from user query data for each modality to obtain at least one key semantic information.

12. The video retrieval method according to claim 11, wherein, Extracting key semantic information from user query data in text modalities includes: The user query data of the text modality is standardized; Remove redundant text from the user query data of the text modality; and Extract key semantic information from the user query data of the text modality.

13. The video retrieval method according to claim 11, wherein, The at least one key semantic information includes action key semantic information, object key semantic information, and constraint key semantic information.

14. The video retrieval method according to claim 3, wherein, The knowledge base includes a knowledge graph, and identifying at least one video or video segment that matches the at least one key semantic information includes: The at least one video or video segment is identified by traversing the nodes and edges of the knowledge graph to query nodes and / or edges that match all key semantic information in the at least one key semantic information, starting from a node or edge that matches any of the key semantic information in the at least one key semantic information.

15. The video retrieval method according to claim 1, further comprising: Output the natural language response; or The output is a natural language response that embeds a link to at least one identified video or video segment.

16. The video retrieval method according to claim 1, further comprising: The knowledge base of the video set is updated in response to an update of the video set.

17. The video retrieval method according to claim 1, wherein, The content feature information includes at least one of the following: Information indicating objects in the video; Information indicating events in the video; Information indicating the time in the video; Indicates information from the markers in the video; Information indicating semantics in the video; and Multimodal information in video.

18. An electronic device comprising: At least one memory on which instructions are stored; At least one processor, coupled to the at least one memory, causes the video retrieval method of any one of claims 1-17 to be performed when the instructions are executed by the at least one processor.

19. A computer program product having instructions stored thereon, which, when executed by a processor, perform the video retrieval method according to any one of claims 1-17.

20. A computer-readable storage medium having instructions stored thereon, which, when executed by a processor, perform the video retrieval method according to any one of claims 1-17.

Images