A video question and answer method and device, electronic equipment and storage medium

Abstract

The application discloses a video question and answer method and device, electronic equipment and storage medium, and relates to the technical field of question and answer systems. The method comprises the following steps: acquiring natural language query information input by a user and queryer information corresponding to the user, and analyzing the natural language query information to obtain query text and query intention; determining a family scene entity and a timestamp interval according to the query text to determine a query object identifier; determining a queryer role and a queried person role based on a preset family knowledge graph, and performing a privacy authorization decision based on the queryer role, the queried person role, the family scene entity and the timestamp interval; when the privacy authorization decision is passed, determining a multi-dimensional query granularity according to the query intention, and determining a candidate video segment set according to the multi-dimensional query granularity, the preset knowledge graph and video segment metadata; determining original video data corresponding to the candidate video segment set, and generating a multi-modal evidence chain matching the natural language query information based on the candidate video segment and the original video data.

Description

Technical Field

[0001] This invention relates to the field of question-and-answer system technology, and in particular to a video question-and-answer method, apparatus, electronic device, and storage medium. Background Technology

[0002] With the widespread adoption of smart home cameras and whole-house monitoring devices, the storage of massive amounts of surveillance video, semantic querying, and privacy and security management in home scenarios have become essential industry requirements. Currently, the industry has three main existing technical solutions for video retrieval, knowledge-based question answering, and data access control, each with inherent shortcomings and unable to meet the comprehensive business needs of home video question answering.

[0003] Video question answering technology has been widely used in home security and smart home care in recent years. Users expect to quickly obtain specific event information from home videos through natural language queries, rather than browsing massive amounts of raw video data. However, existing general-purpose video question answering systems are mainly designed for open domain or service robot scenarios, making it difficult to directly adapt to the unique entity types, member relationships, and privacy protection needs of home scenarios. Therefore, there is an urgent need for a multimodal video question answering method for home scenarios to solve the current problems of home surveillance video querying, privacy protection, and multimodal evidence output. Summary of the Invention

[0004] This invention provides a video question-and-answer method, apparatus, electronic device, and storage medium to address the problem that existing video question-and-answer methods cannot directly adapt to the unique entity types, member relationships, and privacy protection needs of family scenarios.

[0005] According to one aspect of the present invention, a video question-answering method is provided, wherein the method includes: Obtain the natural language query information input by the user and the corresponding user's query information, and parse the natural language query information to obtain the query text and query intent; Based on the query text, determine the family scene entity and timestamp range, and determine the query object identifier according to the query intent and the family scene entity; Based on a preset family knowledge graph, the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier are determined, and a privacy authorization decision is performed based on the queryer role, the queryee role, the family scene entity, and the timestamp interval. When the privacy authorization decision is approved, multi-dimensional query granularity parsing is performed on the query intent to obtain multi-dimensional query granularity, and a candidate video segment set is determined according to the multi-dimensional query granularity, the preset knowledge graph and video segment metadata; Determine the original video data corresponding to the candidate video segment set, and generate a multimodal evidence chain that matches the natural language query information based on the candidate video segments and the original video data.

[0006] According to another aspect of the present invention, a video question-and-answer device is provided, wherein the device comprises: The data parsing module is used to obtain the natural language query information input by the user and the corresponding user's query information, and to parse the natural language query information to obtain the query text and query intent; The information query module is used to determine the family scene entity and timestamp range based on the query text, and to determine the query object identifier according to the query intent and the family scene entity; The authorization decision module is used to determine the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier based on a preset family knowledge graph, and to perform privacy authorization decisions based on the queryer role, the queryee role, the family scene entity and the timestamp interval. The video recognition module is used to perform multi-dimensional query granularity parsing on the query intent to obtain multi-dimensional query granularity when the privacy authorization decision is approved, and to determine a set of candidate video segments according to the multi-dimensional query granularity, the preset knowledge graph and video segment metadata; The question-and-answer feedback module is used to determine the original video data corresponding to the candidate video segment set, and to generate a multimodal evidence chain that matches the natural language query information based on the candidate video segments and the original video data.

[0007] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video question-answering method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video question-answering method according to any embodiment of the present invention.

[0009] According to another aspect of the present invention, embodiments of the present invention also provide a computer program product, the computer program product including a computer program, which, when executed by a processor, implements the video question-and-answer method of any embodiment of the present invention.

[0010] The technical solution of this invention obtains natural language query information input by the user and the corresponding queryer information of the user, parses the natural language query information to obtain query text and query intent, determines the family scene entity and timestamp interval based on the query text, determines the query object identifier according to the query intent and family scene entity, determines the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier based on a preset family knowledge graph, and performs privacy authorization decisions based on the queryer role, queryee role, family scene entity and timestamp interval to solve the risk of family video privacy leakage and adapt to the family differentiated permission system; when the privacy authorization decision is approved, multi-dimensional query granularity is obtained by parsing the query intent to obtain multi-dimensional query granularity, and a candidate video segment set is determined according to the multi-dimensional query granularity, the preset knowledge graph and video segment metadata, which greatly improves the matching accuracy of complex family queries. By determining the original video data corresponding to the candidate video segment set, a multimodal evidence chain matching natural language query information is generated based on the candidate video segments and the original video data, and deep retrieval and evidence generation process is initiated only for compliant queries. It effectively reduces unnecessary calculations and video parsing overhead, improving overall system efficiency while ensuring query accuracy, and is more suitable for lightweight deployment needs of home terminals.

[0011] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0012] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0013] Figure 1 This is a flowchart of a video question-and-answer method provided in Embodiment 1 of the present invention; Figure 2 This is a flowchart of a video question-and-answer method provided in Embodiment 2 of the present invention; Figure 3 This is a schematic diagram of the structure of a video question-and-answer device according to Embodiment 4 of the present invention; Figure 4 This is a schematic diagram of the structure of an electronic device that implements the video question-answering method of this invention. Detailed Implementation

[0014] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0015] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "including" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that includes a series of steps or units is not necessarily limited to those explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0016] Example 1 Figure 1 This is a flowchart of a video question-and-answer method according to Embodiment 1 of the present invention. This embodiment is applicable to situations where natural language queries are performed on surveillance videos to obtain query results. This method can be executed by a video question-and-answer device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the method includes: S110. Obtain the natural language query information input by the user and the queryer information of the corresponding user, and parse the natural language query information to obtain the query text and query intent.

[0017] Natural language query information can be understood as user-inputted questions about family videos in everyday spoken or written language. Generally, natural language queries have no fixed format and do not require structured editing; for example, a video of a child playing in the living room last night. In practice, natural language queries can be input via voice or text. Queriesr information refers to the identity data used to identify the user initiating the query. This can be obtained through at least one method, such as account reading, device identification, or multimodal biometrics (e.g., facial recognition, voiceprint recognition), and includes core identity identifiers such as user account identifier (ID), device ID, or biometric code. Query text refers to the structured text obtained after correcting and standardizing the original natural language query information, such as natural language query information after removing interjections, redundant words, and invalid punctuation. Query intent refers to the core category of the user's video query request, determined by the system. Generally, query intent can include categories such as video playback queries, event retrieval, people tracking queries, and item tracing queries.

[0018] In this embodiment, natural language query information from user terminals can be received in real time. Natural language query information supports both direct text input and voice input. For voice input, speech-to-text parsing is performed first to obtain the original query statement. The parsed original query statement is then preprocessed to obtain the query text. Simultaneously, the user's queryer information is determined through account reading, device identification, or multimodal biometrics, completing user identity binding. When the natural language query information is text input, the parsed original query statement can be directly preprocessed to obtain semantically complete and uniformly formatted query text. The standardized query text is then parsed using a pre-trained semantic classification model to identify the user's core query intent category, and the classification result is determined as the query intent for this query.

[0019] S120. Determine the family scene entity and timestamp range based on the query text, and determine the query object identifier according to the query intent and the family scene entity.

[0020] Here, "family scene entity" refers to family elements extracted from the query text. These can include family members (such as homeowners, children, visitors, and housekeepers), objects (such as furniture, appliances, and toys), events (such as playing, cooking, and resting), and locations (such as living rooms, bedrooms, bathrooms, and kitchens). "Time stamp range" refers to the standardized start and end time range obtained by matching and converting the time information from the query text. "Query object identifier" is a structured ID corresponding to the family scene entity associated with the user's query intent, used to accurately locate the target retrieval object for this query.

[0021] In this embodiment, entities such as family member entities, object entities, event entities, and location entities in the query text can be identified as family scene entities. In actual operation, a pre-trained entity recognition model or a large language model can be used for recognition. Next, time information in the query text is identified, including specific moments, time periods, periodic periods, or relative times. Pre-set time intervals are matched according to the time information to determine the corresponding time interval as a timestamp interval. Combined with the query intent, the extracted family scene entities are semantically filtered to determine the family scene entities to be retrieved corresponding to the natural language query information as query objects, generating query object identifiers for the target entities.

[0022] S130. Based on the preset family knowledge graph, determine the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier, and perform privacy authorization decisions based on the queryer role, the queryee role, the family scene entity and the timestamp interval.

[0023] The pre-built family knowledge graph refers to a pre-constructed, structured knowledge base specific to the family, used to store related knowledge such as family member roles, item ownership, spatial attributes, and scene sensitivity. It supports role matching, entity constraint mapping, and permission rule association. The queryer role refers to the identity level of the user initiating the query, such as owner, family member, and visitor; the queryee role is the entity role corresponding to the query object, such as family member, public property, private property, and private space. Privacy authorization decision refers to the process of comprehensively determining the user's query permissions based on multi-dimensional attributes. The privacy authorization decision result includes authorization granted or authorization denied.

[0024] In this embodiment, the queryer information can be used as a retrieval index to traverse the user role knowledge base in the preset family knowledge graph, matching the queryer role corresponding to the current query initiator, distinguishing identity levels such as homeowner, ordinary family member, and visitor; simultaneously, the object identifier is used as a retrieval keyword to retrieve the attribute information of the corresponding entity in the preset family knowledge graph, matching the query target's corresponding queryee role. The matched queryer role, queryee role, associated family scene entity, and standardized timestamp interval are then structured and fused to construct an intent feature vector. This constructed intent feature vector is matched according to pre-set authorization decision rules to determine the privacy authorization decision result. In actual operation, a comprehensive matching judgment can be made by combining multiple constraints such as user role permission level, query object privacy sensitivity, query time period compliance, and family scene privacy level, ultimately outputting a unique privacy authorization decision result, i.e., authorization approved or authorization denied. In one embodiment, the preset authorization decision rules may include: authorizing when the queryer is a parent; authorizing when the queryer is a child, the queried party is the child themselves, the query intent is a location query or a time query, and the current time period is daytime; denying authorization when the queryer is a child, the queried party is the child themselves, the query intent is a location query or a time query, and the current time period is nighttime; denying authorization when the queryer is a child and the queried party is a family member other than the child themselves; and denying authorization when the queryer is a visitor.

[0025] S140. When the privacy authorization decision is approved, perform multi-dimensional query granularity parsing on the query intent to obtain multi-dimensional query granularity, and determine the candidate video segment set according to the multi-dimensional query granularity, the preset knowledge graph and the video segment metadata.

[0026] Multi-dimensional query granularity refers to the retrieval dimensions obtained by deeply decomposing the query intent, including time-based query granularity, location-based query granularity, and object-based query granularity, used for filtering video clips. Video clip metadata is pre-stored video attribute data; for example, video clip metadata may include information such as the capture time, shooting location, device number, on-screen entity tags, duration, and storage path of a single video segment. The candidate video clip set refers to the collection of video clips that initially match the query requirements after being filtered using multi-dimensional granularity and preset knowledge graph constraints.

[0027] In this embodiment, provided the privacy authorization decision is approved, the user's query intent can be semantically decomposed into finer details. For example, multi-dimensional query granularities can be formed, such as time query granularity for limiting the search period, location query granularity for limiting the search space, and object query granularity for limiting the target content of the search. A preset family knowledge graph is invoked for association matching according to the multi-dimensional query granularity to obtain the family scene entity constraints that are suitable for the current multi-dimensional query granularity, clarifying the matching criteria for people, locations, objects, and events that this video search needs to meet. All video segment metadata pre-stored locally or in the cloud is traversed, and each video segment is verified to fully match all search conditions in combination with the multi-dimensional query granularity and the family scene entity constraints. All video segments that meet the conditions are then aggregated to obtain a candidate video segment set.

[0028] S150. Determine the original video data corresponding to the candidate video segment set, and generate a multimodal evidence chain that matches the natural language query information based on the candidate video segments and the original video data.

[0029] Here, raw video data refers to the complete raw video stream data corresponding to the candidate video segment. The raw video data can contain complete multimodal information such as frame images, audio, and subtitles. Multimodal evidence chain refers to a structured evidence set that integrates spatiotemporally aligned keyframe images, audio segments, audio text, spatial coordinates, and candidate video segments, which can be directly traced and intuitively matched with the user's natural language query information request.

[0030] In this embodiment, the original video data of each video segment in the candidate video segment set can be extracted. The acquired original video data is then analyzed and extracted from multiple dimensions to extract multimodal materials such as keyframe images, effective audio segments, and speech-to-text (STC) text. In actual operation, original text segments related to the query text can be extracted from the STC text of the original video data. Keyframes are extracted from the original video data, and audio segments related to the query text are extracted from the audio stream of the original video data. To address the temporal deviation and spatial mismatch issues caused by asynchronous acquisition from multiple cameras in a home environment, clock drift correction is performed on all multimodal materials corresponding to different cameras. The system standard timestamps of all materials are unified, and combined with the home spatial topology information, the acquisition spatial coordinates corresponding to each material are accurately matched to achieve spatiotemporal alignment of multi-source materials. The spatiotemporally aligned and corrected target multimodal materials, corresponding spatial coordinate information, and original video data are then structurally associated and integrated, and orderly spliced ​​according to the event occurrence sequence, spatial location, and target object dimensions to construct a complete and traceable multimodal evidence chain.

[0031] This invention, through its embodiments, acquires natural language query information input by the user and the corresponding queryer information. It then parses the natural language query information to obtain query text and query intent. Based on the query text, it determines the family scene entity and timestamp range. According to the query intent and family scene entity, it determines the query object identifier. Based on a preset family knowledge graph, it determines the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier. Based on the queryer role, queryee role, family scene entity, and timestamp range, it performs a privacy authorization decision, mitigating the risk of family video privacy leakage and adapting to differentiated family permission systems. When the privacy authorization decision is approved, it performs multi-dimensional query granularity parsing on the query intent to obtain multi-dimensional query granularity. Based on the multi-dimensional query granularity, the preset knowledge graph, and video segment metadata, it determines a candidate video segment set, significantly improving the matching accuracy of complex family queries. By determining the original video data corresponding to the candidate video segment set, it generates a multimodal evidence chain matching the natural language query information based on the candidate video segments and the original video data. Only compliant queries are subject to deep retrieval and evidence generation processes. It effectively reduces unnecessary calculations and video parsing overhead, improving overall system efficiency while ensuring query accuracy, and is more suitable for lightweight deployment needs of home terminals.

[0032] In one embodiment, after determining the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier based on a preset family knowledge graph, and performing a privacy authorization decision based on the queryer role, the queryee role, the family scene entity, and the timestamp interval, the method further includes: When the privacy authorization decision is to deny authorization, the reason for the denial is provided. In this embodiment, when the privacy authorization decision is determined to be to deny authorization, the query intent can be retrieved, and the queryer role, the queried party role, the family scene entity, and the timestamp interval on which the current permission determination was based can be traced back to locate the core triggering reason for the permission interception. Generally, a multi-dimensional authorization denial rule mapping library can be pre-stored to match the corresponding standardized denial reason for different violation dimensions. For example, if the queryer role is a visitor and there is no preset authorization from the owner, a feedback reason for denial is generated indicating that the current visitor identity does not have preset query permissions; if a family member queries a private family scene entity (bedroom, bathroom, etc.) and fails secondary verification, a feedback reason is generated indicating that the current query scene is a private family scene and requires additional identity verification to query; if a family member queries private video data corresponding to a non-family member, a feedback reason is generated indicating that the non-family member has no right to query the family private video data; if the query period exceeds the legal access time interval of the corresponding role, a feedback reason is generated indicating that the current query period exceeds the authorized access period and the query is denied, ensuring video privacy and security.

[0033] Example 2 Figure 2This is a flowchart of a video question-and-answer method according to Embodiment 2 of the present invention. This embodiment is a further optimization and extension based on the above embodiments, and can be combined with various optional technical solutions in the above embodiments. Figure 2 As shown, the method includes: S201. Receive the natural language query information input by the user and determine the user's query information.

[0034] The queryer information is obtained through at least one of the following methods: account reading, device identification, or multimodal biometrics.

[0035] In this embodiment, the system can monitor user queries via text or voice input in real time. For text input, the query information is directly obtained; for voice input, environmental noise is first removed using a speech denoising algorithm, and then a speech recognition model is used to convert the speech to text, outputting a standardized text version of the query information. Simultaneously, the system initiates a user identification process, which can collect the current user's identity data and generate query information through one or more combinations of methods, including account and password login verification, unique terminal device identifier recognition, and multimodal biometric recognition such as face or voiceprint.

[0036] S202. Convert the natural language query information into query text, identify the query intent category of the query text, and use the query intent category as the query intent.

[0037] In this embodiment, the natural language query information can be preprocessed to remove interfering information such as modal particles, invalid punctuation, repetitive and redundant sentences, and typos, while retaining the complete core semantics and generating a properly formatted query text. The query text is then input into a pre-trained semantic classification model to identify the user's query intent category. For example, query intent categories may include searches for people's activities, family events, item tracing, and time-segment video playback, and the identified query intent category is ultimately determined as the query intent.

[0038] S203. Identify family members, objects, events, and locations in the query text, and treat family members, objects, events, and locations as entities in the family scene.

[0039] In this embodiment, the query text can be semantically parsed and entity labeled word by word and sentence by sentence. All family members, objects, events, and locations contained in the query text are extracted, and the identified family members, objects, events, and locations are treated as family scene entities.

[0040] S204. Extract the time information from the query text and match the time information to obtain the corresponding timestamp range.

[0041] In this embodiment, the query text can be traversed to identify time information, which is then mapped to system start and end timestamps according to the system's unified time conversion rules, and concatenated to form a complete timestamp range. Alternatively, a preset time range corresponding to the time information can be determined, and this preset time range can be used as the timestamp range.

[0042] S205. Select the family scene entities associated with the query intent as query objects, and generate corresponding query object identifiers based on the query objects.

[0043] In this embodiment, the identified query intent can be used as a filtering criterion to perform semantic association matching on the extracted family scene entities, eliminating redundant entities irrelevant to the current query request, and retaining only entities that highly match the query intent as the query objects for this query. Based on the entity encoding rules in the preset family knowledge graph, the target query object is encoded to generate a query object identifier.

[0044] S206. Based on the queryer information, retrieve the preset family knowledge graph to obtain the queryer role corresponding to the queryer information; based on the query object identifier, retrieve the preset family knowledge graph to obtain the queryee role corresponding to the query object identifier.

[0045] Among them, the preset family knowledge graph refers to a pre-built family-specific structured knowledge base used to solidify exclusive knowledge such as family member role relationships, item ownership relationships, space privacy levels, scene attributes, and personnel permission rules, supporting role matching, entity constraint reasoning, and permission rule association.

[0046] In this embodiment, the collected queryer information can be used as a retrieval index to traverse a preset family knowledge graph and match the identity and permission level corresponding to the current query initiator user as the queryer role, distinguishing between three types of identities: owner, ordinary family member, and temporary visitor. Simultaneously, using the query object identifier as the retrieval keyword, the object identity attribute corresponding to the query target is matched in the preset family knowledge graph to determine the queried role.

[0047] S207. Construct an intent feature vector by combining the queryer role, the queryee role, the family scene entity, and the timestamp interval. Match the intent feature vector with the preset authorization decision rules to obtain the privacy authorization decision result.

[0048] The preset authorization decision rules refer to a multi-dimensional joint permission judgment rule library customized for family scenarios. Based on role hierarchy, scenario privacy level, query time period, and object attributes, it determines whether a query is compliant and outputs an authorization approval or denial result. For example, the preset authorization decision rules may include, but are not limited to: authorization approved when the queryer is a parent; authorization approved when the queryer is a child, the queried party is the child themselves, the query intent is location or time query, and the current time period is daytime; authorization denied when the queryer is a child, the queried party is the child themselves, the query intent is location or time query, and the current time period is nighttime; authorization denied when the queryer is a child and the queried party is a family member other than the child themselves; and authorization denied when the queryer is a visitor.

[0049] In this embodiment, the matched queryer role, queryee role, family scene entity, and timestamp interval can be structured and normalized to unify the data format and feature dimensions, resulting in an intent feature vector. The intent feature vector is then determined according to preset authorization decision rules to determine whether it meets privacy authorization conditions. If it does not meet the conditions, the privacy authorization decision is to deny authorization; if it does meet the conditions, the privacy authorization decision is to grant authorization.

[0050] S208. When the privacy authorization decision is approved, the query intent is decomposed into time query granularity, location query granularity and object query granularity, and the time query granularity, location query granularity and object query granularity are used as multi-dimensional query granularity.

[0051] In this embodiment, provided that the privacy authorization decision is approved, the query intent can be broken down into time-based query granularity (limited to the video retrieval period), location-based query granularity (limited to the video capture space), and object-based query granularity (limited to the target content of the video retrieval). These three granularities are then combined and bound together to form a multi-dimensional query granularity.

[0052] S209. Using multi-dimensional query granularity as the retrieval index, the corresponding family scene entity constraints are obtained by matching from the preset family knowledge graph.

[0053] Among them, the family scene entity constraint condition refers to the structured retrieval constraint obtained by reasoning and matching in the family knowledge graph based on multi-dimensional query granularity, which is used to limit the people, locations, objects and event standards that need to be matched in video retrieval.

[0054] In this embodiment, a multi-dimensional query granularity can be used as a retrieval index, which is then input into a preset family knowledge graph for matching. By combining the entity relationships, scene attributes, personnel affiliations, and spatial tags stored in the preset family knowledge graph, family scene entity constraints adapted to the multi-dimensional query granularity are mapped, clarifying the personnel matching criteria, spatial range criteria, and object event matching criteria that this video retrieval needs to meet, thus forming structured and executable retrieval constraint rules.

[0055] S210. Call the pre-stored video segment metadata, use multi-dimensional query granularity and scene entity constraints to filter the video segment metadata, and summarize the video segments that simultaneously meet the multi-dimensional query granularity and family scene entity constraints to generate a candidate video segment set.

[0056] In this embodiment, all video segment metadata pre-stored locally or in the cloud can be retrieved, and the structured attribute information of all video segments can be traversed. Each video segment metadata is validated using multi-dimensional query granularity and family scene entity constraints, filtering out video segments that simultaneously and completely satisfy both types of constraints. All successfully matched video segments are then aggregated to obtain a candidate video segment set.

[0057] S211. Extract the original video data corresponding to each video segment in the candidate video segment set, and extract multimodal materials from the original video data respectively.

[0058] Multimodal material includes at least keyframe images, audio clips, and audio-text. Multimodal material refers to multi-dimensional, valid evidence extracted from raw video data.

[0059] In this embodiment, the corresponding original video data can be retrieved in batches based on the identifier and storage path of each video segment in the candidate video segment set. Multimodal parsing processing is performed on the original video data to intelligently filter key valid frames and extract key frame images that can intuitively verify the queried event; valid audio segments synchronized with the video are captured, and corresponding audio-text is generated using speech-to-text technology. Finally, a complete multimodal material containing key frame images, audio segments, and audio-text is obtained.

[0060] S212. Perform clock drift correction on multimodal materials belonging to different cameras, unify the timestamps of the multimodal materials to obtain the target multimodal material, and match the spatial coordinates corresponding to the target multimodal material.

[0061] Clock drift correction refers to a processing mechanism that uniformly calibrates the timestamps of multi-source video footage to address system time deviations caused by asynchronous acquisition from multiple cameras in a home, thus eliminating timing misalignment issues across multiple devices. Spatial coordinates are standardized spatial point coordinates corresponding to the acquisition locations of each camera in the home, used to bind the acquisition space of multimodal footage to achieve precise alignment in spatiotemporal dimensions.

[0062] In this embodiment, to address the issues of timestamp offsets and timing discrepancies in footage caused by independent asynchronous acquisition from multiple cameras in a home environment, a clock drift correction algorithm can be applied to all multimodal footage belonging to different cameras. The timestamps of all footage are uniformly calibrated based on the system's standard time, eliminating timing deviations across multiple devices. Simultaneously, by combining the home's spatial topology with the fixed installation locations of each camera, the acquisition spatial coordinates corresponding to each multimodal footage are matched.

[0063] S213. Integrate candidate video clips, target multimodal materials, and spatial coordinates to construct a multimodal evidence chain that matches the natural language query information.

[0064] In this embodiment, the selected candidate video clips, target multimodal materials, and spatial coordinate information can be structurally associated and integrated, and then orderly spliced ​​and categorized according to the event sequence, spatial location distribution, and query target object dimension. This constructs a multimodal evidence chain that matches the natural language query information.

[0065] In this embodiment of the invention, natural language query information input by a user is received, and the user's queryer information is determined. The natural language query information is converted into query text, and the query intent category of the query text is identified. The query intent category is taken as the query intent. Family members, objects, events, and locations in the query text are identified and taken as family scene entities. Time information in the query text is extracted, and the corresponding timestamp interval is obtained by matching the time information. Family scene entities associated with the query intent are selected as query objects, and corresponding query object identifiers are generated based on the query objects. The queryer role corresponding to the queryer information is obtained by searching a preset family knowledge graph according to the queryer information. The query object role corresponding to the query object identifier is obtained by searching a preset family knowledge graph according to the query object identifier. The queryer role, query object role, family scene entities, and timestamp intervals are used to construct an intent feature vector. The intent feature vector is matched with preset authorization decision rules to obtain a privacy authorization decision result. When the privacy authorization decision is approved, the query intent is decomposed into a multi-dimensional query granularity to obtain a time query granularity. The system employs location and object query granularities, using these granularities as multidimensional query levels. These granularities serve as the retrieval index, matching corresponding family scene entity constraints from a pre-defined family knowledge graph. Pre-stored video segment metadata is then invoked, and the video segment metadata is filtered using both multidimensional query granularity and scene entity constraints. Video segments that simultaneously meet both multidimensional query granularity and family scene entity constraints are aggregated to generate a candidate video segment set. The original video data corresponding to each video segment within the candidate set is extracted, and multimodal materials are extracted from the original video data. Clock drift correction is performed on multimodal materials belonging to different cameras, and the timestamps of the multimodal materials are unified to obtain the target multimodal material. The spatial coordinates corresponding to the target multimodal material are then matched. The candidate video segments, target multimodal materials, and spatial coordinates are integrated to construct a multimodal evidence chain that matches the natural language query information. This achieves privacy authorization before accurate retrieval, executing subsequent multi-granularity decomposition, video filtering, and multimodal processing only on authorized queries, and terminating the process for unauthorized queries. It avoids the waste of computing power caused by the indiscriminate full-scale search and parsing of traditional solutions, greatly reduces the computing load on home terminals and edge devices, and adapts to the deployment needs of lightweight smart home devices.

[0066] Example 3 In one embodiment, this embodiment uses a video question-answering system to execute a video question-answering method as an example to further illustrate a video question-answering method. Taking a family knowledge graph as a preset family knowledge graph as an example, the video question-answering system includes a multimodal input parsing module, a family scene entity extraction module, a privacy authorization decision module, a multi-granularity spatiotemporal query decomposition and execution module, and a video evidence multimodal fusion return module.

[0067] The multimodal input parsing module receives natural language queries (voice input or text input), calls the multimodal input parsing submodule (speech recognition submodule and text normalization submodule) to convert the query into a structured query intent, and outputs the structured query text and query intent vector.

[0068] Specifically, the multimodal input parsing module can receive natural language queries and convert them into structured query intents. The input information for the multimodal input parsing module is the user's natural language query (voice input or text input). It employs industry-standard speech recognition algorithms to convert the voice input into text, and uses text normalization algorithms to normalize colloquial expressions, typos, and duplicate words in the query text. Based on the normalized query text, it identifies the query intent category, such as time query, location query, object query, event query, and behavior query, and outputs the structured query intent query text and the query intent itself.

[0069] The family scene entity extraction module is used to extract family scene entities based on structured query text, extracting four types of entities: family members, objects, events, and locations, and binding them to the query object. It also assists in calling the family knowledge graph to output the query object ID, entity type, and timestamp range.

[0070] Specifically, the family scene entity extraction module can perform family scene entity extraction based on structured query text and bind it to the query object. The input to the family scene entity extraction module is a structured query intent and a family knowledge graph. Based on the structured query text, the module identifies family members (identified by member names, titles, or pronouns), objects (identified by object names or attribute words), events (identified by event verbs, such as falling, eating, playing, etc.), and locations (identified by location names and directional words, such as living room, balcony, bedroom). Each extracted entity is labeled with a type, i.e., labeled as a family member, object, event, or location. Based on the intent category, the query object is determined; for example, the query object for a time query is a time range, for a location query it is a location range, for an object query it is a specific entity, for an event query it is an event type, and for a behavior query it is a behavior pattern. Based on time words in the query text, such as today, yesterday, last week, or 3 PM, the timestamp range is estimated. The knowledge graph then assists in verifying the legality of the extracted entities.

[0071] The privacy authorization decision module is used to identify the roles of the queryer and the queryee based on edge traversal of the family knowledge graph. The three role categories are homeowner, family member, and visitor. Based on the intent feature vector and role type, a privacy authorization decision is made. The output is either authorization approved or denied, along with the reason. If authorization is approved, the module proceeds to a multi-granularity spatiotemporal query decomposition and execution module; if authorization is denied, the reason for denial is returned to the user.

[0072] Specifically, the privacy authorization decision module can identify the roles of the queryer and the queried party based on edge traversal of the family knowledge graph, and execute privacy authorization decisions based on intent feature vectors. The inputs to the privacy authorization decision module include the query object ID, entity type, and timestamp range. The queryer's identity information includes user login, voiceprint, or mobile phone binding.

[0073] The privacy authorization decision module can identify the queryer's role by traversing the edges of the family knowledge graph based on the queryer's identity information. This role could be the homeowner (the member with the highest family privileges), a family member (a member with partial family privileges), or a visitor (a non-family member temporarily entering the home). Based on the query object ID, it can identify the queried party's role by traversing the edges of the family knowledge graph, including involved members (queries involving specific family members) and uninvolved members (queries only involving objects, events, or locations without specific family members). An intent feature vector is constructed based on the query object, timestamp range, and role type. This intent feature vector includes the queryer's role, the queried party's role, the query object type, the time range, and the location range. Based on the intent feature vector, a privacy authorization decision is executed. Privacy authorization decisions include authorization (query approved, proceeding to the query decomposition module) and denial (query rejected, returning a reason for denial). Decision rules may include, but are not limited to: default authorization for the homeowner; default authorization for both the homeowner and the queried object (special scenarios such as private spaces require additional verification); default denial for both the homeowner and the queried object; default denial for the visitor; and the homeowner can pre-authorize visitors to access specific areas.

[0074] The multi-granularity spatiotemporal query decomposition and execution module is used to perform three-level granularity query decomposition based on the authorized query intent: time-based query granularity, location-based query granularity, and object-based query granularity. Combined with cross-video entity tracking, namely person re-identification (ReID) and behavior pattern matching, it retrieves relevant video segments from the family knowledge graph and outputs a candidate video segment set. This set can include video segment IDs, cross-video timelines, and behavior pattern annotations.

[0075] Specifically, the multi-granularity spatiotemporal query decomposition and execution module performs a three-layer granularity query decomposition based on the authorized query intent, and retrieves relevant video clips from the family knowledge graph by combining cross-video entity tracking. The inputs to the multi-granularity spatiotemporal query decomposition and execution module are the authorized query intent, the family knowledge graph, and video clip metadata.

[0076] The multi-granularity spatiotemporal query decomposition and execution module can decompose queries into three granularities based on the query intent: time granularity (decomposing time terms in the query intent into timestamp intervals, supporting three granularities: specific time, time period, and duration; e.g., 3 PM is converted to [15:00, 15:05]; AM to [06:00, 12:00]; and within five minutes to [t, t+5min]); location granularity (decomposing location terms in the query intent into three granularities: camera number, room, and area coordinates; e.g., living room is converted to camera #2 and room = living room + area coordinates = (x1, y1, x2, y2); and object granularity (decomposing object terms in the query intent into entity ID and entity type; e.g., Xiaoming is converted to entity ID = member_001 + type = family member). For queries where the query object is a family member, cross-video entity tracking is performed. This includes ReID (Re-identification of People), which identifies the same entity across video segments using appearance and behavioral features. Then, through behavioral pattern recognition and annotation, the behavior of the same entity is tracked across video segments. Video segments matching the query intent are retrieved from the family knowledge graph. Specifically, filtering can be performed based on three dimensions: timestamp interval, location range, and object ID, outputting a candidate video segment set. The candidate video segment set includes video segment ID, cross-video timeline, and behavioral pattern annotation.

[0077] The video evidence multimodal fusion return module is used to perform video evidence multimodal fusion based on a set of candidate video segments, return a multimodal evidence chain consisting of original text citations, video keyframe audio segment timestamps, and spatial coordinates, and output the video evidence multimodal fusion result.

[0078] Specifically, the video evidence multimodal fusion return module performs multimodal fusion of video evidence based on the candidate video segment set and returns a multimodal evidence chain. The input consists of the candidate video segment set and the original video segment data, including keyframes, audio, timestamps, and coordinates.

[0079] The system can extract relevant original text segments from the speech-to-text transcripts of candidate video clips, extract keyframes (the most representative keyframes) from the candidate video clips, and extract relevant audio segments from the audio stream of the candidate video clips. Based on multi-camera clock drift correction, the timestamps of the candidate video clips are aligned to a unified time base, and coordinate positioning information, including camera ID, room and area coordinates, is extracted from the candidate video clips. Original text citations, keyframes, audio, timestamps, and coordinate positioning are fused into a multimodal evidence chain. The multimodal evidence chain is returned to the user, including original text citations (segments relevant to the query text), keyframes (keyframe ID and keyframe screenshot), audio segments (speech-to-text transcripts and audio IDs), timestamps (start and end timestamps), and coordinate positioning (camera ID, room and area coordinates).

[0080] In one embodiment, using a question-and-answer system for identifying dangerous behaviors in children as an example, the scenario is a three-bedroom, two-living-room family home with six cameras deployed. The permanent residents include the father, mother, 6-year-old Xiaoming, and 4-year-old Xiaohong. The system has been running for three months. Currently, the father uses a family app to check if Xiaoming took a knife in the kitchen this afternoon.

[0081] The querier's identity information is the father's mobile phone bound ID = user_father. In the family knowledge graph, Xiaoming = 6-year-old child (entity ID = member_xiaoming); Kitchen = location 4 (entity ID = room_kitchen); Knife = object (entity ID = object_knife); Father = owner (entity ID = member_father). The provided video clip metadata includes a list of video clips from the kitchen area during the time range of [14:00, 18:00] this afternoon.

[0082] Step 1: Multimodal Input Parsing. Speech Recognition: Did Xiaoming take a knife in the kitchen this afternoon? Text Normalization: No spoken words; Intent Classification: Object Query (the query object is Xiaoming's action of taking a knife); Structured Output: Query Text + Intent Category = Object Query + Key Parameters = {Member = Xiaoming, Location = Kitchen, Time = This afternoon, Object = Knife}.

[0083] Step 2: Entity Extraction in the Family Scene. Entity Recognition: Family Member = Xiaoming (Entity ID = member_xiaoming) + Object = Knife (Entity ID = object_knife) + Event = Taking (implying the act of taking a knife) + Location = Kitchen (Entity ID = room_kitchen); Entity Type Labeling: Family Member, Object, Event, Location; Query Object Binding: Query Object = Xiaoming's act of taking a knife; Timestamp Range Estimation: This afternoon = [14:00, 18:00]; Knowledge Graph Assisted Verification: Through the family knowledge graph, it is confirmed that Xiaoming's kitchen knife are all legitimate entities.

[0084] Step 3, Privacy Authorization Decision. Queries' Role Identification: Father = Owner (highest family authority) based on family knowledge graph traversal; Queries' Role Identification: Xiaoming = Involved member (query involves specific family members); Intent Feature Vector Encoding: Queries' Role = Owner + Queries' Role = Involved Member + Query Object Type = Behavioral Query + Time Range = This Afternoon + Location Range = Kitchen; Privacy Authorization Decision: Authorize (Queries are owners, default authorization); Reason for Decision: Queries are owners and have the highest family authority.

[0085] Step 4: Multi-granularity spatiotemporal query decomposition. Time granularity: Today afternoon = [14:00, 18:00]; Location granularity: Kitchen = Camera #4 + Room = Kitchen + Area coordinates = (x1, y1, x2, y2); Object granularity: Xiaoming = Entity ID = member_xiaoming.

[0086] Step 5: Cross-video entity tracking. ReID pedestrian re-identification: Track Xiaoming's location trajectory across multiple video clips within the time range of [14:00, 18:00]; Behavior pattern matching: Identify Xiaoming's behavior pattern of taking a knife in the kitchen.

[0087] Step 6: Video Segment Retrieval. Retrieve video segments involving Xiaoming in the kitchen area within the time range of [14:00, 18:00] from the family knowledge graph; output a set of candidate video segments (3 candidate video segments, each segment labeled with video segment ID + cross-video timeline + behavior pattern label).

[0088] Step 7: Multimodal Fusion and Return of Video Evidence. Original Citation Extraction: Speech transcription of Xiaoming taking a knife in the kitchen (extracted from the audio stream of candidate video clips); Keyframe Extraction: Keyframes of Xiaoming taking the knife (1-2 keyframe screenshots extracted from candidate video clips); Audio Segment Extraction: Audio segment of Xiaoming in the kitchen (extracted from the audio stream of candidate video clips); Timestamp Alignment: Based on D3 multi-camera clock drift correction, timestamps of all video clips are aligned to a unified time base; Coordinate Location Extraction: Camera #4 + Room = Kitchen + Area Coordinates; Multimodal Evidence Chain Fusion: Fusion of original citation + keyframe + audio + timestamp + coordinate location into a multimodal evidence chain; Returning the multimodal evidence chain to the father (original citation of Xiaoming taking a knife in the kitchen + keyframe screenshot + audio segment + timestamp [16:35, 16:38] + coordinate location of Camera #4, kitchen, and area coordinates).

[0089] Output multimodal evidence chain: Original quote: Xiaoming takes a knife in the kitchen (from the speech transcription of 16:35-16:38); Keyframes: 3 keyframe screenshots (showing Xiaoming's continuous actions of taking a knife in the kitchen); Audio clip: Audio clip from 16:35-16:38 (including Xiaoming's conversation in the kitchen); Timestamps: [16:35, 16:38]; Coordinates: Camera #4, kitchen, area coordinates; Decision basis: Father is the owner, default authorization; Xiaoming is the member involved; Response time: <1s from receiving the query to receiving the evidence.

[0090] In one embodiment, taking a fall detection questionnaire for the elderly as an example, the scenario is the aforementioned family residence, with an additional 70-year-old resident, Mr. Li. The mother currently uses a family application (App) to check whether Mr. Li fell in the living room yesterday morning.

[0091] The input includes a natural language query (text input): Did Lao Li fall in the living room yesterday morning?; Querent identity information: Mother's mobile phone ID = user_mother; Family knowledge graph: Lao Li = 70-year-old (entity ID = member_laoli); Living room = location 1 (entity ID = room_livingroom); Fall = event verb; Mother = family member (entity ID = member_mother); Video clip metadata: A list of video clips from the living room area in the time range [06:00, 12:00] yesterday morning.

[0092] Step 1: Multimodal Input Parsing. Text Normalization: No colloquial words; Intent Classification: Event Query (query object is the fall event); Structured Output: Query Text + Intent Classification = Event Query + Key Parameters = {Member = Lao Li, Location = Living Room, Time = Yesterday Morning, Event = Fall}.

[0093] Step 2: Entity Extraction in the Family Scene. Entity Recognition: Family Member = Lao Li (Entity ID = member_laoli) + Event = Falling (implying the act of falling) + Location = Living Room (Entity ID = room_livingroom); Entity Type Labeling: Family Member, Event, Location; Query Object Binding: Query Object = Lao Li Falling Event; Timestamp Range Estimation: Yesterday Morning = [xxxx-06-02 06:00, xxxx-06-02 12:00]; Knowledge Graph Assisted Verification: Through the family knowledge graph, it is confirmed that Lao Li's falling in the living room is a legitimate entity.

[0094] Step 3, Privacy Authorization Decision. Queries Role Identification: Mother = Family Knowledge Graph Traversal as a Family Member (Non-Head); Queries Role Identification: Mr. Li = Family Knowledge Graph Traversal as the Involved Member (Query Involves a Specific Family Member); Intent Feature Vector Encoding: Queries Role = Family Member + Queries Role = Involved Member + Query Object Type = Event Query + Time Range = Yesterday Morning + Location Range = Living Room; Privacy Authorization Decision: Authorized (If both the querier and the query object are family members, authorization is granted by default); Reason for Decision: If both the querier and the query object are family members, authorization is granted by default.

[0095] Step 4: Multi-granularity spatiotemporal query decomposition. Time granularity: Yesterday morning = [xxxx-06-02 06:00, xxxx-06-02 12:00] (Three-level granularity: specific time, time period, and duration; in this example, it is the time period granularity); Location granularity: Living room = Camera #1 + Room = Living room + Area coordinates = (x1, y1, x2, y2) (Three-level granularity: Camera number + Room + Area coordinates); Object granularity: Lao Li = Entity ID = member_laoli (Three-level granularity: Entity ID + Entity type).

[0096] Step 5: Cross-video entity tracking. ReID pedestrian re-identification: Track Lao Li's location trajectory across multiple video clips within the time range of [xxxx-06-02 06:00, xxxx-06-02 12:00]; Behavior pattern matching: Identify Lao Li's falling behavior pattern (behavior recognition algorithm identifies the elderly man's sudden loss of balance and falling pattern).

[0097] Step 6: Video clip retrieval. Retrieve video clips from the family knowledge graph that involve Mr. Li's fall in the living room area within the time range of [xxxx-06-02 06:00, xxxx-06-02 12:00]; output a set of candidate video clips (1 candidate video clip, labeled with video clip ID + cross-video timeline + behavior pattern label = elderly fall).

[0098] Step 7: Multimodal fusion of video evidence. Original text citation extraction: Speech transcription of Lao Li falling in the living room (extracted from the audio stream of candidate video clips, possibly Lao Li's shouts of "Ouch!"); Keyframe extraction: Keyframes of Lao Li's fall (two keyframe screenshots extracted from candidate video clips, showing before and after the fall); Audio clip extraction: Audio clip of Lao Li falling (extracted from the audio stream of candidate video clips); Timestamp alignment: Based on D3 multi-camera clock drift correction, timestamps of all video clips are aligned to a unified time base; Coordinate location extraction: Camera #1 + Room = Living Room + Area coordinates; Multimodal evidence chain fusion: Merging original text citation + keyframe + audio + timestamp + coordinate location into a multimodal evidence chain; Returning the multimodal evidence chain to the mother (original text citation: Lao Li fell in the living room + "Ouch!" + keyframe screenshot + audio clip + timestamp [09:23, 09:24] + coordinate location: Camera #1, living room, area coordinates).

[0099] Output: Multimodal evidence chain: Original quote: Lao Li fell down in the living room + ouch (from the 09:23-09:24 speech transcription); Keyframes: 2 keyframe screenshots (showing Lao Li before and after falling); Audio clips: Audio clips from 09:23-09:24 (including Lao Li's ouch shouts); Timestamps: [xxxx-06-02 09:23, xxxx-06-02 09:24]; Coordinates: Camera #1, living room, area coordinates; Decision basis: Mother is a family member, Lao Li is the involved member, default authorization; Response time: <1s from receiving the query to receiving the evidence.

[0100] In one embodiment, taking cross-video entity tracking question answering as an example, the scenario is as described above: a family residence. Currently, the father is using a family app to check which rooms his son, Xiaoming, has visited this week.

[0101] Input includes natural language query (voice input): Which rooms did Xiaoming visit this week?; Querent identity information: Father's mobile phone ID = user_father; Family knowledge graph: Xiaoming = 6-year-old child (entity ID = member_xiaoming); Father = owner (entity ID = member_father); Video clip metadata: List of video clips from all areas this week (xxxx-05-27~xxxx-06-02).

[0102] Step 1: Multimodal Input Parsing. Speech Recognition: Which rooms did Xiaoming visit this week? Text Normalization: No spoken words; Intent Classification: Cross-video entity tracking query (the query object is Xiaoming's location trajectory this week); Structured Output: Query Text + Intent Category = Cross-video entity tracking query + Key Parameters = {Member = Xiaoming, Time = This week, Query Type = Location Trajectory}.

[0103] Step 2, Family Scene Entity Extraction. Entity Recognition: Family Member = Xiaoming (Entity ID = member_xiaoming); Entity Type Labeling: Family Member; Query Object Binding: Query Object = Xiaoming's Location Trajectory This Week; Timestamp Interval Estimation: This Week = [xxxx-05-27 00:00, xxxx-06-02 23:59]; Knowledge Graph Assisted Verification: Through the family knowledge graph, Xiaoming is confirmed as a legitimate entity.

[0104] Step 3, Privacy Authorization Decision. Queries' Role Identification: Father = Owner based on family knowledge graph traversal; Queries' Role Identification: Xiaoming = Involved member based on family knowledge graph traversal (query involves specific family members); Intent Feature Vector Encoding: Queries' Role = Owner + Queries' Role = Involved Member + Query Object Type = Cross-Video Entity Tracking Query + Time Range = This Week; Privacy Authorization Decision: Authorized (Queries are owners, default authorization); Reason for Decision: Queries are owners and have the highest family privileges.

[0105] Step 4: Multi-granularity spatiotemporal query decomposition. Time granularity: This week = [xxxx-05-27 00:00, xxxx-06-02 23:59] (Three-level granularity: In this example, it is the duration granularity, spanning 7 days); Location granularity: All rooms (living room, kitchen, bedroom, study, balcony, bathroom); Object granularity: Xiaoming = Entity ID = member_xiaoming.

[0106] Step 5: Cross-video entity tracking. ReID pedestrian re-identification: Track Xiaoming's location trajectory across multiple video clips within the time range of [xxxx-05-27 00:00, xxxx-06-02 23:59]; Behavior pattern matching: Identify Xiaoming's activity behavior patterns in various rooms (studying, playing, resting, eating, etc.); Cross-video timeline construction: Construct Xiaoming's trajectory in various rooms within this week into a cross-video timeline (living room [Monday 10:00-12:00] → kitchen [Monday 12:00-12:30] → bedroom [Monday 12:30-13:30] → living room [Monday 15:00-17:00] → ......).

[0107] Step 6: Video Segment Retrieval. Retrieve all video segments involving Xiaoming from the family knowledge graph within this week; group by location (living room, kitchen, bedroom, study, balcony, bathroom); output a set of 28 candidate video segments (labeled with video segment ID + cross-video timeline + location + behavior pattern).

[0108] Step 7: Multimodal Fusion and Return of Video Evidence. Original Citation Extraction: Speech transcription of Xiaoming's activities this week (extracted from the audio stream of candidate video clips, possibly including dialogue such as Xiaoming: "Mom, I want to go play in the living room," etc.); Keyframe Extraction: Keyframes of Xiaoming's activities this week (multiple keyframe screenshots extracted from candidate video clips, grouped by location); Audio Segment Extraction: Audio segments of Xiaoming's activities this week (extracted from the audio stream of candidate video clips); Timestamp Alignment: Based on D3 multi-camera clock drift correction, timestamps of all video clips are aligned to a unified time base; Coordinate Location Extraction: Camera number + room + area coordinates (grouped by location); Multimodal Evidence Chain Fusion: Fusion of original citation + keyframe + audio + timestamp + coordinate location into a multimodal evidence chain; Returning the multimodal evidence chain to the father (original citation Xiaoming: "Mom, I want to go play in the living room" + keyframe screenshots (grouped by location) + audio segments (grouped by location) + cross-video timeline + coordinate location: camera number, room, area coordinates).

[0109] Output: Multimodal evidence chain: Across video timeline: Xiaoming's location trajectory this week (living room → kitchen → bedroom → living room → study → balcony, arranged in chronological order); Original quote: Xiaoming: Mom, I want to go play in the living room (speech transcription from multiple video clips); Keyframes: Multiple keyframe screenshots (grouped by location: 5 for living room, 4 for kitchen, 6 for bedroom, 3 for study, 2 for balcony); Audio clips: Multiple audio clips (grouped by location, showing Xiaoming's conversations in various rooms); Timestamps: Across video timeline (precise timestamps for each clip this week); Coordinate positioning: Camera number, room, and area coordinates (grouped by location); Decision basis: Father is the owner, default authorization; Response time: From query to evidence return < 3s (slower processing across videos).

[0110] This embodiment realizes entity extraction and query object identification in a family scene, member type information acquisition and three-role classification, privacy permission verification, and multimodal fusion return of video evidence.

[0111] Example 4 Figure 3 This is a schematic diagram of the structure of a video question-and-answer device according to Embodiment 4 of the present invention. Figure 3 As shown, the device includes: a data parsing module 31, an information query module 32, an authorization decision module 33, a video recognition module 34, and a question-and-answer feedback module 35.

[0112] The data parsing module 31 is used to obtain the natural language query information input by the user and the queryer information of the corresponding user, and to parse the natural language query information to obtain the query text and query intent.

[0113] The information query module 32 is used to determine the family scene entity and timestamp range based on the query text, and to determine the query object identifier according to the query intent and the family scene entity.

[0114] The authorization decision module 33 is used to determine the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier based on the preset family knowledge graph, and to perform privacy authorization decisions based on the queryer role, the queryee role, the family scene entity and the timestamp interval.

[0115] The video recognition module 34 is used to perform multi-dimensional query granularity parsing on the query intent to obtain multi-dimensional query granularity when the privacy authorization decision is approved, and to determine the candidate video segment set according to the multi-dimensional query granularity, the preset knowledge graph and the video segment metadata.

[0116] The question-and-answer feedback module 35 is used to determine the original video data corresponding to the candidate video segment set, and to generate a multimodal evidence chain that matches the natural language query information based on the candidate video segments and the original video data.

[0117] In one embodiment, the data parsing module 31 includes: An information receiving unit is used to receive natural language query information input by a user and determine the user's query information; wherein the query information is obtained through at least one of account reading, device identification, or multimodal biometrics. The data parsing unit is used to convert natural language query information into query text, identify the query intent category of the query text, and use the query intent category as the query intent.

[0118] In one embodiment, the information query module 32 includes: The entity extraction unit is used to identify family members, objects, events, and locations in the query text, and to treat family members, objects, events, and locations as entities in the family scene. The interval determination unit is used to extract time information from the query text and match the corresponding timestamp interval according to the time information; The identifier determination unit is used to filter out family scene entities associated with the query intent as query objects, and generate corresponding query object identifiers based on the query objects.

[0119] In one embodiment, the authorization decision module 33 includes: The role determination unit is used to retrieve the queryer role corresponding to the queryer information by searching the preset family knowledge graph and to retrieve the query object role corresponding to the query object identifier by searching the preset family knowledge graph. The authorization decision unit is used to construct an intent feature vector by combining the queryer role, the queryee role, the family scene entity, and the timestamp interval. The intent feature vector is then matched with a preset authorization decision rule to obtain a privacy authorization decision result.

[0120] In one embodiment, the video recognition module 34 includes: The granularity determination unit is used to decompose the query intent into multi-dimensional query granularity to obtain time query granularity, location query granularity and object query granularity, and use time query granularity, location query granularity and object query granularity as multi-dimensional query granularity; The condition determination unit is used to match the corresponding family scene entity constraints from the preset family knowledge graph using multi-dimensional query granularity as the retrieval index. The collection generation unit is used to call the pre-stored video segment metadata, filter the video segment metadata using multi-dimensional query granularity and scene entity constraints, and aggregate the video segments that simultaneously meet the multi-dimensional query granularity and home scene entity constraints to generate a candidate video segment set.

[0121] In one embodiment, the question-and-answer feedback module 35 includes: The material extraction unit is used to extract the original video data corresponding to each video segment in the candidate video segment set, and extract multimodal materials from the original video data; wherein, the multimodal materials include at least keyframe images, audio segments and audio text; The left-hand determination unit is used to perform clock drift correction on multimodal footage belonging to different cameras, unify the timestamps of the multimodal footage to obtain the target multimodal footage, and match the spatial coordinates corresponding to the target multimodal footage; The evidence chain determination unit is used to integrate candidate video clips, target multimodal materials, and spatial coordinates to construct a multimodal evidence chain that matches the natural language query information.

[0122] In one embodiment, the video question-and-answer device further includes: The rejection module is used to provide feedback on the reason for the rejection when the privacy authorization decision is to deny authorization. The video question-answering device provided in this embodiment of the invention can execute the method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method.

[0123] Example 5 Figure 4This is a schematic diagram of the structure of an electronic device implementing the video question-answering method of this invention. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (such as helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0124] like Figure 4 As shown, the electronic device 10 includes at least one processor 11 and a memory, such as a read-only memory (ROM) 12 or a random access memory (RAM) 13, communicatively connected to the at least one processor 11. The memory stores computer programs executable by the at least one processor. The processor 11 can perform various appropriate actions and processes based on the computer program stored in the ROM 12 or loaded from storage unit 18 into the RAM 13. The RAM 13 can also store various programs and data required for the operation of the electronic device 10. The processor 11, ROM 12, and RAM 13 are interconnected via a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0125] Multiple components in electronic device 10 are connected to I / O interface 15, including: input unit 16, such as keyboard, mouse, etc.; output unit 17, such as various types of displays, speakers, etc.; storage unit 18, such as disk, optical disk, etc.; and communication unit 19, such as network card, modem, wireless transceiver, etc. Communication unit 19 allows electronic device 10 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0126] Processor 11 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 11 performs the various methods and processes described above, such as video question answering methods.

[0127] In some embodiments, the video question-answering method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 18. In some embodiments, part or all of the computer program may be loaded and / or mounted on electronic device 10 via ROM 12 and / or communication unit 19. When the computer program is loaded into RAM 13 and executed by processor 11, one or more steps of the video question-answering method described above may be performed. Alternatively, in other embodiments, processor 11 may be configured to perform the video question-answering method by any other suitable means (e.g., by means of firmware).

[0128] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0129] Computer programs used to implement the methods of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer programs cause the functions / operations specified in the flowcharts and / or block diagrams to be performed. The computer programs may be executed entirely on a machine, partially on a machine, or as a standalone software package, partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0130] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0131] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0132] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0133] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0134] In one embodiment, the present invention further includes a computer program product, which includes a computer program that, when executed by a processor, implements the video question-and-answer method of any embodiment of the present invention.

[0135] In implementing the computer program product, computer program code for performing the operations of this invention can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages ​​such as Java, Smalltalk, and C++, as well as conventional procedural programming languages ​​such as C or similar languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0136] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0137] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A method of video question answering, the method comprising: include: Obtain the natural language query information input by the user and the corresponding user's query information, and parse the natural language query information to obtain the query text and query intent; Based on the query text, determine the family scene entity and timestamp range, and determine the query object identifier according to the query intent and the family scene entity; Based on a preset family knowledge graph, the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier are determined, and a privacy authorization decision is performed based on the queryer role, the queryee role, the family scene entity, and the timestamp interval. When the privacy authorization decision is approved, multi-dimensional query granularity parsing is performed on the query intent to obtain multi-dimensional query granularity, and a candidate video segment set is determined according to the multi-dimensional query granularity, the preset knowledge graph and video segment metadata; Determine the original video data corresponding to the candidate video segment set, and generate a multimodal evidence chain that matches the natural language query information based on the candidate video segments and the original video data.

2. The method according to claim 1, characterized in that, The process of obtaining natural language query information input by the user and the corresponding user's query information, and parsing the natural language query information to obtain query text and query intent, includes: The system receives natural language query information input by a user and determines the queryer information of the user; wherein the queryer information is obtained through at least one of account reading, device identification, or multimodal biometrics. The natural language query information is converted into query text, the query intent category of the query text is identified, and the query intent category is taken as the query intent.

3. The method according to claim 1, characterized in that, The step of determining the family scene entity and timestamp range based on the query text, and determining the query object identifier according to the query intent and the family scene entity, includes: Identify family members, objects, events, and locations in the query text, and treat the family members, objects, events, and locations as family scene entities; Extract the time information from the query text, and match the corresponding timestamp range according to the time information; Filter out the family scene entities associated with the query intent as query objects, and generate corresponding query object identifiers based on the query objects.

4. The method according to claim 1, characterized in that, The process of determining the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier based on a preset family knowledge graph, and performing privacy authorization decisions based on the queryer role, the queryee role, the family scene entity, and the timestamp interval, includes: The queryer role corresponding to the queryer information is obtained by searching a preset family knowledge graph based on the queryer information, and the queryee role corresponding to the queryee identifier is obtained by searching a preset family knowledge graph based on the queryee identifier. An intent feature vector is constructed by the queryer role, the queryee role, the family scene entity, and the timestamp interval. The intent feature vector is then matched with a preset authorization decision rule to obtain a privacy authorization decision result.

5. The method according to claim 1, characterized in that, The process of performing multi-dimensional query granularity parsing on the query intent to obtain multi-dimensional query granularity, and determining a candidate video segment set according to the multi-dimensional query granularity, the preset knowledge graph, and video segment metadata, includes: The query intent is decomposed into time query granularity, location query granularity and object query granularity, and the time query granularity, the location query granularity and the object query granularity are used as multi-dimensional query granularity; Using the multidimensional query granularity as the retrieval index, the corresponding family scene entity constraints are matched from the preset family knowledge graph; The video segment metadata is retrieved from the pre-stored metadata and filtered using the multidimensional query granularity and scene entity constraints. Video segments that simultaneously satisfy the multidimensional query granularity and the family scene entity constraints are then aggregated to generate a candidate video segment set.

6. The method according to claim 1, characterized in that, The step of determining the original video data corresponding to the candidate video segment set, and generating a multimodal evidence chain matching the natural language query information based on the candidate video segments and the original video data, includes: Extract the original video data corresponding to each video segment in the candidate video segment set, and extract multimodal materials from the original video data respectively; wherein, the multimodal materials include at least keyframe images, audio segments and audio text; Clock drift correction is performed on multimodal footage belonging to different cameras to unify the timestamps of the multimodal footage to obtain the target multimodal footage, and the spatial coordinates corresponding to the target multimodal footage are matched. The candidate video clips, the target multimodal materials, and the spatial coordinates are integrated to construct a multimodal evidence chain that matches the natural language query information.

7. The method according to claim 1, characterized in that, After determining the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier based on the preset family knowledge graph, and performing a privacy authorization decision based on the queryer role, the queryee role, the family scene entity, and the timestamp interval, the process further includes: When the privacy authorization decision is to deny authorization, the reason for the authorization decision to deny authorization is provided.

8. A video question-and-answer device, characterized in that, include: The data parsing module is used to obtain the natural language query information input by the user and the corresponding user's query information, and to parse the natural language query information to obtain the query text and query intent; The information query module is used to determine the family scene entity and timestamp range based on the query text, and to determine the query object identifier according to the query intent and the family scene entity; The authorization decision module is used to determine the queryer role corresponding to the queryer information and the queryee role corresponding to the query object identifier based on a preset family knowledge graph, and to perform privacy authorization decisions based on the queryer role, the queryee role, the family scene entity and the timestamp interval. The video recognition module is used to perform multi-dimensional query granularity parsing on the query intent to obtain multi-dimensional query granularity when the privacy authorization decision is approved, and to determine a set of candidate video segments according to the multi-dimensional query granularity, the preset knowledge graph and video segment metadata; The question-and-answer feedback module is used to determine the original video data corresponding to the candidate video segment set, and to generate a multimodal evidence chain that matches the natural language query information based on the candidate video segments and the original video data.

9. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video question-answering method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video question-answering method according to any one of claims 1-7.

Images