A video retrieval method and device, electronic equipment, storage medium and product

Abstract

The application discloses a video retrieval method and device, electronic equipment, storage medium and product. The method comprises the following steps: acquiring a query statement input by a user and an identity of the user; performing semantic analysis on the query statement to obtain retrieval semantic elements; determining a privacy permission level corresponding to the user according to the identity; determining retrieval weights of various indexes according to the privacy permission level; performing retrieval processing according to the retrieval semantic elements and the retrieval weights of the indexes to obtain multiple candidate video clips; and fusing and sorting the candidate video clips to obtain a final retrieval result. According to the application, the retrieval weights of various indexes are determined according to the identity difference of the user, so that different users can obtain different retrieval results when performing the same query, unauthorized access is effectively avoided, and the privacy security is improved. Meanwhile, the multi-index collaborative retrieval and fusion sorting can balance the accuracy of the retrieval result and the convenient experience of the natural language interaction.

Description

Technical Field

[0001] This invention relates to the field of video retrieval technology, and in particular to a video retrieval method, apparatus, electronic device, storage medium, and product. Background Technology

[0002] Existing natural language-based video retrieval methods typically treat all video clips as equally accessible, failing to differentiate access permissions based on the query user's identity. Furthermore, various indexes (such as vector indexes, relational indexes, and time-series indexes) are assigned fixed weights during the retrieval process, unable to be dynamically adjusted based on user identity or query sensitivity. This results in indiscriminate search results when different family members execute the same natural language query. Video clips in sensitive areas may be accidentally retrieved by unauthorized users due to insufficient access control or improper index weight allocation, reducing privacy and security in home video surveillance scenarios. Summary of the Invention

[0003] This invention provides a video retrieval method, apparatus, electronic device, storage medium, and product to address the technical problem that existing natural language video retrieval technologies lack differentiated privacy controls based on user identity, resulting in indistinguishable retrieval results for different family members and easy access to sensitive video areas by unauthorized users.

[0004] According to one aspect of the present invention, a video retrieval method is provided, the method comprising: Obtain the user's input query and the user's identity identifier; Semantic parsing of the query statement yields the retrieval semantic elements; Determine the user's corresponding privacy permission level based on their identity identifier; The retrieval weights of various indexes are determined based on privacy permission levels; The retrieval process is performed based on the semantic elements of the search and the retrieval weights of each index to obtain multiple candidate video segments; The candidate video segments are merged and sorted to obtain the final search results.

[0005] According to another aspect of the present invention, a video retrieval device is provided, the device comprising: The acquisition module is used to acquire the query statement entered by the user and the user's identity identifier; The semantic parsing module is used to perform semantic parsing on the query statement to obtain the retrieved semantic elements. The privacy level determination module is used to determine the user's corresponding privacy permission level based on the identity identifier; The retrieval weight determination module is used to determine the retrieval weight of various indexes based on the privacy permission level. The retrieval module is used to perform retrieval processing based on the retrieval semantic elements and the retrieval weights of each index to obtain multiple candidate video segments; The results generation module is used to merge and sort the candidate video segments to obtain the final search results.

[0006] According to another aspect of the present invention, an electronic device is provided, the electronic device comprising: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video retrieval method according to any embodiment of the present invention.

[0007] According to another aspect of the present invention, a computer-readable storage medium is provided, the computer-readable storage medium storing computer instructions for causing a processor to execute and implement the video retrieval method according to any embodiment of the present invention.

[0008] According to another aspect of the present invention, a computer program product is provided, the computer program product comprising a computer program that, when executed by a processor, implements the video retrieval method described in any embodiment of the present invention.

[0009] The technical solution of this invention involves obtaining a user's input query and user identity; performing semantic parsing on the query to obtain retrieval semantic elements; determining the user's corresponding privacy permission level based on the identity; determining the retrieval weights of various indexes based on the privacy permission level; performing retrieval processing based on the retrieval semantic elements and the retrieval weights of each index to obtain multiple candidate video segments; and fusing and sorting the candidate video segments to obtain the final retrieval result. This solution obtains the user's identity and determines the corresponding privacy permission level accordingly, then determines different retrieval weights for various indexes based on this privacy permission level. This ensures that when different family members execute the same natural language query, the retrieval weights of the underlying indexes are already differentiated. After weighted fusion and sorting, differentiated filtered retrieval results are obtained, effectively avoiding the risk of unauthorized users accidentally retrieving videos in sensitive areas and improving the privacy and security of video retrieval. Simultaneously, by converting natural language queries into retrieval semantic elements through semantic parsing and performing collaborative retrieval and weighted fusion sorting based on the differentiated multi-index weights, the relevance and ranking of retrieval results are considered while ensuring privacy compliance, maintaining a convenient natural language interaction experience.

[0010] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of the present invention, nor is it intended to limit the scope of the invention. Other features of the invention will become readily apparent from the following description. Attached Figure Description

[0011] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0012] Figure 1 A flowchart of a video retrieval method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the query statement semantic parsing process provided in an embodiment of the present invention; Figure 3 A flowchart of another video retrieval method provided in an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a video retrieval device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of an electronic device that implements the video retrieval method of this invention. Detailed Implementation

[0013] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0014] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this invention are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of the invention described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0015] Figure 1 This is a flowchart illustrating a video retrieval method provided in an embodiment of the present invention. This embodiment is applicable to privacy-aware video retrieval based on natural language in home video surveillance systems. The method can be executed by a video retrieval device, which can be implemented in hardware and / or software and can be configured in an electronic device. Figure 1 As shown, the video retrieval method provided in this embodiment specifically includes the following steps: S110. Obtain the query statement entered by the user and the user's identity identifier.

[0016] The query statement can refer to the search request text entered by the user in natural language, such as "find a video of my baby playing in the living room last week". The identity identifier can refer to the identification information used to uniquely identify the user, such as user_id.

[0017] In this embodiment of the invention, a natural language query statement input by the user can be received, and the identity recognition module can be invoked to obtain the identity identifier of the current operator. If a valid identity identifier is not obtained, the process is terminated and an identity recognition failure status is returned.

[0018] S120. Perform semantic parsing on the query statement to obtain the retrieval semantic elements.

[0019] The retrieval semantic elements refer to semantic information units extracted from the query statement that describe the retrieval conditions. For example, retrieval semantic elements may include at least one of object elements, location elements, time elements, and event elements. Object elements may refer to people or things mentioned in the query, such as a baby, a stranger, or a pet. Location elements may refer to spatial locations mentioned in the query, such as a living room, study, or doorway. Time elements may refer to time information mentioned in the query, such as last week or yesterday afternoon. Event elements may refer to activities or behaviors mentioned in the query, such as playing, studying, or walking.

[0020] In this embodiment of the invention, natural language understanding can be performed on the obtained query statement to extract retrieval semantic elements such as object elements, location elements, time elements, and event elements, so as to accurately identify the retrieval conditions explicitly expressed by the user.

[0021] S130. Determine the user's corresponding privacy permission level based on the identity identifier.

[0022] The privacy permission level can refer to a level identifier that is pre-set according to the user's identity and used to control the user's access to video data; for example, the privacy permission level can include multiple levels such as completely public, family, privacy zone, and personal privacy zone.

[0023] In this embodiment of the invention, a pre-configured permission mapping table can be queried according to the obtained identity identifier to obtain the privacy permission level corresponding to the user. This privacy permission level remains stable throughout the current retrieval session and will not be modified in subsequent steps.

[0024] S140. Determine the retrieval weights of various indexes based on their privacy permission levels.

[0025] An index can refer to the underlying data structure built for video data to support efficient retrieval. Different types of indexes can describe the attributes of video segments from different dimensions. For example, indexes include at least vector indexes, relational indexes, and time-series indexes. Retrieval weight can refer to the parameters assigned to each index to control its influence on retrieval recall and result ranking. The higher the weight, the greater the decisive role of the index in the final retrieval results. A weight of zero indicates that the index does not participate in this retrieval.

[0026] In this embodiment of the invention, various indexes participating in the video retrieval can be determined first, such as vector indexes, relational indexes, and time-series indexes; then, the pre-configured weight mapping table is queried according to the user privacy permission level determined above, so as to obtain the retrieval weight corresponding to each index.

[0027] S150. Based on the retrieval semantic elements and the retrieval weights of each index, perform retrieval processing to obtain multiple candidate video segments.

[0028] Candidate video segments can refer to video segments that are initially selected from the video database and match the search criteria. Each candidate video segment can include information such as segment identifier, source camera identifier, start timestamp, end timestamp, and original score under each index.

[0029] In this embodiment of the invention, retrieval operations for multiple indexes can be performed simultaneously in parallel based on the retrieval weights of each index. Each index independently completes the query and scoring of its own dimension according to its assigned weight, generating corresponding intermediate candidate sets. Then, the intermediate candidate sets corresponding to each type of index can be deduplicated and merged to obtain multiple candidate video segments and their corresponding original scores, providing complete input data for subsequent fusion and ranking.

[0030] S160. Merge and sort the candidate video segments to obtain the final search results.

[0031] The fusion ranking process refers to the process of comprehensively calculating the original scores of each candidate video segment under multiple indices and arranging them from highest to lowest based on the comprehensive score. The final retrieval result can be a list of video segments or a corresponding status result output after fusion ranking.

[0032] In this embodiment of the invention, the original scores obtained for each candidate video segment in indexes such as vector index, relation index, and time series index can be acquired. Then, for these original scores from different sources and with different dimensions, a normalization transformation is applied to reduce them to the same numerical scale range to eliminate the differences in score dimensions between indexes. Next, based on the retrieval weights of each index determined in the preceding steps, a weighted summation is performed on the normalized scores of each candidate video segment to obtain a comprehensive score for measuring the overall relevance of the segment. Finally, based on this comprehensive score, all candidate video segments are sorted in descending order, and the top-ranked segments are used as the output results of this retrieval.

[0033] The technical solution of this invention obtains user identity identifiers and determines corresponding privacy permission levels accordingly. Then, based on these privacy permission levels, it differentiates the retrieval weights of various indexes. This allows different family members to obtain retrieval results with differentiated weight configurations when performing the same natural language query. Since the retrieval weights directly affect the influence of various index results in the fusion ranking, videos in sensitive areas are naturally at a disadvantage in the overall score due to their corresponding index weights being suppressed. This effectively avoids the risk of unauthorized users accidentally retrieving videos in sensitive areas, thus improving the privacy and security of video retrieval. Simultaneously, semantic parsing converts natural language queries into retrieval semantic elements, and collaborative retrieval and fusion ranking are performed based on the differentiated multi-index weights. This ensures both privacy compliance and the relevance and ranking rationality of the retrieval results, maintaining a convenient natural language interaction experience.

[0034] Furthermore, based on the above embodiments of the invention, such as Figure 2 As shown, semantic parsing of the query statement yields the retrieval semantic elements, specifically including: S210. Generate the retrieval session status and record the identity and query statement in the retrieval session status.

[0035] The retrieval session state can refer to the context data object created for a single video retrieval request and that runs through the entire retrieval process. It is used to centrally store information such as identity information, query content, semantic integrity status, privacy permission level, sensitive area set, and verification progress required for this retrieval.

[0036] In this embodiment of the invention, after receiving a query statement input by a user, a corresponding retrieval session state can be generated, and the identity identifier and the original query text can be recorded in this state. This state object is also used to record information such as the clarification round (initially 0), filled fields, missing fields, ambiguous fields, privacy permission level, sensitive area set, and verification status.

[0037] S220. Based on the preset set of semantic element types, perform semantic parsing on the query statement to obtain the retrieved semantic elements and their corresponding element confidence scores; the set of semantic element types includes object elements, location elements, time elements, and event elements.

[0038] The preset set of semantic element types can refer to a predefined data structure used to limit the categories of information to be extracted in natural language queries. For example, it can include four types: object elements, location elements, time elements, and event elements, guiding the extraction scope and processing boundaries of the semantic parsing model. Element confidence can be a numerical indicator used to characterize the reliability of the semantic parsing results, ranging from 0 to 1. A higher value indicates more accurate and reliable extracted semantic elements, while a lower value indicates vague expression or unreliable parsing results. For example, a clear time expression "May 28, 2026" can obtain a high confidence score (e.g., 0.95), while a vague time expression "last time" has a lower confidence score (e.g., 0.42).

[0039] In this embodiment of the invention, a pre-configured set of semantic element types can be invoked. This set defines the scope to be extracted from the natural language query statement for this parsing, namely, four major categories: object, location, time, and event. Subsequently, a semantic parsing model is used to perform deep parsing on the user-input query statement, performing information matching and extraction for the four preset types respectively, to obtain specific retrieval semantic elements under each category. For example, from the query statement "find a video of my baby playing in the living room last week", the object element "baby", the location element "living room", the time element "last week", and the event element "playing" are extracted. At the same time, the semantic parsing model calculates a confidence score for each extracted element to quantify the reliability of the parsing result, and finally forms a set of structured retrieval semantic data with confidence labels, which is then passed to the subsequent integrity and privacy determination stages.

[0040] S230. Record the successfully extracted semantic elements with corresponding element confidence scores not lower than the preset confidence threshold as filled fields in the retrieval session status.

[0041] The pre-set reliability threshold refers to a critical value set by the system to distinguish between reliable and unreliable parsing results. An element is considered reliable when its confidence level is not lower than this threshold, and unreliable when it is lower. The populated field refers to a retrieval semantic element in the query statement that has been successfully extracted and whose corresponding element confidence level is not lower than the pre-set reliability threshold. This represents a clear and reliable expression of intent by the user in that retrieval dimension.

[0042] In this embodiment of the invention, the extraction results can be compared with a preset confidence threshold, and the retrieved semantic elements that are successfully extracted and whose corresponding element confidence is not lower than the preset confidence threshold are recorded as filled fields in the retrieval session status. This step can accurately identify the clear and unambiguous parts of the user's intent, ensuring that the retrieval system gives the highest priority to these highly certain conditions, avoiding unnecessary follow-up questions, and ensuring retrieval efficiency.

[0043] S240. Record the retrieved semantic elements that were successfully extracted but whose corresponding element confidence is lower than the preset confidence threshold as fuzzy fields in the retrieval session status.

[0044] Among them, fuzzy fields can refer to retrieval semantic elements in the query statement that have been extracted but whose corresponding element confidence is lower than the preset confidence threshold. They are used to characterize semantic information that the user has mentioned in this retrieval dimension but whose expression is not clear enough and has multiple possible interpretations.

[0045] In this embodiment of the invention, if the confidence level of the element corresponding to the extracted retrieval semantic element is lower than a preset confidence threshold, it is recorded as a fuzzy field in the retrieval session state. This step can identify the ambiguous parts of the query (such as "last time," "over there," etc.) and mark them separately instead of discarding them directly or forcibly using them. This effectively avoids the drawback of the system forcibly guessing the meaning of fuzzy words, resulting in incorrect search results, and provides a precise trigger point for subsequent targeted disambiguation clarification.

[0046] S250. Record the semantic element types that were not matched in the semantic element type set as missing fields in the retrieval session state.

[0047] Among them, missing fields can refer to semantic element types in the set of semantic element types that are not hit by the user's query, which are used to represent the blank state in which the user has not provided any information at all in this retrieval dimension.

[0048] In this embodiment of the invention, if there is a semantic element type in the semantic element type set that is not matched (i.e., not mentioned by the user at all), it is recorded as a missing field in the retrieval session state. This step accurately locates the "vacuum zone" of the retrieval conditions, that is, the retrieval dimensions that the user has completely overlooked. By clearly distinguishing between "said but not clear" and "did not say at all," the system can prioritize guiding the user to supplement the missing hard conditions in the subsequent clarification interaction process. Logically, this is more in line with the human habit of completing dialogues, greatly improving the interaction efficiency and user satisfaction of multi-round clarification.

[0049] This embodiment achieves accurate assessment of query semantic completeness by refining semantic parsing results into three categories: filled fields, ambiguous fields, and missing fields. Its technical advantages are twofold: firstly, the confidence threshold filtering mechanism avoids false recalls caused by low-quality parsing results directly participating in the retrieval process, thus improving retrieval accuracy; secondly, by clearly distinguishing between the two defect types of "ambiguity" and "missing," the system can accurately pinpoint the specific problem in the user's query, thereby guiding the user to complete the information in targeted subsequent rounds of clarification, effectively improving human-computer interaction efficiency and the accuracy of retrieval results. This mechanism ensures that the retrieval system only initiates interaction when necessary, avoiding excessive disruption to the user while ensuring effective convergence of fuzzy queries.

[0050] Furthermore, based on the above embodiments of the invention, determining the user's corresponding privacy permission level according to the identity identifier specifically includes: The user's privacy permission level is obtained by querying the pre-configured permission mapping table based on their identity.

[0051] The permission mapping table can be a data structure (such as a key-value pair set or a database table) that is pre-stored in the retrieval system to record the mapping relationship between each identity and the corresponding privacy permission level. It can be dynamically configured and maintained by the system administrator according to the actual situation of family members.

[0052] In this embodiment of the invention, the identity of the user currently initiating the search request can be matched with the previously stored permission mapping table to obtain the privacy permission level corresponding to the current search user. The privacy permission levels can include PL0 Fully Public, PL1 Family Internal, PL2 Privacy Zone, and PL3 Personal Privacy. Furthermore, the permission mapping table supports dynamic configuration, allowing family administrators to flexibly adjust permissions based on changes in family members (such as new visitors or children's growth), enhancing the system's scenario adaptability.

[0053] Furthermore, based on the above embodiments of the invention, the retrieval weights of various indexes are determined according to the privacy permission level, specifically including: The pre-configured basic weight table is queried according to the privacy permission level to obtain the basic weight of each index, and the basic weight is used as the retrieval weight; the index includes at least vector index, relation index and time sequence index.

[0054] The basic weight table can refer to a data structure (such as a multidimensional array or database table) that is pre-stored in the retrieval system to record the correspondence between each privacy permission level and the basic weight of each index. The table is configured with a set of benchmark weight values ​​for different index types for each privacy permission level.

[0055] Vector indexes can refer to index structures built by encoding the visual / semantic features of video frames or video segments into high-dimensional vectors, supporting semantic matching retrieval by calculating the cosine similarity or approximate nearest neighbor between query text and video vectors.

[0056] Relational indexes can refer to index structures built on graph databases or knowledge graphs, which model entities (such as people and objects) in video content and their semantic relationships (such as "baby in the living room") as nodes and edges, supporting retrieval based on entity relationship matching.

[0057] A time-series index can refer to an index structure built based on the time attributes of video segments (such as start timestamp, end timestamp, recording date, etc.), which supports fast filtering and retrieval by time range, time point, or relative time expression (such as "last week").

[0058] In this embodiment of the invention, the aforementioned determined user privacy permission level can be matched with a pre-stored basic weight table to obtain the basic weights configured for vector index, relation index, and time-series index under that permission level. Specifically, the vector index is responsible for content matching retrieval based on semantic similarity, the relation index is responsible for structured matching retrieval based on entity relationship graphs, and the time-series index is responsible for range filtering retrieval based on time attributes. These three types of indexes work collaboratively under unified weight control, comprehensively covering the user's retrieval needs from different dimensions.

[0059] Furthermore, based on the above embodiments of the invention, this method also includes: Determine whether the semantic elements retrieved include location elements; If so, the sensitive areas and their corresponding regional privacy levels are determined based on location elements, the basic weights of each index are adjusted according to the regional privacy levels, and the adjusted basic weights are used as retrieval weights. If not, then the basic weight of each index is maintained as the retrieval weight.

[0060] Sensitive areas can refer to video surveillance areas in a home environment that are pre-marked by the system as having privacy attributes (such as studies, bedrooms, children's rooms, etc.). Access to video data in such areas requires access restrictions or weight reduction processing. Area privacy level can refer to the pre-configured quantitative indicators of privacy protection strength for each video capture area in the home (such as different levels such as public areas, private areas, and personal privacy areas), used to define the strictness of data access in that area independently of the user's privacy permission level (PL).

[0061] In this embodiment of the invention, the solution also includes a linkage correction mechanism of "location-sensitive area-weight penalty", which specifically includes: first, determining whether the retrieval semantic elements contain location elements. If they do, then further querying the privacy level of the area corresponding to the location in the family area metadata database based on the location elements to determine whether the location is a sensitive area. If it is a sensitive area, then adding a negative adjustment term to the basic weight of each index according to the privacy level of the area, and using the adjusted weight as the retrieval weight; if it is not a sensitive area or the location element does not exist, then maintaining the basic weight unchanged. Wherein, if the adjusted weight exceeds the preset valid value range [0,1], then the following boundary pruning operation needs to be performed: when the adjusted weight value is lower than 0, it is pulled up to 0, and when it is higher than 1, it is pulled down to 1, to ensure that the retrieval weight of each index is always within an executable and physically meaningful range; this operation also undertakes the final judgment function of privacy hard isolation, that is, if any weight is zero, the corresponding index channel is closed, and if all three types of index weights are zero, the retrieval is directly terminated and an inaccessible state is returned, thereby providing the final quantitative execution guarantee for the mandatory isolation of privacy permissions from the numerical calculation level.

[0062] This embodiment achieves differentiated control of index weights by determining whether the query contains location elements. Specifically, when the query points to a privacy region, the system automatically reduces the retrieval weight of each index, proactively weakening the exposure risk of sensitive videos at the ranking level and compensating for the security blind spot of traditional methods that rely solely on post-event filtering. When the query does not contain location information, the weights remain unchanged, ensuring that regular searches are not penalized. This mechanism balances privacy and security with retrieval efficiency and recall, achieving an adaptive balance between security and retrieval quality.

[0063] Furthermore, based on the above embodiments, before fusing and sorting the candidate video segments, the method further includes: If the missing or ambiguous field in the search session is not empty, a clarification interaction question will be pushed to the user; the clarification interaction question is used to guide the user to complete the missing or ambiguous search semantic elements. Receive the user's initial feedback regarding clarifying interactive issues; Update the filled, missing, and ambiguous fields in the retrieval session status according to the first feedback information; Determine whether both the missing and ambiguous fields are empty after the update; If so, then the steps of performing the retrieval processing based on the updated retrieval semantic elements and the retrieval weights of each index will be re-executed. If not, continue to initiate the next round of clarification interaction until both the missing field and the ambiguous field are empty, or the preset maximum number of clarification rounds is reached.

[0064] Clarification interaction questions refer to conversational questions automatically generated by the system to guide users in supplementing missing or ambiguous information. These questions can take the form of multiple-choice questions (providing candidate options) or short-answer open-ended questions, aiming to obtain the most crucial missing semantic information with minimal user intervention. The first feedback information refers to the user's response to the clarification interaction questions pushed by the system, including the user's choice of options in multiple-choice questions or supplementary text entered in short-answer questions. The preset maximum number of clarification rounds refers to the maximum number of clarification interactions allowed in a single retrieval session (e.g., 5 rounds), set by the system to prevent infinite loops or deadlocks caused by users repeatedly failing to provide valid information.

[0065] In this embodiment of the invention, after candidate retrieval is completed and before result fusion and sorting, a semantic integrity check gate is inserted. The specific execution process may include the following steps: S1. First, read the missing and ambiguous fields in the current search session state. If any non-empty item is detected in the missing or ambiguous fields, it is determined that the current search conditions are incomplete. The system pauses the result return process and enters the clarification interaction sub-process.

[0066] S2. Based on the specific content of the currently missing and ambiguous fields, the system generates targeted clarification questions according to a preset priority strategy (e.g., prioritizing the completion of completely missing fields, followed by disambiguating ambiguous fields). For example, if the event field is missing, the system generates "What type of activity do you want to find for this location? (1) Learning (2) Leisure (3) Other"; if the time field is ambiguous, the system generates "When you say 'last time', do you mean (1) yesterday (2) last week (3) last month?". After the question is generated, the system pushes the question to the user for confirmation.

[0067] S3. The system waits for and receives the user's first feedback (such as option selection or text input) regarding the clarification question. It then parses the feedback, extracts the user's supplementary or explicit semantic content, and updates the sets of filled, missing, and ambiguous fields in the retrieval session state accordingly. For example, if the user selects "learn," the event element is moved from the missing field to the filled field.

[0068] S4. The system performs a second check on the updated missing and ambiguous fields. If both are empty, it is determined that the current retrieval semantic elements have completely covered the minimum retrieval conditions, and the system proceeds to step S5; if they are not empty, it proceeds to step S6.

[0069] S5. The system takes the updated complete retrieval semantic elements as input, jumps back to the candidate retrieval step (i.e. step S150), re-executes the multi-level index retrieval based on the completed conditions, and uses the newly generated candidate fragments for subsequent fusion sorting.

[0070] S6. If the missing or ambiguous field is still not empty, the system checks whether the currently executed clarification round has reached the preset maximum clarification round. If not, the clarification round is incremented by 1, and the system returns to step S2 to initiate a new round of clarification; if the maximum round has been reached, clarification is stopped, and fields that have not yet converged are treated as "arbitrary matching conditions" (i.e., no restrictions are imposed on this dimension), and the system directly enters the final retrieval and sorting stage.

[0071] This embodiment effectively addresses the retrieval inaccuracies caused by inherent ambiguity and missing information in natural language queries by introducing a proactive multi-round clarification mechanism, significantly improving retrieval accuracy and user experience in fuzzy query scenarios. Simultaneously, field-level state tracking and the closed-loop design of "completeness followed by backfeedback" ensure that each round of user interaction directly translates into optimization of actual search conditions, greatly enhancing the system's conversational intelligence and result relevance. Furthermore, the preset maximum number of clarification rounds acts as a reliable safety valve, effectively suppressing interaction overhead in extreme cases while pursuing semantic completeness, avoiding process deadlock, and ultimately achieving a high level of balance between retrieval accuracy, user-friendliness, and system robustness.

[0072] Furthermore, based on the above embodiments of the invention, before re-executing the step of performing retrieval processing based on retrieval semantic elements and the retrieval weights of each index, the method further includes: If new location features are added to the updated populated fields, the sensitive areas and their corresponding regional privacy levels are determined based on the new location features, and the base weights of each index are adjusted according to the regional privacy levels.

[0073] In this embodiment of the invention, the method further includes the following in a multi-round clarification dynamic scenario: After the user completes multiple rounds of clarification interactions, the system can determine whether the location element has changed from a missing or ambiguous state to a filled state, i.e., whether the location element was initially not extracted from the user's query, or whether the extracted location element's confidence level is lower than a preset confidence threshold. If a new location element is identified, a query matching is performed in the family area metadata database based on the new location field (e.g., the user added "study" in the clarification) to determine whether the area corresponding to the location belongs to a sensitive area (e.g., the study is marked as a private area), and the pre-configured area privacy level is read. Then, based on the determined area privacy level, the corresponding negative adjustment items are re-added to the base weights of the current indexes according to the dynamic adjustment method of retrieval weights described in the previous embodiments, and boundary pruning is performed to generate updated retrieval weights. If no new location element is detected, the system maintains the original adjusted retrieval weights unchanged.

[0074] This embodiment cleverly solves the technical risk of the original privacy protection strategy failing due to user-completed location information in multi-round clarification scenarios by employing a pre-review dynamic weight adjustment mechanism. This ensures that the configuration of search weights remains strictly synchronized with the latest complete search semantics. In particular, when a user adds highly privacy-sensitive location elements during the clarification process, the system can proactively and instantly adjust the index weights at the logical entry point of re-executing the search. This blocks unexpected high-confidence recall of sensitive video clips at the source of the search, significantly improving the dynamic adaptability and immediate response capability of privacy protection in multi-round interaction scenarios. Furthermore, by accurately identifying adjustments triggered only when there is a change in location, unnecessary redundant computational overhead is avoided.

[0075] Furthermore, based on the above embodiments, before fusing and sorting the candidate video segments, the method further includes: In response to the preset sensitive area verification trigger condition, a sensitive area verification request is pushed to the user; the sensitive area verification request is used to ask the user to confirm whether to access candidate video clips that belong to sensitive areas; Receive the second feedback information from the user regarding the verification request for sensitive areas; If the second feedback message is a confirmation of access by the user, then the candidate video segments belonging to the sensitive area are retained; If the second feedback message is that the user refuses to confirm, then the candidate video segments belonging to the sensitive area will be filtered out. If the second feedback message indicates that the user refuses to answer, the search will be terminated and the status will be returned as terminated.

[0076] The preset sensitive area verification trigger conditions can refer to a set of pre-defined rules for determining whether a secondary user confirmation interaction is required for candidate video clips. These rules may include, but are not limited to, the privacy level of the video capture area to which the candidate clip belongs being higher than the user's privacy permission level, the location indicated by the location element in the retrieval semantic elements belonging to a privacy area or a personal privacy area, and the user's first access to the video capture area to which the candidate clip belongs. The sensitive area verification request can refer to an interactive query message pushed by the system to the user, requesting explicit authorization to access video clips involving privacy areas (such as a study or bedroom). This request includes a clear identification of the sensitive area and a warning of the consequences of access. The second feedback information can refer to the explicit selection signal returned by the user in response to the sensitive area verification request. Examples of feedback types include user confirmation of access, user refusal of confirmation, or user refusal to answer (such as no response after timeout, clicking close, or explicitly skipping).

[0077] In this embodiment of the invention, after candidate retrieval is completed and before result fusion and sorting, this solution also embeds a mandatory access control for sensitive areas, which may specifically include the following steps: S1. The system can scan the regional privacy level of the video capture area to which each candidate video clip belongs and compare it with the current user's privacy permission level; at the same time, it checks whether the location explicitly specified in the query text belongs to a privacy area and queries the current user's historical access records. If any of the above conditions are met (for example, a candidate clip comes from a camera in a study, and the regional privacy level of the study is higher than the user's privacy permission level), the system determines that the verification process is triggered.

[0078] S2. After determining that verification has been triggered, the system immediately generates a sensitive area verification request. This request clearly informs the user of the name of the sensitive area to which the video clip to be accessed belongs and its privacy attributes, along with explicit question options, such as: "The study is a private area, accessible only to authorized members. Confirm to view the video related to the study? [Confirm] [Cancel] [Skip]", and then pushes the request to the current operator.

[0079] S3. The system enters a waiting state, listens for and captures the second feedback information from the user, and classifies it into three situations according to the signal type: confirmation of access, denial of confirmation, or denial of response.

[0080] S4. If the second feedback information is a user confirmation of access, the system determines that the user has clearly been informed and authorized, and retains all candidate video segments belonging to sensitive areas, allowing them to continue to participate in the subsequent fusion and sorting process.

[0081] If the second feedback message indicates that the user refuses to confirm, the system determines that the user does not want to view sensitive content, immediately filters out all candidate segments belonging to sensitive areas from the candidate segment set, retains only candidate segments from non-sensitive areas to participate in subsequent sorting, and returns non-sensitive results normally.

[0082] If the second feedback message indicates that the user refuses to answer (such as no response after timeout, closing the pop-up window, or choosing to skip), the system determines that the user has not given valid authorization. In accordance with the principle of privacy and security, the entire search process is immediately terminated, and no further fusion, sorting, or result return operations are performed. The system then returns a terminated status to the user interface.

[0083] In addition, regardless of the user's feedback, the system will write the verification event (including user identification, sensitive area name, verification time, user feedback type, etc.) into the session audit log for subsequent security auditing and tracing.

[0084] This embodiment completely changes the default behavior of existing systems that simply "return upon hit" by embedding a mandatory user secondary confirmation mechanism at the end of the retrieval process, thus giving the final decision-making power for accessing sensitive videos entirely to the user on-site. Simultaneously, through differentiated diversion strategies of "refuse confirmation → filtering" and "refuse response → suspension," it preserves the normal availability of non-sensitive results when the user explicitly refuses access, while providing a safety net by completely suspending access when the user's attitude is ambiguous. From an interaction design perspective, this systematically reduces the risk of passive leakage of home privacy videos due to accidental touches, hasty operations, or incorrect permission mapping, significantly improving the fault tolerance, security, and user trust of natural language retrieval in home video surveillance scenarios.

[0085] Furthermore, based on the above embodiments of the invention, the candidate video segments are fused and sorted to obtain the final retrieval results, specifically including: Obtain the original scores of each candidate video segment under each index; The original scores are normalized to obtain the normalized scores for each candidate video segment; The normalized scores are weighted and summed according to the retrieval weight of each index to obtain the comprehensive score of each candidate video segment. The candidate video segments are sorted according to their comprehensive scores, and the sorted candidate video segments are output as the final search results.

[0086] Normalization refers to the process of mapping the original scores, which have different dimensions and value ranges from the output of each index, to a unified numerical range (such as between 0 and 1) through mathematical transformation. It aims to eliminate the differences in scoring scales between different indexes and make the scores of each dimension comparable.

[0087] In this embodiment of the invention, the process of fusing and sorting candidate video segments to output the final retrieval result specifically includes: S1. After completing the multi-level index retrieval, all candidate video segments can be traversed, and the original score corresponding to each segment can be extracted from the retrieval results of the vector index, relation index, and time sequence index respectively.

[0088] S2. Due to the different scoring mechanisms of each index—for example, the vector index outputs cosine similarity (ranging from -1 to 1), the relation index outputs matching path length (smaller values ​​indicate greater relevance), and the temporal index outputs temporal overlap ratio (ranging from 0 to 1)—the system can use a unified normalization algorithm (such as min-max normalization or function mapping) to map these original scores with different dimensions and value ranges to the same standard numerical range [0,1], thus obtaining the normalized score for each candidate video segment under each index. The normalization process does not change the relative order of merit of candidate segments within each index; it only unifies the scoring scale across indices.

[0089] S3. The normalized scores of each index of each candidate segment obtained in step S2 are weighted and summed according to the retrieval weights of the corresponding indexes determined in the aforementioned embodiments to obtain the comprehensive score of each candidate video segment. Since the retrieval weights are dynamically adjusted based on the user's privacy permission level and sensitive areas, the influence of privacy permissions is naturally incorporated into the comprehensive score during the weighted summation process. That is, candidate segments in sensitive areas are naturally at a disadvantage in the comprehensive score because their corresponding index weights are lowered.

[0090] S4. The system can sort all candidate segments in descending order of comprehensive score to form an ordered candidate list. Then, based on the preset return quantity threshold (such as Top-5 or Top-10), it can extract the top-ranked segments or directly output all sorted results that have passed privacy filtering as the final search results and return them to the user.

[0091] This embodiment effectively eliminates the differences in scoring dimensions between multi-source heterogeneous indexes through normalization processing, ensuring the rigor of weighted fusion. More importantly, it directly embeds the retrieval weights, which are pre-determined based on user privacy permissions, into the core of the comprehensive scoring calculation. This allows privacy awareness to penetrate from the hard shielding layer before retrieval to the ranking algorithm. Candidates in highly sensitive areas are naturally at a disadvantage in the comprehensive score and are ranked at the end because their weights are dynamically reduced. Privacy-friendly ranking can be achieved without adding additional post-filtering logic. Privacy awareness is further strengthened at the end of the retrieval process, ensuring that the final output results are highly relevant in content and strictly compliant within the permission boundaries.

[0092] Figure 3 A flowchart illustrating another video retrieval method provided in an embodiment of the present invention. For example... Figure 3 As shown, the method includes: first, obtaining the user's input query statement and user identity identifier; performing semantic parsing on the query statement to obtain retrieval semantic elements; querying a pre-configured permission mapping table based on the identity identifier to determine the user's corresponding privacy permission level; querying a pre-configured basic weight table based on the privacy permission level to obtain the basic weight of each index, and combining the location elements in the retrieval semantic elements to determine sensitive areas and their corresponding regional privacy levels, adjusting the basic weights of each index according to the regional privacy levels to obtain the retrieval weight of each index; performing retrieval processing based on the retrieval semantic elements and the retrieval weights of each index to obtain multiple candidate video segments; if missing or ambiguous fields are detected in the retrieval semantic elements, pushing clarification interaction questions to the user and receiving feedback information to complete the retrieval conditions, and re-executing the retrieval processing according to the updated retrieval semantic elements; finally, merging and sorting the candidate video segments to obtain the final retrieval result.

[0093] This embodiment determines the privacy permission level based on the user's identity identifier and dynamically adjusts the retrieval weights of various indexes based on this level and sensitive areas. This allows different family members to obtain differentiated filtered retrieval results when executing the same query, solving the problem of lack of privacy awareness in natural language retrieval. By coupling privacy permissions with index weights, dynamic adjustment of index weights is achieved, improving retrieval security and query response flexibility. By extracting retrieval semantic elements through semantic parsing and combining a multi-round retrieval clarification mechanism, users are guided to gradually improve their retrieval conditions when the retrieval semantic elements are incomplete, improving the retrieval accuracy of fuzzy queries. Finally, by fusion and sorting, video clips that comply with privacy permissions and are highly relevant to the user's retrieval intent are output, ensuring both the privacy and security of family video retrieval and preserving the flexibility of natural language interaction and the relevance of retrieval results.

[0094] Figure 4 This is a schematic diagram of the structure of a video retrieval device provided in an embodiment of the present invention. Figure 4 As shown, the device includes: The acquisition module 31 is used to acquire the query statement input by the user and the user's identity identifier; Semantic parsing module 32 is used to perform semantic parsing on the query statement to obtain the retrieval semantic elements; Privacy level determination module 33 is used to determine the privacy permission level corresponding to the user based on the identity identifier; The retrieval weight determination module 34 is used to determine the retrieval weight of various indexes based on the privacy permission level. The retrieval module 35 is used to perform retrieval processing based on the retrieval semantic elements and the retrieval weights of each index to obtain multiple candidate video segments; The result generation module 36 is used to merge and sort the candidate video segments to obtain the final search results.

[0095] Furthermore, based on the above embodiments of the invention, the semantic parsing module 32 is specifically used for: Generate a search session state and record the identity and query statement in the search session state; Based on a pre-defined set of semantic element types, the query statement is semantically parsed to obtain the retrieved semantic elements and their corresponding element confidence scores; the set of semantic element types includes object elements, location elements, time elements, and event elements; The retrieved semantic elements that are successfully extracted and whose corresponding element confidence is not lower than the preset confidence threshold are recorded as populated fields in the retrieval session status. Search semantic elements that are successfully extracted but whose corresponding element confidence is lower than the preset confidence threshold are recorded as fuzzy fields in the search session status; Record any missing semantic element types in the semantic element type set as missing fields in the retrieval session state.

[0096] Furthermore, based on the above embodiments of the invention, the privacy level determination module 33 is specifically used for: The user's privacy permission level is obtained by querying the pre-configured permission mapping table based on their identity.

[0097] Furthermore, based on the above embodiments of the invention, the retrieval weight determination module 34 is specifically used for: The pre-configured basic weight table is queried according to the privacy permission level to obtain the basic weight of each index, and the basic weight is used as the retrieval weight; the index includes at least vector index, relation index and time sequence index.

[0098] Furthermore, based on the above embodiments of the invention, the retrieval weight determination module 34 is also used for: Determine whether the semantic elements retrieved include location elements; If so, the sensitive areas and their corresponding regional privacy levels are determined based on location elements, the basic weights of each index are adjusted according to the regional privacy levels, and the adjusted basic weights are used as retrieval weights. If not, then the basic weight of each index is maintained as the retrieval weight.

[0099] Furthermore, based on the above embodiments of the invention, the device further includes a clarification interaction module, used for: Before merging and sorting the candidate video segments, if the missing or ambiguous fields in the search session state are not empty, a clarification interaction question is pushed to the user; the clarification interaction question is used to guide the user to complete the missing or ambiguous search semantic elements. Receive the user's initial feedback regarding clarifying interactive issues; Update the filled, missing, and ambiguous fields in the retrieval session status according to the first feedback information; Determine whether both the missing and ambiguous fields are empty after the update; If so, then the steps of performing the retrieval processing based on the updated retrieval semantic elements and the retrieval weights of each index will be re-executed. If not, continue to initiate the next round of clarification interaction until both the missing field and the ambiguous field are empty, or the preset maximum number of clarification rounds is reached.

[0100] Furthermore, based on the above embodiments of the invention, the retrieval weight determination module 34 is also used for: Before re-executing the retrieval processing steps based on the retrieval semantic elements and the retrieval weights of each index, if location elements are added to the updated populated fields, the sensitive areas and their corresponding regional privacy levels are determined based on the added location elements, and the basic weights of each index are adjusted according to the regional privacy levels.

[0101] Furthermore, based on the above embodiments of the invention, the device further includes a sensitive area verification module, used for: Before merging and sorting the candidate video segments, a sensitive area verification request is pushed to the user in response to the preset sensitive area verification trigger condition. The sensitive area verification request is used to ask the user to confirm whether to access the candidate video segment that belongs to the sensitive area. Receive the second feedback information from the user regarding the verification request for sensitive areas; If the second feedback message is a confirmation of access by the user, then the candidate video segments belonging to the sensitive area are retained; If the second feedback message is that the user refuses to confirm, then the candidate video segments belonging to the sensitive area will be filtered out. If the second feedback message indicates that the user refuses to answer, the search will be terminated and the status will be returned as terminated.

[0102] Furthermore, based on the above embodiments of the invention, the preset sensitive area verification trigger condition includes at least one of the following: The regional privacy level of the video capture area to which the candidate video clip belongs is higher than the privacy permission level; The location indicated by the location element in the retrieved semantic elements belongs to a privacy zone or a personal privacy zone; The user visits the video capture area of ​​the candidate video clip for the first time.

[0103] Furthermore, based on the above embodiments of the invention, the result generation module 36 is specifically used for: Obtain the original scores of each candidate video segment under each index; The original scores are normalized to obtain the normalized scores for each candidate video segment; The normalized scores are weighted and summed according to the retrieval weight of each index to obtain the comprehensive score of each candidate video segment. The candidate video segments are sorted according to their comprehensive scores, and the sorted candidate video segments are output as the final search results.

[0104] The video retrieval device provided in the embodiments of the present invention can execute the video retrieval method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of executing the method.

[0105] Figure 5A schematic diagram of an electronic device 40 that can be used to implement embodiments of the present invention is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices (e.g., helmets, glasses, watches, etc.), and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the invention described and / or claimed herein.

[0106] like Figure 5 As shown, the electronic device 40 includes at least one processor 41 and a memory, such as a read-only memory (ROM) 42 or a random access memory (RAM) 43, communicatively connected to the at least one processor 41. The memory stores computer programs executable by the at least one processor. The processor 41 can perform various appropriate actions and processes based on the computer program stored in the ROM 42 or loaded from storage unit 48 into the RAM 43. The RAM 43 may also store various programs and data required for the operation of the electronic device 40. The processor 41, ROM 42, and RAM 43 are interconnected via a bus 44. An input / output (I / O) interface 45 is also connected to the bus 44.

[0107] Multiple components in electronic device 40 are connected to I / O interface 45, including: input unit 46, such as keyboard, mouse, etc.; output unit 47, such as various types of monitors, speakers, etc.; storage unit 48, such as disk, optical disk, etc.; and communication unit 49, such as network card, modem, wireless transceiver, etc. Communication unit 49 allows electronic device 40 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0108] Processor 41 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of processor 41 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. Processor 41 performs the various methods and processes described above, such as video retrieval methods.

[0109] In some embodiments, the video retrieval method may be implemented as a computer program tangibly contained in a computer-readable storage medium, such as storage unit 48. In some embodiments, part or all of the computer program may be loaded and / or installed on electronic device 40 via ROM 42 and / or communication unit 49. When the computer program is loaded into RAM 43 and executed by processor 41, one or more steps of the video retrieval method described above may be performed. Alternatively, in other embodiments, processor 41 may be configured to perform the video retrieval method by any other suitable means (e.g., by means of firmware).

[0110] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0111] In some embodiments, the video retrieval method may be implemented as a computer program, which is implicitly included in a computer program product. When executed by a processor, the computer program implements the video retrieval method of the present invention. The computer program product can be understood as a software product that primarily implements its solution through a computer program. The computer program used to implement the method of the present invention may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the processor, the computer program causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The computer program may be executed entirely on a machine, partially on a machine, partially on a remote machine as a standalone software package, or entirely on a remote machine or server.

[0112] In the context of this invention, a computer-readable storage medium can be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination thereof. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fibers, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof.

[0113] To provide interaction with a user, the systems and techniques described herein can be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the electronic device. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0114] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as data servers), or middleware components (e.g., application servers), or frontend components (e.g., user computers with graphical user interfaces or web browsers through which users can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., communication networks). Examples of communication networks include local area networks (LANs), wide area networks (WANs), blockchain networks, and the Internet.

[0115] A computing system can include clients and servers. Clients and servers are generally located far apart and typically interact through a communication network. The client-server relationship is created by computer programs running on the respective computers and having a client-server relationship with each other. The server can be a cloud server, also known as a cloud computing server or cloud host, which is a hosting product within the cloud computing service system to address the shortcomings of traditional physical hosts and VPS services, such as high management difficulty and weak business scalability.

[0116] It should be understood that the various forms of processes shown above can be used, with steps reordered, added, or deleted. For example, the steps described in this invention can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution of this invention can be achieved, and this is not limited herein.

[0117] The specific embodiments described above do not constitute a limitation on the scope of protection of this invention. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A video retrieval method, characterized in that, The method includes: Obtain the query statement entered by the user and the user's identity identifier; The query statement is semantically parsed to obtain the retrieval semantic elements; The privacy permission level corresponding to the user is determined based on the identity identifier; The retrieval weights of various indexes are determined based on the privacy permission levels described above. The retrieval process is performed based on the retrieval semantic elements and the retrieval weights of each index to obtain multiple candidate video segments; The candidate video segments are fused and sorted to obtain the final search results.

2. The method according to claim 1, characterized in that, The semantic parsing of the query statement to obtain the retrieval semantic elements includes: Generate a retrieval session state, and record the identity identifier and the query statement in the retrieval session state; Based on a preset set of semantic element types, the query statement is semantically parsed to obtain the retrieved semantic elements and their corresponding element confidence scores; the set of semantic element types includes object elements, location elements, time elements, and event elements. The retrieved semantic elements that are successfully extracted and whose corresponding element confidence is not lower than the preset confidence threshold are recorded as the filled fields in the retrieval session status. The retrieved semantic elements that were successfully extracted but whose corresponding element confidence scores were lower than the preset confidence threshold were recorded as fuzzy fields in the retrieval session status. The semantic element types that are not matched in the set of semantic element types are recorded as missing fields in the retrieval session state.

3. The method according to claim 1, characterized in that, The step of determining the privacy permission level corresponding to the user based on the identity identifier includes: The user's privacy permission level is obtained by querying the pre-configured permission mapping table based on the identity identifier.

4. The method according to claim 1, characterized in that, The determination of the retrieval weights for various indexes based on the privacy permission level includes: The pre-configured basic weight table is queried according to the privacy permission level to obtain the basic weight of each index, and the basic weight is used as the retrieval weight; the index includes at least a vector index, a relation index and a time-series index.

5. The method according to claim 4, characterized in that, Also includes: Determine whether the retrieved semantic elements include location elements; If so, the sensitive area and its corresponding regional privacy level are determined according to the location elements, the basic weight of each index is adjusted according to the regional privacy level, and the adjusted basic weight is used as the retrieval weight. If not, then the base weight of each index is maintained as the retrieval weight.

6. The method according to claim 2, characterized in that, Before performing the fusion and sorting of the candidate video segments, the method further includes: If the missing field or the ambiguous field in the search session state is not empty, a clarification interaction question is pushed to the user; the clarification interaction question is used to guide the user to complete the missing or ambiguous search semantic elements; Receive the user's first feedback information regarding the clarification interaction question; Update the filled fields, missing fields, and ambiguous fields in the retrieval session status according to the first feedback information; Determine whether both the missing field and the ambiguous field are empty after the update; If so, then according to the updated retrieval semantic elements, the step of performing the retrieval processing based on the retrieval semantic elements and the retrieval weights of each index shall be re-executed; If not, the next round of clarification interaction will continue until both the missing field and the ambiguous field are empty, or the preset maximum number of clarification rounds is reached.

7. The method according to claim 6, characterized in that, Before re-executing the step of performing the retrieval process based on the retrieval semantic elements and the retrieval weights of each of the indexes, the method further includes: If a location element is added to the updated populated field, the sensitive area and its corresponding regional privacy level are determined based on the added location element, and the basic weight of each index is adjusted according to the regional privacy level.

8. The method according to claim 5, characterized in that, Before performing the fusion and sorting of the candidate video segments, the method further includes: In response to a preset sensitive area verification trigger condition, a sensitive area verification request is pushed to the user; the sensitive area verification request is used to ask the user to confirm whether to access candidate video clips belonging to sensitive areas. Receive second feedback information from the user regarding the verification request for the sensitive area; If the second feedback information is a user confirmation of access, then the candidate video segments belonging to the sensitive area are retained; If the second feedback message indicates that the user refuses to confirm, then candidate video segments belonging to the sensitive area are filtered out. If the second feedback message indicates that the user refuses to answer, the search will be terminated and the status will be returned as terminated.

9. The method according to claim 8, characterized in that, The preset sensitive area verification trigger condition includes at least one of the following: The regional privacy level of the video capture area to which the candidate video segment belongs is higher than the privacy permission level; The location indicated by the location element in the retrieval semantic elements belongs to a privacy area or a personal privacy area; The user visits the video capture area to which the candidate video segment belongs for the first time.

10. The method according to claim 1, characterized in that, The process of fusing and sorting the candidate video segments to obtain the final search results includes: Obtain the original score of each candidate video segment under each index; The original scores are normalized to obtain the normalized scores of the candidate video segments. The normalized scores are weighted and summed according to the retrieval weight of each index to obtain the comprehensive score of each candidate video segment; The candidate video segments are sorted according to the comprehensive score, and the sorted candidate video segments are output as the final search result.

11. A video retrieval device, characterized in that, The device includes: The acquisition module is used to acquire the query statement input by the user and the user's identity identifier; The semantic parsing module is used to perform semantic parsing on the query statement to obtain the retrieval semantic elements; A privacy level determination module is used to determine the privacy permission level corresponding to the user based on the identity identifier; The retrieval weight determination module is used to determine the retrieval weight of each of the various indexes based on the privacy permission level. The retrieval module is used to perform retrieval processing based on the retrieval semantic elements and the retrieval weights of each index to obtain multiple candidate video segments; The result generation module is used to fuse and sort the candidate video segments to obtain the final search results.

12. An electronic device, characterized in that, The electronic device includes: At least one processor; and A memory communicatively connected to the at least one processor; wherein, The memory stores a computer program that can be executed by the at least one processor, the computer program being executed by the at least one processor to enable the at least one processor to perform the video retrieval method according to any one of claims 1-10.

13. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions that cause a processor to execute the video retrieval method according to any one of claims 1-10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor, implements the video retrieval method according to any one of claims 1-10.

Images