Video retrieval method and apparatus, electronic device, and storage medium

Abstract

This disclosure provides a video retrieval method, apparatus, electronic device, and storage medium, relating to the field of computer technology, and particularly to the field of multimodal model technology and video processing technology. The specific implementation scheme is as follows: At least one semantic unit for each query event is determined from the retrieval text, wherein the semantic unit includes at least one of time information, subject information, and behavioral description information; based on the semantic unit, target video frames of the query event are retrieved from a video library, wherein the video library includes multiple video frames, each video frame being associated with a timestamp, subject tag, and visual feature vector; based on the semantics of the retrieval text, the target video frames are analyzed to generate retrieval results.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and more particularly to the field of multimodal modeling technology and video processing technology. More specifically, this disclosure provides a video retrieval method, apparatus, electronic device, storage medium, and computer program product. Background Technology

[0002] With the increasing application of network cameras (IPCs), the demand for accurate retrieval of massive amounts of video data is growing rapidly. Current video retrieval technologies primarily rely on multimodal vector matching or deep analysis of large models. However, traditional multimodal vector models have shallow semantic understanding, making it difficult to handle complex search intents; single-frame retrieval based on large image-text models easily loses temporal correlation information in the video, leading to the omission of long-term continuous actions; and directly using large video models to process the entire dataset faces technical bottlenecks such as extremely high computational costs and slow response times, making large-scale deployment difficult. Therefore, current video retrieval technologies cannot simultaneously achieve deep semantic understanding, high real-time retrieval performance, and low computational deployment costs in massive IPC application scenarios. Summary of the Invention

[0003] This disclosure provides a video retrieval method, apparatus, electronic device, storage medium, and computer program product.

[0004] According to the first aspect, a video retrieval method is provided, the method comprising: determining at least one semantic unit for each of a query event from a retrieval text, wherein the semantic unit includes at least one of time information, subject information, and behavioral description information; retrieving target video frames of the query event from a video library based on the semantic units, wherein the video library includes multiple video frames, each video frame being associated with a timestamp, a subject tag, and a visual feature vector; and analyzing the target video frames based on the semantics of the retrieval text to generate retrieval results.

[0005] According to a second aspect, a video retrieval apparatus is provided, comprising: a determining module for determining at least one semantic unit of a query event from a retrieval text, wherein the semantic unit includes at least one of time information, subject information, and behavioral description information; a retrieval module for retrieving target video frames of the query event from a video library based on the semantic units, wherein the video library includes multiple video frames, each video frame being associated with a timestamp, a subject tag, and a visual feature vector; and an analysis module for analyzing the target video frames based on the semantics of the retrieval text to generate retrieval results.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a method provided according to the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium is provided that stores computer instructions for causing a computer to perform the methods provided in this disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program stored on at least one of a readable storage medium and an electronic device, wherein the computer program, when executed by a processor, implements the method provided in this disclosure.

[0009] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0010] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0011] Figure 1 This is an exemplary system architecture diagram of a video retrieval method and apparatus applicable according to an embodiment of the present disclosure;

[0012] Figure 2 This is a flowchart of a video retrieval method according to an embodiment of the present disclosure;

[0013] Figure 3 This is a flowchart of a video processing method according to an embodiment of the present disclosure;

[0014] Figure 4 This is a flowchart of a video retrieval method according to another embodiment of the present disclosure;

[0015] Figure 5 This is a block diagram of a video retrieval device according to an embodiment of the present disclosure; and

[0016] Figure 6 This is a block diagram of an electronic device for a video retrieval method according to an embodiment of the present disclosure. Detailed Implementation

[0017] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0018] For IPC video retrieval and general video retrieval scenarios, there are currently three main implementation schemes, each based on different technical paths to match videos with retrieval requirements.

[0019] The first type of solution is the multimodal vector model retrieval scheme. This scheme primarily uses a multimodal vector model to extract features from image frames or video clips collected by IPC and the user-input search text, transforming the various data types into high-dimensional vectors. By calculating the similarity between vectors, it selects the image or video clip most similar to the search text vector, thus completing the retrieval match. This scheme mainly focuses on aligning the basic features of text with images / videos and is one of the commonly used lightweight retrieval implementation methods. In some scenarios, it is combined with a vector database to optimize retrieval speed.

[0020] However, this solution has a low level of intelligence, only capable of basic feature matching between text and video / images, and cannot achieve a deep understanding of complex semantic information. For complex retrieval needs common in IPC retrieval scenarios (such as "retrieving video clips of people bending down to pick up items in a certain area"), this solution cannot accurately interpret the semantic associations of actions, scenes, and subjects in the retrieval text, easily leading to matching biases. At the same time, its feature extraction dimensions are limited, resulting in low differentiation between similar scenes and similar subjects, and the retrieval accuracy is insufficient to meet the high precision requirements of scenarios such as security monitoring.

[0021] The second type of approach is the image-text large-scale model retrieval scheme. This scheme relies on an image-text large-scale model for retrieval. The core of this scheme is to break down IPC videos into individual frames, use the large-scale model to achieve semantic matching between these individual frames and the search text, and then integrate the matching results to obtain the video retrieval conclusion. This scheme focuses on improving the matching accuracy between individual frames and text, leveraging the semantic understanding capabilities of the large-scale model to compensate for the shortcomings of traditional vector models. It is suitable for retrieval scenarios with high requirements for the features of individual frames.

[0022] However, this approach is essentially based on single-frame image retrieval, completely failing to understand the temporal information of the video. It can only capture static features in a single frame and cannot reconstruct the complete process of an event. In IPC retrieval scenarios, many critical events require long-term video changes for identification (such as people continuously following or objects being moved multiple times). This approach will miss such long-term events due to the loss of temporal correlation information, resulting in extremely poor retrieval completeness. Furthermore, breaking down video into single frames increases data processing volume, indirectly affecting retrieval efficiency, and fails to reflect the core characteristics of video as dynamic content.

[0023] The third type of solution is the video-text large-scale model retrieval solution. This solution directly uses a large video-text model to process complete video segments captured by IPC end-to-end, without needing to break them down into individual frames. It directly achieves semantic alignment and matching between video content and searched text. This solution attempts to balance intra-frame features and basic temporal information of the video, relying on the deep semantic understanding capabilities of the large model to improve the accuracy of search results. It is mainly applied to niche scenarios with extremely high requirements for search accuracy and low sensitivity to cost and speed.

[0024] While this solution can balance video semantic understanding and temporal information processing, it suffers from fatal flaws when dealing with massive video inputs in IPC scenarios: extremely high cost, slow speed, and unstable results. On one hand, training and inference of the large video-text model requires significant computing resources, making large-scale deployment difficult due to the high computational costs. On the other hand, the large model's processing speed for video data is slow, failing to meet the core real-time retrieval requirements of IPC query systems. Furthermore, when faced with IPC videos of varying resolutions and shooting environments (such as nighttime or backlighting), the semantic understanding accuracy fluctuates significantly, resulting in insufficient stability of retrieval performance and making it unsuitable for complex and ever-changing IPC application scenarios.

[0025] In summary, none of the three existing technical solutions can balance the core requirements of IPC query systems for real-time retrieval, semantic understanding depth, retrieval accuracy, and low cost. They have obvious technical shortcomings and urgently need a new video retrieval technology solution adapted to IPC scenarios.

[0026] The collection, storage, use, processing, transmission, provision, and disclosure of any type of information, such as user personal information, in this technical solution comply with relevant laws and regulations and do not violate public order and good morals.

[0027] In the technical solution disclosed herein, the user's authorization or consent is obtained before acquiring or collecting the user's personal information.

[0028] Figure 1This is a schematic diagram of an exemplary system architecture for applying video retrieval methods and apparatus according to an embodiment of this disclosure. It should be noted that... Figure 1 The examples shown are merely examples of system architectures that can be applied to the embodiments of this disclosure, in order to help those skilled in the art understand the technical content of this disclosure, but do not mean that the embodiments of this disclosure cannot be used in other devices, systems, environments or scenarios.

[0029] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, and 103, a network 104, and a server 105. The network 104 serves as a medium for providing a communication link between the terminal devices 101, 102, and 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0030] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Terminal devices 101, 102, and 103 can be various electronic devices, including but not limited to smartphones, tablets, laptops, etc.

[0031] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using terminal devices 101, 102, and 103 (for example only). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as determining the target video frame from the video library based on the search text, or the output result determined by semantic analysis of the search text and the target video frame) to the terminal devices.

[0032] The video retrieval method provided in this embodiment can generally be executed by server 105. Accordingly, the video retrieval device provided in this embodiment can generally be installed in server 105.

[0033] Figure 2 This is a flowchart of a video retrieval method according to an embodiment of the present disclosure.

[0034] like Figure 2 As shown, the video retrieval method 200 includes operations S210 to S230.

[0035] In operation S210, at least one semantic unit for each of the events to be queried is determined from the retrieved text, wherein the semantic unit includes at least one of time information, subject information, and behavioral description information.

[0036] In this embodiment of the disclosure, the search text may refer to the natural language query text input by the user, which may come from the user's text input or text converted from the user's speech. The query event may refer to the independent events included in the search text. For example, "Did the child do homework or watch TV today?" includes two query events: the child did homework today and the child watched TV today. The semantic unit may refer to information extracted from the query time for video retrieval.

[0037] Specifically, the search text can be segmented and semantically parsed to extract at least one query event. By extracting information from each query event, semantic units including time information, subject information, and behavioral description information can be obtained.

[0038] In operation S220, the target video frame of the event to be queried is retrieved from the video library based on semantic units.

[0039] In this embodiment of the disclosure, the video library includes multiple video frames, each video frame being associated with a timestamp, a subject tag, and a visual feature vector.

[0040] In this embodiment, the video library can refer to a structured database that stores multiple video frames and their associated information. For example, this disclosure detects real-time video frames acquired by an IPC device. If the detected real-time video frame meets the triggering conditions for video recording, video recording can be performed, for example, recording the video from 1 second before the frame to 10 seconds after the frame as a video segment. Video frames are then extracted from the video segment, and the associated information between the video frames is stored in the video library. This associated information includes timestamps, subject tags, and visual feature vectors, and may also include video IDs, IPC device numbers, video frame IDs, etc. The video ID can be used to retrieve the corresponding video from the cloud, the IPC device number can be used to query the video frame recorded by the IPC device corresponding to that IPC device in the video library, and the video frame ID is used to index the video frame.

[0041] During the retrieval process, semantic units can be matched with the association information of video frames in the video library to retrieve the target video frames for the query event. For example, video frames that match time information with timestamps, subject information with subject tags, and behavioral description information with visual feature vectors can be used as target video frames.

[0042] In operation S230, the target video frame is analyzed based on the semantics of the retrieved text to generate retrieval results.

[0043] In this embodiment of the disclosure, a multimodal model can be used to analyze the target video frame based on the semantics of the retrieved text to generate search results. For example, if the search text is "Did your child watch TV today?", the target video frame could be a video frame showing the child watching TV, and the corresponding search result could be the time period during which the child watched TV today, along with a link to the video clip stored in the cloud.

[0044] This embodiment of the disclosure reduces the computational pressure of online retrieval and reduces the storage space occupied by redundant data by searching in a video library that includes video frames and related information. By searching in the video library using the semantic units of the event to be queried, it is possible to achieve accurate semantic matching between text queries and video content, thereby improving the accuracy of retrieval results.

[0045] According to embodiments of this disclosure, retrieving target video frames of a query event from a video library based on semantic units includes: determining a first candidate video frame set from the video library based on time information and the timestamp of each video frame in the video library; determining a second candidate video frame set from the first candidate video frame set based on subject information and the subject tag of each video frame in the first candidate video frame set; and determining the target video frame from the second candidate video frame set based on behavior description information and the visual feature vector of each video frame in the second candidate video frame set.

[0046] In this embodiment of the disclosure, the association information of video frames in the video library can be filtered at multiple levels based on semantic units. The multiple levels of filtering can include time range filtering, subject tag filtering, and image-text similarity filtering.

[0047] Specifically, time range filtering refers to retrieving video frames from a video library whose timestamps fall within the time range indicated by the time information, filtering out data irrelevant to the time information, and initially narrowing down the data range. For example, if the time information is "today," then the corresponding time range could be from 00:00 to 24:00 on [date]. Through time range filtering, a first candidate video frame set can be determined from the video library, and the first candidate video frame set includes at least one first candidate video frame.

[0048] Subject tag filtering refers to filtering first candidate video frames based on subject information and subject tags associated with them. For example, if the subject information is "pet," video frames whose subject tag is not "pet" can be filtered out. Through subject tag filtering, a second set of candidate video frames can be determined from the first set of candidate video frames, wherein the second set of candidate video frames includes at least one second candidate video frame.

[0049] Image-text similarity filtering can be achieved by vectorizing behavioral description information to obtain behavioral feature vectors, calculating the similarity between the behavioral feature vectors and visual feature vectors, selecting second candidate video frames whose similarity meets the similarity threshold as target video frames, and then determining the target video frame from the set of second candidate video frames through image-text similarity filtering.

[0050] This disclosure embodiment reduces the amount of data retrieved layer by layer through multi-level filtering, improves the efficiency of subsequent processing, and enhances the retrieval speed and accuracy, thus meeting the needs of efficient retrieval of massive amounts of IPC videos.

[0051] According to embodiments of this disclosure, retrieving the target video frame of the query event from the video library based on the semantic unit further includes: in response to detecting that the semantic unit lacks at least one of time information, subject information, and behavioral description information, obtaining historical search text associated with the search text; extracting historical information from the historical search text to supplement the semantic unit; and retrieving the target video frame of the query event from the video library based on the supplemented semantic unit.

[0052] In this embodiment of the disclosure, if there is a missing semantic unit, it can be supplemented based on historical information. The historical search text associated with the search text can refer to the historical search text initiated by the same user, or it can refer to the historical search text whose reception time is close to that of the search text.

[0053] For example, if the query event is "Have you eaten?", the semantic unit determined from the query event is the behavioral description information of eating, lacking subject information and time information. Suppose the retrieved historical search text is: "Did your child watch TV today?", the extracted historical information includes "child" (subject information), "today" (time information), and "watching TV" (behavioral description information). Based on the historical information, the semantic information of the query event is supplemented, and the supplemented semantic units include "child" (subject information), "today" (time information), and "eating" (behavioral description information).

[0054] This embodiment of the disclosure, by supplementing semantic units, can avoid the problems of missing semantic units and incomplete elements in the search text, ensure that multi-level filtering can be executed normally, and improve the accuracy of retrieval.

[0055] According to embodiments of this disclosure, determining a target video frame from a second candidate video frame set includes: vectorizing behavior description information to generate behavior feature vectors; and determining a target video frame from the second candidate video frame set based on the behavior feature vectors and the visual feature vectors of each video frame in the second candidate video frame set.

[0056] In this embodiment of the disclosure, behavioral description information can be vectorized using a graph-text vector generation model. This model can also be used to generate visual feature vectors for video frames. Both behavioral feature vectors and visual feature vectors are, for example, 1536-dimensional normalized vectors.

[0057] After vectorization, the cosine similarity between the behavioral feature vector and the visual feature vector can be calculated. Then, video frames with a similarity greater than the similarity threshold are used as target video frames to achieve semantic alignment between behavioral and visual information.

[0058] The embodiments of this disclosure can improve the accuracy of retrieval by performing cross-modal semantic alignment of behavioral description information and visual feature vectors between text and vision.

[0059] According to embodiments of this disclosure, the method further includes: determining video frames that meet the video recording trigger conditions from video stream data, and recording video segments within a preset time period based on the timestamps of the video frames that meet the video recording trigger conditions; determining keyframes from the video segments based on the degree of difference between each video frame in the video segment; performing subject detection on the keyframes to determine the subject labels of the keyframes; performing feature encoding on the keyframes to determine the visual feature vectors of the keyframes; and associating and storing the keyframes, their timestamps, subject labels, and visual feature vectors in a video library.

[0060] In this embodiment of the disclosure, video stream data can refer to video stream data collected in real time by the IPC device. The video recording trigger condition can be used as a rule to determine whether to start recording. For example, it can be achieved through real-time detection, i.e., obtaining real-time video frames and determining whether the real-time video frames meet the trigger condition. If the real-time video frames are detected to meet the trigger condition, then video recording is triggered.

[0061] Specifically, video recording can be based on the real-time video frame, and the recording duration can be set to an extended mode of "1 second before trigger - 10 seconds after trigger" to avoid missing key actions. For example, if the timestamp of the video frame is 09:06:25, then the recording time segment can be from 09:06:24 to 09:06:35.

[0062] After acquiring the video clip, the video frames with the greatest differences can be selected as keyframes based on the degree of difference between the various video frames in the video clip, such as the degree of difference in pixel changes.

[0063] After determining the keyframes, a lightweight subject detection model with zero-shot detection capability can be used to detect subjects in the keyframes and output subject labels (including people, pets, cats, dogs, etc., with a customizable labeling system). The output can also include subject bounding boxes to determine whether a subject exists within the bounding box. The model also outputs the recognition confidence score for each subject; subject labels with confidence scores greater than a threshold are retained, while those with confidence scores less than the threshold are not. The pre-trained model used can be deployed in the cloud.

[0064] In addition, this disclosure also performs feature encoding on keyframes to determine the visual feature vector of the keyframes. For example, a multimodal feature extraction model can be used to first encode the keyframes to generate a 1536-dimensional image vector. Then, the image vector is normalized to obtain the visual feature vector, for example, by normalizing based on the L2 norm.

[0065] After determining the subject label and visual feature vector, the keyframes, along with their timestamps, subject labels, and visual feature vectors, can be associated and stored in a structured video library that can be searched online.

[0066] This embodiment of the disclosure achieves lightweight processing and structured construction of video data through triggered recording, keyframe extraction, subject detection, feature encoding, and structured associated storage. This avoids storage pressure caused by redundant video data, reduces computational pressure and latency for online retrieval, and improves retrieval speed.

[0067] According to embodiments of this disclosure, determining a video frame that meets the video recording triggering conditions from video stream data includes: for any video frame in the video stream data, in response to a pixel difference between the video frame and a video frame preceding the video frame being greater than a first preset threshold, determining the video frame as a video frame that meets the video recording triggering conditions.

[0068] In this embodiment of the disclosure, video frames in video stream data can be detected in real time. Specifically, the frame difference method can be used to detect whether a video frame meets the triggering conditions for video recording. That is, the pixel difference between the real-time video frame and the previous video frame can be calculated. When the pixel difference is detected to be greater than a first preset threshold, the real-time video frame is a video frame that meets the triggering conditions for video recording.

[0069] This embodiment of the disclosure uses the frame difference method to detect screen motion, which can quickly and accurately determine whether there is obvious screen motion in the video frame, avoid redundant video data caused by invalid recording and reduce storage pressure.

[0070] According to embodiments of this disclosure, the method further includes: obtaining multiple historical pixel differences for historical video frames, and calculating the mean and standard deviation of the multiple historical pixel differences; and adjusting a first preset threshold based on the mean and standard deviation of the multiple historical pixel differences.

[0071] In this embodiment of the disclosure, the first preset threshold can be a dynamically adjusted threshold. Specifically, multiple historical pixel differences for historical video frames can be obtained. For example, the first preset threshold can be adjusted every hour. The historical pixel differences of each historical video frame within one hour before the current time can be obtained, and then the mean and standard deviation of the multiple historical pixel differences can be calculated. Then, the first preset threshold can be adjusted based on the mean and standard deviation of the multiple historical pixel differences.

[0072] For example, it can be done according to The threshold is adjusted in the following manner, where, This represents the first preset threshold. These are the weighting coefficients. This represents the mean of historical pixel differences. This represents the adjustment coefficient. The standard deviation of historical pixel differences.

[0073] This embodiment of the present disclosure can improve the ability to resist light interference by adjusting the first preset threshold, and can still accurately determine the presence of abnormalities in the image under scenarios such as light changes at different times and static interference in the image.

[0074] According to embodiments of this disclosure, determining a video frame that meets the video recording triggering conditions from video stream data includes: for any video frame in the video stream data, in response to the probability that a target subject appears in the video frame being greater than a second preset threshold, determining the video frame as a video frame that meets the video recording triggering conditions.

[0075] In this embodiment of the disclosure, the target subject can be a pre-defined object to be monitored, such as a cat, dog, or person. Specifically, a lightweight convolutional neural network model can be used to infer the content of video frames to detect whether a target subject exists in the video frame and output the probability of the target subject appearing in the video frame.

[0076] If a target subject is detected in a video frame and the probability of the target subject appearing is greater than the second preset threshold, the video frame can be considered to meet the video recording trigger condition.

[0077] This embodiment of the disclosure uses a lightweight model to detect target subjects in video frames. When the probability of a target subject appearing is greater than a second preset threshold, the video is recorded, which can reduce the storage of invalid data in the video library and reduce storage pressure.

[0078] According to embodiments of this disclosure, determining keyframes from a video segment based on the degree of difference between each video frame in the video segment includes: for each video frame in the video segment, calculating the variance of pixel grayscale values ​​between the video frame and adjacent video frames as an evaluation index of the degree of difference between the video frame and adjacent video frames; and in response to the evaluation index being greater than a third preset threshold, and the evaluation index of the video frame being greater than the evaluation index of the video frame's adjacent video frames, designating the video frame as a keyframe.

[0079] In this embodiment of the disclosure, for a video segment, a frame splitting tool can be used to split the video segment into frames. The frame splitting frame rate can be set to 1fps. Through frame splitting, multiple video frames included in the video segment can be obtained.

[0080] For each video frame in a video segment, keyframes can be extracted using peak detection. Specifically, the variance of pixel grayscale values ​​between a video frame and its adjacent video frames can be calculated. Adjacent video frames are, for example, the three frames before and after the video frame in the time series. The larger the variance, the more obvious the pixel differences between the frames, indicating more obvious changes in motion. Therefore, the variance of pixel grayscale values ​​can be used as an evaluation index of the degree of difference between a video frame and its adjacent video frames.

[0081] If the evaluation index is greater than the third preset threshold, the evaluation index can be compared with the evaluation index of the adjacent video frame. If the evaluation index is greater than the evaluation index of the adjacent video frame, the video frame can be considered as the peak frame in the local sequence and the video frame is used as the key frame.

[0082] Compared to methods that extract keyframes at fixed intervals (e.g., extracting 1 frame every 10 frames), the embodiments disclosed herein are less likely to miss key scenes with dense action and can avoid extracting too many redundant keyframes, thus improving retrieval accuracy and storage efficiency. Compared to schemes that use inter-frame similarity to determine keyframes, the peak detection method uses intra-frame pixel differences as the core evaluation criterion, which has stronger resistance to light interference, can more accurately capture the peak moments of action changes, and effectively compresses the number of keyframes, reducing data processing volume for subsequent online retrieval and improving retrieval speed.

[0083] According to embodiments of this disclosure, analyzing target video frames based on the semantics of the retrieved text to generate search results includes: performing intent analysis on the query event to determine the intent type of the query event; generating a target video based on the target video frame and its associated video frames in response to the intent type being a summary type; performing at least one of subject trajectory analysis and subject behavior analysis on the target video according to the semantics of the query event to generate an output result corresponding to the query event; and integrating at least one output result corresponding to at least one query event to generate search results.

[0084] In this embodiment of the disclosure, intent analysis can be performed on the query event to determine the intent type of the query event. The intent type can include summary type and retrieval type. The difference between summary type events and ordinary retrieval type events is whether there is a specific event. If there is no specific event, it is a summary type event, such as what happened today and what happened yesterday. If there is a specific event, it is an ordinary retrieval type event, such as how long the child watched TV or how many times the child played on the phone today.

[0085] If the intent type of the event to be queried is a summary type, since summary events usually cannot generate the required results from a single image frame, it is necessary to generate the target video based on the target video frame and its associated video frames. The associated video frames are those that are close to the target video frame in the time sequence. The target video frame and the associated video frames are then recombined according to their original time sequence to obtain the target video.

[0086] After generating the target video, at least one of the following can be performed on the target video based on the semantics of the event to be queried: subject trajectory analysis and subject behavior analysis. Subject trajectory analysis can be used to analyze the subject's location, movement route, area of ​​stay, and duration. Subject behavior analysis can be used to identify the subject's behavior type, frequency, and start and end times. The output corresponding to the event to be queried can include event statistics such as the number of times the subject's behavior occurred, the subject's activity trajectory, and key time nodes such as the start and end times of the subject's behavior.

[0087] After obtaining at least one output result corresponding to at least one query event, the output results can be integrated according to information such as time sequence, and duplicate content can be removed and similar information can be merged to generate search results.

[0088] This disclosure embodiment, by merging target video frames and associated video frames for analysis in summary-type events, avoids the deficiency that a single frame image cannot accurately reflect the search results, improves the comprehensiveness and accuracy of the search results, and enhances the summarization capability in complex monitoring scenarios.

[0089] According to embodiments of this disclosure, the analysis of target video frames based on the semantics of the retrieved text and the generation of retrieval results further include: in response to the fact that the intent type of at least one query event is a retrieval type, for each query event, based on the semantics of the query event, outputting at least one of the following: video segment link, behavioral description information, timestamp, and integrity score of the target video frame corresponding to the query event.

[0090] In this embodiment of the disclosure, if at least one query event is a retrieval type, such as a query event that is to accurately find a specific behavior, image, or event without needing to summarize the whole, then the semantics of the query event can be analyzed to output the output result corresponding to the query event. The output result may include at least one of the following: video segment link of the target video frame, behavior description information, timestamp, and integrity score.

[0091] Among them, video clip links can refer to the links of video clips to which the target video frame belongs in the cloud, behavioral description information can be a textual description of the main behavior in the video frame, and completeness score can be used to evaluate whether the search results are complete.

[0092] Understandably, the integrated search results for the query event will also include at least one of the following: video clip links, behavioral descriptions, timestamps, and completeness scores. Furthermore, the output format supports both text and video clip lists to accommodate different user needs.

[0093] According to embodiments of this disclosure, integrating at least one output result corresponding to at least one query event to generate search results includes: in response to the existence of multiple output results, sorting the multiple output results according to the event occurrence time of each of the multiple output results; and merging the sorted multiple output results based on the consistency of subject behavior information to generate search results.

[0094] In this embodiment of the disclosure, if there are multiple output results, they need to be integrated to generate search results. Specifically, the multiple output results can be sorted according to the event occurrence time in each output result. The event occurrence time can be a timestamp related to the output result, such as the timestamp of a video frame related to the output result, or time information extracted from the text description of the output result. For example, if the output result is "Child watches TV at 10:00", the extracted time information is "10 o'clock". For the sorted multiple output results, the main behavior information can be extracted from each output result. Then, the results are merged based on the consistency of the main behavior information, that is, it is determined whether there are output results with consistent main behavior information. The output results with consistent main behavior information are merged to generate search results.

[0095] For example, the sorted output results include: the child does homework at 10:00, does homework at 10:10, does homework at 10:20, and does homework at 10:30. The main behavioral information of these four output results is consistent: the child does homework. Therefore, these four output results can be merged. The merged search result can be: the child does homework from 10:00 to 10:30.

[0096] The embodiments of this disclosure sort the output results based on the event occurrence information and merge them based on the consistency of the subject behavior information to obtain concise and logically coherent search results, thereby improving the user experience.

[0097] Figure 3 This is a flowchart of a video processing method according to an embodiment of the present disclosure.

[0098] like Figure 3 As shown, the video processing method 300 includes operations S310 to S350.

[0099] In operation S310, in response to detecting that a real-time acquired video frame meets the preset trigger conditions, a video segment within a preset time period is recorded based on the timestamp of the video frame, and the video segment is uploaded to the cloud.

[0100] In this embodiment of the disclosure, the detection of video frames may include two types: anomaly detection and subject detection. The corresponding preset triggering conditions may include at least one of the following: the pixel difference between the video frame and the previous video frame is greater than a first preset threshold, and the probability of a target subject appearing in the video frame is greater than a second preset threshold.

[0101] If the preset trigger conditions are met, the video frame can be considered a relatively important video frame. For example, if the video frame contains an object that needs to be monitored or there is a relatively obvious screen movement, then video segments within a preset time period can be recorded based on the timestamp of the video frame and uploaded to the cloud.

[0102] In operation S320, for each video frame in the video segment, the variance of the pixel grayscale value between the video frame and its neighboring video frames is calculated as an evaluation index of the degree of difference between the video frame and its neighboring video frames.

[0103] In this embodiment of the disclosure, the variance of pixel grayscale values ​​between a video frame and its adjacent video frames is calculated. The larger the variance, the more obvious the pixel differences between frames, indicating that the action changes are more obvious. Therefore, the variance of pixel grayscale values ​​can be used as an evaluation index of the degree of difference between a video frame and its adjacent video frames.

[0104] In operation S330, in response to the evaluation index being greater than a third preset threshold, and the evaluation index of the video frame being greater than the evaluation index of the adjacent video frames, the video frame is designated as a key frame.

[0105] In this embodiment of the disclosure, if the evaluation index of a video frame is greater than a third preset threshold, it is necessary to further determine whether the video frame is a peak frame in the local sequence, that is, to determine whether the evaluation index of the video frame is greater than the evaluation index of the adjacent video frames. For example, there are three frames before and after the adjacent video frames. If it is further determined that the evaluation index of the video frame is greater than the evaluation index of the adjacent video frames, then the video frame can be regarded as a key frame.

[0106] In operation S340, subject detection is performed on keyframes to determine the subject labels of the keyframes; feature encoding is performed on keyframes to determine the visual feature vectors of the keyframes.

[0107] In this embodiment of the disclosure, a lightweight subject detection model with zero-sample detection capability can be used to detect the subject and obtain the subject label of the key frame. A multimodal feature extraction model can be used to encode and normalize the key frame to obtain the visual feature vector of the key frame.

[0108] When operating the S350, keyframes, along with their timestamps, subject labels, and visual feature vectors, are associated and stored in the video library.

[0109] In this embodiment of the disclosure, in addition to the keyframe timestamp, subject tag and visual feature vector, information such as IPC device number, keyframe ID and video ID can also be associated with the keyframe and stored in the video library as association information.

[0110] Figure 4 This is a flowchart of a video retrieval method according to another embodiment of the present disclosure.

[0111] like Figure 4 As shown, the video retrieval method 400 includes operations S410 to S480.

[0112] In operation S410, at least one query event is split from the retrieved text, and the semantic unit of each query event is determined.

[0113] The semantic unit can include at least one of time information, subject information, and behavioral description information. Specifically, the search text can be split into events, and information can be extracted from the split events to determine the semantic unit for each query event. If a semantic unit is missing, such as time information, it can be supplemented based on historical search text.

[0114] In operation S420, for each query event, the first candidate video frame set is determined from the video library based on the time information and the timestamp of each video frame in the video library.

[0115] Specifically, based on time information and the timestamp of each video frame in the video library, the video frame whose timestamp corresponds to the time information can be retrieved from the video library (for example, the time represented by the timestamp is within the time range represented by the time information), and data that is irrelevant to the time information can be filtered out.

[0116] In operation S430, based on the subject information and the subject tag of each video frame in the first candidate video frame set, a second candidate video frame set is determined from the first candidate video frame set.

[0117] For example, if the subject information is a cat, then for at least one first candidate video frame included in the first candidate video frame set, first candidate video frames whose subject label is not a cat can be filtered out to obtain a second candidate video frame set, wherein the second candidate video frame set includes at least one second candidate video frame.

[0118] In operation S440, the target video frame is determined from the second candidate video frame set based on the behavior description information and the visual feature vector of each video frame in the second candidate video frame set.

[0119] Specifically, the behavioral description information can be vectorized to obtain behavioral feature vectors, and the similarity between the behavioral feature vectors and the visual feature vectors can be calculated. The second candidate video frame whose similarity meets the similarity threshold is taken as the target video frame.

[0120] In operation S450, determine whether the event to be queried is a summary event.

[0121] Specifically, intent analysis can be performed on the query event to determine the event type. If the query event is not a summary event, operation S460 is executed; if the query event is a summary event, operation S470 is executed.

[0122] In operation S460, output results corresponding to the query event are generated based on the target video frame.

[0123] Specifically, multimodal joint semantic analysis can be performed on the query event and the target video frame to generate output results corresponding to the query event. For example, if the query event is "Did the child do homework today?", and the target video frame includes multiple scenes of children doing homework, the output result could be "The child did homework from XX time to XX time today (the specific time can be determined based on the timestamp of the target video frame)", and provide a link to the video segment where the target video frame is located.

[0124] In operation S470, a target video is generated based on the target video frame and its associated video frames, and an output result corresponding to the query event is generated based on the target video.

[0125] Specifically, for summary-type events, video frames that are similar to the target video frame in the time series can be used as associated video frames. The target video frame and associated video frames are then integrated and combined to obtain the target video. Cross-modal joint semantic analysis is then performed on the target video and the event to be queried to generate the output results corresponding to the event to be queried.

[0126] In operation S480, the output results corresponding to each query event are integrated to generate the output results corresponding to the retrieval task.

[0127] Specifically, after obtaining the output results for each query event, the output results can be integrated according to information such as time sequence, duplicate content can be removed, and similar information can be merged to generate search results corresponding to the search task.

[0128] According to embodiments of this disclosure, this disclosure also provides a video retrieval device.

[0129] Figure 5 This is a block diagram of a video retrieval device according to an embodiment of the present disclosure.

[0130] like Figure 5 As shown, the video retrieval device 500 includes a determining module 510, a retrieval module 520, and an analysis module 530. The determining module is used to determine at least one semantic unit for each query event from the retrieval text, wherein the semantic unit includes at least one of time information, subject information, and behavioral description information; the retrieval module is used to retrieve target video frames of the query event from a video library based on the semantic units, wherein the video library includes multiple video frames, each video frame being associated with a timestamp, a subject tag, and a visual feature vector; the analysis module is used to analyze the target video frames based on the semantics of the retrieval text and generate retrieval results.

[0131] According to embodiments of this disclosure, the retrieval module includes a first candidate video frame set determination module, a second candidate video frame set determination module, and a target video frame determination module. The first candidate video frame set determination module determines a first candidate video frame set from the video library based on time information and the timestamp of each video frame in the video library. The second candidate video frame set determination module determines a second candidate video frame set from the first candidate video frame set based on subject information and the subject tag of each video frame in the first candidate video frame set. The target video frame determination module determines a target video frame from the second candidate video frame set based on behavior description information and the visual feature vector of each video frame in the second candidate video frame set.

[0132] According to embodiments of this disclosure, the retrieval module 520 further includes a historical retrieval text acquisition module, a supplementation module, and a retrieval submodule. The historical retrieval text acquisition module is used to acquire historical retrieval text associated with the retrieval text in response to detecting that the semantic unit lacks at least one of time information, subject information, and behavioral description information. The supplementation module is used to extract historical information from the historical retrieval text to supplement the semantic unit. The retrieval submodule is used to retrieve the target video frame of the event to be queried from the video library based on the supplemented semantic unit.

[0133] According to embodiments of this disclosure, the target video frame determination module includes a vectorization processing module and a target video frame determination submodule. The vectorization processing module is used to perform vectorization processing on the behavior description information to generate behavior feature vectors; the target video frame determination submodule is used to determine the target video frame from the second candidate video frame set based on the behavior feature vectors and the visual feature vectors of each video frame in the second candidate video frame set.

[0134] According to embodiments of this disclosure, the video retrieval device 500 further includes a video frame determination module, a recording module, a key frame determination module, a subject detection module, a feature encoding module, and an association storage module. The video frame determination module is used to determine video frames from video stream data that meet video recording trigger conditions; the recording module is used to record video segments within a preset time period based on the timestamps of the video frames that meet the video recording trigger conditions; the key frame determination module is used to determine key frames from the video segments based on the degree of difference between each video frame in the video segment; the subject detection module is used to perform subject detection on the key frames to determine the subject tags of the key frames; the feature encoding module is used to perform feature encoding on the key frames to determine the visual feature vectors of the key frames; and the association storage module is used to associate and store the key frames, their timestamps, subject tags, and visual feature vectors in a video library.

[0135] According to an embodiment of the present disclosure, the video frame determination module includes a first video frame determination module, which is used to determine, for any video frame in the video stream data, a video frame that satisfies the video recording triggering condition in response to a pixel difference between the video frame and a video frame preceding the video frame being greater than a first preset threshold.

[0136] According to an embodiment of this disclosure, the video retrieval device 500 further includes a processing module and an adjustment module. The processing module is used to acquire multiple historical pixel differences for historical video frames and calculate the mean and standard deviation of the multiple historical pixel differences. The adjustment module is used to adjust a first preset threshold based on the mean and standard deviation of the multiple historical pixel differences.

[0137] According to an embodiment of this disclosure, the video frame determination module further includes a second video frame determination module, which is used to determine, for any video frame in the video stream data, that the probability of a target subject appearing in the video frame is greater than a second preset threshold, that the video frame meets the video recording triggering condition.

[0138] According to embodiments of this disclosure, the keyframe determination module includes a calculation module and a keyframe determination submodule. The calculation module is used to calculate the variance of pixel grayscale values ​​between each video frame and its neighboring video frames for each video frame in a video segment, as an evaluation index of the degree of difference between the video frame and its neighboring video frames; the keyframe determination submodule is used to determine the video frame as a keyframe in response to the evaluation index being greater than a third preset threshold and the evaluation index of the video frame being greater than the evaluation index of its neighboring video frames.

[0139] According to embodiments of this disclosure, the analysis module 530 includes an intent type determination module, a target video generation module, an output result generation module, and a search result generation module. The intent type determination module performs intent analysis on the query event to determine the intent type of the query event; the target video generation module generates a target video based on the target video frame and its associated video frames in response to the intent type being a summary type; the output result generation module performs at least one of subject trajectory analysis and subject behavior analysis on the target video according to the semantics of the query event, generating an output result corresponding to the query event; and the search result generation module integrates at least one output result corresponding to at least one query event to generate a search result.

[0140] According to embodiments of this disclosure, the retrieval result generation module further includes an output module, which is configured to respond to at least one of the following in response to the intent type of at least one query event being a retrieval type: for each query event, based on the semantics of the query event, output at least one of the following: a video segment link, behavioral description information, timestamp, and integrity score of the target video frame corresponding to the query event.

[0141] According to embodiments of this disclosure, the retrieval result generation module further includes a sorting module and a retrieval result generation submodule. The sorting module is used to sort the multiple output results according to their respective event occurrence times in response to the existence of multiple output results; the retrieval result generation submodule is used to merge the sorted output results based on the consistency of the subject's behavioral information to generate retrieval results.

[0142] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0143] Figure 6A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0144] like Figure 6 As shown, device 600 includes a computing unit 601, which can perform various appropriate actions and processes based on a computer program stored in read-only memory (ROM) 602 or a computer program loaded into random access memory (RAM) 603 from storage unit 608. RAM 603 may also store various programs and data required for the operation of device 600. The computing unit 601, ROM 602, and RAM 603 are interconnected via bus 604. Input / output (I / O) interface 605 is also connected to bus 604.

[0145] Multiple components in device 600 are connected to I / O interface 605, including: input unit 606, such as keyboard, mouse, etc.; output unit 607, such as various types of monitors, speakers, etc.; storage unit 608, such as disk, optical disk, etc.; and communication unit 609, such as network card, modem, wireless transceiver, etc. Communication unit 609 allows device 600 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0146] The computing unit 601 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 601 performs the various methods and processes described above, such as video retrieval methods. For example, in some embodiments, the video retrieval method may be implemented as a computer software program tangibly contained in a machine-readable medium, such as storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the video retrieval method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the video retrieval method by any other suitable means (e.g., by means of firmware).

[0147] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0148] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0149] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0150] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0151] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0152] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact through communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other.

[0153] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0154] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A video retrieval method, comprising: Determine at least one semantic unit for each of the events to be queried from the retrieved text, wherein the semantic unit includes at least one of time information, subject information, and behavioral description information; Based on the semantic unit, the target video frame of the query event is retrieved from the video library, wherein the video library includes multiple video frames, and each video frame is associated with a timestamp, a subject tag, and a visual feature vector; Based on the semantics of the search text, the target video frame is analyzed to generate search results.

2. The method according to claim 1, wherein retrieving the target video frame of the query event from the video library based on the semantic unit comprises: Based on the time information and the timestamp of each video frame in the video library, a first candidate video frame set is determined from the video library; Based on the subject information and the subject tag of each video frame in the first candidate video frame set, a second candidate video frame set is determined from the first candidate video frame set; The target video frame is determined from the second candidate video frame set based on the behavior description information and the visual feature vector of each video frame in the second candidate video frame set.

3. The method according to claim 2, wherein, The step of retrieving the target video frame of the query event from the video library based on the semantic unit further includes: In response to detecting that the semantic unit lacks at least one of time information, subject information, and behavioral description information, historical search text associated with the search text is obtained; Historical information is extracted from the historical search text to supplement the semantic unit; The target video frame of the query event is retrieved from the video library based on the supplemented semantic units.

4. The method according to claim 2, wherein, Determining the target video frame from the second candidate video frame set includes: The behavioral description information is vectorized to generate behavioral feature vectors; The target video frame is determined from the second candidate video frame set based on the behavioral feature vector and the visual feature vector of each video frame in the second candidate video frame set.

5. The method according to claim 1, further comprising: The video frames that meet the video recording trigger conditions are identified from the video stream data, and video segments within a preset time period are recorded based on the timestamps of the video frames that meet the video recording trigger conditions. Keyframes are determined from the video segment based on the degree of difference between the various video frames in the video segment; Perform subject detection on the keyframes to determine the subject labels of the keyframes; The keyframes are feature-encoded to determine their visual feature vectors. The keyframes, along with their timestamps, subject tags, and visual feature vectors, are associated and stored in the video library.

6. The method according to claim 5, wherein, The process of determining video frames that meet the video recording trigger conditions from video stream data includes: For any video frame in the video stream data, in response to the pixel difference between the video frame and the video frame preceding the video frame being greater than a first preset threshold, the video frame is determined to be a video frame that meets the video recording trigger condition.

7. The method according to claim 6, wherein, The method further includes; Obtain multiple historical pixel differences for historical video frames, and calculate the mean and standard deviation of the multiple historical pixel differences; The first preset threshold is adjusted based on the mean and standard deviation of the multiple historical pixel differences.

8. The method according to claim 5, wherein, The process of determining video frames that meet the video recording trigger conditions from video stream data includes: For any video frame in the video stream data, if the probability of a target subject appearing in the video frame is greater than a second preset threshold, the video frame is determined to be a video frame that meets the video recording triggering condition.

9. The method according to claim 5, wherein, Determining keyframes from a video segment based on the degree of difference between the individual video frames includes: For each video frame in the video segment, the variance of the pixel grayscale value between the video frame and its neighboring video frames is calculated as an evaluation index of the degree of difference between the video frame and its neighboring video frames. In response to the evaluation index being greater than a third preset threshold, and the evaluation index of the video frame being greater than the evaluation index of the adjacent video frames, the video frame is designated as a keyframe.

10. The method according to claim 1, wherein, The step of analyzing the target video frame based on the semantics of the retrieved text to generate retrieval results includes: Perform intent analysis on the event to be queried to determine the intent type of the event to be queried; In response to the intent type being a summary type, a target video is generated based on the target video frame and its associated video frames; Based on the semantics of the event to be queried, perform at least one of subject trajectory analysis and subject behavior analysis on the target video to generate an output result corresponding to the event to be queried; The search results are generated by integrating at least one output result corresponding to each of the at least one query event.

11. The method according to claim 10, wherein analyzing the target video frame based on the semantics of the retrieved text to generate the retrieval result further includes: In response to the fact that the intent type of the at least one query event is a retrieval type, for each query event, based on the semantics of the query event, at least one of the following is output: video segment link, behavior description information, timestamp, and integrity score of the target video frame corresponding to the query event.

12. The method according to claim 10, wherein, The step of integrating at least one output result corresponding to each of the at least one query event to generate the search result includes: In response to the existence of multiple output results, the multiple output results are sorted according to the event occurrence time of each of the multiple output results; For the multiple sorted output results, a merging process is performed based on the consistency of the subject's behavioral information to generate search results.

13. A video retrieval device, comprising: A determination module is used to determine at least one semantic unit of each query event from the retrieved text, wherein the semantic unit includes at least one of time information, subject information, and behavioral description information; The retrieval module is used to retrieve the target video frame of the query event from the video library based on the semantic unit, wherein the video library includes multiple video frames extracted from video segments recorded in response to video recording trigger conditions, and each video frame is associated with a timestamp, a subject tag and a visual feature vector. The analysis module is used to analyze the target video frame based on the semantics of the search text and generate search results.

14. An electronic device comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1 to 12.

15. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1 to 12.

16. A computer program product comprising a computer program stored on at least one of a readable storage medium and an electronic device, the computer program implementing the method according to any one of claims 1 to 12 when executed by a processor.

Images